10.5.3 – ksqlDB Architecture

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.5 – ksqlDB Fundamentals
Goal: Understand ksqlDB's internal architecture, components, and how it works under the hood

Introduction

ksqlDB is built on top of Kafka Streams, providing a SQL interface with automatic state management, distributed processing, and fault tolerance.

Why This Module Matters: Understanding the architecture helps with deployment, scaling, troubleshooting, and performance tuning.

High-Level Architecture

┌────────────────────────────────────────────────────────────┐
│                    ksqlDB Client Layer                     │
│   ksqlDB CLI  |  REST API Client  |  Confluent Control Center │
└────────────────────────────────────────────────────────────┘
                          ↓ (HTTP)
┌────────────────────────────────────────────────────────────┐
│                  ksqlDB Server Cluster                     │
│  ┌──────────────────┐  ┌──────────────────┐              │
│  │  ksqlDB Server 1 │  │  ksqlDB Server 2 │              │
│  │                  │  │                  │              │
│  │ - REST API:8088  │  │ - REST API:8088  │              │
│  │ - SQL Parser     │  │ - SQL Parser     │              │
│  │ - Query Executor │  │ - Query Executor │              │
│  │ - State Stores   │  │ - State Stores   │              │
│  │ - Kafka Streams  │  │ - Kafka Streams  │              │
│  └──────────────────┘  └──────────────────┘              │
└────────────────────────────────────────────────────────────┘
          ↓ (reads/writes)          ↓
┌────────────────────────────────────────────────────────────┐
│                     Kafka Cluster                          │
│  Topics: orders, users, enriched_orders, _command_topic   │
└────────────────────────────────────────────────────────────┘

ksqlDB Server Components

1. REST API Layer

Purpose: Accept SQL queries via HTTP.

Default Port: 8088

Endpoints:

POST /ksql - Execute SQL statements
POST /query - Run queries
GET /info - Server information
GET /healthcheck - Health status

Example:

curl -X POST http://localhost:8088/ksql \
  -H "Content-Type: application/vnd.ksql.v1+json" \
  -d '{
    "ksql": "SELECT * FROM users EMIT CHANGES;",
    "streamsProperties": {}
  }'

2. SQL Parser

Purpose: Parse SQL statements into internal representation.

Flow:

SQL Statement → Lexer → Parser → AST → Logical Plan → Physical Plan

Example:

SELECT user_id, COUNT(*) AS click_count
FROM user_clicks
GROUP BY user_id
EMIT CHANGES;

Parsed Into:

Source: user_clicks stream
Projection: user_id, COUNT(*)
Aggregation: GROUP BY user_id
Emit: Continuous

3. Query Executor

Purpose: Execute queries using Kafka Streams.

Types of Queries:

Push Query (Continuous):

SELECT * FROM orders EMIT CHANGES;

Runs forever
Returns continuous stream of results
Uses Kafka Streams topology

Pull Query (Point-in-Time):

SELECT * FROM users WHERE user_id = 123;

Returns immediate result
Queries local state store
Does not use Kafka Streams

4. State Stores (RocksDB)

Purpose: Store materialized state for tables and aggregations.

Storage:

Local disk (RocksDB)
Replicated via Kafka changelog topics
Automatically managed by ksqlDB

Example: User click counts table.

CREATE TABLE user_click_counts AS
SELECT user_id, COUNT(*) AS clicks
FROM user_clicks
GROUP BY user_id
EMIT CHANGES;

Storage:

State store: user_click_counts (RocksDB)
Changelog topic: user_click_counts-changelog

5. Kafka Streams Engine

Purpose: Actual stream processing.

Key Point: ksqlDB is a wrapper around Kafka Streams.

Under the Hood:

CREATE STREAM enriched_orders AS
SELECT o.*, u.username
FROM orders o
LEFT JOIN users u ON o.user_id = u.user_id
EMIT CHANGES;

Compiles to Kafka Streams code:

StreamsBuilder builder = new StreamsBuilder();
KStream<String, Order> orders = builder.stream("orders");
KTable<String, User> users = builder.table("users");
orders.leftJoin(users, ...)
  .to("enriched_orders");

ksqlDB Server Modes

1. Interactive Mode (Default)

Characteristics:

Accepts ad-hoc SQL queries
Interactive CLI/REST API
Shared server for all queries

Use Case: Development, exploration, debugging.

2. Headless Mode

Characteristics:

Executes predefined SQL script
No interactive queries
Deterministic deployment

Deployment:

ksql-server-start /etc/ksqldb/ksqldb-server.properties \
  --queries-file /path/to/queries.sql

Use Case: Production deployments, CI/CD pipelines.

Persistent Queries vs Transient Queries

Persistent Queries

Definition: Long-running queries that create new streams/tables.

Example:

CREATE STREAM enriched_orders AS
SELECT * FROM orders
LEFT JOIN users ON orders.user_id = users.user_id
EMIT CHANGES;

Characteristics:

Runs forever (until terminated)
Creates new Kafka topics
Survives server restart
Distributed across ksqlDB servers

Storage:

Metadata stored in _command_topic
Results written to Kafka topic

Transient Queries

Definition: Temporary queries for exploration.

Example:

SELECT * FROM orders EMIT CHANGES LIMIT 10;

Characteristics:

Runs until client disconnects
No new topics created
Does not survive restart
Runs on single server

The `_command_topic`

Purpose: Store ksqlDB cluster state (persistent queries, streams, tables).

Topic Name: _confluent-ksql-<ksql.service.id>command_topic

Contents:

CREATE STREAM orders (...);
CREATE TABLE users (...);
CREATE STREAM enriched_orders AS SELECT ...;

Why It Matters:

All ksqlDB servers consume this topic
Ensures all servers have same state
Enables fault tolerance (restart recovers state)

ksqlDB Cluster Coordination

Server Discovery

Configuration:

ksql.service.id=my-ksql-cluster
bootstrap.servers=localhost:9092

All servers with same ksql.service.id form a cluster.

Query Distribution

Persistent Queries: Distributed across all servers (via Kafka Streams).

Example: 3 ksqlDB servers, 6 partitions.

Server 1: Processes partitions 0, 1
Server 2: Processes partitions 2, 3
Server 3: Processes partitions 4, 5

Automatic Rebalancing: When server added/removed, partitions reassigned.

State Replication

Changelog Topics:

users-changelog
user_click_counts-changelog

Flow:

1. Server 1 processes partition 0 → Updates local state store
2. State change written to changelog topic
3. Server 2 (standby) consumes changelog → Replicates state
4. If Server 1 crashes, Server 2 takes over partition 0 (state already replicated)

ksqlDB vs Kafka Streams

Aspect	ksqlDB	Kafka Streams
Interface	SQL	Java API
Deployment	Server	Embedded library
State Management	Automatic	Manual
Scalability	Add ksqlDB servers	Add app instances
Fault Tolerance	Built-in	Built-in
Flexibility	Limited to SQL	Full programmatic control
Learning Curve	Low	High

Performance Considerations

1. State Store Size

Issue: Large state stores slow down recovery.

Example: User click counts for 1 billion users = 10+ GB state store.

Solution: Partition state, use windowed aggregations.

2. Join Performance

Issue: Stream-table join requires full table in memory.

Example: Orders (stream) join Users (table with 10M users).

Solution: Ensure tables fit in memory or use RocksDB.

3. Rebalancing Impact

Issue: Adding/removing servers triggers rebalancing (temporary query pause).

Mitigation: Plan capacity, avoid frequent scaling.

Key Takeaways

ksqlDB Server: REST API + SQL Parser + Query Executor + Kafka Streams.
Built on Kafka Streams: ksqlDB compiles SQL to Kafka Streams code.
State Stores: RocksDB for local state, changelog topics for replication.
Interactive vs Headless: Interactive for development, headless for production.
Persistent vs Transient: Persistent queries create topics, transient are temporary.
_command_topic: Stores cluster state (streams, tables, queries).
Cluster Coordination: All servers with same ksql.service.id form a cluster.
Query Distribution: Partitions distributed across servers (like Kafka Streams).
Fault Tolerance: State replicated via changelog topics.
Scaling: Add ksqlDB servers to increase throughput (automatic rebalancing).

What's Next?

Next Module: 10.6 – Querying & Processing with ksqlDB

What We Covered:

✅ ksqlDB architecture
✅ Server components
✅ Interactive vs headless mode
✅ Persistent vs transient queries
✅ Cluster coordination

What's Coming:

❌ Creating streams and tables
❌ Transformations and filters
❌ Aggregations and windows

References:

Introduction → 10.5.1: Introduction to ksqlDB
Streams vs tables → 10.5.2: Streams vs Tables
Operations → 10.8: ksqlDB in Production

10.5.3 – ksqlDB Architecture

Introduction

High-Level Architecture

ksqlDB Server Components

1. REST API Layer

2. SQL Parser

3. Query Executor

4. State Stores (RocksDB)

5. Kafka Streams Engine

ksqlDB Server Modes

1. Interactive Mode (Default)

2. Headless Mode

Persistent Queries vs Transient Queries

Persistent Queries

Transient Queries

The _command_topic

ksqlDB Cluster Coordination

Server Discovery

Query Distribution

State Replication

ksqlDB vs Kafka Streams

Performance Considerations

1. State Store Size

2. Join Performance

3. Rebalancing Impact

Key Takeaways

What's Next?

The `_command_topic`