KafkaIQ

10.8.1 – Deployment Models

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.8 – ksqlDB in Production
Goal: Understand deployment options for ksqlDB

Interactive Mode

For: Development, exploration

ksql.service.id=my-ksql-cluster
bootstrap.servers=localhost:9092
listeners=http://0.0.0.0:8088

ksql-server-start /etc/ksqldb/ksqldb-server.properties

Usage: ksqlDB CLI, REST API

Headless Mode

For: Production deployments

ksql-server-start /etc/ksqldb/ksqldb-server.properties \
  --queries-file /app/queries.sql

queries.sql:

CREATE STREAM orders (...) WITH (...);
CREATE TABLE users (...) WITH (...);
CREATE STREAM enriched_orders AS
SELECT * FROM orders LEFT JOIN users ...;

Benefits: Deterministic, version-controlled, CI/CD friendly

High Availability Setup

3 ksqlDB servers:

Server 1: ksql.service.id=prod-ksql, bootstrap.servers=kafka:9092
Server 2: ksql.service.id=prod-ksql, bootstrap.servers=kafka:9092
Server 3: ksql.service.id=prod-ksql, bootstrap.servers=kafka:9092

Load Balancer → [Server 1, Server 2, Server 3]

Automatic failover: If server crashes, others take over partitions

Docker Deployment

version: '3'
services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:latest
    ports:
      - "8088:8088"
    environment:
      KSQL_BOOTSTRAP_SERVERS: kafka:9092
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_KSQL_SERVICE_ID: docker-ksql

Key Takeaways

  1. Interactive: Development, ad-hoc queries
  2. Headless: Production, predefined queries
  3. HA: Multiple servers with same service ID
  4. Docker: Containerized deployment

What's Next?

Next Module: 10.8.2 – Scaling & Performance