10.8.2 – Scaling & Performance

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.8 – ksqlDB in Production
Goal: Scale ksqlDB and optimize performance

Horizontal Scaling

Add more ksqlDB servers:

Before: 1 server, 6 partitions → Server processes all 6 partitions
After:  3 servers, 6 partitions → Each server processes 2 partitions

Result: 3x throughput increase

Performance Tuning

Increase thread count:

ksql.streams.num.stream.threads=4

Increase heap:

export KSQL_HEAP_OPTS="-Xms4g -Xmx4g"

Increase state store cache:

ksql.streams.cache.max.bytes.buffering=100000000

State Store Sizing

Estimate:

Users table: 1M users × 500 bytes = 500 MB
Hourly aggregations: 24 hours × 10 MB = 240 MB
Total: ~750 MB per server

Recommendation: 2x-3x overhead for RocksDB

Key Takeaways

1. Horizontal Scaling: Add servers for more throughput
2. Threads: Increase num.stream.threads
3. Heap: 4-8 GB typical
4. State Stores: Plan for RocksDB overhead

What's Next?

Next Module: 10.8.3 – Fault Tolerance