KafkaIQ

13.1 Advanced Configuration & Choosing the Right Tool

Module Overview

Goal: Master advanced configuration and learn when to use Kafka Streams vs alternatives.

What You'll Learn:

  • Advanced configuration tuning
  • Performance optimization
  • Resource tuning (threads, memory, cache)
  • RocksDB tuning
  • When to use Kafka Streams
  • Kafka Streams vs alternatives (ksqlDB, Flink, Spark)
  • Use case patterns
  • Decision framework

What's NOT Covered Here:

  • Basic configuration (covered in Module 3.1)
  • Monitoring (covered in Module 12.1)
  • Specific operations (covered in earlier modules)

Key Insight: Configuration tuning can dramatically improve performance — choose the right tool for your use case.

1. Thread Configuration

Number of Threads

// Default: 1 thread per instance
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);

// More threads = more parallelism within instance
// But: Max parallelism = partition count

Example:
├─ 10 partitions, 1 instance, 4 threads
│  └─ Each thread handles 2-3 tasks
├─ 10 partitions, 1 instance, 1 thread
│  └─ Thread handles all 10 tasks
└─ 10 partitions, 2 instances, 4 threads each
   └─ Each instance handles 5 tasks across 4 threads

Choosing Thread Count

GUIDELINES:

Too few threads (1-2):
├─ Underutilized CPU
├─ Lower throughput
└─ Use when: single-threaded processing needed

Optimal (4-8):
├─ Good CPU utilization
├─ Balanced resource usage
└─ Use when: standard workload

Too many (16+):
├─ Context switching overhead
├─ Diminishing returns
└─ Use when: many partitions (> 50)

Rule of thumb:
└─ num_threads = min(num_partitions, num_cpu_cores)

2. Memory Configuration

Heap Memory

JVM Heap = Java objects (application logic, buffers)

Recommended heap:
├─ Small workload: 2-4 GB
├─ Medium workload: 4-8 GB
├─ Large workload: 8-16 GB
└─ Very large: 16-32 GB

JVM options:
-Xms8g -Xmx8g  # 8 GB heap (min = max, avoid resizing)

Avoid:
└─ > 32 GB heap (GC pauses)

Off-Heap Memory (RocksDB)

RocksDB uses off-heap (direct memory)

Estimated off-heap usage:
├─ State store size (GB)
├─ + RocksDB overhead (20-30%)
└─ = Total off-heap

Example:
├─ State stores: 50 GB
├─ Overhead: 15 GB
└─ Total: 65 GB off-heap

Container memory:
├─ Heap: 8 GB
├─ Off-heap: 65 GB
├─ OS overhead: 2 GB
└─ Total: 75 GB

Cache Configuration

// Per-task cache for state updates
props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 
    10 * 1024 * 1024);  // 10 MB per task

// Global = cache_per_task × num_tasks

Purpose:
├─ Buffer state updates
├─ Reduce changelog writes
└─ Reduce downstream output

Trade-off:
├─ Larger cache = fewer writes, more memory
└─ Smaller cache = more writes, less memory

Recommendations:
├─ High throughput: 50-100 MB
├─ Low latency: 1-10 MB
└─ Balanced: 10-50 MB

3. Commit Interval

Configuration

// How often to commit offsets/transactions
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 30000);  // 30 seconds (default)

Impact:
├─ Shorter interval (5-10s): Lower latency, higher overhead
└─ Longer interval (60s+): Higher latency, lower overhead

Choosing Commit Interval

LOW LATENCY (5-10 seconds):
├─ Data visible quickly
├─ Fast recovery (less replay)
├─ Higher broker load
└─ Use when: Real-time dashboards, low-latency requirements

BALANCED (30 seconds):
├─ Good trade-off
├─ Default, works for most cases
└─ Use when: Standard workload

HIGH THROUGHPUT (60+ seconds):
├─ Lower overhead
├─ Higher throughput
├─ Longer recovery time
└─ Use when: Batch processing, high-volume aggregations

4. RocksDB Configuration

What is RocksDB?

RocksDB = embedded key-value store (LSM tree)

Used by Kafka Streams for:
└─ Persistent state stores

Characteristics:
├─ Off-heap (direct memory)
├─ Disk-backed
├─ Fast reads and writes
└─ Tunable performance

Custom RocksDBConfigSetter

public class CustomRocksDBConfig implements RocksDBConfigSetter {
    @Override
    public void setConfig(String storeName, Options options, Map<String, Object> configs) {
        // Increase write buffer for better write performance
        options.setWriteBufferSize(64 * 1024 * 1024);  // 64 MB
        options.setMaxWriteBufferNumber(3);

        // Optimize for point lookups
        BlockBasedTableConfig tableConfig = new BlockBasedTableConfig();
        tableConfig.setBlockSize(16 * 1024);  // 16 KB blocks
        tableConfig.setCacheIndexAndFilterBlocks(true);
        options.setTableFormatConfig(tableConfig);

        // Compaction settings
        options.setCompactionStyle(CompactionStyle.LEVEL);
        options.setMaxBackgroundCompactions(4);

        // Memory usage
        options.setMaxTotalWalSize(100 * 1024 * 1024);  // 100 MB WAL
    }

    @Override
    public void close(String storeName, Options options) {
        // Cleanup if needed
    }
}

Apply Configuration

props.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, 
    CustomRocksDBConfig.class);

RocksDB Tuning Guidelines

WRITE-HEAVY WORKLOAD:
├─ Increase write_buffer_size (64-128 MB)
├─ Increase max_write_buffer_number (3-5)
└─ Use more compaction threads

READ-HEAVY WORKLOAD:
├─ Increase block_cache_size (100-500 MB)
├─ Use bloom filters
└─ Optimize block_size

LARGE STATE (> 100 GB):
├─ Use SSD (not HDD)
├─ Increase compaction threads
└─ Monitor disk I/O

SMALL STATE (< 10 GB):
├─ Use defaults (usually sufficient)
└─ Consider in-memory stores

5. Producer & Consumer Overrides

Producer Configuration

// Override producer configs
props.put(StreamsConfig.PRODUCER_PREFIX + ProducerConfig.BATCH_SIZE_CONFIG, 
    32 * 1024);  // 32 KB batches

props.put(StreamsConfig.PRODUCER_PREFIX + ProducerConfig.LINGER_MS_CONFIG, 
    10);  // Wait 10ms for batching

props.put(StreamsConfig.PRODUCER_PREFIX + ProducerConfig.COMPRESSION_TYPE_CONFIG, 
    "lz4");  // Fast compression

// For exactly-once (EOS), these are set automatically:
// enable.idempotence=true
// acks=all
// max.in.flight.requests.per.connection=5

Consumer Configuration

// Override consumer configs
props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 
    52428800);  // 50 MB

props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 
    1000);  // 1000 records per poll

props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 
    45000);  // 45 seconds

props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 
    300000);  // 5 minutes

6. Performance Optimization Checklist

1. Topology Optimization

✅ Minimize repartitions
   └─ Use groupByKey() instead of groupBy() when possible

✅ Reduce state store count
   └─ Combine aggregations

✅ Use stateless operations when possible
   └─ map, filter (no state overhead)

✅ Avoid unnecessary materializations
   └─ Only materialize what needs querying

2. Parallelism

✅ Partition input topics appropriately
   └─ More partitions = more parallelism

✅ Set appropriate thread count
   └─ 4-8 threads per instance

✅ Scale horizontally
   └─ More instances = more throughput

3. Batching & Buffering

✅ Enable caching
   └─ props.put(CACHE_MAX_BYTES_BUFFERING_CONFIG, 50 MB)

✅ Increase batch sizes
   └─ producer: batch.size, linger.ms
   └─ consumer: fetch.max.bytes, max.poll.records

✅ Use appropriate commit interval
   └─ 30-60 seconds for high throughput

4. Compression

✅ Enable compression
   └─ props.put(PRODUCER_PREFIX + COMPRESSION_TYPE_CONFIG, "lz4")

Compression types:
├─ lz4: Fast, moderate ratio (recommended)
├─ zstd: Slower, better ratio
├─ snappy: Fast, lower ratio
└─ gzip: Slowest, best ratio

Choose based on:
├─ CPU availability (lz4 if CPU-constrained)
└─ Network bandwidth (zstd if network-constrained)

5. Resource Allocation

✅ Use SSD for state stores
   └─ 10-100× faster than HDD

✅ Allocate sufficient memory
   └─ Heap + off-heap (RocksDB)

✅ Provision adequate CPU
   └─ 4-8 vCPUs per instance

✅ Ensure network bandwidth
   └─ 1-10 Gbps for high throughput

7. When to Use Kafka Streams

Ideal Use Cases

✅ Event-driven microservices
   └─ Process events, emit results

✅ Real-time aggregations
   └─ Counts, sums, windowed metrics

✅ Stream enrichment
   └─ Join streams with reference data

✅ Stateful transformations
   └─ Sessionization, deduplication

✅ Event sourcing
   └─ Maintain materialized views

✅ Real-time analytics
   └─ Dashboards, monitoring

When NOT to Use Kafka Streams

❌ Complex SQL queries
   └─ Use ksqlDB or external DB

❌ Batch processing (historical data)
   └─ Use Spark, Flink batch mode

❌ Machine learning inference (heavy CPU)
   └─ Use dedicated ML serving platform

❌ Ad-hoc queries
   └─ Use ksqlDB or data warehouse

❌ Very large state (> 1 TB per instance)
   └─ Use Flink with external state backend

❌ Sub-millisecond latency
   └─ Use in-memory cache (Redis)

8. Kafka Streams vs Alternatives

Kafka Streams vs ksqlDB

KAFKA STREAMS:
✅ Full Java/Scala API (flexibility)
✅ Programmatic control
✅ Embedded in application
✅ Lower latency (no extra hop)
❌ Requires coding
❌ No ad-hoc queries

KSQLDB:
✅ SQL interface (easier)
✅ Ad-hoc queries
✅ Standalone service
❌ Less flexible than code
❌ Additional infrastructure
❌ Higher latency (network hop)

When to choose:
├─ Kafka Streams: Complex logic, embedded processing
└─ ksqlDB: SQL users, ad-hoc queries, simpler use cases

KAFKA STREAMS:
✅ Simpler (no cluster management)
✅ Embedded in application
✅ Tightly integrated with Kafka
✅ Lower operational overhead
❌ Limited to Kafka (no other sources)
❌ State size limited by disk
❌ No native ML support

APACHE FLINK:
✅ Multiple sources (Kafka, DB, files, etc.)
✅ Advanced features (CEP, ML, etc.)
✅ Large-scale state (external backends)
✅ Powerful windowing
❌ Complex (cluster management)
❌ Higher operational overhead
❌ Steeper learning curve

When to choose:
├─ Kafka Streams: Kafka-centric, simpler operations
└─ Flink: Multiple sources, advanced features, large scale

Kafka Streams vs Spark Streaming

KAFKA STREAMS:
✅ True streaming (record-at-a-time)
✅ Lower latency (milliseconds)
✅ Simpler operations
✅ Tightly integrated with Kafka
❌ Limited to streaming
❌ No batch processing
❌ Smaller ecosystem

SPARK STREAMING:
✅ Unified batch + streaming
✅ Rich ecosystem (ML, graph, etc.)
✅ Structured Streaming (micro-batches)
❌ Higher latency (seconds)
❌ More complex (cluster management)
❌ Micro-batching (not true streaming)

When to choose:
├─ Kafka Streams: Low latency, pure streaming
└─ Spark: Batch + streaming, ML, larger ecosystem

9. Decision Framework

Question 1: What's Your Data Source?

Only Kafka?
└─ Kafka Streams ✅

Kafka + other sources (DB, files, etc.)?
└─ Flink, Spark

SQL queries on Kafka?
└─ ksqlDB

Question 2: What's Your Latency Requirement?

Sub-second (< 1s)?
└─ Kafka Streams ✅

Seconds (1-10s)?
└─ Kafka Streams, Flink, ksqlDB

Minutes (batch)?
└─ Spark, Flink batch mode

Question 3: What's Your State Size?

Small (< 100 GB per instance)?
└─ Kafka Streams ✅

Large (100 GB - 1 TB)?
└─ Kafka Streams with large instances

Very large (> 1 TB)?
└─ Flink with external state backend

Question 4: What's Your Team's Skill Set?

Java/Scala developers?
└─ Kafka Streams ✅

SQL users (analysts)?
└─ ksqlDB

Data engineers (big data)?
└─ Flink, Spark

Question 5: Operational Complexity?

Simple (embedded, no cluster)?
└─ Kafka Streams ✅

Acceptable (standalone cluster)?
└─ ksqlDB, Flink, Spark

10. Architecture Patterns

Pattern 1: Microservice Stream Processor

Use case: Order processing service

Architecture:
├─ Kafka Streams embedded in microservice
├─ Reads from "orders" topic
├─ Validates, enriches, transforms
├─ Writes to "validated-orders" topic
└─ Exposes REST API for queries

Why Kafka Streams:
├─ Tight integration with service
├─ Low latency
└─ Simple deployment

Pattern 2: Real-Time Analytics Pipeline

Use case: Dashboard metrics (counts, sums)

Architecture:
├─ Kafka Streams application
├─ Reads events from multiple topics
├─ Windowed aggregations
├─ Exposes metrics via REST API
└─ Materialized views for queries

Why Kafka Streams:
├─ Real-time aggregations
├─ Interactive queries
└─ Stateful processing

Pattern 3: Stream Enrichment

Use case: Enrich events with reference data

Architecture:
├─ Kafka Streams application
├─ KStream: Event stream (orders)
├─ KTable: Reference data (customers)
├─ Join: Enrich orders with customer data
└─ Write to enriched topic

Why Kafka Streams:
├─ Stream-table join
├─ Low latency
└─ Automatic state management

Pattern 4: Event Sourcing

Use case: Maintain materialized views

Architecture:
├─ Kafka Streams application
├─ Reads event log (source of truth)
├─ Builds materialized views (aggregates)
├─ Exposes views via interactive queries
└─ Views queryable in real-time

Why Kafka Streams:
├─ Stateful aggregations
├─ Interactive queries
└─ Event-driven

11. Migration Strategies

From Batch to Streaming

Step 1: Identify use case
└─ Can it be done incrementally (streaming)?

Step 2: Start small
├─ Single topic, simple transformation
└─ Validate in staging

Step 3: Gradually migrate
├─ One use case at a time
└─ Run batch + streaming in parallel (validate)

Step 4: Retire batch
└─ Once streaming proven reliable

From Another Stream Processor

From Flink/Spark to Kafka Streams:

1. Evaluate fit
   └─ Is Kafka the only source?
   └─ Is state size manageable?

2. Rewrite topology
   └─ Translate Flink/Spark code to Kafka Streams API

3. Test thoroughly
   └─ Unit tests, integration tests

4. Deploy canary
   └─ Run in parallel, compare results

5. Full migration
   └─ Switch traffic, retire old system

12. Cost Optimization

Compute Costs

Reduce instances:
├─ Increase threads per instance
├─ Optimize processing logic
└─ Use larger instances (fewer total)

Example:
├─ Before: 10× small instances (2 vCPU each) = 20 vCPU
└─ After:  5× medium instances (4 vCPU each) = 20 vCPU (cheaper)

Storage Costs

Reduce state store size:
├─ Use compaction (keep latest only)
├─ Set TTL on state (expire old data)
├─ Use in-memory stores (if state small)
└─ Offload to external DB (if state large)

Example:
├─ Before: 100 GB state per instance × 10 instances = 1 TB
└─ After:  10 GB state per instance × 10 instances = 100 GB (90% savings)

Network Costs

Reduce network I/O:
├─ Enable compression (lz4, zstd)
├─ Increase commit interval (fewer writes)
├─ Use caching (buffer updates)
└─ Co-locate Kafka and Streams (same datacenter)

Example:
├─ Before: 10 GB/hour network transfer
└─ After:  5 GB/hour (50% savings with compression)

13. Advanced Patterns

Pattern: Time-Travel Queries

// Query historical state (windowed stores)
ReadOnlyWindowStore<String, Long> store = streams.store(...);

// Query specific time in past
Instant past = Instant.now().minus(Duration.ofHours(24));
WindowStoreIterator<Long> results = store.fetch("key", past, past);

// Use case: "What was the count 24 hours ago?"

Pattern: Multi-Region Deployment

Scenario: Deploy across regions

Architecture:
├─ Kafka cluster per region (mirrored)
├─ Kafka Streams in each region (reads local cluster)
├─ MirrorMaker2 for cross-region replication
└─ Regional failover

Benefits:
├─ Low latency (local reads)
├─ Disaster recovery
└─ Data locality

Pattern: Blue-Green Deployment

Strategy: Zero-downtime deployments

Steps:
1. Deploy new version (green) alongside old (blue)
2. Green joins consumer group → rebalance
3. Both versions process (gradually shift traffic)
4. Validate green version
5. Shut down blue version

Benefit:
└─ Zero downtime, easy rollback

14. Best Practices Summary

Configuration

✅ Set num.stream.threads (4-8)
✅ Enable caching (10-50 MB)
✅ Use EOS v2 for critical apps
✅ Configure commit interval (30-60s)
✅ Tune RocksDB (if large state)
✅ Set appropriate timeouts (session, poll)
✅ Use compression (lz4)

Operations

✅ Use standby replicas (production)
✅ Set uncaught exception handler
✅ Graceful shutdown
✅ Monitor key metrics (state, lag, throughput)
✅ Set up alerts (lag, errors, rebalances)
✅ Log structured events
✅ Test in staging (load, failure, rebalance)

Topology Design

✅ Minimize repartitions
✅ Name state stores explicitly
✅ Reduce state store count
✅ Use appropriate abstractions (KStream vs KTable)
✅ Design for idempotence (external writes)

15. Mental Model Summary

When to Use Kafka Streams

┌───────────────────────────────────────────────────┐
│  KAFKA STREAMS SWEET SPOT                         │
│                                                   │
│  ✅ Kafka as primary source                       │
│  ✅ Real-time processing (< 1s latency)           │
│  ✅ Stateful transformations                      │
│  ✅ Stream joins and aggregations                 │
│  ✅ Embedded in application                       │
│  ✅ Simple operations (no cluster management)     │
│  ✅ Moderate state size (< 100 GB/instance)       │
│                                                   │
│  ❌ Multiple sources (not just Kafka)             │
│  ❌ Batch processing (historical data)            │
│  ❌ Very large state (> 1 TB/instance)            │
│  ❌ Complex SQL queries                           │
│  ❌ Sub-millisecond latency                       │
└───────────────────────────────────────────────────┘

Tool Comparison

KAFKA STREAMS:
├─ Embedded library
├─ Kafka-native
├─ Millisecond latency
└─ Use when: Simple, Kafka-centric, low-latency

KSQLDB:
├─ Standalone service
├─ SQL interface
├─ Second latency
└─ Use when: SQL users, ad-hoc queries

APACHE FLINK:
├─ Cluster-based
├─ Multiple sources
├─ Advanced features
└─ Use when: Large scale, complex CEP, ML

SPARK STREAMING:
├─ Cluster-based
├─ Batch + streaming
├─ Second latency
└─ Use when: Unified batch/stream, ML

16. Questions You Should Be Able to Answer

After this module, you should be able to answer:

✅ How many threads should I configure?
✅ How do I tune RocksDB performance?
✅ When should I use Kafka Streams vs ksqlDB?
✅ When should I use Kafka Streams vs Flink?
✅ How do I optimize for high throughput?
✅ How do I optimize for low latency?
✅ What are ideal use cases for Kafka Streams?
✅ How do I reduce costs?

17. Conclusion

You've Completed the Kafka Streams Course!

You now understand:

  • ✅ Core abstractions (KStream, KTable, GlobalKTable)
  • ✅ Stateless and stateful operations
  • ✅ Windowing and time-based processing
  • ✅ State stores and fault tolerance
  • ✅ Exactly-once semantics
  • ✅ Joins (stream-stream, stream-table, table-table)
  • ✅ Error handling and resilience
  • ✅ Interactive queries and testing
  • ✅ Monitoring and operations
  • ✅ Advanced configuration and tuning
  • ✅ When to use Kafka Streams vs alternatives

Next Steps:

  1. Build a real project
  2. Deploy to production
  3. Monitor and iterate
  4. Contribute to open source
  5. Share your knowledge

Thank you for learning Kafka Streams!

Summary

You learned:

  • ✅ Thread config: 4-8 threads per instance (based on partitions and CPU)
  • ✅ Memory: Heap (8-16 GB) + off-heap (state size + 30%)
  • ✅ Cache: 10-50 MB per task (balance writes vs memory)
  • ✅ Commit interval: 30-60s (balance latency vs throughput)
  • ✅ RocksDB tuning: write_buffer_size, block_cache, compaction threads
  • ✅ Use Kafka Streams when: Kafka-native, real-time, moderate state, embedded
  • ✅ Avoid Kafka Streams when: Multiple sources, batch, very large state, complex SQL
  • ✅ Kafka Streams vs ksqlDB: Code vs SQL, embedded vs standalone
  • ✅ Kafka Streams vs Flink: Simpler vs advanced features, Kafka-only vs multi-source
  • ✅ Optimize: Minimize repartitions, enable compression, scale horizontally, tune resources

This concludes the Kafka Streams course. Go build amazing streaming applications!