10.5.1 – JVM Tuning for Kafka

Module: M10 – Advanced Topics & Architecture
Section: 10.5 – Advanced Performance Tuning Layer
Goal: Optimize JVM heap, GC, and runtime settings for Kafka brokers

Introduction

Kafka brokers run on the JVM. Proper JVM tuning reduces GC pauses, improves throughput, and prevents OutOfMemoryErrors.

Note: This covers JVM-level tuning. Kafka config tuning covered in M6: Operations.

Heap Size Tuning

Recommended Heap Size

Rule of Thumb: 6-8 GB heap for most brokers.

Why Not Larger?:

• Kafka relies on OS page cache (not JVM heap) for message buffering
• Larger heap = longer GC pauses
• Diminishing returns beyond 8 GB

Setting Heap Size

Environment Variable:

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"

Why Same -Xms and -Xmx?:

• Prevents heap resizing (avoids GC pauses during resize)
• JVM allocates full heap upfront

Heap Size by Workload

Garbage Collection Tuning

GC Algorithm Selection

Kafka 2.8+: G1GC (default, recommended)

Config:

export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=20"

Why G1GC?:

• Low-latency (pausetimes < 20ms)
• Predictable (goal-based tuning)
• Handles large heaps well

G1GC Tuning Parameters

1. MaxGCPauseMillis

-XX:MaxGCPauseMillis=20  # Target: 20ms pauses

Default: 200ms (too high for Kafka)
Recommended: 20ms (balance throughput vs latency)

2. G1HeapRegionSize

-XX:G1HeapRegionSize=16m  # 16 MB regions

Default: Calculated from heap size
Tuning: Increase if many large objects (e.g., large messages)

3. InitiatingHeapOccupancyPercent

-XX:InitiatingHeapOccupancyPercent=35  # Start GC at 35% heap usage

Default: 45%
Tuning: Lower if frequent long GC pauses (more aggressive GC)

Alternative: ZGC (Kafka 3.0+, Java 11+)

Config:

export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseZGC"

Benefits:

• Ultra-low latency (< 1ms pauses)
• Scales to TB-size heaps

Drawbacks:

• Requires Java 11+
• Higher CPU overhead (~10-15%)

Use Case: Ultra-low latency requirements (< 1ms p99).

JVM Performance Flags

Full JVM Config Example

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"

export KAFKA_JVM_PERFORMANCE_OPTS="
  -XX:+UseG1GC
  -XX:MaxGCPauseMillis=20
  -XX:InitiatingHeapOccupancyPercent=35
  -XX:G1HeapRegionSize=16m
  -XX:+ParallelRefProcEnabled
  -XX:+UnlockExperimentalVMOptions
  -XX:+DisableExplicitGC
  -Djava.awt.headless=true
  -Dcom.sun.management.jmxremote
  -Dcom.sun.management.jmxremote.port=9999
  -Dcom.sun.management.jmxremote.authenticate=false
  -Dcom.sun.management.jmxremote.ssl=false
"

Key Flags Explained

1. -XX:+ParallelRefProcEnabled

• Parallelizes reference processing (finalization, weak refs)
• Reduces GC pause time

2. -XX:+DisableExplicitGC

• Ignores System.gc() calls (prevents manual GC triggers)
• Kafka code doesn't call System.gc(), but libraries might

3. -Djava.awt.headless=true

• Disables AWT (graphics library, not needed for Kafka)
• Reduces memory overhead

4. JMX Flags

• Enables JMX monitoring on port 9999
• No authentication (secure with firewall or SSH tunnel in production)

Monitoring JVM Performance

Key Metrics

1. GC Pause Time

jvm.gc.pause.duration (p99, p999)

Target: < 20ms p99, < 50ms p999
Alert: > 100ms p99 (impacts throughput)

2. GC Frequency

jvm.gc.count (per minute)

Healthy: < 10 GCs/minute
Alert: > 100 GCs/minute (heap too small or memory leak)

3. Heap Usage

jvm.memory.heap.used / jvm.memory.heap.max

Healthy: 50-70% average
Alert: > 90% (heap too small)

4. Old Generation Usage

jvm.memory.pool.used{pool="G1 Old Gen"}

Healthy: Gradual growth, periodic resets after full GC
Alert: Continuous growth (memory leak)

Monitoring Tools

1. JConsole (built-in)

jconsole localhost:9999

2. VisualVM (open-source)

visualvm --openjmx localhost:9999

3. Prometheus + Grafana (production)

• JMX Exporter exports JVM metrics to Prometheus
• Grafana dashboards visualize GC performance

Common JVM Issues

Issue 1: Long GC Pauses (> 100ms)

Symptoms: High p99 latency, producer timeouts

Causes:

• Heap too large (> 12 GB)
• Too many objects in old generation

Solutions:

• Reduce heap size to 6-8 GB
• Lower -XX:MaxGCPauseMillis to 20ms
• Lower -XX:InitiatingHeapOccupancyPercent to 35%

Issue 2: OutOfMemoryError: Java Heap Space

Symptoms: Broker crashes with OOM error

Causes:

• Heap too small
• Memory leak (e.g., accumulating request/response buffers)

Solutions:

• Increase heap size (but not > 12 GB)
• Check for memory leaks (heap dump analysis)

Heap Dump:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/kafka/heapdump.hprof

Issue 3: High CPU Usage from GC

Symptoms: 50-100% CPU usage, mostly in GC threads

Causes:

• Heap too small (constant GC)
• Too many short-lived objects (e.g., large producer batches)

Solutions:

• Increase heap size
• Reduce object allocation (tune Kafka configs, not JVM)

Key Takeaways

1. Heap Size: 6-8 GB for most brokers (not > 12 GB).
2. GC Algorithm: G1GC (default, recommended), ZGC for ultra-low latency.
3. GC Target: -XX:MaxGCPauseMillis=20 (20ms pauses).
4. Monitoring: Track GC pause time (p99 < 20ms), frequency (< 10/min), heap usage (50-70%).
5. Issues: Long pauses → reduce heap, OOM → increase heap or fix leak, high CPU → increase heap.
6. Flags: -Xms6g -Xmx6g, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20.

What's Next?

Next Module: 10.5.2 – OS and Disk Tuning

References:

• Broker configs → M6: Operations
• Performance tuning basics → M6: Operations