5.1 Performance Tuning & Capacity Planning

Module Overview

Goal: One authoritative place for performance decisions and capacity planning.

What You'll Learn:

• Broker-level tuning configurations
• Disk vs network constraint identification
• JVM tuning for Kafka
• Capacity planning methodology
• Scaling strategies (horizontal vs vertical)
• Performance testing and benchmarking
• Common performance bottlenecks and solutions

What's NOT Covered Here:

• Topic design fundamentals (covered in Module 2.2)
• Client-side tuning (covered in Producer/Consumer modules)
• Monitoring metrics (covered in Module 4.x)

Key Insight: Performance tuning is about identifying bottlenecks and systematically removing them. Capacity planning is about staying ahead of growth.

1. Performance Tuning Philosophy

The Tuning Process

1. MEASURE (Baseline)
   └─ Collect current performance metrics

2. IDENTIFY (Bottleneck)
   └─ Find the limiting factor (disk, network, CPU)

3. TUNE (Single Change)
   └─ Change one config at a time

4. MEASURE (Validate)
   └─ Did performance improve?

5. REPEAT
   └─ Move to next bottleneck

NEVER:
└─ Change multiple configs simultaneously
└─ Tune without measuring impact

Common Bottlenecks

DISK I/O:
├─ Symptoms: High I/O wait, slow flush times
└─ Solution: SSDs, more disks, tune flush settings

NETWORK:
├─ Symptoms: High network utilization, dropped packets
└─ Solution: Faster network, compression, add brokers

CPU:
├─ Symptoms: High CPU usage, slow request processing
└─ Solution: Tune compression, add brokers

MEMORY:
├─ Symptoms: High GC time, OOM errors
└─ Solution: Increase heap, tune GC

COORDINATION:
├─ Symptoms: Slow controller operations, high ZK latency
└─ Solution: Tune ZK, migrate to KRaft

2. Broker-Level Tuning

Thread Pool Configuration

num.network.threads

# Network threads handling requests/responses
num.network.threads=8  # Default: 3

Recommendation:
└─ Set to number of CPU cores (or 2x for high network traffic)

Impact:
├─ More threads = better network concurrency
└─ Too many = context switching overhead

num.io.threads

# Threads handling disk I/O
num.io.threads=16  # Default: 8

Recommendation:
└─ Set to 2x number of disks (for HDDs)
└─ Can be lower for SSDs (less disk seek time)

Impact:
├─ More threads = better disk I/O concurrency
└─ Must balance with actual disk capacity

num.replica.fetchers

# Threads per broker for follower replication
num.replica.fetchers=4  # Default: 1

Recommendation:
└─ Increase if replication lag is high

Impact:
├─ More threads = faster replication
└─ Higher network and disk usage

Socket Buffer Configuration

socket.send.buffer.bytes & socket.receive.buffer.bytes

# Socket buffer sizes (bytes)
socket.send.buffer.bytes=102400     # Default: 100 KB
socket.receive.buffer.bytes=102400  # Default: 100 KB

Recommendation:
└─ Increase for high-throughput, high-latency networks
└─ 1 MB (1048576) for 10 Gbps network

Impact:
├─ Larger buffers = better throughput on fast networks
└─ More memory usage per connection

socket.request.max.bytes

# Max size of request broker will accept
socket.request.max.bytes=104857600  # Default: 100 MB

Recommendation:
└─ Match with max.message.bytes if large messages

Impact:
└─ Prevents oversized requests from consuming resources

Background Thread Configuration

background.threads

# Threads for background tasks (log cleanup, etc.)
background.threads=10  # Default: 10

Recommendation:
└─ Default is usually sufficient
└─ Increase if log cleanup lagging

Impact:
└─ More threads = faster background processing

log.cleaner.threads

# Threads for log compaction
log.cleaner.threads=1  # Default: 1

Recommendation:
└─ Increase to 2-4 if using many compacted topics

Impact:
└─ More threads = faster compaction
└─ Higher CPU and disk usage

3. Disk Performance Tuning

Use SSDs

HDD vs SSD COMPARISON:

HDD:
├─ IOPS: 100-200
├─ Latency: 10-20 ms
├─ Cost: Low
└─ Use case: Archival, low-throughput

SSD:
├─ IOPS: 10,000+
├─ Latency: < 1 ms
├─ Cost: Higher
└─ Use case: Production, high-throughput

RECOMMENDATION:
└─ Use SSDs for Kafka brokers in production
└─ 10-100x performance improvement

Filesystem Choice

EXT4 vs XFS:

EXT4:
├─ Widely used, stable
├─ Good performance
└─ Default for many Linux distributions

XFS:
├─ Better for large files
├─ Better parallel I/O
├─ Recommended by Confluent
└─ Use with: mkfs.xfs /dev/sdX

RECOMMENDATION:
└─ Use XFS for Kafka data directories

Mount Options

# /etc/fstab entry for Kafka data directory
/dev/sdX  /var/kafka  xfs  defaults,noatime,nodiratime  0  2

Key options:
├─ noatime: Don't update access time (less I/O)
├─ nodiratime: Don't update directory access time
└─ defaults: Standard mount options

Flush Settings

log.flush.interval.messages & log.flush.interval.ms

# Flush to disk after N messages or T milliseconds
log.flush.interval.messages=10000  # Default: Long.MAX_VALUE (disabled)
log.flush.interval.ms=1000         # Default: null (disabled)

Recommendation:
└─ Leave DISABLED (rely on OS page cache and replication)
└─ Flushing is expensive (reduces throughput)

When to enable:
└─ Only if replication factor = 1 (not recommended!)
└─ Compliance requirements (rare)

4. Network Performance Tuning

Enable Compression

# Broker-side compression (if producer doesn't compress)
compression.type=producer  # Default: producer

Options:
├─ producer: Use producer's compression
├─ none: No compression
├─ gzip, snappy, lz4, zstd: Force recompression

Recommendation:
└─ Use 'producer' (let producers compress)
└─ If mixed producers, enforce 'lz4' or 'zstd' at broker

Impact:

Compression reduces:
├─ Network bandwidth (3-5x)
├─ Disk usage (3-5x)
└─ Replication traffic

Trade-off:
└─ Higher CPU usage (compression/decompression)

Network Tuning (OS-Level)

# /etc/sysctl.conf (Linux kernel tuning)

# Increase network buffer sizes
net.core.rmem_max=134217728        # 128 MB
net.core.wmem_max=134217728        # 128 MB
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728

# Increase max connections
net.core.somaxconn=4096
net.ipv4.tcp_max_syn_backlog=4096

# Enable TCP window scaling
net.ipv4.tcp_window_scaling=1

# Increase connection tracking
net.netfilter.nf_conntrack_max=1000000

# Apply changes
sysctl -p

5. OS-Level Optimizations

File Descriptor Limits

Why File Descriptors Matter:

Kafka brokers need many open files simultaneously:

• Log segment files (per partition)
• Index files
• Client connections (sockets)
• Inter-broker connections

# Check current limit
ulimit -n
# Common default: 1024 (way too low for Kafka!)

# Check Kafka process file descriptor usage
lsof -p $(pgrep -f kafka.Kafka) | wc -l

Setting File Descriptor Limits:

# Temporary (current session only)
ulimit -n 100000

# Permanent: Edit /etc/security/limits.conf
kafka  soft  nofile  100000
kafka  hard  nofile  100000

# OR for systemd service: Edit /etc/systemd/system/kafka.service
[Service]
LimitNOFILE=100000

# Reload systemd and restart
systemctl daemon-reload
systemctl restart kafka

# Verify after restart
cat /proc/$(pgrep -f kafka.Kafka)/limits | grep "open files"
# Should show: 100000 (soft) and 100000 (hard)

Recommended Limits:

Small cluster (< 10 brokers, < 1000 partitions):
└─ 50,000 file descriptors

Medium cluster (10-50 brokers, 1000-10000 partitions):
└─ 100,000 file descriptors

Large cluster (> 50 brokers, > 10000 partitions):
└─ 200,000+ file descriptors

Formula (rough estimate):
└─ FD limit = (partitions × 3) + (connections × 2) + 10000
   Example: (5000 × 3) + (1000 × 2) + 10000 = 27,000

Disk I/O Scheduler

Why It Matters:

Traditional HDDs need I/O scheduling to minimize disk seeks. SSDs and NVMe drives don't benefit from this — in fact, scheduling adds unnecessary latency!

# Check current scheduler
cat /sys/block/sda/queue/scheduler
# Output: [mq-deadline] kyber none
# [brackets] indicate active scheduler

Recommended Schedulers:

FOR SSDs:
└─ Use: noop or none
└─ Why: SSDs have no seek time, scheduling adds latency

FOR NVMe:
└─ Use: none
└─ Why: NVMe is extremely fast, any scheduling is overhead

FOR HDDs (if you must):
└─ Use: deadline
└─ Why: Minimizes latency while preventing starvation

Setting I/O Scheduler:

# Temporary (until reboot)
echo none > /sys/block/nvme0n1/queue/scheduler  # For NVMe
echo noop > /sys/block/sda/queue/scheduler      # For SSD

# Permanent: Add to kernel boot parameters
# Edit /etc/default/grub:
GRUB_CMDLINE_LINUX="elevator=noop"  # For SSDs
# OR
GRUB_CMDLINE_LINUX="elevator=none"  # For NVMe

# Update grub and reboot
update-grub
reboot

# Verify after reboot
cat /sys/block/sda/queue/scheduler
# Should show: [none] or [noop]

Performance Impact:

BENCHMARK RESULTS (typical):

HDD with deadline scheduler:
└─ Latency: 10-20ms (acceptable for HDD)

SSD with default scheduler (mq-deadline):
└─ Latency: 1-5ms (scheduler overhead)

SSD with noop/none:
└─ Latency: < 1ms (10x improvement!)
└─ Throughput: +30-50% in some workloads

NVMe with none:
└─ Latency: < 0.1ms (critical for low-latency topics)

Memory Swappiness

What is Swappiness?

vm.swappiness controls how aggressively the Linux kernel swaps memory to disk:

├─ 0: Avoid swapping except absolute emergency
├─ 1: Minimal swapping (recommended for Kafka)
├─ 10: Low swapping preference
├─ 60: Default on most systems
└─ 100: Aggressive swapping (bad for Kafka!)

Why Low Swappiness Matters for Kafka:

Kafka relies heavily on OS page cache:
├─ Kafka writes to page cache (in RAM)
├─ OS flushes to disk asynchronously
├─ Kafka reads from page cache (zero-copy)
└─ Page cache = performance critical!

If memory is swapped:
├─ Page cache evicted to make room for swap
├─ Disk I/O increases dramatically
├─ Latency spikes (10-100x slower)
└─ Throughput collapses

Setting Swappiness:

# Check current value
cat /proc/sys/vm/swappiness
# Default: 60 (too high for Kafka)

# Temporary (until reboot)
sysctl vm.swappiness=1

# Permanent: Add to /etc/sysctl.conf
vm.swappiness=1

# Apply immediately
sysctl -p

# Verify
cat /proc/sys/vm/swappiness
# Should show: 1

Recommended Values:

vm.swappiness=1 (RECOMMENDED)
├─ Minimal swapping
├─ Maximizes page cache usage
└─ Emergency swapping still available

vm.swappiness=0 (AGGRESSIVE)
├─ Disable swapping entirely (except OOM)
├─ Risk: Out-of-memory killer may terminate processes
└─ Use only if you're confident in memory sizing

Monitoring Swap Usage:

# Check current swap usage
free -h
#               total        used        free      shared  buff/cache   available
# Mem:           62Gi       30Gi       10Gi       1.0Gi        22Gi        30Gi
# Swap:         8.0Gi          0B       8.0Gi  ← Should be 0!

# If swap usage > 0, investigate memory pressure
vmstat 1 5
# Watch "si" (swap in) and "so" (swap out) columns
# Should be 0 or very low

# Check which processes are swapping
for file in /proc/*/status ; do 
  awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; 
done | sort -k 2 -n -r | head -10

Complete OS Tuning Script

#!/bin/bash
# kafka-os-tuning.sh - Apply OS-level optimizations for Kafka

set -e

echo "=== Kafka OS Tuning Script ==="

# 1. File Descriptor Limits
echo "Setting file descriptor limits..."
cat >> /etc/security/limits.conf << EOF
kafka  soft  nofile  100000
kafka  hard  nofile  100000
EOF

# 2. Swappiness
echo "Setting vm.swappiness=1..."
echo "vm.swappiness=1" >> /etc/sysctl.conf
sysctl -p

# 3. Network Buffers (already in section 4)
echo "Optimizing network buffers..."
cat >> /etc/sysctl.conf << EOF
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728
net.core.somaxconn=4096
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.tcp_window_scaling=1
EOF
sysctl -p

# 4. Disk Scheduler (for SSDs/NVMe)
echo "Setting I/O scheduler..."
DISK=$(lsblk -d -o NAME,TYPE | grep disk | head -1 | awk '{print $1}')
if [[ $DISK == nvme* ]]; then
    echo "Detected NVMe drive, setting scheduler to 'none'"
    echo none > /sys/block/$DISK/queue/scheduler
else
    echo "Detected SSD/HDD, setting scheduler to 'noop'"
    echo noop > /sys/block/$DISK/queue/scheduler 2>/dev/null || echo none > /sys/block/$DISK/queue/scheduler
fi

# Make permanent via grub
sed -i 's/GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="elevator=none /' /etc/default/grub
update-grub

echo "✓ OS tuning complete!"
echo "✓ Reboot required for all changes to take effect"

OS Tuning Verification Checklist

# After applying OS tuning, verify:

# 1. File descriptors
ulimit -n  # Should show 100000+

# 2. Swappiness
cat /proc/sys/vm/swappiness  # Should show 1

# 3. Disk scheduler
cat /sys/block/sda/queue/scheduler  # Should show [none] or [noop]

# 4. No swap usage
free -h | grep Swap  # used should be 0B or minimal

# 5. Network buffers
sysctl net.core.rmem_max  # Should show 134217728

# All checks pass? Ready for production! ✓

6. JVM Tuning

Heap Size Configuration

# Set heap size (4-16 GB recommended)
export KAFKA_HEAP_OPTS="-Xms8g -Xmx8g"

# Smaller brokers (light traffic)
export KAFKA_HEAP_OPTS="-Xms4g -Xmx4g"

# Larger brokers (heavy traffic)
export KAFKA_HEAP_OPTS="-Xms16g -Xmx16g"

RULES:
├─ Set -Xms == -Xmx (avoid resizing)
├─ Don't exceed 32 GB (compressed pointers disabled)
├─ Leave at least 50% RAM for OS page cache
└─ Monitor heap usage and GC time

Garbage Collection Tuning

G1GC (Recommended)

# G1 Garbage Collector (default in Java 9+)
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=20 \
  -XX:InitiatingHeapOccupancyPercent=35 \
  -XX:G1ReservePercent=20"

Benefits:
├─ Low pause times (< 20ms typical)
├─ Predictable latency
├─ Good throughput
└─ Default for Java 9+

Target:
└─ GC pause time < 100ms
└─ < 1% time spent in GC

GC Logging

# Enable GC logging (Java 8)
export KAFKA_GC_LOG_OPTS="-Xloggc:/var/log/kafka/gc.log \
  -XX:+PrintGCDetails \
  -XX:+PrintGCDateStamps \
  -XX:+PrintGCTimeStamps \
  -XX:+UseGCLogFileRotation \
  -XX:NumberOfGCLogFiles=10 \
  -XX:GCLogFileSize=100M"

# Enable GC logging (Java 11+)
export KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=/var/log/kafka/gc.log:time,tags:filecount=10,filesize=100M"

Use GC logs for:
├─ Analyzing GC pause times
├─ Tuning heap size
├─ Identifying memory leaks
└─ Troubleshooting performance issues

JVM Monitoring

# Enable JMX for monitoring
export JMX_PORT=9999
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.rmi.server.hostname=<broker-hostname>"

Monitor:
├─ Heap usage (target: < 70% of max)
├─ GC count and time
├─ Thread count
└─ Class loading stats

6. Capacity Planning

Capacity Planning Methodology

1. DEFINE REQUIREMENTS:
   ├─ Target throughput (MB/s, messages/s)
   ├─ Retention period
   ├─ Replication factor
   ├─ Number of topics/partitions
   └─ Expected growth rate

2. CALCULATE RESOURCES:
   ├─ Network bandwidth
   ├─ Disk capacity
   ├─ Disk throughput
   └─ CPU and memory

3. SIZE CLUSTER:
   └─ Number of brokers needed

4. ADD HEADROOM:
   └─ 30-50% buffer for growth and failures

5. VALIDATE:
   └─ Test with realistic workload

Disk Capacity Calculation

FORMULA:

Disk Capacity = (Daily Ingress Rate) × (Retention Days) × (Replication Factor) × (Growth Buffer)

Example:
├─ Daily ingress: 1 TB/day
├─ Retention: 7 days
├─ Replication factor: 3
├─ Growth buffer: 1.5x (50% buffer)

Calculation:
1 TB × 7 × 3 × 1.5 = 31.5 TB per broker

With 10 brokers:
└─ Total cluster capacity: 315 TB raw
└─ Effective capacity (after replication): 105 TB

Network Bandwidth Calculation

FORMULA:

Network Bandwidth = (Peak Ingress) × (Replication Factor + 1)

The "+1" accounts for consumer traffic

Example:
├─ Peak ingress: 100 MB/s
├─ Replication factor: 3
├─ Consumer traffic: ~1x ingress

Network needed:
100 MB/s × (3 + 1) = 400 MB/s (3.2 Gbps)

Recommendation:
└─ Use 10 Gbps network for production
└─ Ensure network has headroom (< 70% utilization)

Partition Count Planning

FORMULA:

Partitions = max(
    Throughput Target / Single Partition Throughput,
    Max Consumers Needed
)

Single partition throughput:
├─ Typical: 10-20 MB/s
└─ Depends on message size, compression

Example:
├─ Target throughput: 100 MB/s
├─ Single partition: 10 MB/s
└─ Partitions needed: 10

But also consider:
├─ Consumer parallelism (max consumers = partitions)
├─ Total partitions per broker (< 4000 recommended)
└─ Leadership distribution

Tiered Storage for Capacity Optimization

Modern Approach: Reduce Local Disk Requirements

Instead of storing all data on expensive local broker disks, Tiered Storage (Kafka 3.6+) allows offloading older data to cheap remote storage (S3, Azure Blob, GCS).

TRADITIONAL CAPACITY PLANNING:

Requirements:
├─ Daily ingress: 1 TB/day
├─ Retention: 365 days (1 year)
├─ Replication: 3x
└─ Growth buffer: 1.5x

Disk needed per broker:
1 TB × 365 × 3 × 1.5 = 1,642 TB per broker ⚠️
└─ Cost: Very expensive (SSD required for performance)

TIERED STORAGE APPROACH:

Same requirements, but:
├─ Local retention: 7 days (hot data on SSD)
├─ Remote retention: 358 days (cold data on S3)
└─ Total retention: 365 days

Local disk needed per broker:
1 TB × 7 × 3 × 1.5 = 31.5 TB per broker ✓
└─ Cost: 50x less disk per broker!

Remote storage needed (cluster-wide):
1 TB × 358 days × 1 replication (S3 already durable) = 358 TB
└─ Cost: 90% cheaper than SSD (S3 pricing)

Total savings:
├─ Local disk: 1,642 TB → 31.5 TB (98% reduction!)
├─ Cost: ~10x cheaper overall
└─ Performance: Hot data still fast (on local SSD)

Tiered Storage Capacity Formula:

FORMULA:

Local Disk = (Daily Ingress) × (Local Retention Days) × (Replication Factor) × (Buffer)

Remote Storage = (Daily Ingress) × (Remote Retention Days) × (1)
                 # No replication needed (S3/Azure handles durability)

Example Calculation:

Requirements:
├─ Daily ingress: 500 GB/day
├─ Total retention: 180 days (6 months)
├─ Replication factor: 3
├─ Local retention (hot): 14 days
├─ Remote retention (cold): 166 days
└─ Growth buffer: 1.3x

Local Disk per Broker:
500 GB × 14 × 3 × 1.3 = 27.3 TB

Remote Storage (Cluster-Wide):
500 GB × 166 = 83 TB
└─ S3 Standard cost: ~$2,000/month
└─ vs SSD cost for same data: ~$20,000/month

Savings: 90% cost reduction! ✓

Capacity Planning Decision Tree:

WHEN TO USE TIERED STORAGE:

✅ Long retention requirements (> 7 days)
   └─ Example: Compliance, auditing, analytics

✅ Cost is a major concern
   └─ Tiered storage = 10x cheaper for cold data

✅ Consumers rarely access old data
   └─ Acceptable latency: 50-200ms for historical data

✅ Want elastic storage without adding brokers
   └─ Remote storage grows independently

WHEN TO USE TRADITIONAL (LOCAL ONLY):

✅ Short retention (< 7 days)
   └─ All data is "hot", tiered storage adds complexity

✅ All data needs low latency (< 1ms)
   └─ Consumers frequently replay historical data

✅ Kafka version < 3.6
   └─ Tiered storage not available

✅ Want simplicity over cost
   └─ Tiered storage adds operational complexity

Capacity Planning Comparison:

SCENARIO: E-Commerce Platform
├─ Daily traffic: 2 TB/day
├─ Retention requirement: 90 days
├─ Replication factor: 3

┌──────────────────────┬─────────────────┬──────────────────────┐
│ Approach             │ Traditional     │ Tiered Storage       │
├──────────────────────┼─────────────────┼──────────────────────┤
│ Local retention      │ 90 days         │ 7 days (hot)         │
│ Remote retention     │ N/A             │ 83 days (cold)       │
│                      │                 │                      │
│ Local disk/broker    │ 810 TB          │ 63 TB                │
│ Remote storage       │ N/A             │ 166 TB (S3)          │
│                      │                 │                      │
│ Brokers needed (10TB │ 81 brokers ⚠️   │ 7 brokers ✓          │
│ disks each)          │                 │                      │
│                      │                 │                      │
│ Monthly cost         │ ~$200,000       │ ~$25,000             │
│ (estimate)           │                 │ (8x cheaper!)        │
│                      │                 │                      │
│ Broker recovery time │ Hours (810 TB)  │ Minutes (63 TB)      │
│ Operational          │ Complex         │ Moderate             │
│ complexity           │ (many brokers)  │ (tiered config)      │
└──────────────────────┴─────────────────┴──────────────────────┘

Best Practice: Hybrid Approach

RECOMMENDED STRATEGY:

Critical Topics (low-latency, high-value):
├─ Use: Traditional local storage only
├─ Retention: 7-14 days
└─ Example: order-events, payment-transactions

Analytics Topics (high-volume, cold access):
├─ Use: Tiered storage
├─ Local retention: 1-3 days
├─ Remote retention: 30-365 days
└─ Example: clickstream-events, log-aggregation

Audit Topics (compliance, rarely accessed):
├─ Use: Tiered storage
├─ Local retention: 6 hours
├─ Remote retention: 1-7 years
└─ Example: audit-logs, financial-records

RESULT:
├─ Optimize cost where it matters
├─ Keep performance where needed
└─ Meet compliance affordably

Capacity Planning with Tiered Storage Checklist:

☑ Define hot vs cold data boundaries (local.retention.ms)
☑ Calculate local disk for hot data only
☑ Calculate remote storage for cold data
☑ Factor in S3/object storage costs ($0.023/GB/month)
☑ Plan for egress costs if consumers access old data frequently
☑ Consider lifecycle policies (S3 Glacier for very old data)
☑ Monitor remote fetch latency (should be < 200ms)
☑ Test disaster recovery (what if S3 unavailable?)
☑ Document which topics use tiered storage
☑ Review and adjust local retention based on access patterns

7. Scaling Strategies

Horizontal Scaling (Add Brokers)

WHEN TO SCALE HORIZONTALLY:

├─ Disk capacity exhausted
├─ Network saturated
├─ CPU overloaded on all brokers
├─ Need higher throughput
└─ Need more fault tolerance

PROCESS:
1. Add new brokers to cluster
2. Reassign partitions to new brokers
3. Wait for replication to complete
4. Verify load distributed

BENEFITS:
├─ Increases total capacity
├─ Improves fault tolerance
├─ Distributes load
└─ Kafka's native scaling method

CHALLENGES:
└─ Partition reassignment takes time
└─ Temporary increased load during rebalance

Vertical Scaling (Bigger Machines)

WHEN TO SCALE VERTICALLY:

├─ Single broker bottleneck (not cluster-wide)
├─ Need more disk per broker
├─ Need faster network per broker
└─ Short-term quick fix

PROCESS:
1. Decommission broker (move partitions)
2. Replace with larger machine
3. Add back to cluster
4. Reassign partitions

BENEFITS:
├─ Fewer brokers to manage
├─ Less network chatter
└─ Simpler operations

CHALLENGES:
├─ Limited by single-machine limits
├─ More expensive per unit capacity
└─ Harder to scale incrementally

RECOMMENDATION:
└─ Prefer horizontal scaling (Kafka's design)

8. Performance Testing

Benchmarking Tools

kafka-producer-perf-test

# Benchmark producer throughput
kafka-producer-perf-test.sh \
  --topic test-topic \
  --num-records 10000000 \
  --record-size 1000 \
  --throughput -1 \
  --producer-props \
    bootstrap.servers=localhost:9092 \
    acks=all \
    compression.type=lz4

# Output:
# 500000 records sent, 100000 records/sec (95.37 MB/sec)
# Average latency: 5.00 ms
# Max latency: 100.00 ms

kafka-consumer-perf-test

# Benchmark consumer throughput
kafka-consumer-perf-test.sh \
  --topic test-topic \
  --messages 10000000 \
  --bootstrap-server localhost:9092 \
  --group test-group \
  --threads 1

# Output:
# 10000000 records consumed in 60 seconds
# 166666.67 records/sec (158.63 MB/sec)

End-to-End Testing

# Test producer → broker → consumer latency
kafka-run-class.sh kafka.tools.EndToEndLatency \
  localhost:9092 \
  test-topic \
  10000 \
  1 \
  1000

# Output:
# Avg latency: 5.0 ms
# Percentiles: 50th: 3 ms, 99th: 15 ms, 99.9th: 50 ms

9. Common Performance Issues and Solutions

Issue 1: High Producer Latency

SYMPTOM:
└─ request-latency-avg > 100ms

INVESTIGATION:
1. Check broker TotalTimeMs metric
2. Check broker CPU, disk I/O, network
3. Check if acks=all (replication latency)
4. Check if batching enabled (linger.ms)

SOLUTIONS:
├─ If broker overloaded: Add brokers
├─ If disk slow: Use SSDs
├─ If replication slow: Check follower brokers
├─ If network slow: Enable compression
└─ If batching too aggressive: Tune linger.ms

Issue 2: Consumer Lag Growing

SYMPTOM:
└─ Consumer lag increasing steadily

INVESTIGATION:
1. Check consumer processing time
2. Check consumer fetch settings
3. Check partition count vs consumers
4. Check broker throughput capacity

SOLUTIONS:
├─ Slow processing: Optimize app, add consumers
├─ Too few consumers: Add consumers (up to partition count)
├─ Fetch settings: Tune fetch.min.bytes, fetch.max.wait.ms
├─ Broker slow: Scale cluster
└─ Partition bottleneck: Increase partitions (if possible)

Issue 3: Disk Space Filling Up

SYMPTOM:
└─ Disk usage > 85%

INVESTIGATION:
1. Check topic retention settings
2. Check log cleanup running
3. Check if compaction working (if applicable)
4. Identify largest topics

SOLUTIONS:
├─ Reduce retention.ms for non-critical topics
├─ Enable compression (reduces disk usage)
├─ Add disk capacity
├─ Scale brokers horizontally
└─ Review data retention policies

10. Key Takeaways

You've learned:

✅ Performance tuning is iterative: measure, identify, tune, measure
✅ Common bottlenecks: disk I/O, network, CPU, memory, coordination
✅ Thread pools control concurrency (network, I/O, replication)
✅ SSDs provide 10-100x performance over HDDs
✅ JVM heap should be 4-16 GB with G1GC
✅ Capacity planning calculates disk, network, partitions needed
✅ Horizontal scaling is Kafka's native scaling method
✅ Compression reduces network and disk by 3-5x
✅ Benchmarking validates performance improvements
✅ Monitoring is essential for identifying bottlenecks

11. Performance Tuning Checklist

BROKER CONFIGURATION:

☑ num.network.threads = CPU cores
☑ num.io.threads = 2x disks (HDD) or 8-16 (SSD)
☑ socket.send/receive.buffer.bytes = 1 MB (fast network)
☑ compression.type = producer (or lz4/zstd)
☑ log.flush.* = disabled (rely on replication)

INFRASTRUCTURE:

☑ SSDs for data directories
☑ XFS filesystem
☑ noatime mount option
☑ 10 Gbps network
☑ Kernel network tuning applied

JVM:

☑ Heap size: 4-16 GB
☑ -Xms == -Xmx
☑ G1GC enabled
☑ GC logging enabled
☑ < 1% time in GC

CAPACITY:

☑ Disk capacity planned for retention + growth
☑ Network headroom > 30%
☑ Partition count appropriate for throughput
☑ Broker count allows for 1-2 failures

Summary

Performance optimization is about removing bottlenecks:

Tuning Process =
├─ Identify the bottleneck (disk, network, CPU)
├─ Apply targeted fix
├─ Measure improvement
└─ Repeat for next bottleneck

Capacity Planning =
├─ Understand requirements (throughput, retention)
├─ Calculate resources (disk, network, partitions)
├─ Size cluster appropriately
└─ Add 30-50% headroom for growth

Kafka Performance =
├─ Use SSDs (biggest impact)
├─ Tune JVM carefully (avoid over-tuning)
├─ Enable compression (reduces I/O)
├─ Scale horizontally (Kafka's strength)
└─ Monitor continuously (catch issues early)

Remember: Performance tuning is not one-time. As traffic grows and patterns change, revisit tuning decisions. Always measure before and after changes.

Next up: Module 6.1 — Quotas, Throttling & Resource Management — Protect your cluster from abuse and ensure fair resource usage.