KafkaIQ

10.5.2 – OS and Disk Tuning

Module: M10 – Advanced Topics & Architecture
Section: 10.5 – Advanced Performance Tuning Layer
Goal: Optimize operating system and disk subsystem for Kafka

Introduction

Kafka relies heavily on the OS page cache and disk I/O. Proper OS/disk tuning improves throughput and reduces latency.

Page Cache Tuning

What Is Page Cache?

Purpose: OS caches recently read/written data in RAM (avoid disk reads).

Why Important for Kafka?:

  • Kafka doesn't cache messages in JVM heap
  • Recent messages served from page cache (zero-copy)
  • Page cache size = Total RAM - JVM heap - OS overhead

Example:

Server: 64 GB RAM
JVM Heap: 6 GB
OS Overhead: 2 GB
Page Cache: 64 - 6 - 2 = 56 GB

Maximizing Page Cache

Rule: More RAM = larger page cache = better performance.

Recommendation: 64-128 GB RAM per broker (general purpose).

Page Cache Monitoring

Command:

free -h

Output:

              total        used        free      shared  buff/cache   available
Mem:          64Gi        8Gi        10Gi       100Mi        45Gi        54Gi

Key: buff/cache = page cache size (45 GB in example).

Healthy: Page cache ≥ 50% of total RAM.

File Descriptor Limits

Why Important?

Kafka opens many files:

  • Partition log files (.log, .index, .timeindex)
  • Network connections (sockets)
  • Each partition = 3-4 open files

Example:

1000 partitions × 4 files = 4,000 open files
+ 1,000 network connections = 5,000 total

Default Linux Limit: 1,024 (too low!).

Increasing File Descriptor Limit

Edit /etc/security/limits.conf:

kafka soft nofile 100000
kafka hard nofile 100000

Verify:

ulimit -n

Expected: 100,000

Disk I/O Tuning

Disk Scheduler

Purpose: Controls order of disk I/O requests.

Recommended: deadline or noop (for SSD/NVMe).

Why:

  • cfq (default) optimizes for interactive workloads (not Kafka)
  • deadline optimizes for throughput (sequential writes)
  • noop for SSD/NVMe (no mechanical seek time)

Check Current Scheduler:

cat /sys/block/sda/queue/scheduler

Output:

noop [deadline] cfq

Change Scheduler:

echo deadline > /sys/block/sda/queue/scheduler

Persist (add to /etc/rc.local or systemd):

echo deadline > /sys/block/sda/queue/scheduler

Filesystem Selection

Recommended: XFS (default for Kafka).

Why XFS?:

  • Better performance for large files
  • Better allocation groups (parallel I/O)
  • Industry standard for Kafka

Alternative: ext4 (acceptable, but XFS preferred).

Mount Options (/etc/fstab):

/dev/sda1 /var/kafka/data xfs noatime,nodiratime 0 2

Flags:

  • noatime: Don't update file access time (reduces writes)
  • nodiratime: Don't update directory access time

Filesystem Flush Interval

Purpose: Control how often dirty pages flushed to disk.

Default: 30 seconds (too aggressive for Kafka).

Tuning (/etc/sysctl.conf):

vm.dirty_background_ratio = 5   # Start flushing at 5% dirty memory
vm.dirty_ratio = 10              # Block writes at 10% dirty memory
vm.dirty_writeback_centisecs = 100  # Flush interval: 1 second
vm.dirty_expire_centisecs = 3000    # Max age before flush: 30 seconds

Why:

  • Allows larger write buffers
  • Reduces flush frequency (improves throughput)

Apply:

sysctl -p

Swappiness

What Is Swappiness?

Purpose: Controls how aggressively OS swaps memory to disk.

Default: 60 (too aggressive for Kafka).

Problem: Page cache evicted to swap → disk reads instead of cache hits.

Tuning Swappiness

Recommended: 1 (only swap to avoid OOM).

Set (/etc/sysctl.conf):

vm.swappiness = 1

Apply:

sysctl -p

NUMA Awareness

What Is NUMA?

Non-Uniform Memory Access: Multi-socket servers where each CPU has local memory.

Problem: Cross-socket memory access is slower (latency penalty).

Disabling NUMA (Simplified Approach)

Option 1: Disable NUMA in BIOS (if available).

Option 2: Use numactl to bind Kafka to one NUMA node.

numactl --cpunodebind=0 --membind=0 /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties

Why: Keeps Kafka on one NUMA node (avoids cross-socket latency).

Disk Performance Testing

Sequential Write Test

Tool: dd

dd if=/dev/zero of=/var/kafka/data/testfile bs=1M count=10000 oflag=direct

Expected (SSD): 300-500 MB/sec
Expected (NVMe): 1-3 GB/sec

Random I/O Test

Tool: fio

fio --name=random-write --ioengine=libaio --rw=randwrite --bs=4k \
    --size=4G --numjobs=4 --runtime=60 --time_based --group_reporting \
    --directory=/var/kafka/data

Expected (SSD): 50k-100k IOPS
Expected (NVMe): 200k-500k IOPS

Key Takeaways

  1. Page Cache: Maximize RAM (64-128 GB), page cache = RAM - JVM heap - OS overhead.
  2. File Descriptors: Increase limit to 100,000 (ulimit -n).
  3. Disk Scheduler: Use deadline (HDD) or noop (SSD/NVMe).
  4. Filesystem: XFS with noatime,nodiratime mount options.
  5. Flush Interval: vm.dirty_background_ratio=5, vm.dirty_ratio=10.
  6. Swappiness: Set to 1 (avoid swapping page cache).
  7. NUMA: Disable in BIOS or bind Kafka to one NUMA node.

What's Next?

Next Module: 10.5.3 – Network Stack Optimization

References:

  • Broker configs → M6: Operations