KafkaIQ

10.5.3 – Network Stack Optimization

Module: M10 – Advanced Topics & Architecture
Section: 10.5 – Advanced Performance Tuning Layer
Goal: Optimize Linux network stack for high-throughput Kafka clusters

Introduction

Kafka is network-intensive. Proper network tuning improves throughput, reduces latency, and prevents packet loss.

TCP Tuning

Socket Buffer Sizes

Purpose: Control how much data can be buffered before send/receive.

Defaults: Too small for high-throughput Kafka (87380 bytes).

Recommended (/etc/sysctl.conf):

net.core.rmem_default = 262144    # 256 KB default receive buffer
net.core.rmem_max = 134217728     # 128 MB max receive buffer
net.core.wmem_default = 262144    # 256 KB default send buffer
net.core.wmem_max = 134217728     # 128 MB max send buffer

net.ipv4.tcp_rmem = 4096 87380 134217728   # min, default, max (receive)
net.ipv4.tcp_wmem = 4096 87380 134217728   # min, default, max (send)

Apply:

sysctl -p

TCP Window Scaling

Purpose: Allow TCP windows > 64 KB (required for high-bandwidth, high-latency links).

Enable (/etc/sysctl.conf):

net.ipv4.tcp_window_scaling = 1

Why: Cross-region replication (100ms RTT) needs large windows for throughput.

TCP Congestion Control

Purpose: Control how TCP reacts to congestion.

Recommended: bbr (Bottleneck Bandwidth and RTT).

Set (/etc/sysctl.conf):

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Why BBR?:

  • Better throughput on high-latency links (e.g., cross-region)
  • Lower latency (fewer retransmits)

Verify:

sysctl net.ipv4.tcp_congestion_control

Expected: bbr

Connection Tuning

Connection Backlog

Purpose: Queue size for pending connections.

Defaults: Too small for high connection rate.

Recommended (/etc/sysctl.conf):

net.core.somaxconn = 4096        # Max connection backlog
net.ipv4.tcp_max_syn_backlog = 8192  # SYN queue size

Apply:

sysctl -p

Connection Reuse

Purpose: Reuse TIME_WAIT connections faster.

Recommended (/etc/sysctl.conf):

net.ipv4.tcp_tw_reuse = 1        # Allow reuse of TIME_WAIT sockets
net.ipv4.tcp_fin_timeout = 30    # Reduce FIN_WAIT timeout (default: 60)

Why: Kafka opens many short-lived connections (producer retries, consumer rebalancing).

Network Interface Tuning

Ring Buffer Size

Purpose: NIC hardware buffer for incoming/outgoing packets.

Check Current:

ethtool -g eth0

Increase:

ethtool -G eth0 rx 4096 tx 4096

Persist (add to /etc/rc.local or systemd).

Offloading Features

Purpose: Offload TCP/IP processing to NIC (reduces CPU usage).

Enable:

ethtool -K eth0 tso on      # TCP Segmentation Offload
ethtool -K eth0 gso on      # Generic Segmentation Offload
ethtool -K eth0 gro on      # Generic Receive Offload

Verify:

ethtool -k eth0 | grep segmentation

Firewall and iptables

Disable Firewall (if secure network)

Production: Use security groups/VPCs, not host firewall.

systemctl stop firewalld
systemctl disable firewalld

Why: Firewall adds latency (~100μs per packet).

If Firewall Required

Allow Kafka Ports:

iptables -A INPUT -p tcp --dport 9092 -j ACCEPT  # Broker
iptables -A INPUT -p tcp --dport 9999 -j ACCEPT  # JMX

Network Monitoring

Key Metrics

1. Network Throughput

sar -n DEV 1 10

Output:

IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s
eth0   250000.00 250000.00 2000000.00 2000000.00

Alert: If rxkB/s or txkB/s near line rate (e.g., 1.25 GB/sec for 10 Gbps).

2. Packet Loss

netstat -s | grep -i "packet loss"

Healthy: 0 losses
Alert: > 0 (network overload or misconfiguration)

3. Connection Count

ss -s

Output:

TCP:   1234 (estab 1200, closed 20, orphaned 0)

Alert: If estab > 10,000 (may need to increase somaxconn).

Tuning for Cross-Region Replication

Bandwidth-Delay Product (BDP)

Formula:

BDP = Bandwidth × RTT

Example:
  Bandwidth = 1 Gbps = 125 MB/sec
  RTT = 100ms = 0.1 sec
  BDP = 125 MB/sec × 0.1 sec = 12.5 MB

TCP Buffer Size = BDP (to fully utilize bandwidth).

Set:

net.ipv4.tcp_rmem = 4096 87380 13421772  # 12.8 MB max
net.ipv4.tcp_wmem = 4096 87380 13421772

Key Takeaways

  1. Socket Buffers: Increase to 128 MB (net.core.rmem_max, net.core.wmem_max).
  2. TCP Window Scaling: Enable for high-bandwidth links (net.ipv4.tcp_window_scaling=1).
  3. Congestion Control: Use BBR (net.ipv4.tcp_congestion_control=bbr).
  4. Connection Backlog: Increase to 4096 (net.core.somaxconn=4096).
  5. Connection Reuse: Enable TIME_WAIT reuse (net.ipv4.tcp_tw_reuse=1).
  6. NIC Offloading: Enable TSO, GSO, GRO (ethtool -K).
  7. Cross-Region: Set TCP buffer = BDP (bandwidth × RTT).

What's Next?

Next Module: 10.6 – Graceful Lifecycle and Testing Strategies

References:

  • Broker configs → M6: Operations
  • Multi-datacenter → 10.2