8.1 Batching & Buffering

Module Overview

Goal: Master batching and buffering for optimal producer performance.

What You'll Learn:

• How batch.size affects throughput and latency
• Tuning linger.ms for batching efficiency
• Memory buffer (buffer.memory) configuration
• Batching strategies for different workloads
• Monitoring batch metrics
• Common batching anti-patterns

Prerequisites:

• Module 7.1 (Producer Internals) — understand RecordAccumulator and Sender thread

1. Batching Fundamentals

Why Batching Matters

Without batching:

send(record1) → Network request
send(record2) → Network request
send(record3) → Network request

Result: 3 network requests (high overhead)

With batching:

send(record1) ─┐
send(record2) ─┼→ [Batch] → Single network request
send(record3) ─┘

Result: 1 network request (3x more efficient)

Benefits of Batching

✅ Higher Throughput — More records per request
✅ Better Compression — Compress multiple records together
✅ Lower Network Overhead — Fewer TCP round-trips
✅ More Efficient Broker Processing — Fewer writes
✅ Better Resource Utilization — Amortize fixed costs

2. The batch.size Configuration

What is batch.size?

Definition: Maximum size (in bytes) of a batch before it's sent.

props.put("batch.size", 16384); // 16 KB (default)

How It Works

Producer accumulates records into batch:

Record 1: 1 KB  → Total: 1 KB
Record 2: 2 KB  → Total: 3 KB
Record 3: 3 KB  → Total: 6 KB
Record 4: 4 KB  → Total: 10 KB
Record 5: 5 KB  → Total: 15 KB
Record 6: 6 KB  → Total: 21 KB (exceeds batch.size!)

Action: Send batch with records 1-5 (15 KB)
Start new batch with record 6

Key Point: batch.size is a threshold, not a guarantee.

Tuning batch.size

Small batch.size (e.g., 1 KB)

props.put("batch.size", 1024); // 1 KB

Effects:

• ✅ Lower latency — Batches fill quickly
• ❌ Lower throughput — More frequent sends, more overhead
• ❌ Less compression — Smaller batches compress less efficiently

Use When:

• Latency is critical
• Low message rate (batches won't fill anyway)
• Real-time applications

Large batch.size (e.g., 128 KB)

props.put("batch.size", 131072); // 128 KB

Effects:

• ✅ Higher throughput — Fewer, larger requests
• ✅ Better compression — More data to compress
• ❌ Higher latency — Wait longer to fill batch
• ❌ More memory — Larger buffer allocations

Use When:

• Throughput is critical
• High message rate
• Large messages
• Batch processing

Default batch.size (16 KB)

props.put("batch.size", 16384); // 16 KB (default)

Good starting point for most use cases.

Calculating Optimal batch.size

Formula:

batch.size ≈ 10 × average_message_size

Or:

batch.size ≈ average_message_size × messages_per_batch_target

Examples:

3. The linger.ms Configuration

What is linger.ms?

Definition: How long (in milliseconds) to wait for more records before sending a batch.

props.put("linger.ms", 10); // Wait 10ms

How It Works

Config: linger.ms = 10, batch.size = 16 KB

Timeline:
0ms    │ Record 1 arrives (1 KB) → Start linger timer
1ms    │ Record 2 arrives (2 KB) → Total: 3 KB
5ms    │ Record 3 arrives (3 KB) → Total: 6 KB
10ms   │ Linger timer expires!
       │ → Send batch (6 KB, 3 records)
       │ Even though batch.size (16 KB) not reached

Key Point: Batch sent when either condition met:

• Batch size ≥ batch.size OR
• Time since first record ≥ linger.ms

Tuning linger.ms

linger.ms = 0 (Default)

props.put("linger.ms", 0); // Send immediately

Effects:

• ✅ Lowest latency — No waiting
• ❌ Lower throughput — Batches may be small
• ❌ More overhead — More frequent sends

Behavior: Send as soon as Sender thread is ready (not literally instant, but no artificial delay).

Use When:

• Ultra-low latency required
• Interactive applications
• Real-time updates

linger.ms > 0 (e.g., 10ms)

props.put("linger.ms", 10); // Wait 10ms

Effects:

• ✅ Higher throughput — Larger batches
• ✅ Better compression — More records per batch
• ❌ Higher latency — Artificial delay

Use When:

• Throughput more important than latency
• High message rate
• Batch processing
• Can tolerate 10-100ms latency

Recommended Values

linger.ms with High Load

Important: At high message rates, linger.ms has minimal impact.

Scenario: 10,000 msg/sec, avg message size 1 KB

Without linger:
  Batches fill to batch.size (16 KB) in < 1ms
  → linger.ms irrelevant (batches already full)

With linger.ms = 10:
  Same result — batches fill before linger expires

Takeaway: linger.ms matters most at low to medium message rates.

4. The buffer.memory Configuration

What is buffer.memory?

Definition: Total memory (in bytes) for the RecordAccumulator buffer.

props.put("buffer.memory", 33554432); // 32 MB (default)

How It Works

buffer.memory = 32 MB

Current usage:
  Partition 0: 3 batches × 16 KB = 48 KB
  Partition 1: 2 batches × 16 KB = 32 KB
  Partition 2: 5 batches × 16 KB = 80 KB
  ...
  
Total: 10 MB used
Available: 22 MB

When buffer full:

producer.send(record); // Blocks up to max.block.ms

Tuning buffer.memory

Small Buffer (e.g., 16 MB)

props.put("buffer.memory", 16777216); // 16 MB

Effects:

• ✅ Lower memory usage
• ❌ More frequent blocking (if high message rate)
• ❌ Backpressure under load

Use When:

• Low message rate
• Memory constrained
• Want natural backpressure

Large Buffer (e.g., 64 MB)

props.put("buffer.memory", 67108864); // 64 MB

Effects:

• ✅ More headroom under spiky load
• ✅ Less blocking
• ❌ Higher memory usage
• ⚠️ May hide issues (broker slow, network issues)

Use When:

• High message rate
• Spiky traffic
• Want to absorb bursts

Calculating Buffer Size

Formula:

buffer.memory ≥ batch.size × partitions × batches_per_partition

Example:

batch.size = 16 KB
partitions = 10 (across all topics)
batches_per_partition = 2 (buffer 2 batches per partition)

buffer.memory = 16 KB × 10 × 2 = 320 KB (minimum)

Recommended: 10-20x minimum
buffer.memory = 320 KB × 10 = 3.2 MB

Default 32 MB is usually sufficient for most use cases.

5. Batching Strategies

Strategy 1: Latency-Optimized

Goal: Minimize latency, accept lower throughput.

props.put("batch.size", 8192);         // 8 KB (small)
props.put("linger.ms", 0);             // No waiting
props.put("buffer.memory", 16777216);  // 16 MB
props.put("compression.type", "none"); // No compression overhead

Use Case: Real-time applications, interactive systems.

Strategy 2: Throughput-Optimized

Goal: Maximize throughput, accept higher latency.

props.put("batch.size", 65536);        // 64 KB (large)
props.put("linger.ms", 20);            // Wait 20ms
props.put("buffer.memory", 67108864);  // 64 MB
props.put("compression.type", "lz4");  // Compress for efficiency

Use Case: Batch processing, analytics, high-volume ingestion.

Strategy 3: Balanced

Goal: Balance latency and throughput.

props.put("batch.size", 16384);        // 16 KB (default)
props.put("linger.ms", 10);            // Wait 10ms
props.put("buffer.memory", 33554432);  // 32 MB (default)
props.put("compression.type", "lz4");  // Efficient compression

Use Case: General-purpose applications.

6. Monitoring Batch Performance

Key Metrics

1. Batch Size Metrics

kafka.producer:type=producer-metrics,client-id=<id>,name=batch-size-avg
  → Average batch size (bytes)

kafka.producer:type=producer-metrics,client-id=<id>,name=batch-size-max
  → Maximum batch size

kafka.producer:type=producer-metrics,client-id=<id>,name=records-per-request-avg
  → Average records per request

What to look for:

• batch-size-avg << batch.size → Batches not filling (increase linger.ms or decrease batch.size)
• records-per-request-avg = 1 → No batching (problem!)

2. Buffer Utilization

kafka.producer:type=producer-metrics,client-id=<id>,name=buffer-available-bytes
  → Free buffer space

kafka.producer:type=producer-metrics,client-id=<id>,name=buffer-exhausted-rate
  → Rate of buffer exhaustion (backpressure)

What to look for:

• buffer-available-bytes consistently low → Increase buffer.memory or improve throughput
• buffer-exhausted-rate > 0 → Producer blocking (backpressure)

3. Request Metrics

kafka.producer:type=producer-metrics,client-id=<id>,name=request-rate
  → Requests per second

kafka.producer:type=producer-metrics,client-id=<id>,name=request-size-avg
  → Average request size

What to look for:

• request-rate very high → Poor batching
• request-size-avg << batch.size → Increase linger.ms

Example: Diagnosing Batching Issues

Observed metrics:
  batch-size-avg: 2 KB
  records-per-request-avg: 2
  batch.size config: 16 KB
  linger.ms config: 0

Diagnosis: Batches not filling (only 2 records, 2 KB)

Solution: Increase linger.ms to allow more batching
  props.put("linger.ms", 10);

Result after change:
  batch-size-avg: 14 KB
  records-per-request-avg: 14
  → Much better batching!

7. Common Anti-Patterns

❌ Anti-Pattern 1: Tiny Batches

props.put("batch.size", 100);  // Too small!
props.put("linger.ms", 0);

Problem: One record per request (no batching benefit).

Fix:

props.put("batch.size", 16384);  // Reasonable size
props.put("linger.ms", 10);      // Allow batching

❌ Anti-Pattern 2: Excessive Linger

props.put("linger.ms", 5000); // 5 seconds!

Problem: Adds 5s latency even for first record in batch.

Fix:

props.put("linger.ms", 10); // 10ms is usually sufficient

❌ Anti-Pattern 3: Undersized Buffer

props.put("buffer.memory", 1048576);  // 1 MB (too small)
props.put("batch.size", 65536);       // 64 KB batches
// Only 16 batches fit in buffer!

Problem: Frequent blocking under load.

Fix:

props.put("buffer.memory", 33554432); // 32 MB (default)

❌ Anti-Pattern 4: Ignoring Metrics

Problem: Not monitoring batch effectiveness.

Fix: Monitor these metrics:

• batch-size-avg
• records-per-request-avg
• buffer-available-bytes

8. Practical Examples

Example 1: High-Volume Logging

Requirements:

• 100,000 messages/sec
• Average message: 500 bytes
• Latency: Acceptable (< 100ms)

Configuration:

props.put("batch.size", 65536);       // 64 KB (130 msgs)
props.put("linger.ms", 20);           // Wait 20ms for batching
props.put("buffer.memory", 67108864); // 64 MB
props.put("compression.type", "lz4"); // Compress logs

Result:

• Batches fill to ~64 KB (130 messages)
• ~770 requests/sec (vs 100,000 without batching)
• 99th percentile latency: ~25ms

Example 2: Real-Time User Events

Requirements:

• 1,000 messages/sec
• Average message: 1 KB
• Latency: Critical (< 10ms)

Configuration:

props.put("batch.size", 8192);        // 8 KB (8 msgs)
props.put("linger.ms", 1);            // Minimal wait
props.put("buffer.memory", 16777216); // 16 MB
props.put("compression.type", "none"); // No compression overhead

Result:

• Small batches (2-8 messages)
• Low latency (~5ms p99)
• Still get some batching benefit

Example 3: Batch ETL Process

Requirements:

• Process files every hour
• 10 million messages per file
• Average message: 2 KB
• Latency: Not important

Configuration:

props.put("batch.size", 131072);       // 128 KB (64 msgs)
props.put("linger.ms", 100);           // Wait 100ms
props.put("buffer.memory", 134217728); // 128 MB
props.put("compression.type", "zstd"); // Max compression

Result:

• Very large batches (~128 KB)
• Maximum compression
• Highest throughput

9. Tuning Workflow

Step-by-Step Tuning Process

1. Baseline Measurement

Run with defaults:

batch.size = 16 KB
linger.ms = 0
buffer.memory = 32 MB

Measure:

• Throughput (msgs/sec)
• Latency (p50, p99)
• batch-size-avg
• records-per-request-avg

2. Identify Bottleneck

If batch-size-avg is low:

• Batches not filling
• → Increase linger.ms or decrease batch.size

If buffer-exhausted-rate > 0:

• Buffer too small or broker slow
• → Increase buffer.memory or investigate broker

If latency too high:

• Over-batching
• → Decrease linger.ms or batch.size

3. Adjust One Parameter

Change one config, measure again:

// Try increasing linger
props.put("linger.ms", 10);

Measure impact:

• Did batch-size-avg increase?
• Did throughput improve?
• What happened to latency?

4. Iterate

Repeat until you hit your target metrics.

Summary

Key Takeaways

✅ Batching improves throughput by amortizing overhead
✅ batch.size controls maximum batch size (threshold)
✅ linger.ms controls how long to wait for more records
✅ buffer.memory controls total buffer space
✅ Small batches → Low latency, lower throughput
✅ Large batches → High throughput, higher latency
✅ Monitor metrics to understand batching effectiveness
✅ Tune iteratively — measure, adjust, repeat

Configuration Quick Reference

Tuning Checklist

• Understand workload (message rate, size, latency requirements)
• Start with defaults, measure baseline
• Monitor batch metrics (batch-size-avg, records-per-request-avg)
• Adjust linger.ms if batches not filling
• Adjust batch.size for message size
• Ensure buffer.memory sufficient for workload
• Test under peak load
• Document final configuration and rationale

What's Next

• Module 8.2: Latency vs Throughput Tuning
  Understand and optimize the latency/throughput trade-off

• Module 8.3: Compression Strategies
  Choose and tune compression for your workload

Previous Module: ← 7.1 - Producer Internals
Next Module: 8.2 - Latency vs Throughput Tuning →