8.3 Compression Strategies

Module Overview

Goal: Choose and tune compression for optimal performance and cost.

What You'll Learn:

• How compression works in Kafka producers
• Compression algorithms: gzip, snappy, lz4, zstd
• Trade-offs: CPU vs network vs disk
• When to use compression
• Measuring compression effectiveness
• Tuning compression for your workload

Prerequisites:

• Module 7.1 (Producer Internals) — understand batching
• Module 8.1 (Batching & Buffering)

1. How Compression Works in Kafka

Compression Lifecycle

Application Thread                 Sender Thread
      │                                 │
      ├─ send(record1)                  │
      ├─ send(record2)                  │
      ├─ send(record3)                  │
      │                                 │
      │  [Records added to batch, UNCOMPRESSED]
      │                                 │
      │                                 ├─ Fetch batch
      │                                 ├─ Compress entire batch
      │                                 │  (gzip/snappy/lz4/zstd)
      │                                 └─ Send compressed bytes
                                        
Broker                              Consumer
   │                                    │
   ├─ Store compressed                 │
   │  (saves disk space)                │
   │                                    │
   │──────── compressed data ──────────→│
                                        │
                                        ├─ Decompress
                                        └─ Process records

Key Points:

• Compression happens per batch (not per record)
• Producer compresses, broker stores compressed, consumer decompresses
• Larger batches → Better compression ratios

Configuration

props.put("compression.type", "lz4"); // or "none", "gzip", "snappy", "zstd"

2. Compression Algorithms

Overview Table

Algorithm 1: None (No Compression)

props.put("compression.type", "none");

Characteristics:

• No CPU overhead
• Maximum network bandwidth usage
• Fastest (no processing)

Use When:

• Messages already compressed (images, videos)
• Network bandwidth abundant
• CPU constrained
• Ultra-low latency required

Algorithm 2: LZ4

props.put("compression.type", "lz4");

Characteristics:

• Excellent balance: speed vs compression
• Very low CPU overhead
• Good compression ratio (2-3:1 for text)
• Fast decompression

Use When:

• Default choice for most use cases
• Need good compression without CPU cost
• Mixed workload (text + binary)

Benchmark (1MB text data):

• Compression time: ~10ms
• Compressed size: ~350 KB (2.8:1)
• Decompression time: ~5ms

Algorithm 3: Snappy

props.put("compression.type", "snappy");

Characteristics:

• Similar to LZ4 (slightly older)
• Fast compression/decompression
• Good compression ratio (2-3:1)
• Widely supported

Use When:

• Need proven, stable compression
• Compatibility with older Kafka versions
• Similar performance to LZ4

Benchmark (1MB text data):

• Compression time: ~12ms
• Compressed size: ~400 KB (2.5:1)
• Decompression time: ~6ms

Algorithm 4: GZIP

props.put("compression.type", "gzip");

Characteristics:

• Best compression ratio (3-5:1)
• Slowest compression
• High CPU usage
• Slow decompression

Use When:

• Network bandwidth very limited
• Storage cost is critical
• Latency not important
• CPU abundant

Benchmark (1MB text data):

• Compression time: ~80ms
• Compressed size: ~250 KB (4:1)
• Decompression time: ~30ms

Algorithm 5: ZSTD (Zstandard)

props.put("compression.type", "zstd");

Characteristics:

• Best balance: compression ratio + speed
• High compression ratio (similar to gzip)
• Much faster than gzip
• Tunable compression level (see below)

Use When:

• Need high compression without gzip slowness
• Kafka 2.1.0+
• Best choice if version supports it

Benchmark (1MB text data):

• Compression time: ~25ms
• Compressed size: ~280 KB (3.6:1)
• Decompression time: ~10ms

Tuning Compression Level (GZIP and ZSTD)

For gzip and zstd, you can tune the compression level to balance speed vs. ratio.

GZIP Compression Level

props.put("compression.type", "gzip");
props.put("compression.gzip.level", 6); // Range: 1-9 (default: 6)

Levels:

• 1: Fastest compression, lowest ratio (~2:1)
• 6: Default balance (~4:1)
• 9: Best compression, slowest (~4.5:1)

Example:

// Fast gzip (less CPU, lower ratio)
props.put("compression.gzip.level", 1);

// Default gzip (balanced)
props.put("compression.gzip.level", 6);

// Maximum gzip (high CPU, best ratio)
props.put("compression.gzip.level", 9);

When to tune:

• Use level 1-3 if CPU is constrained
• Use level 6 (default) for most cases
• Use level 9 if network/storage is very expensive

ZSTD Compression Level

props.put("compression.type", "zstd");
props.put("compression.zstd.level", 3); // Range: -5 to 22 (default: 3)

Levels:

• -5 to 0: Ultra-fast, lower ratio (rarely used)
• 1-3: Fast, good ratio (recommended)
• 4-9: Slower, better ratio
• 10-22: Very slow, maximum ratio (rarely useful)

Example:

// Fast zstd (low latency)
props.put("compression.zstd.level", 1);

// Default zstd (balanced)
props.put("compression.zstd.level", 3);

// High compression zstd
props.put("compression.zstd.level", 9);

Recommendation:

• Level 1-3: Best for real-time systems (default 3 is great)
• Level 4-6: If you need more compression and can tolerate latency
• Level 7+: Rarely useful (gzip-level slowness)

Comparison: GZIP vs ZSTD Levels

Key Insight: ZSTD level 3 achieves similar compression to GZIP level 6, but 3x faster.

When NOT to Tune Compression Level

Don't tune if:

• You're using lz4 or snappy (no tunable levels)
• You're using default settings and they work fine
• You haven't measured a bottleneck

Do tune if:

• CPU is a bottleneck (lower level)
• Network/storage is a bottleneck (higher level)
• You've benchmarked and know the impact

3. Compression Effectiveness by Data Type

Text Data (JSON, Logs, CSV)

Compression Ratios:

Type: JSON
Original: 1000 bytes

none:   1000 bytes (1.0:1)
lz4:     350 bytes (2.9:1) ✅ Good
snappy:  380 bytes (2.6:1) ✅ Good
gzip:    220 bytes (4.5:1) ✅ Best ratio
zstd:    240 bytes (4.2:1) ✅ Best balance

Recommendation: Use lz4 or zstd (balance speed + ratio).

Binary Data (Protobuf, Avro)

Compression Ratios:

Type: Protobuf/Avro
Original: 1000 bytes

none:   1000 bytes (1.0:1)
lz4:     750 bytes (1.3:1) ⚠️ Limited
snappy:  780 bytes (1.3:1) ⚠️ Limited
gzip:    650 bytes (1.5:1) ⚠️ Limited
zstd:    680 bytes (1.5:1) ⚠️ Limited

Recommendation: Binary formats already compact. Use lz4 for modest gains or none if CPU matters.

Already Compressed (Images, Videos)

Compression Ratios:

Type: JPEG image
Original: 100 KB

none:   100 KB (1.0:1)
lz4:    100 KB (1.0:1) ❌ No gain
snappy: 100 KB (1.0:1) ❌ No gain
gzip:   101 KB (0.99:1) ❌ Actually larger!
zstd:   100 KB (1.0:1) ❌ No gain

Recommendation: Use none (waste of CPU).

4. Trade-Off Analysis

CPU vs Network Trade-Off

Scenario: 1 GB of data to send

No Compression (none):
  Compression CPU: 0 ms
  Network transfer (1 Gbps): 8 seconds
  Total: 8 seconds

LZ4 Compression:
  Compression CPU: 100 ms
  Compressed size: 350 MB
  Network transfer: 2.8 seconds
  Total: 2.9 seconds
  ✅ 5.1 seconds saved (63% faster)

GZIP Compression:
  Compression CPU: 800 ms
  Compressed size: 250 MB
  Network transfer: 2 seconds
  Total: 2.8 seconds
  ✅ 5.2 seconds saved (65% faster)
  ❌ But 8x more CPU than lz4

Conclusion: Compression usually worth it unless network is very fast or CPU very limited.

Network Speed Impact

5. Measuring Compression Effectiveness

Key Metrics

1. Compression Ratio

kafka.producer:type=producer-topic-metrics,client-id=<id>,topic=<topic>,name=compression-rate
  → Compression ratio (< 1.0 = effective)

// Example values:
// 0.3 = 3.3:1 compression (70% reduction)
// 0.5 = 2:1 compression (50% reduction)
// 1.0 = No compression

2. Request Size

kafka.producer:type=producer-metrics,client-id=<id>,name=request-size-avg
  → Average request size (bytes)

// Compare with/without compression:
// No compression: 100 KB avg request
// LZ4 compression: 35 KB avg request
// → 65% reduction ✅

3. Record Send Rate

kafka.producer:type=producer-metrics,client-id=<id>,name=record-send-rate
  → Records per second

// With compression, throughput may increase
// (network no longer bottleneck)

Benchmarking Compression

public class CompressionBenchmark {
    
    public static void main(String[] args) {
        String[] compressionTypes = {"none", "lz4", "snappy", "gzip", "zstd"};
        
        for (String compression : compressionTypes) {
            System.out.println("\n=== Testing: " + compression + " ===");
            
            Properties props = createProps();
            props.put("compression.type", compression);
            
            KafkaProducer<String, String> producer = new KafkaProducer<>(props);
            
            long start = System.currentTimeMillis();
            
            for (int i = 0; i < 10000; i++) {
                String data = generateTestData(1000); // 1 KB
                producer.send(new ProducerRecord<>("test", "key-" + i, data));
            }
            
            producer.flush();
            long duration = System.currentTimeMillis() - start;
            
            System.out.println("Duration: " + duration + " ms");
            System.out.println("Throughput: " + (10000.0 / duration * 1000) + " msgs/sec");
            
            producer.close();
        }
    }
}

6. Best Practices

✅ Best Practice 1: Start with LZ4

props.put("compression.type", "lz4");

Rationale:

• Excellent default choice
• Low CPU overhead
• Good compression ratio
• Fast decompression

✅ Best Practice 2: Enable Compression for Text Data

// JSON, logs, text messages
props.put("compression.type", "lz4"); // or zstd

Savings: 50-70% bandwidth reduction.

✅ Best Practice 3: Disable Compression for Pre-Compressed Data

// Images, videos, already compressed formats
props.put("compression.type", "none");

Rationale: No benefit, wastes CPU.

✅ Best Practice 4: Use ZSTD for Maximum Efficiency (if available)

// Kafka 2.1.0+
props.put("compression.type", "zstd");

Best of both worlds: High compression, reasonable speed.

✅ Best Practice 5: Monitor Compression Ratio

// Alert if compression ineffective
if (compressionRate > 0.9) {
    logger.warn("Compression not effective: {}", compressionRate);
    // Consider disabling or investigating data
}

7. Compression + Batching Synergy

Why Larger Batches Compress Better

Small batch (1 record, 1 KB):
  Compressed: 950 bytes (1.05:1) ❌ Poor

Medium batch (10 records, 10 KB):
  Compressed: 4 KB (2.5:1) ✅ Good

Large batch (100 records, 100 KB):
  Compressed: 30 KB (3.3:1) ✅ Better

Reason: Compression algorithms find more patterns in larger data.

Optimal Configuration

// Combine large batches with compression
props.put("batch.size", 65536);       // 64 KB
props.put("linger.ms", 10);           // Allow batching
props.put("compression.type", "lz4"); // Compress large batches

// Result: Maximum compression effectiveness

8. Compression Strategies by Use Case

Use Case 1: High-Volume Logs

Requirements:

• 500,000 msgs/sec
• Average size: 500 bytes (text)
• Network: 1 Gbps

Configuration:

props.put("compression.type", "lz4");
props.put("batch.size", 65536);
props.put("linger.ms", 10);

Result:

• Compression ratio: 3:1
• Network usage: 83 MB/sec (instead of 250 MB/sec)
• CPU overhead: < 5%

Use Case 2: Real-Time Metrics

Requirements:

• 10,000 msgs/sec
• Average size: 200 bytes
• Latency: < 10ms

Configuration:

props.put("compression.type", "none");  // Save CPU, prioritize latency
props.put("batch.size", 8192);
props.put("linger.ms", 0);

Rationale: Low volume, latency critical, compression overhead not worth it.

Use Case 3: JSON API Events

Requirements:

• 50,000 msgs/sec
• Average size: 2 KB (JSON)
• Network: 100 Mbps

Configuration:

props.put("compression.type", "zstd");   // Max compression (network constrained)
props.put("batch.size", 131072);         // 128 KB
props.put("linger.ms", 20);

Result:

• Compression ratio: 4:1
• Network usage: 25 MB/sec (instead of 100 MB/sec)
• Fits within network capacity ✅

9. Common Mistakes

❌ Mistake 1: Compressing Already Compressed Data

// Images, videos
props.put("compression.type", "gzip"); // Wastes CPU, no benefit

Fix:

props.put("compression.type", "none");

❌ Mistake 2: Using GZIP for Latency-Sensitive Applications

props.put("compression.type", "gzip"); // Slow compression

Fix:

props.put("compression.type", "lz4"); // Fast compression

❌ Mistake 3: Not Monitoring Compression Effectiveness

Problem: Compression enabled but not working.

Fix: Monitor compression-rate metric:

if (compressionRate > 0.8) {
    // Compression not effective — investigate
}

❌ Mistake 4: Small Batches with Compression

props.put("batch.size", 1024);        // 1 KB batches
props.put("compression.type", "gzip"); // Won't compress well

Fix: Increase batch size for better compression:

props.put("batch.size", 16384);       // 16 KB

10. Advanced: Compression Level Tuning (ZSTD)

ZSTD Compression Levels

ZSTD supports configurable compression levels (1-22).

Note: Kafka doesn't expose this directly, but you can set via JVM properties.

Level 1:  Fastest, lowest compression (~2.5:1)
Level 3:  Default (~3.5:1)
Level 9:  Balanced (~4:1)
Level 19: Maximum compression (~5:1, very slow)

Trade-off:

• Higher level → Better compression, slower
• Lower level → Faster, less compression

Summary

Key Takeaways

✅ Compression saves network and disk at the cost of CPU
✅ LZ4: Best default choice (fast, good ratio)
✅ ZSTD: Best overall (if Kafka 2.1.0+)
✅ GZIP: Maximum compression, but slow
✅ Snappy: Similar to LZ4, older
✅ None: For pre-compressed data or ultra-low latency
✅ Larger batches compress better — combine with batching
✅ Text data compresses well (3-5:1), binary less so
✅ Monitor compression-rate to verify effectiveness

Compression Decision Tree

Data type?
  ├─ Text (JSON, logs, CSV)
  │   ├─ Network fast (> 1 Gbps) → lz4
  │   ├─ Network medium → zstd
  │   └─ Network slow → gzip
  │
  ├─ Binary (Protobuf, Avro)
  │   └─ lz4 (modest gains) or none
  │
  └─ Pre-compressed (images, videos)
      └─ none

Quick Reference

Configuration Checklist

• Identify data type (text, binary, pre-compressed)
• Measure baseline (no compression)
• Enable compression (start with lz4)
• Monitor compression-rate metric
• Measure impact on throughput and latency
• Adjust if needed (try zstd for better ratio)
• Document final choice and rationale

What's Next

• Module 9.1: Practical Producer Patterns & Real-World Scenarios
  Apply all tuning knowledge to production patterns

Previous Module: ← 8.2 - Latency vs Throughput Tuning
Next Module: 9.1 - Practical Producer Patterns & Real-World Scenarios →