4.2 – Log Compaction

What is Log Compaction?

Log compaction is a cleanup policy that retains only the latest value for each key in a topic, discarding older values for the same key.

Key Difference from Retention:

Why Log Compaction?

Use Case: Modeling a Table in Kafka

Problem: You want to store a table (e.g., user profiles) in Kafka.

Without Compaction:

Key: user-123, Value: {name: "Alice", age: 25}  (offset 0)
Key: user-123, Value: {name: "Alice", age: 26}  (offset 1)  ← Update
Key: user-123, Value: {name: "Alice", age: 27}  (offset 2)  ← Update

After 7 days (retention policy), all messages are deleted, including the latest value.

Problem: New consumers can't see the current state (table is empty after retention).

With Compaction:

Key: user-123, Value: {name: "Alice", age: 25}  (offset 0)
Key: user-123, Value: {name: "Alice", age: 26}  (offset 1)
Key: user-123, Value: {name: "Alice", age: 27}  (offset 2)

After compaction:
Key: user-123, Value: {name: "Alice", age: 27}  (offset 2)  ← Latest value retained

Benefit: The latest value is always available, even after compaction. New consumers can rebuild the table.

How Log Compaction Works

Conceptual Model

Compaction Rule: For each key, keep only the latest value (highest offset).

Before Compaction:

Segment 0:
  offset 0: key=A, value=1
  offset 1: key=B, value=2
  offset 2: key=A, value=3  ← Newer value for key=A

Segment 1:
  offset 3: key=A, value=4  ← Newest value for key=A
  offset 4: key=C, value=5

After Compaction:

Segment 0:
  offset 1: key=B, value=2  ← Only message for key=B

Segment 1:
  offset 3: key=A, value=4  ← Latest value for key=A
  offset 4: key=C, value=5  ← Only message for key=C

Result: Offsets 0 and 2 (old values for key=A) are deleted. Disk space is reclaimed.

Compacted Topic Structure

A compacted topic has two logical sections:

┌─────────────────────────────────────────────────┐
│              Compacted Topic                    │
│                                                 │
│  ┌──────────────┐  ┌──────────────────────────┐│
│  │  Clean Tail  │  │   Dirty Head (Active)    ││
│  │  (Compacted) │  │   (Not Yet Compacted)    ││
│  │              │  │                          ││
│  │ Latest value │  │ All messages             ││
│  │ per key only │  │ (including duplicates)   ││
│  └──────────────┘  └──────────────────────────┘│
│                                                 │
└─────────────────────────────────────────────────┘

1. Clean Tail (Compacted Section)

Characteristics:

• Contains only the latest value for each key
• No duplicates (old values have been removed)
• Read-only (closed segments)

Example:

Segment 0 (clean):
  key=A, value=100 (latest)
  key=B, value=200 (latest)

2. Dirty Head (Uncompacted Section)

Characteristics:

• Contains all recent messages (including duplicates)
• Active segment + recent closed segments
• Not yet compacted (compaction hasn't run yet)

Example:

Segment 2 (dirty):
  key=A, value=98
  key=A, value=99
  key=A, value=100  ← Latest value (will be kept after compaction)
  key=B, value=200

Key Point: Compaction runs asynchronously. New messages go to the dirty head. Periodically, Kafka's log cleaner compacts the dirty head and moves it to the clean tail.

Log Cleaner Thread

Kafka runs a background thread called the log cleaner to compact topics.

Process:

1. Scan dirty head: Identify all messages in uncompacted segments
2. Build offset map: For each key, track the latest offset
3. Rewrite segments: Copy only the latest value for each key to new segments
4. Swap segments: Replace old segments with compacted segments
5. Delete old segments: Free disk space

Config: log.cleaner.enable=true (default: true)

Compaction Frequency

Config: log.cleaner.min.compaction.lag.ms (default: 0)

Behavior: Kafka waits at least N milliseconds before compacting a segment.

Why Wait?

If compaction runs immediately:

• Producer sends: key=A, value=1
• Compaction runs, keeps key=A, value=1
• Producer sends: key=A, value=2 (updated value)
• Consumers who read before update see value=1, after update see value=2 (inconsistent)

Solution: Wait a bit (e.g., 10 minutes) to allow all updates for a key to arrive before compacting.

Compaction Ratio

Config: log.cleaner.min.cleanable.ratio (default: 0.5)

Behavior: Kafka only compacts a segment if dirty head size ≥ 50% of total size.

Why?

Compaction is expensive (CPU, I/O). If the dirty head is small, compaction won't reclaim much space. Better to wait until there's enough duplication to make compaction worthwhile.

Example:

Segment size: 1 GB
Dirty head: 200 MB (20% of total)
Result: Compaction doesn't run (ratio < 50%)

Dirty head: 600 MB (60% of total)
Result: Compaction runs (ratio ≥ 50%)

Tombstones (Deleting Keys)

The Delete Problem

Problem: How do you delete a key from a compacted topic?

Naive Approach: Just stop sending messages for that key.

Issue: The last value for that key remains forever (compaction keeps the latest value).

Example:

key=user-123, value={"name": "Alice"}  (offset 100)

... time passes ...

After compaction: key=user-123 still exists (latest value retained)

How to delete?: Send a tombstone (message with key but null value).

Tombstone Semantics

Tombstone: A message with a key but null value.

Example:

producer.send(new ProducerRecord<>("users", "user-123", null));  // Tombstone

Behavior:

1. Tombstone is written to the log
2. After compaction, the tombstone deletes all previous values for that key
3. After a retention period (delete.retention.ms), the tombstone itself is deleted

Timeline:

Time 0:
  key=user-123, value={"name": "Alice"}  (offset 100)

Time 1: Send tombstone
  key=user-123, value=null  (offset 101)

Time 2: Compaction runs
  key=user-123, value=null  (offset 101)  ← Previous value deleted

Time 3 (after delete.retention.ms): Tombstone deleted
  key=user-123 no longer exists in log

Tombstone Retention

Config: delete.retention.ms (default: 86400000 = 24 hours)

Purpose: Keep tombstones for at least 24 hours so consumers can see the delete event.

Why?

If tombstones are deleted immediately:

• Consumer reads: key=user-123, value={"name": "Alice"}
• Tombstone arrives and is immediately deleted
• Consumer misses the delete event
• Consumer thinks user-123 still exists (stale data)

Solution: Retain tombstones for 24 hours so consumers have time to see the delete.

Compaction and Ordering

Important: Compaction preserves message order (by offset) within a key.

Example:

key=A, value=1, offset=10
key=A, value=2, offset=20
key=A, value=3, offset=30

After compaction:
key=A, value=3, offset=30  ← Latest value retained

Key Point: The offset of the latest value is preserved. Consumers reading in offset order will see the correct sequence.

Use Cases for Compacted Topics

1. Change Data Capture (CDC)

Scenario: Stream database table changes to Kafka.

Example:

key=user-123, value={"name": "Alice", "age": 25}  (insert)
key=user-123, value={"name": "Alice", "age": 26}  (update)
key=user-123, value={"name": "Alice", "age": 27}  (update)

With Compaction: Only the latest state is retained. New consumers can rebuild the table without replaying all history.

2. Kafka Streams State Stores

Scenario: Kafka Streams applications store state in changelog topics (which are compacted).

Example:

Application maintains a counter: key=word, value=count

key=hello, value=1
key=hello, value=2
key=hello, value=3

After compaction: key=hello, value=3  ← Latest count retained

Benefit: If the application restarts, it can restore state from the compacted changelog (without replaying all increments).

3. Configuration Management

Scenario: Store application configs in Kafka.

Example:

key=max-connections, value=100
key=timeout, value=30
key=max-connections, value=200  ← Config update

After compaction:
key=max-connections, value=200  ← Latest config
key=timeout, value=30

Benefit: New instances read the latest config from Kafka.

4. User Profiles / Entity State

Scenario: Store user profiles, device state, etc.

Example:

key=device-123, value={"status": "online", "location": "US"}
key=device-123, value={"status": "offline", "location": "US"}

After compaction:
key=device-123, value={"status": "offline", "location": "US"}  ← Latest state

Compaction Configuration

Enabling Compaction

Config (topic-level):

cleanup.policy=compact

Create compacted topic:

kafka-topics --create --topic users \
  --partitions 10 \
  --replication-factor 3 \
  --config cleanup.policy=compact

Combined Retention + Compaction

Config:

cleanup.policy=compact,delete
retention.ms=604800000  # 7 days

Behavior:

• Compaction runs continuously (keeps latest value per key)
• Retention deletes segments older than 7 days (even if they contain latest values)

Use Case: Keep latest state, but don't keep data forever (e.g., delete inactive users after 7 days).

Key Compaction Parameters

Compaction Guarantees

What Compaction Guarantees

1. Latest Value Retained: For each key, at least one message (the latest) is retained.

2. Ordering Preserved: The latest value for a key has the highest offset for that key.

3. Deletes Work: Tombstones delete all previous values for a key (after delete.retention.ms).

What Compaction Does NOT Guarantee

1. Instant Compaction: Compaction runs asynchronously. Old values remain in the dirty head until compaction runs.

2. Unique Keys in Log: Multiple values for the same key can coexist in the dirty head (not yet compacted).

3. Consistent Snapshot: Different partitions may compact at different times. Reading from multiple partitions may see inconsistent states.

Compaction vs Retention (Summary)

Compaction Performance

Compaction Overhead

Compaction is CPU and I/O intensive:

• Must scan all messages in dirty head
• Must rewrite segments (copy latest values)

Config: log.cleaner.threads (default: 1)

Recommendation: Use 2-4 threads for high-throughput compacted topics.

Monitoring Compaction

Key Metrics:

• kafka.log:type=LogCleaner,name=cleaner-recopy-percent: Percentage of messages rewritten during compaction
• kafka.log:type=LogCleanerManager,name=max-dirty-percent: Max dirty head size (as percentage)

Alert: If max-dirty-percent > 80%, compaction is falling behind. Increase log.cleaner.threads or tune log.cleaner.min.cleanable.ratio.

Compaction Trade-offs Summary

Quick Reference Table

When to Use Each Policy

Retention (Delete):

• ✅ Event logs (e.g., clickstreams, application logs)
• ✅ Time-series data (e.g., metrics, sensor readings)
• ✅ Audit trails (need full history)
• ✅ Predictable storage (retention.ms controls disk usage)

Compaction:

• ✅ Database changelogs (CDC)
• ✅ Kafka Streams state stores
• ✅ Configuration management (latest config per service)
• ✅ User profiles (latest state per user)
• ✅ Device state (latest state per device)

Hybrid (Compact + Delete):

• ✅ User sessions (latest state, expire after 30 days)
• ✅ Shopping carts (latest cart, expire after 7 days)
• ✅ Temporary state (latest value, but not forever)

Storage Implications

Retention (Delete):

Storage = message_rate * avg_message_size * retention.ms

Example:
  1000 msg/sec * 1 KB * 7 days = 605 GB

Compaction:

Storage = unique_key_count * avg_message_size

Example:
  1 million users * 1 KB = 1 GB (regardless of update rate)

Hybrid (Compact + Delete):

Storage = min(
  unique_key_count * avg_message_size,
  message_rate * avg_message_size * retention.ms
)

Example:
  1 million users * 1 KB = 1 GB (if updates are frequent)
  OR
  1000 msg/sec * 1 KB * 30 days = 2.6 TB (if updates are rare)
  
  Result: 1 GB (compaction dominates)

Performance Implications

Retention (Delete):

• Write: Fast (append-only, no compaction overhead)
• Read: Fast (sequential reads)
• CPU: Low (no compaction threads)
• Disk I/O: Low (just segment deletion)

Compaction:

• Write: Fast (append-only)
• Read: Slower (consumers may read dirty head + clean tail)
• CPU: High (compaction threads rewrite segments)
• Disk I/O: High (compaction reads + rewrites segments)

Hybrid (Compact + Delete):

• Write: Fast (append-only)
• Read: Slower (compaction overhead)
• CPU: High (compaction + segment deletion)
• Disk I/O: High (compaction + segment deletion)

Operational Complexity

Retention (Delete):

• ✅ Simple (set retention.ms, done)
• ✅ Predictable (storage = retention * rate)
• ✅ Low overhead (no compaction threads)

Compaction:

• ⚠️ Complex (tune log.cleaner.threads, min.cleanable.ratio, etc.)
• ⚠️ Unpredictable storage (depends on key cardinality)
• ⚠️ High overhead (compaction threads compete with production traffic)
• ⚠️ Monitoring required (max-dirty-percent, compaction lag)

Hybrid (Compact + Delete):

• ⚠️ Most complex (tune both compaction and retention)
• ⚠️ Requires understanding of both policies
• ⚠️ Highest overhead (compaction + segment deletion)

Key Takeaways

1. Log Compaction: Retains only the latest value for each key, discarding old values. Use for tables, changelogs, and state.

2. Clean Tail vs Dirty Head: Compacted topics have a clean tail (compacted) and dirty head (not yet compacted).

3. Tombstones: Delete a key by sending a message with null value. Tombstones are retained for delete.retention.ms (default: 24 hours).

4. Compaction is Asynchronous: Compaction runs periodically (not instantly). Old values remain in the dirty head until compaction runs.

5. Use Cases: CDC, Kafka Streams state stores, configuration management, user profiles, device state.

6. Compaction + Retention: Can combine both policies (cleanup.policy=compact,delete) to keep latest state but delete old data after N days.

7. Compaction Overhead: Compaction is CPU/I/O intensive. Monitor max-dirty-percent and tune log.cleaner.threads for high-throughput topics.

8. Ordering Preserved: Compaction preserves message order (by offset) for each key.

What's Next?

Next Module: 4.3 – Cleanup Tradeoffs & Guarantees

What We Covered:

• ✅ Log compaction model (latest value per key)
• ✅ Clean tail vs dirty head
• ✅ Tombstones (deleting keys)
• ✅ Use cases (CDC, state stores, configs, profiles)
• ✅ Compaction configuration and tuning
• ✅ Compaction vs retention

What We Didn't Cover (Explicitly Deferred):

• ❌ Choosing between retention and compaction → Module 4.3
• ❌ Operational impact of compaction → M6: Operations

Log Compaction: Complete ✅