KafkaIQ

2.3 Compaction & Retention Mechanics

Module Overview

Goal: Explain Kafka's log cleanup mechanisms at the engine level.

What You'll Learn:

  • Delete-based retention (time and size)
  • Log compaction algorithm internals
  • Tomb

stones and deletion markers

  • Cleaner thread mechanics
  • Compaction guarantees and edge cases
  • Cleanup coordination

What's NOT Covered Here:

  • Topic configuration strategy (covered in Operations)
  • When to use compaction vs deletion (design decision, covered in Operations)
  • Performance tuning of cleanup (covered in Operations)

Key Insight: Kafka provides two cleanup strategies — deletion (time/size-based) and compaction (key-based). Understanding the internal mechanics explains their guarantees and limitations.

1. Why Cleanup?

The Disk Space Problem

WITHOUT CLEANUP:

Log grows forever:
├─ Day 1: 100 GB
├─ Day 30: 3 TB
├─ Day 365: 36 TB
└─ Eventually: Disk full, broker crash

NEED CLEANUP MECHANISMS:

1. Delete old data (retention)
   └─ Keep last N days or last N GB

2. Delete old values per key (compaction)
   └─ Keep only latest value for each key

2. Cleanup Policies

Two Modes

cleanup.policy:

"delete":
├─ Time-based: Delete segments older than retention.ms
├─ Size-based: Delete oldest segments when size > retention.bytes
└─ Result: Data eventually deleted

"compact":
├─ Key-based: Delete old values for duplicate keys
├─ Keep latest value per key
└─ Result: Data never deleted (per key)

"compact,delete":
├─ Compaction first (remove old values)
├─ Then deletion (remove old segments)
└─ Hybrid: Keep latest values, but not forever

3. Delete-Based Retention

Time-Based Deletion

RETENTION BY TIME (retention.ms):

Configuration:
└─ retention.ms=604800000  # 7 days

Behavior:
├─ For each closed segment:
│   └─ Check: max timestamp in segment
├─ If (now - max timestamp) > retention.ms:
│   └─ Delete segment files (.log, .index, .timeindex)
└─ Active segment never deleted

Example:
├─ Segment 1: max timestamp = Jan 1, 2024
├─ Segment 2: max timestamp = Jan 5, 2024
├─ Segment 3: max timestamp = Jan 8, 2024 (active)
├─ Current time: Jan 10, 2024
├─ Retention: 7 days
└─ Action: Delete Segment 1 (> 7 days old)

Size-Based Deletion

RETENTION BY SIZE (retention.bytes):

Configuration:
└─ retention.bytes=10737418240  # 10 GB per partition

Behavior:
├─ Calculate total partition size
├─ If total size > retention.bytes:
│   └─ Delete oldest closed segments
│   └─ Until size <= retention.bytes
└─ Active segment never deleted

Example:
├─ Partition total size: 12 GB
├─ Retention limit: 10 GB
├─ Segments:
│   ├─ Segment 1: 1 GB (oldest)
│   ├─ Segment 2: 1 GB
│   ├─ Segment 3: 1 GB
│   └─ ... (10 segments)
└─ Action: Delete Segments 1 and 2 (reduce to 10 GB)

Combined Retention

IF BOTH retention.ms AND retention.bytes CONFIGURED:

Deletion occurs when EITHER condition met:
├─ Segment is older than retention.ms
└─ OR total size exceeds retention.bytes

Whichever happens first triggers deletion

Example:
├─ retention.ms: 7 days
├─ retention.bytes: 10 GB
├─ Segment 1: 5 days old, but total size = 11 GB
└─ Action: Delete Segment 1 (size limit hit)

Deletion Process

LOG CLEANER THREAD (for delete policy):

1. Scan all partitions
2. For each partition:
   a. Check closed segments
   b. Identify segments eligible for deletion
   c. Delete files atomically:
      ├─ rm *.log
      ├─ rm *.index
      └─ rm *.timeindex
   d. Update partition start offset
3. Sleep (log.retention.check.interval.ms)
4. Repeat

Atomic deletion:
└─ All segment files deleted together
└─ No partial deletion (corrupted state)

4. Log Compaction Overview

Key-Based Retention

COMPACTION CONCEPT:

For each unique key:
└─ Keep only the latest value
└─ Delete all older values

Before compaction:
[k1:v1(0), k2:v2(1), k1:v3(2), k2:v4(3), k3:v5(4)]

After compaction:
[k1:v3(2), k2:v4(3), k3:v5(4)]

Offsets preserved:
└─ Offsets 0 and 1 are "deleted" (gaps appear)
└─ But offsets 2, 3, 4 remain

Use Cases

WHY COMPACTION?

Changelog topics:
├─ Database table replicated to Kafka
├─ Each record = current row state
├─ Only latest state matters
└─ Example: user profiles, product catalog

State stores:
├─ Kafka Streams state stores backed by Kafka
├─ Each key = state for an entity
└─ Old states can be discarded

Configuration stores:
├─ Application configs keyed by config name
└─ Only latest config value matters

5. Compaction Algorithm Internals

Log Structure for Compaction

LOG DIVIDED INTO TWO SECTIONS:

┌───────────────────────┬─────────────────────────┐
│   CLEAN               │   DIRTY                 │
│   (already compacted) │   (needs compaction)    │
└───────────────────────┴─────────────────────────┘
                        ↑
                Clean Point (cleaner-offset-checkpoint)

Clean section:
├─ Previously compacted
├─ Only latest value per key
└─ Immutable (won't be re-compacted)

Dirty section:
├─ Recently written records
├─ May have duplicate keys
└─ Target for next compaction

Compaction Process

CLEANER THREAD ALGORITHM:

1. Select partition for compaction
   └─ Based on "dirty ratio" (dirty size / total size)

2. Build offset map:
   ├─ Scan dirty section
   ├─ For each key: record latest offset
   └─ Map: key → latest offset

3. Create new compacted segment:
   ├─ Scan all segments (clean + dirty)
   ├─ For each record:
   │   ├─ If key in map AND offset == map[key]:
   │   │   └─ Keep record (latest value)
   │   └─ Else: Skip record (older value)
   └─ Write to new segment

4. Replace old segments:
   ├─ Atomically swap in new segments
   └─ Delete old segments

5. Update clean point
   └─ Mark compacted area as "clean"

Offset Map (In-Memory)

OFFSET MAP STRUCTURE:

HashMap: MD5(key) → offset

Why hash instead of full key?
└─ Saves memory (16 bytes vs full key size)
└─ Collision risk very low

Memory usage:
├─ 24 bytes per unique key (hash + offset + metadata)
├─ log.cleaner.dedupe.buffer.size: 128 MB default
└─ Can compact ~5 million unique keys with default

If map fills up:
└─ Compact partial dirty section
└─ Multiple compaction passes

Compaction Example

BEFORE COMPACTION:

Segment 1 (clean):
[k1:v1(100), k2:v2(101)]

Segment 2 (dirty):
[k1:v3(200), k3:v4(201), k2:v5(202)]

Segment 3 (dirty):
[k1:v6(300), k3:v7(301)]

OFFSET MAP (from dirty sections):
k1 → 300 (latest)
k2 → 202 (latest)
k3 → 301 (latest)

AFTER COMPACTION:

New Segment (clean):
[k1:v6(300), k2:v5(202), k3:v7(301)]

Result:
└─ Offsets 100, 101, 200, 201 deleted (gaps)
└─ Offsets 300, 202, 301 retained

6. Tombstones

Deletion Markers

TOMBSTONE = Record with null value

Purpose:
└─ Signal deletion of a key in compacted log

Example:
Produce: key="user123", value=null

Effect:
└─ After compaction, "user123" removed entirely

Timeline:
1. Record exists: [user123:profile_data]
2. Tombstone written: [user123:null]
3. Compaction runs: Both records deleted
4. Result: user123 completely removed

Tombstone Retention

TOMBSTONE DELAY (delete.retention.ms):

Configuration:
└─ delete.retention.ms=86400000  # 1 day

Behavior:
├─ Tombstone must exist for at least delete.retention.ms
├─ Ensures consumers see the deletion
└─ After delay, tombstone itself deleted

Why delay?
├─ Consumer may be offline during tombstone write
├─ Consumer restarts and reads from old offset
└─ Must see tombstone to know key was deleted

Without delay:
└─ Consumer could miss deletion (never see tombstone)

Compaction with Tombstones

PROCESS:

1. Producer writes tombstone: [k1:null]
2. Tombstone remains for delete.retention.ms
3. Compaction runs:
   ├─ Removes all old values for k1
   ├─ Keeps tombstone (if within retention)
4. After delete.retention.ms:
   └─ Next compaction removes tombstone too
5. Result: k1 completely gone

Timeline:
├─ T=0: [k1:v1(0)]
├─ T=1: [k1:v1(0), k1:null(1)]
├─ T=2 (compaction): [k1:null(1)]
├─ T=3 (after retention): []
└─ k1 fully deleted

7. Compaction Guarantees

What Compaction Guarantees

GUARANTEES:

1. At least one value per key retained:
   └─ If key exists, latest value is kept

2. Offsets are not reused:
   └─ Deleted offsets leave gaps (never filled)

3. Ordering per key preserved:
   └─ Later writes always have higher offsets

4. Consumer sees all writes:
   └─ Compaction doesn't affect reads from HW
   └─ Only cleaned logs affected

5. Transactional atomicity respected:
   └─ All records in transaction kept or deleted together

What Compaction Does NOT Guarantee

NO GUARANTEES:

1. Timing:
   └─ Compaction is asynchronous (not immediate)
   └─ May take hours for dirty section to compact

2. Exact deduplication:
   └─ Multiple values may exist during dirty phase
   └─ Only cleaned log has single value per key

3. Deleted keys immediately gone:
   └─ Tombstones retained for delete.retention.ms
   └─ Key fully deleted only after compaction + retention

4. Ordering across keys:
   └─ No guarantee on order of different keys

8. Cleaner Thread Internals

Cleaner Configuration

CLEANER THREADS:

log.cleaner.threads=1  # Number of cleaner threads

Each thread:
├─ Selects partitions for compaction
├─ Runs compaction algorithm
└─ Manages offset map

More threads:
├─ More partitions compacted in parallel
└─ Higher CPU usage

Compaction Trigger

WHEN TO COMPACT?

Dirty ratio:
└─ dirty size / (clean size + dirty size)

Configuration:
└─ min.cleanable.dirty.ratio=0.5  # 50%

If dirty ratio >= threshold:
└─ Partition eligible for compaction

Example:
├─ Clean size: 600 MB
├─ Dirty size: 400 MB
├─ Dirty ratio: 400/(600+400) = 0.4 (40%)
└─ Below 50% threshold, not yet compacted

├─ Dirty size grows to: 600 MB
├─ Dirty ratio: 600/(600+600) = 0.5 (50%)
└─ Now eligible for compaction

Compaction Priority

CLEANER SELECTS PARTITIONS:

Priority based on:
├─ Dirty ratio (higher = higher priority)
├─ Time since last compaction
└─ log.cleaner.min.compaction.lag.ms (delay before compaction)

Backoff:
└─ Recently compacted partitions lower priority
└─ Ensures all partitions get compacted eventually

9. Compaction and Replication

Interaction

COMPACTION IS LOCAL TO EACH REPLICA:

Leader:
├─ Compaction runs independently
└─ Creates compacted segments

Followers:
├─ Replicate raw data from leader (not compacted data)
├─ Run own compaction independently
└─ May compact at different times

Result:
├─ Each replica has its own compaction schedule
└─ All replicas eventually converge to same data
└─ Compaction state NOT replicated

10. Edge Cases and Gotchas

Active Segment Never Compacted

ACTIVE SEGMENT EXEMPTION:

Active segment:
└─ Never compacted (currently being written)
└─ Always in "dirty" section

Implication:
├─ Latest writes may have duplicates
├─ Only visible after segment rolls
└─ Then becomes eligible for compaction

Workaround:
└─ Force segment roll with segment.ms

Compaction Lag

COMPACTION IS ASYNC:

Write to compacted topic:
├─ T=0: Write [k1:v1]
├─ T=1: Write [k1:v2]
├─ T=2: Reads see both k1:v1 and k1:v2
└─ T=100: Compaction runs, only k1:v2 remains

Consumer reading at T=2:
└─ Sees duplicates (k1:v1 and k1:v2)
└─ Application must handle (use latest)

Why async?
└─ Compaction is expensive (CPU, I/O)
└─ Can't block writes

Transactional Records

TRANSACTIONS AND COMPACTION:

Records in transaction:
├─ Kept or deleted as a unit
└─ Partial transaction never compacted

If transaction includes:
├─ [k1:v1(txn1), k2:v2(txn1)]

Both records kept until compaction determines:
└─ Keep (if latest) or delete (if superseded)
└─ But always together

11. Cleanup Coordination

Controller-Side Logic

CONTROLLER DOES NOT TRIGGER CLEANUP:

Cleanup is broker-local:
├─ Each broker runs own cleaner threads
├─ Independently decides when to clean
└─ No coordination with controller

Controller only:
└─ Provides metadata (partition assignments)
└─ Cleanup execution is autonomous

Cleanup During Rebalance

IF PARTITION LEADERSHIP CHANGES:

Old leader:
├─ Stops compaction (if running)
└─ Closes partition cleanly

New leader:
├─ Starts accepting writes immediately
├─ Compaction resumes independently
└─ No continuity of compaction state

Result:
└─ Compaction may restart from scratch on new leader
└─ But semantics remain correct

12. Key Takeaways

You've learned:

✅ Two cleanup policies: delete (time/size-based) and compact (key-based)
✅ Delete retention removes old segments based on time or size
✅ Compaction keeps only the latest value per key
✅ Offset map (in-memory) tracks latest offset per key during compaction
✅ Tombstones (null values) signal key deletion in compacted logs
✅ delete.retention.ms keeps tombstones visible to consumers
✅ Compaction is async — duplicates may exist until compaction runs
✅ Offsets are preserved — compaction creates gaps, doesn't renumber
✅ Cleaner threads run compaction based on dirty ratio
✅ Compaction is per-replica — each broker compacts independently

13. Cleanup Mechanics Checklist

DELETE-BASED RETENTION:

☑ retention.ms: Time-based deletion
☑ retention.bytes: Size-based deletion
☑ Closed segments deleted atomically
☑ Active segment never deleted

LOG COMPACTION:

☑ Keep latest value per key
☑ Offset map built from dirty section
☑ Compaction triggered by dirty ratio
☑ Offsets preserved (gaps appear)

TOMBSTONES:

☑ null value signals deletion
☑ Retained for delete.retention.ms
☑ Ensures consumers see deletion
☑ Eventually deleted by compaction

CLEANER THREADS:

☑ log.cleaner.threads (default: 1)
☑ min.cleanable.dirty.ratio (trigger)
☑ Runs asynchronously
☑ Per-broker, not coordinated

Summary

Kafka provides flexible log cleanup mechanisms:

Delete Retention =
├─ Time-based (retention.ms)
├─ Size-based (retention.bytes)
├─ Removes old segments entirely
└─ Simple, predictable

Log Compaction =
├─ Key-based (latest value per key)
├─ Tombstones for deletion
├─ Offset map tracks latest offsets
├─ Async (duplicates until compaction)
└─ Preserves offsets (creates gaps)

Cleaner Threads =
├─ Run independently per broker
├─ Triggered by dirty ratio
├─ Use in-memory offset map
└─ Atomic segment replacement

Remember: Retention controls disk usage, compaction controls data semantics. Delete retention is straightforward (remove old data). Compaction is complex but powerful (changelog semantics, state stores). Choose based on your use case.

Next up: Module 3.1 — Replication Protocol Explained — One authoritative explanation of Kafka's replication mechanics.