4.3 – Cleanup Tradeoffs & Guarantees

Cleanup Policy Decision Framework

Choosing between retention (delete) and compaction (compact) depends on your data model and use case.

Decision Tree:

Is your data event-based or state-based?
   │
   ├─ Event-based (immutable facts)
   │    → Use retention (cleanup.policy=delete)
   │    Examples: Logs, clickstreams, transactions
   │
   └─ State-based (mutable entities)
        → Use compaction (cleanup.policy=compact)
        Examples: User profiles, device state, tables

Retention vs Compaction: Comparison

Use Case: When to Use Retention

Scenario 1: Immutable Events

Example: User clickstream

{user: "alice", action: "click", page: "/home", timestamp: "2023-01-01"}
{user: "alice", action: "click", page: "/about", timestamp: "2023-01-02"}
{user: "alice", action: "purchase", item: "product-123", timestamp: "2023-01-03"}

Why Retention?

• Each event is a distinct fact (no "updates")
• You need full history for analytics (count clicks per day)
• Events are independent (no relationship between consecutive events)

Config:

cleanup.policy=delete
retention.ms=604800000  # 7 days

Scenario 2: High-Volume Time-Series Data

Example: IoT sensor data

{sensor: "temp-sensor-1", value: 72.5, timestamp: "2023-01-01 10:00:00"}
{sensor: "temp-sensor-1", value: 73.1, timestamp: "2023-01-01 10:00:05"}
{sensor: "temp-sensor-1", value: 72.8, timestamp: "2023-01-01 10:00:10"}

Why Retention?

• High throughput (millions of messages/sec)
• Each reading is a separate measurement (not an update)
• Compaction would be too expensive (CPU overhead)

Config:

cleanup.policy=delete
retention.ms=86400000  # 1 day (shorter retention for high volume)

Scenario 3: Append-Only Logs

Example: Application logs

{level: "INFO", message: "User logged in", timestamp: "2023-01-01 10:00:00"}
{level: "ERROR", message: "Database connection failed", timestamp: "2023-01-01 10:00:05"}
{level: "INFO", message: "Retrying connection", timestamp: "2023-01-01 10:00:10"}

Why Retention?

• Logs are append-only (no updates)
• Need full log history for debugging
• No concept of "latest value" per key

Config:

cleanup.policy=delete
retention.ms=259200000  # 3 days

Use Case: When to Use Compaction

Scenario 1: Database Change Data Capture (CDC)

Example: Syncing a database table to Kafka

{id: 123, name: "Alice", age: 25}  (insert)
{id: 123, name: "Alice", age: 26}  (update)
{id: 123, name: "Alice", age: 27}  (update)

Why Compaction?

• Each message represents current state of row 123
• Only the latest state matters (old states are outdated)
• New consumers can rebuild the table from compacted log

Config:

cleanup.policy=compact
min.compaction.lag.ms=60000  # Wait 1 minute before compacting (allow late updates)

Scenario 2: Kafka Streams State Stores

Example: Word count application

{word: "hello", count: 1}
{word: "hello", count: 2}
{word: "hello", count: 3}

Why Compaction?

• Each message represents current count for word "hello"
• Only the latest count matters
• If app restarts, it restores state from compacted changelog (no need to replay all increments)

Config:

cleanup.policy=compact

Scenario 3: Configuration Store

Example: Application configs

{config: "max-connections", value: "100"}
{config: "timeout", value: "30"}
{config: "max-connections", value: "200"}  (update)

Why Compaction?

• Each message represents current config value
• Only the latest config matters
• New instances read latest config from compacted topic

Config:

cleanup.policy=compact

Combined Retention + Compaction (Hybrid Cleanup)

Use Case: Keep latest state, but don't keep data forever.

Config:

cleanup.policy=compact,delete
retention.ms=2592000000  # 30 days
min.compaction.lag.ms=60000  # 1 minute

Behavior:

1. Compaction runs continuously (keeps latest value per key)
2. Retention deletes segments older than 30 days (even if they contain latest values)

Hybrid Cleanup Example 1: User Sessions

Use Case: Track user sessions (latest state per user), but expire inactive users after 30 days.

Config:

cleanup.policy=compact,delete
retention.ms=2592000000  # 30 days
min.compaction.lag.ms=60000  # 1 minute

Timeline:

Day 0: {userId: 123, sessionId: "abc", lastActive: "2023-01-01"}
Day 5: {userId: 123, sessionId: "def", lastActive: "2023-01-05"}  (update)
Day 10: {userId: 456, sessionId: "xyz", lastActive: "2023-01-10"}

After compaction (Day 10):
  {userId: 123, sessionId: "def", lastActive: "2023-01-05"}  (latest)
  {userId: 456, sessionId: "xyz", lastActive: "2023-01-10"}  (latest)

Day 31 (retention expires):
  {userId: 123, ...}  ← Deleted (segment older than 30 days)
  {userId: 456, ...}  ← Retained (only 21 days old)

Result: User 123's session is deleted after 30 days of inactivity, even though it's the latest value.

Hybrid Cleanup Example 2: Shopping Carts

Use Case: Track shopping carts (latest cart per user), but expire carts after 7 days.

Config:

cleanup.policy=compact,delete
retention.ms=604800000  # 7 days
min.compaction.lag.ms=60000  # 1 minute

Timeline:

Day 0: {cartId: "cart-123", items: ["item1", "item2"]}
Day 2: {cartId: "cart-123", items: ["item1", "item2", "item3"]}  (update)
Day 4: {cartId: "cart-456", items: ["item4"]}

After compaction (Day 4):
  {cartId: "cart-123", items: ["item1", "item2", "item3"]}  (latest)
  {cartId: "cart-456", items: ["item4"]}  (latest)

Day 8 (retention expires):
  {cartId: "cart-123", ...}  ← Deleted (segment older than 7 days)
  {cartId: "cart-456", ...}  ← Retained (only 4 days old)

Result: Cart 123 is deleted after 7 days, even though it's the latest cart state.

Hybrid Cleanup Example 3: Device State with Expiration

Use Case: Track IoT device state (latest state per device), but expire devices that haven't reported in 90 days.

Config:

cleanup.policy=compact,delete
retention.ms=7776000000  # 90 days
min.compaction.lag.ms=300000  # 5 minutes

Timeline:

Day 0: {deviceId: "sensor-123", temp: 22.5, battery: 80}
Day 30: {deviceId: "sensor-123", temp: 23.1, battery: 75}  (update)
Day 60: {deviceId: "sensor-456", temp: 21.0, battery: 90}

After compaction (Day 60):
  {deviceId: "sensor-123", temp: 23.1, battery: 75}  (latest, Day 30)
  {deviceId: "sensor-456", temp: 21.0, battery: 90}  (latest, Day 60)

Day 91 (retention expires):
  {deviceId: "sensor-123", ...}  ← Deleted (segment older than 90 days)
  {deviceId: "sensor-456", ...}  ← Retained (only 31 days old)

Result: Sensor 123's state is deleted after 90 days, even though it's the latest state. This is useful for decommissioned or inactive devices.

Hybrid Cleanup Example 4: User Profiles with GDPR Compliance

Use Case: Track user profiles (latest profile per user), but automatically delete profiles after 2 years of inactivity (GDPR compliance).

Config:

cleanup.policy=compact,delete
retention.ms=63072000000  # 2 years (730 days)
min.compaction.lag.ms=60000  # 1 minute

Timeline:

Day 0: {userId: 123, name: "Alice", email: "alice@example.com"}
Day 100: {userId: 123, name: "Alice Smith", email: "alice@example.com"}  (update)
Day 200: {userId: 456, name: "Bob", email: "bob@example.com"}

After compaction (Day 200):
  {userId: 123, name: "Alice Smith", email: "alice@example.com"}  (latest)
  {userId: 456, name: "Bob", email: "bob@example.com"}  (latest)

Day 731 (retention expires):
  {userId: 123, ...}  ← Deleted (segment older than 2 years)
  {userId: 456, ...}  ← Retained (only 531 days old)

Result: User 123's profile is automatically deleted after 2 years, ensuring GDPR compliance (right to be forgotten).

When to Use Hybrid Cleanup

Use Cases:

• ✅ User sessions (latest state, expire after inactivity)
• ✅ Shopping carts (latest cart, expire after N days)
• ✅ Temporary state (latest value, but not forever)
• ✅ IoT device state (latest state, expire inactive devices)
• ✅ GDPR compliance (latest profile, auto-delete after N years)

Trade-offs:

• ⚠️ Complexity: Must tune both compaction and retention configs
• ⚠️ Unpredictable deletion: Latest values are deleted if segment is old
• ⚠️ Higher overhead: Both compaction and segment deletion run

Storage Impact

Retention: Predictable Disk Usage

Formula:

Disk Usage = throughput × retention × replication_factor

Example:

Throughput: 10 MB/sec
Retention: 7 days = 604,800 seconds
Replication Factor: 3

Disk Usage = 10 MB/sec × 604,800 sec × 3 = 18 TB

Key Point: Disk usage is predictable and linear with throughput.

Compaction: Unbounded Disk Usage (Worst Case)

Formula:

Disk Usage = num_unique_keys × message_size × replication_factor

Example:

Unique Keys: 1 million users
Message Size: 1 KB per user
Replication Factor: 3

Disk Usage = 1M × 1 KB × 3 = 3 GB

Key Point: Disk usage grows with number of unique keys, not throughput.

Worst Case: If keys are never reused (e.g., key=timestamp), compaction degrades to retention (every message is the "latest" for its key).

Consumer Impact

Retention: Consumers Read All Events

Behavior:

• Consumers read all messages within retention period
• Consumers see full history (all inserts, updates, deletes)

Example:

Consumer reads:
  {user: 123, name: "Alice"}
  {user: 123, name: "Alice Smith"}  (sees both insert and update)

Use Case: Stream processing (aggregate events, count clicks, etc.)

Compaction: Consumers Read Latest State

Behavior:

• Consumers read latest value per key (after compaction)
• Consumers miss intermediate states (old values are deleted)

Example:

Consumer reads:
  {user: 123, name: "Alice Smith"}  (only sees latest state, misses initial "Alice")

Use Case: Table snapshot (rebuild database table, restore Kafka Streams state)

Operational Risks

Retention Risks

1. Consumer Lag Exceeds Retention

Scenario: Consumer lags by 8 days, retention is 7 days.

Result: Consumer gets OffsetOutOfRangeException (messages are deleted before consumer reads them).

Mitigation:

• Set retention ≥ 2x max acceptable lag
• Monitor consumer lag (records-lag-max metric)
• Alert if lag > 50% of retention period

2. Disk Full

Scenario: High throughput + long retention → disk runs out of space.

Result: Broker refuses writes (DISK_FULL error).

Mitigation:

• Monitor disk usage (df -h)
• Set disk usage alerts (warn at 70%, critical at 85%)
• Reduce retention or add more brokers

Compaction Risks

1. Compaction Falling Behind

Scenario: High write throughput → dirty head grows faster than compaction can clean.

Result: Disk usage grows unbounded.

Symptoms:

• max-dirty-percent metric > 80%
• Disk usage increasing despite compaction

Mitigation:

• Increase log.cleaner.threads (default: 1, try 2-4)
• Reduce write throughput (if possible)
• Add more brokers (distribute partitions)

2. Tombstone Retention Too Short

Scenario: delete.retention.ms=1000 (1 second), consumers lag by 10 seconds.

Result: Consumers miss tombstones (don't see deletes).

Example:

Time 0: {user: 123, name: "Alice"}
Time 1: {user: 123, value: null}  (tombstone, consumer hasn't read yet)
Time 2: Tombstone deleted (after 1 second)
Time 10: Consumer reads, sees {user: 123, name: "Alice"} (stale data, missed delete)

Mitigation:

• Set delete.retention.ms ≥ max consumer lag (default: 24 hours is usually safe)

3. Compaction CPU Overhead

Scenario: Compacted topic with high throughput → compaction consumes 50%+ CPU.

Result: Slower broker, higher latencies.

Mitigation:

• Tune log.cleaner.threads (balance between throughput and CPU)
• Use fewer partitions (reduces compaction parallelism)
• Consider retention instead of compaction if events are truly immutable

Guarantees and Edge Cases

Retention Guarantees

Compaction Guarantees

What Compaction Does NOT Guarantee:

• ❌ Instant compaction (old values remain until compaction runs)
• ❌ Exactly-one value per key (multiple values can coexist in dirty head)

Decision Matrix

Use this matrix to choose the right cleanup policy:

Best Practices

1. Use Retention for Events

Examples: Logs, clickstreams, transactions, metrics

Why: Events are immutable facts. Full history is valuable. Retention is operationally simpler.

2. Use Compaction for State

Examples: User profiles, device state, CDC, Kafka Streams state stores

Why: Only latest state matters. Compaction reduces disk usage and enables table snapshots.

3. Combine Retention + Compaction for Bounded State

Config:

cleanup.policy=compact,delete
retention.ms=2592000000  # 30 days

Use Case: Keep latest state, but delete inactive entities after 30 days (e.g., delete inactive users).

4. Monitor Cleanup Health

Key Metrics:

• Retention: Disk usage, consumer lag
• Compaction: max-dirty-percent, cleaner-recopy-percent, CPU usage

Alerts:

• Disk usage > 80%
• Consumer lag > 50% of retention
• max-dirty-percent > 80% (compaction falling behind)

5. Test Cleanup Policies in Staging

Scenario: You change retention from 7 days → 1 day.

Risk: Consumers that lag > 1 day will break (OffsetOutOfRangeException).

Mitigation: Test cleanup policy changes in staging first. Monitor consumer lag before reducing retention.

Cleanup Policy Tuning Cheat Sheet

Quick Decision Tree

Start Here: What is your use case?

Is each message an independent event?
├─ YES → Use Retention (cleanup.policy=delete)
│   └─ Set retention.ms based on replay needs
│
└─ NO → Does each message represent the latest state for a key?
    ├─ YES → Use Compaction (cleanup.policy=compact)
    │   └─ Should state expire after N days?
    │       ├─ YES → Use Hybrid (cleanup.policy=compact,delete)
    │       └─ NO → Use Compaction only
    │
    └─ UNSURE → Default to Retention (safer, simpler)

Retention (Delete) Tuning

Example Config (7-day retention, fine-grained):

cleanup.policy=delete
retention.ms=604800000  # 7 days
segment.bytes=536870912  # 512 MB
segment.ms=86400000  # 1 day

Compaction Tuning

Example Config (compaction with safe tombstone retention):

cleanup.policy=compact
min.compaction.lag.ms=60000  # 1 minute
max.compaction.lag.ms=86400000  # 1 day
min.cleanable.dirty.ratio=0.5  # 50%
delete.retention.ms=86400000  # 24 hours
segment.bytes=268435456  # 256 MB

Hybrid (Compact + Delete) Tuning

Example Config (30-day expiration, compacted state):

cleanup.policy=compact,delete
retention.ms=2592000000  # 30 days
min.compaction.lag.ms=60000  # 1 minute
max.compaction.lag.ms=1296000000  # 15 days (half of retention)
delete.retention.ms=86400000  # 24 hours
segment.bytes=268435456  # 256 MB

Broker-Level Compaction Tuning

Example Config (high-throughput cluster):

log.cleaner.threads=4
log.cleaner.io.max.bytes.per.second=104857600  # 100 MB/sec
log.cleaner.dedupe.buffer.size=268435456  # 256 MB

Common Tuning Scenarios

Scenario 1: High-Throughput Event Log (1000 msg/sec)

Goal: Retain 7 days of events, minimize disk usage.

Config:

cleanup.policy=delete
retention.ms=604800000  # 7 days
segment.bytes=536870912  # 512 MB
segment.ms=86400000  # 1 day (force daily segment roll)
compression.type=lz4  # Reduce disk usage

Scenario 2: Database CDC (Change Data Capture)

Goal: Keep latest state per row, compact frequently.

Config:

cleanup.policy=compact
min.compaction.lag.ms=60000  # 1 minute
max.compaction.lag.ms=3600000  # 1 hour (compact frequently)
min.cleanable.dirty.ratio=0.3  # Compact when 30% dirty (more aggressive)
delete.retention.ms=86400000  # 24 hours
segment.bytes=268435456  # 256 MB

Scenario 3: User Sessions (Expire After 30 Days)

Goal: Keep latest session per user, expire after 30 days of inactivity.

Config:

cleanup.policy=compact,delete
retention.ms=2592000000  # 30 days
min.compaction.lag.ms=60000  # 1 minute
max.compaction.lag.ms=1296000000  # 15 days
delete.retention.ms=86400000  # 24 hours
segment.bytes=268435456  # 256 MB

Scenario 4: Low-Latency Compacted Topic

Goal: Minimize dirty head (fast bootstrap for new consumers).

Config:

cleanup.policy=compact
min.compaction.lag.ms=10000  # 10 seconds (compact quickly)
max.compaction.lag.ms=300000  # 5 minutes (force frequent compaction)
min.cleanable.dirty.ratio=0.2  # Compact when 20% dirty (very aggressive)
delete.retention.ms=86400000  # 24 hours
segment.bytes=134217728  # 128 MB (smaller segments = faster compaction)
log.cleaner.threads=4  # Parallel compaction

Trade-off: High CPU/I/O overhead (compaction runs frequently).

Monitoring Checklist

Retention Topics:

• ✅ Disk usage per partition (kafka.log:type=Log,name=Size)
• ✅ Oldest segment age (should be ≤ retention.ms)
• ✅ Consumer lag (should be < retention.ms)

Compaction Topics:

• ✅ max-dirty-percent (should be < 80%)
• ✅ Compaction lag (kafka.log:type=LogCleanerManager,name=max-dirty-percent)
• ✅ Tombstone retention (consumers should see tombstones)
• ✅ CPU usage (compaction threads)

Hybrid Topics:

• ✅ All of the above (retention + compaction metrics)

Key Takeaways

1. Retention vs Compaction: Use retention for immutable events, compaction for mutable state.

2. Disk Growth: Retention grows with throughput × retention. Compaction grows with num unique keys.

3. Consumer Behavior: Retention → consumers read all events. Compaction → consumers read latest state.

4. Operational Overhead: Retention is simple (file deletion). Compaction is CPU/I/O intensive.

5. Combined Policy: cleanup.policy=compact,delete keeps latest state but deletes old data after N days.

6. Risks: Retention → consumer lag exceeds retention, disk full. Compaction → compaction falling behind, tombstone retention too short.

7. Monitoring: Track disk usage, consumer lag, max-dirty-percent, CPU usage.

8. Test in Staging: Always test cleanup policy changes in staging before production.

What's Next?

Next Module: 5.1 – Designing Topics & Partition Strategy

What We Covered:

• ✅ Retention vs compaction decision framework
• ✅ Use cases for retention (events, logs, time-series)
• ✅ Use cases for compaction (state, CDC, configs)
• ✅ Combined retention + compaction
• ✅ Storage impact and disk growth
• ✅ Consumer impact (stream vs snapshot)
• ✅ Operational risks and mitigation

What We Didn't Cover (Explicitly Deferred):

• ❌ Topic design strategy → Module 5.1
• ❌ Performance tuning → M6: Operations

Cleanup Tradeoffs & Guarantees: Complete ✅