6.2 – Topic-Level Configurations

What are Topic-Level Configurations?

Topic-level configurations are settings that override broker defaults for a specific topic.

Key Concept: Every topic inherits broker-level defaults. Topic-level configs allow you to customize behavior per topic.

Example:

Broker default: log.retention.ms=604800000 (7 days)

Topic "user-events": retention.ms=2592000000 (30 days)  ← Override
Topic "logs": retention.ms=86400000 (1 day)            ← Override

Setting Topic Configs

Method 1: At Topic Creation

kafka-topics --create --topic orders \
  --partitions 10 \
  --replication-factor 3 \
  --config retention.ms=2592000000 \
  --config compression.type=lz4 \
  --bootstrap-server localhost:9092

Method 2: Alter Existing Topic

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Method 3: Dynamic Update (Kafka 1.1+)

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Result: Config update takes effect immediately (no broker restart needed).

Common Topic-Level Configurations

1. Retention

Use Case: Short retention for high-volume logs, long retention for audit logs.

Example:

# Short retention for logs (1 day)
kafka-topics --create --topic app-logs --config retention.ms=86400000

# Long retention for audit logs (90 days)
kafka-topics --create --topic audit-logs --config retention.ms=7776000000

2. Cleanup Policy

Use Case: Compaction for tables/state, retention for events.

Example:

# Compacted topic (for CDC)
kafka-topics --create --topic user-profiles --config cleanup.policy=compact

# Retention + compaction (bounded state)
kafka-topics --create --topic device-state \
  --config cleanup.policy=compact,delete \
  --config retention.ms=2592000000  # 30 days

3. Segment Settings

Use Case: Smaller segments for aggressive retention, larger segments for long retention.

Example:

# Small segments for fine-grained retention (100 MB)
kafka-topics --create --topic short-term-logs \
  --config segment.bytes=104857600 \
  --config retention.ms=3600000  # 1 hour

4. Compaction Settings

Use Case: Tune compaction frequency for high-throughput compacted topics.

Example:

# Aggressive compaction (for Kafka Streams state stores)
kafka-topics --create --topic app-state-store \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=10000 \
  --config min.cleanable.dirty.ratio=0.3

5. Replication Settings

Use Case: Override broker default for critical topics.

Example:

# Critical topic (require 3 ISR replicas)
kafka-topics --create --topic payments \
  --replication-factor 3 \
  --config min.insync.replicas=3

Warning: min.insync.replicas=3 with RF=3 means all replicas must be in-sync. If 1 broker is down, writes fail (high durability, lower availability).

6. Message Format Settings

Use Case: Force compression for bandwidth savings, increase max message size for large payloads.

Example:

# Enable compression (reduce disk/network usage)
kafka-topics --create --topic large-messages \
  --config compression.type=lz4

# Allow 100 MB messages
kafka-topics --create --topic file-uploads \
  --config max.message.bytes=104857600

7. Timestamp Settings

Use Case: LogAppendTime for untrusted clients (prevent fake timestamps).

Example:

# Broker-set timestamps (for consistency)
kafka-topics --create --topic logs \
  --config message.timestamp.type=LogAppendTime

8. Index Settings

Use Case: Smaller index interval = faster lookups (but larger index files).

Use Cases for Topic-Level Overrides

Use Case 1: High-Volume Logs (Short Retention)

Scenario: Application logs, high throughput (1 GB/sec), short retention (1 day).

Config:

kafka-topics --create --topic app-logs \
  --partitions 50 \
  --replication-factor 3 \
  --config retention.ms=86400000 \
  --config segment.bytes=536870912 \
  --config compression.type=lz4

Why:

• Short retention (1 day) → saves disk space
• Small segments (512 MB) → fine-grained retention
• LZ4 compression → reduces disk/network usage

Use Case 2: CDC Topic (Compacted)

Scenario: Change Data Capture, replicate database table to Kafka.

Config:

kafka-topics --create --topic user-profiles \
  --partitions 10 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 \
  --config delete.retention.ms=86400000

Why:

• Compaction → keep only latest value per key
• min.compaction.lag.ms=60000 → wait 1 minute (allow late updates)
• delete.retention.ms=86400000 → keep tombstones for 1 day (consumers see deletes)

Use Case 3: Critical Financial Transactions

Scenario: Payment processing, highest durability, low throughput.

Config:

kafka-topics --create --topic payments \
  --partitions 10 \
  --replication-factor 3 \
  --config min.insync.replicas=3 \
  --config retention.ms=31536000000  # 1 year
  --config compression.type=snappy

Why:

• min.insync.replicas=3 → all replicas must ack (highest durability)
• Long retention (1 year) → regulatory compliance
• Snappy compression → balance speed and compression ratio

Use Case 4: IoT Telemetry (High Throughput, No Ordering)

Scenario: Device telemetry, 10 GB/sec, no ordering needed.

Config:

kafka-topics --create --topic iot-telemetry \
  --partitions 100 \
  --replication-factor 3 \
  --config retention.ms=259200000  # 3 days
  --config compression.type=lz4 \
  --config message.timestamp.type=LogAppendTime

Why:

• Many partitions (100) → high throughput
• Short retention (3 days) → disk savings
• LogAppendTime → consistent timestamps (devices may have clock skew)

Viewing Topic Configs

View All Topic Configs

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092

Output:

Configs for topic 'orders' are:
  compression.type=lz4 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=lz4}
  retention.ms=2592000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000, DEFAULT_CONFIG:log.retention.ms=604800000}

Interpretation:

• compression.type=lz4 is a topic-level override
• retention.ms=2592000000 (30 days) overrides broker default log.retention.ms=604800000 (7 days)

View Only Non-Default Configs

kafka-topics --describe --topic orders --bootstrap-server localhost:9092

Output:

Topic: orders
PartitionCount: 10
ReplicationFactor: 3
Configs: retention.ms=2592000000, compression.type=lz4

Removing Topic Configs (Revert to Broker Default)

Remove Override

kafka-configs --alter --entity-type topics --entity-name orders \
  --delete-config retention.ms \
  --bootstrap-server localhost:9092

Result: Topic orders now uses broker default log.retention.ms.

Topic Config Anti-Patterns

Anti-Pattern 1: Setting Same Config on All Topics

Example: Setting retention.ms=604800000 (7 days) on every topic manually.

Problem: Duplicates broker default. If you want to change retention cluster-wide, you must update every topic.

Solution: Use broker default (log.retention.ms=604800000). Only override for topics that need different retention.

Anti-Pattern 2: min.insync.replicas=1 (No Durability)

Example:

kafka-topics --create --topic orders --replication-factor 3 --config min.insync.replicas=1

Problem: With min.insync.replicas=1, only the leader must ack writes. If leader crashes before replication, data loss occurs.

Solution: Use min.insync.replicas=2 (requires at least 2 replicas to ack).

Anti-Pattern 3: Very Short segment.ms (e.g., 1 minute)

Example:

kafka-topics --create --topic logs --config segment.ms=60000  # 1 minute

Problem:

• Too many segment files (one per minute = 1440 files/day/partition)
• High metadata overhead (more files = more inodes, more file handles)
• Slow retention checks (Kafka must check 1440 files/day)

Solution: Use default segment.ms=604800000 (7 days) unless you have a specific reason to roll segments more frequently.

Key Takeaways

1. Topic Configs Override Broker Defaults: Every topic inherits broker-level defaults. Topic configs allow per-topic customization.

2. Common Overrides: Retention (retention.ms), cleanup policy (cleanup.policy), compression (compression.type), min ISR (min.insync.replicas).

3. Use Cases: Short retention for logs, compaction for CDC, strict ISR for critical data.

4. Dynamic Updates: Topic configs can be changed without broker restart (Kafka 1.1+).

5. View Configs: Use kafka-configs --describe to see topic-level overrides and broker defaults.

6. Remove Overrides: Use --delete-config to revert to broker default.

7. Anti-Patterns: Don't duplicate broker defaults on every topic, don't use min.insync.replicas=1 for critical data, don't use very short segment.ms.

8. Best Practice: Only override topic configs when needed. Most topics should use broker defaults.

What's Next?

Next Module: 6.3 – Config Precedence & Overrides

What We Covered:

• ✅ Topic-level configuration overview
• ✅ Common topic configs (retention, cleanup, compaction, replication, compression)
• ✅ Use cases for overrides (logs, CDC, financial transactions, IoT telemetry)
• ✅ Viewing and removing topic configs
• ✅ Anti-patterns (duplicate defaults, min.insync.replicas=1, very short segment.ms)

What We Didn't Cover (Explicitly Deferred):

• ❌ Config precedence rules → Module 6.3
• ❌ Dynamic config updates → Module 7.1
• ❌ CLI management → Module 8.1

Topic-Level Configurations: Complete ✅