KafkaIQ

2.2 Topic Design & Configuration

Module Overview

Goal: Centralize all topic-level decisions.

What You'll Learn:

  • Topic creation and deletion
  • Core topic configurations (retention, cleanup, segments)
  • Default vs per-topic configs
  • Topic-level tradeoffs
  • Topic naming conventions
  • Best practices for topic design

What's NOT Covered Here:

  • Broker configurations (covered in Module 1.1 and 5.1)
  • Partition assignment and replication details (covered in Module 2.3)
  • Performance tuning beyond design-level decisions (covered in Module 5.1)

Key Insight: Topic design decisions made today affect performance, cost, and operations for months or years. Get it right from the start.

1. Topics: The Fundamental Abstraction

What is a Topic?

A Topic is:
├─ A logical channel for messages
├─ Divided into partitions for parallelism
├─ Replicated for durability
├─ Configured with retention and cleanup policies
└─ The primary way applications organize data in Kafka

Topic Anatomy

Topic: "orders"
├─ Partition 0 (Leader: Broker 1, Replicas: [1, 2, 3])
│   └─ Segments: 00000000000000000000.log, 00000000000012345678.log
├─ Partition 1 (Leader: Broker 2, Replicas: [2, 3, 1])
│   └─ Segments: 00000000000000000000.log, 00000000000023456789.log
└─ Partition 2 (Leader: Broker 3, Replicas: [3, 1, 2])
    └─ Segments: 00000000000000000000.log, 00000000000034567890.log

Configuration:
├─ retention.ms = 604800000 (7 days)
├─ cleanup.policy = delete
├─ segment.bytes = 1073741824 (1 GB)
└─ compression.type = producer

2. Topic Creation

Creating a Topic

# Basic topic creation
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3

# With additional configs
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config compression.type=lz4 \
  --config min.insync.replicas=2

Topic Creation: Key Decisions

MUST DECIDE:

1. Topic Name
   └─ Meaningful, consistent naming convention

2. Partition Count
   └─ Based on throughput requirements
   └─ Hard to change later (requires reassignment)

3. Replication Factor
   └─ Typically 3 for production
   └─ Cannot exceed broker count

4. Retention Policy
   └─ How long to keep data

5. Cleanup Policy
   └─ delete (time/size-based) or compact (keep latest per key)

Auto-Creation vs Explicit Creation

AUTO-CREATION (NOT RECOMMENDED for Production):

auto.create.topics.enable=true

When a producer writes to non-existent topic:
├─ Kafka auto-creates it
├─ Uses default configs (usually wrong!)
└─ No validation, no review

EXPLICIT CREATION (RECOMMENDED):

auto.create.topics.enable=false

Benefits:
├─ Deliberate design decisions
├─ Proper partition count
├─ Correct replication factor
├─ Appropriate configs
└─ Prevents typos from creating topics

Best Practice: Disable auto-creation in production. Always create topics explicitly.

3. Topic Naming Conventions

Naming Best Practices

GOOD NAMING:
├─ Descriptive: "customer-orders" not "topic1"
├─ Hierarchical: "ecom.orders.created"
├─ Consistent: Use dots or dashes consistently
├─ Avoid special characters (stick to a-z, 0-9, -, ., _)
└─ Include environment/namespace if needed

EXAMPLES:

By domain:
├─ orders.placed
├─ orders.fulfilled
├─ payments.processed

By application:
├─ myapp.events.user-login
├─ myapp.events.user-logout
├─ myapp.analytics.clicks

By environment:
├─ prod.orders
├─ staging.orders
└─ dev.orders

Topic Naming Anti-Patterns

❌ AVOID:

├─ Generic names: "data", "events", "test"
├─ Abbreviations: "cust_ord" instead of "customer-orders"
├─ Spaces: "my topic" (use "my-topic" or "my.topic")
├─ Special characters: "orders@2024", "topic#1"
├─ Version in name: "orders-v2" (use schema evolution instead)
└─ Mixed conventions: "orders.created" and "payments_processed"

4. Core Topic Configurations

Retention Configuration

Purpose: Control how long messages are kept.

4.1 Time-Based Retention

# Keep messages for 7 days (default)
retention.ms=604800000

# Common values:
# 1 hour:   3600000
# 1 day:    86400000
# 7 days:   604800000
# 30 days:  2592000000
# Forever:  -1 (not recommended)

4.2 Size-Based Retention

# Keep up to 10 GB per partition
retention.bytes=10737418240

# -1 means unlimited (default)
retention.bytes=-1

# How it works:
# Kafka deletes oldest segments when partition exceeds this size

4.3 Retention Precedence

BOTH retention.ms AND retention.bytes configured:

Kafka deletes data when EITHER limit is reached (whichever comes first)

Example:
├─ retention.ms = 7 days
├─ retention.bytes = 10 GB
└─ Data deleted after 7 days OR when partition > 10 GB

Cleanup Policies

Purpose: Define how Kafka removes old data.

4.4 Delete Policy (Default)

cleanup.policy=delete

# Delete old segments based on retention.ms or retention.bytes
# Use for: Event streams, logs, time-series data

Behavior:

Time → 
┌────────────────────────────────────────────┐
│ Active Segment (currently written)        │ ← Never deleted
├────────────────────────────────────────────┤
│ Segment 3 (2 days old)                    │ ← Kept
├────────────────────────────────────────────┤
│ Segment 2 (5 days old)                    │ ← Kept
├────────────────────────────────────────────┤
│ Segment 1 (8 days old)                    │ ← DELETED (> 7 days)
└────────────────────────────────────────────┘

After cleanup:
└─ Only segments within retention window remain

4.5 Compact Policy (Log Compaction)

cleanup.policy=compact

# Keep only the latest value for each key
# Use for: Changelogs, snapshots, KTable topics

Behavior:

BEFORE COMPACTION:

Key: user-123
├─ Offset 10: {name: "Alice", status: "active"}
├─ Offset 25: {name: "Alice", status: "inactive"}
├─ Offset 40: {name: "Alice", status: "active"}
└─ Offset 55: {name: "Alice", status: "deleted"}

AFTER COMPACTION:

Key: user-123
└─ Offset 55: {name: "Alice", status: "deleted"}  ← Only latest kept

Other keys retain their latest values too

4.6 Compact + Delete

cleanup.policy=compact,delete

# Compaction + time-based deletion
# Useful for: Changelogs with eventual purging

Use case:

Scenario: User profile changes
├─ Compact to keep latest state
├─ Delete profiles older than 365 days (GDPR compliance)
└─ Best of both worlds

Segment Configuration

Purpose: Control how data is organized on disk.

4.7 Segment Size

# Create new segment when current reaches 1 GB (default)
segment.bytes=1073741824

# Common values:
# 100 MB:  104857600
# 1 GB:    1073741824 (default)
# 2 GB:    2147483648

Impact:

SMALLER SEGMENTS:
├─ More files on disk
├─ Faster retention cleanup
├─ More granular compaction
└─ Higher overhead

LARGER SEGMENTS:
├─ Fewer files on disk
├─ Slower retention cleanup
├─ Less granular compaction
└─ Lower overhead

4.8 Segment Time

# Create new segment after 7 days (default: 168 hours)
segment.ms=604800000

# Works with segment.bytes: new segment when EITHER limit reached

Compression Configuration

# Compression type: none, gzip, snappy, lz4, zstd
compression.type=producer (default)

# Options:
# producer:  Use producer's compression
# none:      No compression
# gzip:      High compression ratio, slower
# snappy:    Balanced
# lz4:       Fast, low CPU
# zstd:      Best compression, good speed

Compression Comparison:

Type Ratio Speed CPU Use Case
none 1x Fastest None Small messages, network not bottleneck
lz4 3-4x Fast Low General purpose, recommended
snappy 2-3x Fast Medium Legacy, use lz4 instead
gzip 4-5x Slow High Archive, low network bandwidth
zstd 4-5x Medium Medium Best overall, Kafka 2.1+

Recommendation: Use lz4 or zstd for most use cases.

Replication Configuration

# Minimum in-sync replicas required for write success
min.insync.replicas=2

# Common values:
# 1: Accept writes with only leader (risky)
# 2: Leader + 1 follower (recommended)
# 3: Leader + 2 followers (high durability)

How it works with acks:

Producer: acks=all
Topic: min.insync.replicas=2
Replication factor: 3

Write succeeds when:
└─ Leader + at least 1 follower acknowledge

If only leader available (followers down):
└─ Write fails with NOT_ENOUGH_REPLICAS error

Trade-off:
├─ Higher min.insync.replicas = more durability
└─ But lower availability (more failure scenarios)

5. Tiered Storage & Remote Log Storage

What is Tiered Storage?

Tiered Storage (introduced in Kafka 3.6) allows Kafka to offload older log segments to cheaper remote storage (like Amazon S3, Azure Blob, Google Cloud Storage) while keeping recent "hot" data on local broker disks.

TRADITIONAL STORAGE:
┌─────────────────────────────────────┐
│  Broker Local Disk (All Data)      │
│  ├─ Recent segments (hot)           │
│  ├─ Older segments (warm)           │
│  └─ Old segments (cold)             │
│  Cost: High (SSD/NVMe required)     │
└─────────────────────────────────────┘

TIERED STORAGE:
┌─────────────────────────────────────┐
│  Broker Local Disk (Hot Data Only) │
│  ├─ Recent segments (active)        │
│  └─ Segments within local retention │
│  Cost: Moderate (smaller SSD)       │
└─────────────────────────────────────┘
          ↓ (offload old segments)
┌─────────────────────────────────────┐
│  Remote Storage (S3, etc.)          │
│  ├─ Older segments (archived)       │
│  └─ Historical data (cold)          │
│  Cost: Low (object storage)         │
└─────────────────────────────────────┘

Benefits of Tiered Storage

1. COST REDUCTION:
   ├─ Store hot data on expensive SSD
   ├─ Store cold data on cheap object storage (90% cost savings)
   └─ Keep long retention without huge disk costs

2. SIMPLIFIED OPERATIONS:
   ├─ Smaller local disks per broker
   ├─ Faster broker recovery (less data to replicate)
   └─ Elastic storage (grow without adding brokers)

3. COMPLIANCE & AUDITING:
   ├─ Keep data for years without expensive infrastructure
   ├─ Meet regulatory requirements affordably
   └─ Support long-term analytics

4. PERFORMANCE STABILITY:
   ├─ Brokers handle less disk I/O
   ├─ More predictable latency
   └─ Hot data remains fast

Enabling Tiered Storage

Broker Configuration:

# Enable remote storage
remote.storage.enable=true

# Remote log metadata manager (topic-based)
remote.log.metadata.manager.class=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager

# Remote storage manager implementation
remote.log.storage.manager.class=org.apache.kafka.server.log.remote.storage.RemoteStorageManager

# Local retention (how much to keep on broker disk)
remote.log.manager.copy.max.bytes.per.second=1048576  # 1 MB/s upload rate
remote.log.manager.fetch.max.bytes.per.second=1048576  # 1 MB/s download rate

# Remote storage system plugin (S3, Azure, GCS)
remote.log.storage.manager.impl.prefix=rsm-config
rsm-config.remote.log.storage.endpoint=s3://my-kafka-bucket
rsm-config.remote.log.storage.region=us-west-2

Topic Configuration:

# Enable tiered storage for a topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name audit-logs \
  --alter \
  --add-config remote.storage.enable=true

# Set local retention (how long to keep on broker disk)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name audit-logs \
  --alter \
  --add-config local.retention.ms=86400000  # 1 day local

# Total retention can be much longer
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name audit-logs \
  --alter \
  --add-config retention.ms=31536000000  # 365 days total

How Tiered Storage Works

DATA LIFECYCLE WITH TIERED STORAGE:

1. NEW MESSAGES ARRIVE:
   ├─ Written to local broker disk (active segment)
   └─ Served from local disk (fast)

2. SEGMENT ROLLS:
   ├─ Active segment becomes inactive
   └─ Still on local disk

3. LOCAL RETENTION THRESHOLD REACHED:
   ├─ Segment older than local.retention.ms
   ├─ Segment uploaded to remote storage (S3)
   └─ Local copy deleted after successful upload

4. CONSUMER REQUESTS OLD DATA:
   ├─ Broker checks: local disk or remote?
   ├─ If remote: Fetches from S3 on demand
   └─ Cached locally temporarily for performance

5. RETENTION THRESHOLD REACHED:
   ├─ Segment exceeds retention.ms
   └─ Deleted from both local and remote storage

Configuration Strategy

EXAMPLE CONFIGURATIONS:

Hot Topic (order-events):
├─ local.retention.ms = 7 days (keep on disk)
├─ retention.ms = 7 days (no remote storage needed)
└─ remote.storage.enable = false
└─ Use case: High throughput, recent data only

Warm Topic (analytics-events):
├─ local.retention.ms = 1 day (hot data on disk)
├─ retention.ms = 30 days (total retention)
└─ remote.storage.enable = true
└─ Use case: Recent data fast, older data acceptable latency

Cold Topic (audit-logs):
├─ local.retention.ms = 6 hours (minimal local storage)
├─ retention.ms = 365 days (1 year compliance)
└─ remote.storage.enable = true
└─ Use case: Compliance, rarely accessed

Performance Considerations

LATENCY CHARACTERISTICS:

Local Disk Read:
└─ Latency: < 1 ms (SSD)

Remote Storage Read (first time):
├─ Latency: 50-200 ms (S3 fetch)
├─ Cached locally after first fetch
└─ Subsequent reads: < 10 ms (from cache)

RECOMMENDATIONS:
├─ Set local.retention.ms based on active consumer pattern
├─ If consumers typically lag < 1 day → local.retention.ms = 2 days
├─ If consumers replay historical data → expect higher latency
└─ Monitor fetch latency metrics

Monitoring Tiered Storage

KEY METRICS:

Remote Log Size:
└─ kafka.log.remote:type=RemoteLogManager,name=RemoteLogSizeBytes

Remote Copy Rate:
└─ kafka.log.remote:type=RemoteLogManager,name=RemoteLogCopyBytesPerSec

Remote Fetch Rate:
└─ kafka.log.remote:type=RemoteLogManager,name=RemoteLogFetchBytesPerSec

Storage Cost Savings:
└─ Compare local disk usage vs previous (without tiered storage)

Alerts to Configure:
├─ Remote upload failures
├─ Slow remote fetch times (> 500ms)
└─ Remote storage unavailable

When to Use Tiered Storage

✅ USE TIERED STORAGE WHEN:
├─ Long retention requirements (> 7 days)
├─ Compliance/audit needs (months to years)
├─ Cost optimization is priority
├─ Consumers rarely need old data
└─ Acceptable latency for historical data (50-200ms)

❌ DON'T USE TIERED STORAGE WHEN:
├─ All data must be low-latency (< 1ms)
├─ Short retention (< 24 hours)
├─ Consumers frequently replay old data at high throughput
├─ Broker disk is not a constraint
└─ Kafka version < 3.6 (not available)

Tiered Storage Best Practices

1. Start with less critical topics
   └─ Test tiered storage behavior before production rollout

2. Set appropriate local.retention.ms
   └─ Based on typical consumer lag (2-3x max lag)

3. Monitor remote storage costs
   └─ S3 storage costs + API calls can add up

4. Configure upload/download rate limits
   └─ Prevent network saturation during bulk operations

5. Test disaster recovery scenarios
   └─ What happens if remote storage is unavailable?

6. Use lifecycle policies on object storage
   └─ Automatically delete very old data (cost optimization)

6. Default vs Per-Topic Configs

Broker-Level Defaults

# In server.properties (applies to all topics)

# Default partition count for auto-created topics
num.partitions=1

# Default replication factor
default.replication.factor=1

# Default retention
log.retention.hours=168  # 7 days
log.retention.bytes=-1   # Unlimited

# Default segment size
log.segment.bytes=1073741824  # 1 GB

# Default cleanup policy
log.cleanup.policy=delete

Note: These apply ONLY to auto-created topics. Explicitly created topics use specified values.

Configuration Scope: Broker vs Topic

Understanding the two levels of configuration:

┌────────┬────────────────────────────────┬─────────────────────────────────────────┐
│ Scope  │ Applies To                     │ Changed Via                             │
├────────┼────────────────────────────────┼─────────────────────────────────────────┤
│ Broker │ Global defaults for all topics │ server.properties OR dynamic config API │
│ Topic  │ Per-topic overrides            │ kafka-configs.sh OR Admin API           │
└────────┴────────────────────────────────┴─────────────────────────────────────────┘

PRECEDENCE RULE:
└─ Topic-level config ALWAYS overrides broker-level default

Example:

# Broker default (in server.properties)
log.retention.hours=168  # 7 days for all topics

# But for topic "important-data":
kafka-configs.sh --alter \
  --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name important-data \
  --add-config retention.ms=86400000  # 1 day override

Result:
├─ All topics: 7 days retention (broker default)
├─ Topic "important-data": 1 day retention (override wins)
└─ Precedence: Topic config > Broker default

Per-Topic Overrides

# Override retention for specific topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config retention.ms=2592000000

# List topic configs
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

When to use per-topic configs:

✅ Use per-topic configs when:
├─ Topic has unique requirements (longer retention, compaction)
├─ Testing new settings before making them default
└─ Compliance requirements (e.g., GDPR deletion)

❌ Avoid per-topic configs when:
├─ Settings should apply cluster-wide
├─ Creates operational complexity
└─ Makes troubleshooting harder

6. Topic-Level Tradeoffs

Retention vs Storage Cost

LONGER RETENTION:
✅ Historical data available
✅ Replay capability
✅ Disaster recovery
❌ Higher storage cost
❌ Slower compaction
❌ More data to backup

SHORTER RETENTION:
✅ Lower storage cost
✅ Faster recovery
✅ Less data to manage
❌ Limited replay window
❌ Data loss if not consumed
❌ Less debugging capability

Partitions vs Overhead

MORE PARTITIONS:
✅ Higher throughput (more parallelism)
✅ More consumers possible
✅ Better load distribution
❌ More files on disk (partition * segments)
❌ Longer leader election
❌ Higher memory overhead

FEWER PARTITIONS:
✅ Simpler management
✅ Less overhead
✅ Faster recovery
❌ Lower throughput
❌ Less consumer parallelism
❌ Harder to rebalance

Replication Factor vs Cost

HIGHER REPLICATION (RF=3 or more):
✅ Better fault tolerance
✅ Can survive multiple failures
✅ Better read scalability (followers can serve fetch)
❌ 3x storage cost
❌ More network bandwidth
❌ Slower writes (more acks)

LOWER REPLICATION (RF=2):
✅ Lower cost (2x storage)
✅ Faster writes
❌ Lower fault tolerance
❌ Risk of data loss
❌ Not recommended for production

7. Topic Design Patterns

Pattern 1: Event Stream Topics

USE CASE: Order events, clickstreams, logs

CONFIGURATION:
├─ cleanup.policy=delete
├─ retention.ms=604800000 (7 days)
├─ compression.type=lz4
├─ High partition count for throughput
└─ Replication factor = 3

EXAMPLE:
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic clickstream \
  --partitions 50 \
  --replication-factor 3 \
  --config retention.ms=259200000 \
  --config compression.type=lz4 \
  --config min.insync.replicas=2

Pattern 2: Changelog Topics (KTable)

USE CASE: User profiles, product catalog, database CDC

CONFIGURATION:
├─ cleanup.policy=compact
├─ min.compaction.lag.ms=60000 (wait 1 min before compact)
├─ segment.ms=86400000 (1 day)
├─ Moderate partition count
└─ Replication factor = 3

EXAMPLE:
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic user-profiles \
  --partitions 12 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 \
  --config segment.ms=86400000

Pattern 3: Time-Series Data

USE CASE: Metrics, sensor data, telemetry

CONFIGURATION:
├─ cleanup.policy=delete
├─ retention.ms=short (hours or days)
├─ retention.bytes=limit per partition
├─ compression.type=zstd (high ratio)
├─ Many partitions for write throughput
└─ Replication factor = 2 (if data not critical)

EXAMPLE:
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic sensor-metrics \
  --partitions 100 \
  --replication-factor 2 \
  --config retention.ms=43200000 \
  --config retention.bytes=10737418240 \
  --config compression.type=zstd

Pattern 4: Dead Letter Queue (DLQ)

USE CASE: Failed/poisoned messages

CONFIGURATION:
├─ cleanup.policy=delete
├─ retention.ms=long (30+ days for investigation)
├─ Low partition count (low volume expected)
├─ Replication factor = 3 (important for debugging)
└─ No compression (preserve original)

EXAMPLE:
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders-dlq \
  --partitions 3 \
  --replication-factor 3 \
  --config retention.ms=2592000000 \
  --config compression.type=none

8. Topic Deletion

Deleting a Topic

# Delete topic
kafka-topics.sh --bootstrap-server localhost:9092 \
  --delete \
  --topic old-topic

# Prerequisites:
# delete.topic.enable=true (broker config)

What Happens During Deletion

DELETION PROCESS:

1. Topic marked for deletion
   ├─ Controller updates metadata
   └─ Brokers notified

2. Partitions stop accepting writes
   ├─ Producers get UNKNOWN_TOPIC_OR_PARTITION
   └─ Consumers see topic disappear

3. Data files deleted from brokers
   ├─ All log segments removed
   └─ All index files removed

4. Metadata cleaned up
   ├─ ZooKeeper: /brokers/topics/{name} deleted
   └─ KRaft: Tombstone record written

5. Deletion complete
   └─ Topic no longer exists

Topic Deletion Caveats & Common Issues

Broker Configuration Requirement:

# Required in server.properties for deletion to work
delete.topic.enable=true  # Default: true in recent Kafka versions

⚠️ If delete.topic.enable=false: The topic will be marked for deletion but NOT actually removed from disk. The deletion command will succeed, but the topic remains!

Common Deletion Issues:

ISSUE 1: Topic marked for deletion but still exists
├─ Cause: delete.topic.enable=false
├─ Symptom: Topic shows in --list but is unusable
└─ Fix: Set delete.topic.enable=true and restart brokers

ISSUE 2: Log files remain on disk after deletion
├─ Cause: Broker crashed during cleanup OR
│         Background deletion still in progress
├─ Symptom: Files in /var/kafka/data/<topic-name>-* still exist
└─ Fix: Wait for background cleanup OR
        Manually remove after verifying topic truly deleted

ISSUE 3: Slow deletion for large topics
├─ Cause: Many segments to delete
├─ Symptom: Deletion takes minutes
└─ Expected behavior: Let Kafka complete cleanup

ISSUE 4: Deletion during rebalance
├─ Cause: Topic deleted while consumers active
├─ Symptom: Consumer rebalance storms, errors
└─ Prevention: Stop all consumers before deletion

Verification Steps After Deletion:

# Step 1: Verify topic no longer in list
kafka-topics.sh --bootstrap-server localhost:9092 --list | grep orders
# Should return nothing

# Step 2: Manually check disk (on each broker)
ls /var/kafka/data/orders-*
# OR
ls /tmp/kafka-logs/orders-*

# If directories still exist:
# - Check if files are being deleted (size decreasing)
# - Check broker logs for deletion progress
# - Wait for background cleanup (can take minutes)

# Step 3: Check metadata (KRaft)
kafka-metadata.sh --bootstrap-server localhost:9092 describe --topics
# Should not show deleted topic

# Step 4: Verify no lingering state
# Check that no consumer groups are stuck on the deleted topic
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Manual Cleanup (Last Resort):

# ONLY if topic is confirmed deleted from metadata but files remain

# 1. Verify topic truly deleted
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
# Should return: "Topic orders does not exist"

# 2. Stop broker (required for safe cleanup)
systemctl stop kafka

# 3. Remove log directories manually
rm -rf /var/kafka/data/orders-*

# 4. Start broker
systemctl start kafka

# ⚠️ NEVER delete files while broker is running!

Safe Topic Deletion

BEFORE DELETING:

✅ Verify no producers writing
✅ Verify no consumers reading  
✅ Check if topic is used by Kafka Streams apps
✅ Backup data if needed
✅ Notify stakeholders
✅ Document reason for deletion
✅ Confirm delete.topic.enable=true

WHAT YOU CANNOT UNDO:
❌ Data is permanently deleted
❌ No way to recover after deletion
❌ Even with backups, offsets are lost

DELETION TIMELINE:
├─ Small topic (< 1 GB): Seconds
├─ Medium topic (10-100 GB): Minutes  
├─ Large topic (> 1 TB): 10-30 minutes
└─ Background cleanup continues after command returns

Alternative: Topic Rename

Note: Kafka does NOT support renaming topics!

WORKAROUND:

1. Create new topic with desired name
2. Dual-write to both topics (producers)
3. Migrate consumers to new topic
4. Stop writing to old topic
5. Delete old topic once all consumers migrated

This is disruptive! Plan carefully.

9. Viewing Topic Configuration

List All Topics

# List all topics
kafka-topics.sh --bootstrap-server localhost:9092 --list

# List with details
kafka-topics.sh --bootstrap-server localhost:9092 --list --exclude-internal

Describe Topic

# Detailed topic information
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

# Output:
# Topic: orders  PartitionCount: 12  ReplicationFactor: 3  Configs: retention.ms=604800000
#   Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
#   Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
#   ...

View Topic Configs

# Show all configs for topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --describe

# Show only non-default configs
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --describe \
  --all | grep "DYNAMIC_TOPIC_CONFIG"

10. Modifying Topic Configuration

Changing Topic Configs

# Add or update config
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config retention.ms=1209600000

# Remove config (revert to default)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --delete-config retention.ms

# Change multiple configs
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config retention.ms=1209600000,compression.type=zstd,min.insync.replicas=2

Configs You CANNOT Change

IMMUTABLE (Cannot Change After Creation):

❌ Partition count (can only increase, see below)
❌ Replication factor (requires reassignment)

Increasing Partition Count (IRREVERSIBLE!)

# Increase partition count
kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter \
  --topic orders \
  --partitions 24  # Must be GREATER than current count

🚨 CRITICAL WARNING: This operation is IRREVERSIBLE. You can ONLY INCREASE partition count, never decrease!

Why This is Dangerous:

IMPACT OF INCREASING PARTITIONS:

1. KEY DISTRIBUTION CHANGES:
   ├─ Existing messages: Remain in original partitions
   ├─ New messages: Distributed across ALL partitions (including new ones)
   └─ BREAKS ORDERING: Messages with same key may go to different partitions

Example:
Before: 3 partitions
├─ Key "user123" → partition 1
└─ Ordering guaranteed for "user123"

After: Increased to 6 partitions
├─ OLD messages for "user123" → still in partition 1
├─ NEW messages for "user123" → might go to partition 4
└─ ORDERING LOST: Can't guarantee order across partitions

2. CONSUMER GROUP REBALANCE:
   ├─ All consumers rebalance immediately
   ├─ Processing pauses during rebalance
   └─ Can take seconds to minutes

3. CANNOT UNDO:
   └─ Once increased, stuck with new partition count forever
   └─ Only option: Create new topic and migrate (very disruptive)

When Increasing Partitions is Safe:

✅ SAFE SCENARIOS:
├─ Topic uses null keys (no ordering requirements)
├─ Topic is new with minimal data
├─ Consumers don't care about message order
└─ Throughput genuinely requires more partitions

❌ RISKY SCENARIOS:
├─ Topic uses keys for ordering
├─ Consumers expect ordered processing per key
├─ Topic has significant data already
└─ "Just trying to see if it helps performance"

Best Practice: Get Partition Count Right from the Start

PLANNING PARTITION COUNT:

1. Calculate based on target throughput:
   Partitions = Target Throughput / Single Partition Throughput

   Example:
   ├─ Target: 100 MB/s
   ├─ Single partition: ~10 MB/s
   └─ Partitions needed: 10

2. Consider consumer parallelism:
   └─ Max consumers = partition count

3. Add buffer (20-30%):
   └─ 10 partitions → create with 12-15 partitions

4. Don't over-partition:
   ├─ Each partition = file handles, memory
   ├─ More partitions = more metadata overhead
   └─ Rule of thumb: < 4000 partitions per broker

Alternative to Increasing Partitions:

INSTEAD OF INCREASING PARTITIONS:

Option 1: Optimize throughput
├─ Tune producer batching
├─ Enable compression
└─ Check for bottlenecks

Option 2: Split into multiple topics
├─ Create new topic with more partitions
├─ Route future data to new topic
└─ Keep old topic for historical data

Option 3: Create new topic + migrate
├─ Create new topic with desired partition count
├─ Migrate consumers to new topic
├─ Dual-write producers (temporary)
├─ Delete old topic after migration
└─ Most disruptive but cleanest solution

11. Topic Best Practices Summary

TOPIC DESIGN CHECKLIST:

☑ Use meaningful naming convention
☑ Disable auto-creation in production
☑ Set explicit partition count (based on throughput)
☑ Use replication factor ≥ 3
☑ Configure appropriate retention
☑ Choose correct cleanup policy (delete vs compact)
☑ Enable compression (lz4 or zstd)
☑ Set min.insync.replicas ≥ 2
☑ Document topic purpose and schema
☑ Monitor disk usage and retention

12. Common Configuration Mistakes

❌ MISTAKE 1: Auto-creation enabled
└─ FIX: Set auto.create.topics.enable=false

❌ MISTAKE 2: Single partition for high-throughput topic
└─ FIX: Increase partitions based on target throughput

❌ MISTAKE 3: Retention too short
└─ FIX: Set retention.ms to allow for replay and recovery

❌ MISTAKE 4: Using compact for event streams
└─ FIX: Use cleanup.policy=delete for event data

❌ MISTAKE 5: No compression
└─ FIX: Enable lz4 or zstd compression

❌ MISTAKE 6: RF=1 in production
└─ FIX: Use replication factor ≥ 3

❌ MISTAKE 7: Too many partitions
└─ FIX: Start conservative, increase only if needed

13. Key Takeaways

You've learned:

✅ Topics are the primary abstraction for organizing data
✅ Explicit creation prevents misconfiguration
✅ Naming conventions improve maintainability
✅ Retention controls how long data is kept (time/size)
✅ Cleanup policies define delete vs compact behavior
✅ Segments organize data on disk
✅ Compression reduces storage and network
✅ Per-topic configs override defaults when needed
✅ Tradeoffs exist for retention, partitions, replication
✅ Topic deletion is permanent and cannot be undone

Summary

Topic design is about making the right tradeoffs:

Good Topic Design =
├─ Appropriate retention for use case
├─ Right partition count for throughput
├─ Proper cleanup policy (delete vs compact)
├─ Sufficient replication for durability
├─ Clear naming for maintainability
└─ Documented decisions for future operators

Remember: Topic design decisions affect cost, performance, and operations. Take time to get it right — changing later is expensive or impossible.

Next up: Module 2.3 — Partitions, Replication & ISR Management — Master replication health and partition leadership.