6.3 – Config Precedence & Overrides

The Configuration Hierarchy

Kafka has a multi-level configuration model:

┌─────────────────────────────────────────────┐
│         Topic-Level Override                │  ← Highest precedence
│  (kafka-configs --alter --entity-type topics)│
└─────────────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────┐
│       Broker-Level Default                  │  ← Medium precedence
│  (server.properties: log.retention.ms)      │
└─────────────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────┐
│       Kafka Built-In Default                │  ← Lowest precedence
│  (hardcoded in Kafka source code)           │
└─────────────────────────────────────────────┘

Key Rule: Topic-level overrides broker-level, which overrides built-in defaults.

Example: Retention Configuration Precedence

Scenario 1: No Overrides (Use Broker Default)

Broker Config (server.properties):

log.retention.ms=604800000  # 7 days

Topic Config:

kafka-topics --create --topic orders  # No retention.ms specified

Effective Config: retention.ms=604800000 (7 days, from broker default)

Scenario 2: Topic Override

Broker Config (server.properties):

log.retention.ms=604800000  # 7 days

Topic Config:

kafka-topics --create --topic orders --config retention.ms=2592000000  # 30 days

Effective Config: retention.ms=2592000000 (30 days, topic override wins)

Scenario 3: Multiple Retention Configs (Precedence Within Broker)

Broker Config (server.properties):

log.retention.ms=604800000       # 7 days (milliseconds, most precise)
log.retention.minutes=10080      # 7 days (minutes)
log.retention.hours=168          # 7 days (hours)

Precedence: log.retention.ms > log.retention.minutes > log.retention.hours

Effective Config: log.retention.ms=604800000 (most precise wins)

Recommendation: Only set one retention config (prefer log.retention.ms for precision).

Config Name Mapping: Broker vs Topic

Important: Broker and topic configs use different names for the same setting.

Why Different Names?

• Broker configs start with log.* (because they apply to all logs/topics)
• Topic configs omit log. prefix (because they're already scoped to a topic)

Viewing Effective Configuration

Show All Configs (Including Defaults)

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092 --all

Output:

Configs for topic 'orders' are:
  compression.type=lz4 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=lz4}
  retention.ms=2592000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000, DEFAULT_CONFIG:log.retention.ms=604800000}
  segment.bytes=1073741824 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.bytes=1073741824}
  min.insync.replicas=2 sensitive=false synonyms={DEFAULT_CONFIG:min.insync.replicas=2}

Interpretation:

• compression.type=lz4: Topic-level override (no broker default shown)
• retention.ms=2592000000: Topic-level override (overrides broker default 604800000)
• segment.bytes=1073741824: Inherited from broker default log.segment.bytes
• min.insync.replicas=2: Inherited from broker default

Show Only Overrides (Compact View)

kafka-topics --describe --topic orders --bootstrap-server localhost:9092

Output:

Topic: orders
PartitionCount: 10
ReplicationFactor: 3
Configs: compression.type=lz4, retention.ms=2592000000

Interpretation: Only compression.type and retention.ms are overridden (other configs use broker defaults).

Precedence Rules by Config Type

1. Retention

Precedence:

1. Topic-level retention.ms (highest)
2. Topic-level retention.bytes
3. Broker-level log.retention.ms (most precise)
4. Broker-level log.retention.minutes
5. Broker-level log.retention.hours (least precise)
6. Kafka built-in default: 7 days (lowest)

Example:

# Broker: log.retention.ms=604800000 (7 days)
# Topic: retention.ms=2592000000 (30 days)
# Effective: 30 days (topic wins)

2. Segment Size

Precedence:

1. Topic-level segment.bytes
2. Topic-level segment.ms
3. Broker-level log.segment.bytes
4. Broker-level log.segment.ms
5. Kafka built-in default: 1 GB / 7 days

Key Point: A segment rolls when either condition is met (size OR time).

3. Cleanup Policy

Precedence:

1. Topic-level cleanup.policy
2. Broker-level log.cleanup.policy
3. Kafka built-in default: delete

Values: delete, compact, or compact,delete

4. Compression

Precedence:

1. Producer-level compression.type (e.g., props.put("compression.type", "lz4"))
2. Topic-level compression.type
3. Broker-level compression.type
4. Kafka built-in default: producer (broker uses whatever producer sent)

Special Value: compression.type=producer (broker doesn't re-compress, uses producer's compression)

5. Min In-Sync Replicas

Precedence:

1. Topic-level min.insync.replicas
2. Broker-level min.insync.replicas
3. Kafka built-in default: 1

Key Point: Must be set explicitly for durability (default 1 is not safe).

Config Resolution Example

Full Scenario

Broker Config (server.properties):

log.retention.ms=604800000       # 7 days
log.segment.bytes=1073741824     # 1 GB
log.cleanup.policy=delete
min.insync.replicas=2

Topic 1 (orders, no overrides):

kafka-topics --create --topic orders --partitions 10 --replication-factor 3

Effective Config for orders:

retention.ms=604800000         ← Broker default
segment.bytes=1073741824       ← Broker default
cleanup.policy=delete          ← Broker default
min.insync.replicas=2          ← Broker default

Topic 2 (user-profiles, with overrides):

kafka-topics --create --topic user-profiles \
  --partitions 10 \
  --replication-factor 3 \
  --config retention.ms=2592000000 \
  --config cleanup.policy=compact \
  --config min.insync.replicas=3

Effective Config for user-profiles:

retention.ms=2592000000        ← Topic override (30 days)
segment.bytes=1073741824       ← Broker default (no override)
cleanup.policy=compact         ← Topic override
min.insync.replicas=3          ← Topic override

Dynamic vs Static Configs

Static Configs (Require Broker Restart)

Examples:

• broker.id
• log.dirs
• listeners
• zookeeper.connect (ZooKeeper mode)

Reason: These configs are read once at startup and cannot be changed without restarting the broker.

Dynamic Configs (Take Effect Immediately)

Examples:

• log.retention.ms
• log.segment.bytes
• log.cleanup.policy
• min.insync.replicas
• compression.type
• Topic-level configs (all topic configs are dynamic)

Reason: These configs are stored in ZooKeeper/KRaft metadata and can be updated at runtime.

How to Update:

# Update broker-level config (all topics)
kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

# Update topic-level config (single topic)
kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Override Strategies

Strategy 1: Use Broker Defaults for Most Topics

Approach: Set sensible broker defaults, only override for special topics.

Broker Config:

log.retention.ms=604800000       # 7 days (good default for most topics)
log.segment.bytes=1073741824     # 1 GB
min.insync.replicas=2

Topic Overrides (only for special cases):

# Long retention for audit logs
kafka-topics --create --topic audit-logs --config retention.ms=31536000000  # 1 year

# Short retention for high-volume logs
kafka-topics --create --topic app-logs --config retention.ms=86400000  # 1 day

Benefit: Most topics use defaults. Easy to change cluster-wide behavior (update broker config).

Strategy 2: Set Everything at Topic Level (Explicit)

Approach: Don't rely on broker defaults. Set every config explicitly per topic.

Broker Config:

# Minimal broker config (rely on topic overrides)

Topic Config (explicit for every topic):

kafka-topics --create --topic orders \
  --config retention.ms=604800000 \
  --config segment.bytes=1073741824 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete

Benefit: Each topic's config is explicit and self-documenting.

Downside: Lots of duplication. Hard to change cluster-wide settings.

Recommendation: Hybrid Approach

Best Practice:

1. Set sensible broker defaults for common use cases (7-day retention, RF=3, min ISR=2)
2. Override only when needed (long retention for audit logs, compaction for CDC, short retention for high-volume logs)
3. Document overrides (use topic naming or tagging to indicate special configs)

Config Synonyms

Kafka tracks "synonyms" (all levels of config that apply to a setting).

Example:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092 --all | grep retention.ms

Output:

retention.ms=2592000000 sensitive=false synonyms={
  DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000,
  DEFAULT_CONFIG:log.retention.ms=604800000,
  STATIC_BROKER_CONFIG:log.retention.hours=168
}

Interpretation:

• DYNAMIC_TOPIC_CONFIG: Topic-level override (retention.ms=2592000000)
• DEFAULT_CONFIG: Broker-level default (log.retention.ms=604800000)
• STATIC_BROKER_CONFIG: Broker-level fallback (log.retention.hours=168)

Effective Config: 2592000000 (topic-level wins)

Common Precedence Pitfalls

Pitfall 1: Forgetting to Remove Override

Scenario:

1. You set topic-level retention.ms=86400000 (1 day) for testing
2. You change broker default to log.retention.ms=2592000000 (30 days) for all topics
3. Topic orders still uses 1-day retention (topic override still active!)

Solution: Always check and remove topic overrides when no longer needed:

kafka-configs --alter --entity-type topics --entity-name orders \
  --delete-config retention.ms \
  --bootstrap-server localhost:9092

Pitfall 2: Mixing retention.ms and retention.hours

Scenario:

log.retention.ms=604800000      # 7 days
log.retention.hours=24          # 1 day

Effective Config: 604800000 (7 days, ms wins over hours)

Problem: Confusing. Looks like 1-day retention, but actually 7 days.

Solution: Only set one retention config (prefer log.retention.ms).

Pitfall 3: Producer Compression vs Broker/Topic Compression

Scenario:

// Producer compresses with gzip
props.put("compression.type", "gzip");

Broker/Topic Config:

compression.type=lz4

What Happens?

• Producer compresses with gzip, sends to broker
• Broker sees compression.type=lz4 (topic config)
• Broker re-compresses messages from gzip → lz4 (CPU overhead!)

Solution: Set broker/topic compression.type=producer (broker doesn't re-compress).

Key Takeaways

1. Config Hierarchy: Topic-level > Broker-level > Built-in default.

2. Topic Overrides Broker: Topic configs always override broker defaults.

3. Name Mapping: Broker configs use log.* prefix, topic configs omit it (e.g., log.retention.ms vs retention.ms).

4. Retention Precedence: retention.ms > retention.minutes > retention.hours (most precise wins).

5. Dynamic Configs: Most configs (retention, segment, cleanup, min ISR) can be changed without broker restart.

6. Viewing Effective Config: Use kafka-configs --describe --all to see topic overrides and broker defaults.

7. Remove Overrides: Use --delete-config to revert to broker default.

8. Best Practice: Set sensible broker defaults, only override for special topics.

9. Config Synonyms: Kafka tracks all levels of config (topic, broker, built-in) for transparency.

10. Pitfalls: Forgetting to remove overrides, mixing multiple retention configs, producer vs broker compression mismatch.

What's Next?

Next Module: 7.1 – Dynamic Configuration Management

What We Covered:

• ✅ Configuration hierarchy (topic > broker > built-in)
• ✅ Config name mapping (broker log.* vs topic)
• ✅ Precedence rules (retention, segment, cleanup, compression, min ISR)
• ✅ Viewing effective configuration and synonyms
• ✅ Dynamic vs static configs
• ✅ Override strategies (broker defaults, explicit topic configs, hybrid)
• ✅ Common pitfalls (forgetting overrides, mixing configs, compression mismatch)

What We Didn't Cover (Explicitly Deferred):

• ❌ Dynamic config updates at runtime → Module 7.1
• ❌ CLI management → Module 8.1

Config Precedence & Overrides: Complete ✅