KafkaIQ

7.1 – Dynamic Configuration Management

What is Dynamic Configuration?

Dynamic configuration allows you to change Kafka settings without restarting brokers.

Before Dynamic Configs (Kafka < 1.1):

  1. Edit server.properties file
  2. Restart broker
  3. Wait for broker to rejoin cluster (~30 seconds)
  4. Clients experience brief unavailability during leader election

After Dynamic Configs (Kafka 1.1+):

  1. Run kafka-configs --alter command
  2. Config takes effect immediately (no restart)
  3. Zero downtime

Types of Configurations

1. Static Configs (Require Restart)

Examples:

  • broker.id
  • log.dirs
  • listeners
  • zookeeper.connect (ZooKeeper mode)

Why Static? These configs are read once at broker startup and affect fundamental broker behavior (identity, network bindings, storage locations).

2. Dynamic Broker Configs (No Restart)

Examples:

  • log.retention.ms
  • log.segment.bytes
  • min.insync.replicas
  • num.io.threads
  • compression.type

Why Dynamic? These configs control runtime behavior and can be hot-reloaded from metadata store (ZooKeeper or KRaft).

3. Topic Configs (Always Dynamic)

All topic-level configs are dynamic.

Examples:

  • retention.ms
  • cleanup.policy
  • segment.bytes
  • compression.type

Why? Topic configs are stored in cluster metadata (not in server.properties), so they can be changed at any time.

Dynamic Config Scopes

Kafka 1.1+ supports three scopes for dynamic broker configs:

Scope Description Applies To
Per-Broker Config applies to a specific broker Only the specified broker (e.g., broker 1)
Cluster-Wide Config applies to all brokers Entire cluster (default for new brokers)
Broker Default Fallback config if no per-broker or cluster-wide config All brokers (unless overridden)

Example: Per-Broker Config

Use Case: Broker 1 has faster disks, so increase its I/O threads.

Command:

kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config num.io.threads=16 \
  --bootstrap-server localhost:9092

Result: Only broker 1 uses num.io.threads=16. Other brokers use their own configs (or cluster default).

Example: Cluster-Wide Config

Use Case: Increase retention cluster-wide (all brokers).

Command:

kafka-configs --alter --entity-type brokers --entity-default \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

Result: All brokers use log.retention.ms=1209600000 (14 days) as the new default.

Dynamic Topic Configuration

Adding/Changing Topic Configs

Command:

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Result: Topic orders retention changes to 30 days immediately (no broker restart).

Removing Topic Configs (Revert to Broker Default)

Command:

kafka-configs --alter --entity-type topics --entity-name orders \
  --delete-config retention.ms \
  --bootstrap-server localhost:9092

Result: Topic orders reverts to broker default log.retention.ms.

Viewing Topic Configs

Command:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092

Output:

Configs for topic 'orders' are:
  compression.type=lz4 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=lz4}
  retention.ms=2592000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=2592000000, DEFAULT_CONFIG:log.retention.ms=604800000}

Dynamic Broker Configuration

Changing Broker Configs (Specific Broker)

Command:

kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

Result: Broker 1 uses 14-day retention. Other brokers use their own configs.

Changing Broker Configs (Cluster-Wide)

Command:

kafka-configs --alter --entity-type brokers --entity-default \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

Result: All brokers (including new brokers) use 14-day retention as default.

Viewing Broker Configs

Command:

kafka-configs --describe --entity-type brokers --entity-name 1 \
  --bootstrap-server localhost:9092 --all

Output:

Configs for broker 1 are:
  log.retention.ms=1209600000 sensitive=false synonyms={DYNAMIC_BROKER_CONFIG:log.retention.ms=1209600000, DEFAULT_CONFIG:log.retention.ms=604800000}
  num.io.threads=8 sensitive=false synonyms={STATIC_BROKER_CONFIG:num.io.threads=8}

Use Cases for Dynamic Configuration

Use Case 1: Emergency Retention Reduction

Scenario: Disk usage is at 95%. Need to free space immediately.

Solution: Reduce retention dynamically (no restart).

Command:

# Reduce retention to 1 day for all brokers
kafka-configs --alter --entity-type brokers --entity-default \
  --add-config log.retention.ms=86400000 \
  --bootstrap-server localhost:9092

Result: Old segments are deleted faster (disk space reclaimed within minutes).

Use Case 2: Gradual Rollout of Config Change

Scenario: You want to test a config change (e.g., more I/O threads) on one broker before applying cluster-wide.

Step 1: Apply to broker 1 (canary):

kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config num.io.threads=16 \
  --bootstrap-server localhost:9092

Step 2: Monitor broker 1 (check throughput, CPU usage)

Step 3: If successful, apply cluster-wide:

kafka-configs --alter --entity-type brokers --entity-default \
  --add-config num.io.threads=16 \
  --bootstrap-server localhost:9092

Use Case 3: Temporary Config Change (Testing)

Scenario: You want to test aggressive compaction for 1 hour.

Step 1: Enable aggressive compaction:

kafka-configs --alter --entity-type topics --entity-name user-profiles \
  --add-config min.cleanable.dirty.ratio=0.2 \
  --bootstrap-server localhost:9092

Step 2: Test for 1 hour

Step 3: Revert to default:

kafka-configs --alter --entity-type topics --entity-name user-profiles \
  --delete-config min.cleanable.dirty.ratio \
  --bootstrap-server localhost:9092

Dynamic Config Best Practices

1. Always Test in Staging First

Why: Config changes can have unintended consequences (e.g., reducing retention too aggressively causes consumer lag).

Workflow:

  1. Test config change in staging
  2. Monitor for 24 hours
  3. If successful, apply to production

2. Use Cluster-Wide Configs for Consistency

Why: Per-broker configs can lead to configuration drift (brokers behave differently).

Recommendation: Prefer cluster-wide configs (--entity-default) over per-broker configs (--entity-name).

Exception: Use per-broker configs only when brokers have different hardware (e.g., fast SSDs on broker 1, slow HDDs on broker 2).

3. Document Config Changes

Why: Dynamic configs are not in server.properties. Easy to forget what changed.

Solution: Maintain a config changelog:

Date: 2023-06-01
Change: Increased log.retention.ms to 14 days (cluster-wide)
Reason: Business requirement for longer data retention
Command: kafka-configs --alter --entity-type brokers --entity-default --add-config log.retention.ms=1209600000

Alternative: Use Infrastructure-as-Code (Terraform, Ansible) to manage configs declaratively.

4. Monitor Config Changes

Kafka emits metrics for config changes:

Key Metrics:

  • kafka.server:type=DynamicBrokerConfig,name=Last* (last config change timestamp)

Alerting: Set up alerts for unexpected config changes (security risk if someone changes configs without authorization).

5. Audit Config Changes

Kafka logs config changes to broker logs:

Example Log:

[2023-06-01 10:00:00,123] INFO Updated broker config: log.retention.ms=1209600000 (kafka.server.DynamicBrokerConfig)

Security: Enable ACLs to restrict who can change configs:

kafka-acls --add --allow-principal User:admin \
  --operation AlterConfigs \
  --cluster \
  --bootstrap-server localhost:9092

Dynamic Config Limitations

1. Static Configs Still Require Restart

Cannot Change Dynamically:

  • broker.id
  • log.dirs
  • listeners
  • zookeeper.connect

Workaround: None. Must restart broker.

2. Config Changes Are Eventual (Not Atomic)

Behavior: When you change a cluster-wide config, each broker applies it independently (not simultaneously).

Example:

  • Time 0:00: You change log.retention.ms=1209600000 (cluster-wide)
  • Time 0:01: Broker 1 applies change
  • Time 0:02: Broker 2 applies change
  • Time 0:03: Broker 3 applies change

Implication: For ~3 seconds, brokers have different retention policies. Usually not a problem, but be aware.

3. No Rollback Mechanism

Problem: If you accidentally change a config to a bad value, you must manually revert.

Example:

# Oops, set retention to 1 second (typo!)
kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=1000

# Immediately revert
kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=604800000

Solution: Always double-check commands before running. Use --describe to verify current value first.

Dynamic vs Static Configuration Workflow

Static Config Workflow (Requires Restart)

Step 1: Edit server.properties:

vi /config/server.properties
# Change: broker.id=2

Step 2: Restart broker:

kafka-server-stop.sh
kafka-server-start.sh /config/server.properties

Step 3: Wait for broker to rejoin (~30 seconds)

Downside: Downtime during restart (clients experience errors until broker is back).

Dynamic Config Workflow (No Restart)

Step 1: Check current value:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092

Step 2: Change config:

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=1209600000 \
  --bootstrap-server localhost:9092

Step 3: Verify change:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092

Downside: None (zero downtime).

Automation with Infrastructure-as-Code

Terraform Example

resource "kafka_topic" "orders" {
  name               = "orders"
  partitions         = 10
  replication_factor = 3

  config = {
    "retention.ms"     = "2592000000"  # 30 days
    "cleanup.policy"   = "delete"
    "compression.type" = "lz4"
  }
}

Benefit: Configs are version-controlled and auditable. Terraform detects drift and applies changes automatically.

Ansible Example

- name: Update topic retention
  kafka_config:
    entity_type: topics
    entity_name: orders
    configs:
      retention.ms: "2592000000"
    bootstrap_servers: "localhost:9092"

Benefit: Idempotent (safe to run multiple times). Ansible tracks config state.

Key Takeaways

  1. Dynamic Configs = No Restart: Most Kafka configs (retention, segment, compaction, min ISR) can be changed without restarting brokers.

  2. Three Scopes: Per-broker (specific broker), cluster-wide (all brokers), broker default (fallback).

  3. Topic Configs Always Dynamic: All topic-level configs can be changed at runtime.

  4. Static Configs Still Exist: broker.id, log.dirs, listeners require restart.

  5. Use Cases: Emergency retention reduction, gradual rollout, temporary testing.

  6. Best Practices: Test in staging, prefer cluster-wide configs, document changes, monitor/audit config changes.

  7. Limitations: Static configs require restart, config changes are eventual (not atomic), no rollback mechanism.

  8. Automation: Use Terraform/Ansible for Infrastructure-as-Code (version control, audit trail).

  9. Workflow: Check current value → Change config → Verify change (zero downtime).

  10. Security: Enable ACLs to restrict who can change configs (AlterConfigs permission).

What's Next?

Next Module: 8.1 – CLI Management for Brokers & Topics

What We Covered:

  • ✅ Dynamic vs static configs
  • ✅ Dynamic config scopes (per-broker, cluster-wide, default)
  • ✅ Changing topic and broker configs dynamically
  • ✅ Use cases (emergency retention reduction, gradual rollout, testing)
  • ✅ Best practices (test in staging, document changes, monitor/audit)
  • ✅ Limitations (static configs, eventual consistency, no rollback)
  • ✅ Automation with IaC (Terraform, Ansible)

What We Didn't Cover (Explicitly Deferred):

  • ❌ CLI management deep dive → Module 8.1
  • ❌ Performance tuning strategy → M6: Operations

Dynamic Configuration Management: Complete ✅