8.1 – CLI Management for Brokers & Topics

Kafka Command-Line Tools Overview

Kafka ships with command-line tools (bin/ directory) for managing brokers and topics.

Key Tools:

• kafka-topics: Create, list, describe, alter, delete topics
• kafka-configs: View and modify configs (broker, topic, client)
• kafka-broker-api-versions: List broker API versions
• kafka-metadata: Inspect cluster metadata (KRaft mode)
• kafka-log-dirs: View log directory usage per broker

Topic Management Commands

1. List Topics

kafka-topics --list --bootstrap-server localhost:9092

Output:

__consumer_offsets
__transaction_state
orders
payments
user-events

2. Create Topic

Basic Creation:

kafka-topics --create --topic orders \
  --partitions 10 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

With Configs:

kafka-topics --create --topic audit-logs \
  --partitions 20 \
  --replication-factor 3 \
  --config retention.ms=31536000000 \
  --config compression.type=lz4 \
  --bootstrap-server localhost:9092

Result:

Created topic audit-logs.

3. Describe Topic

Single Topic:

kafka-topics --describe --topic orders --bootstrap-server localhost:9092

Output:

Topic: orders
TopicId: 8sG2Xj9hRfGkL3mN4oP5q
PartitionCount: 10
ReplicationFactor: 3
Configs: retention.ms=604800000, compression.type=lz4

Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
...

Interpretation:

• Partition 0: Leader is broker 1, replicas on brokers 1, 2, 3
• ISR: All 3 replicas are in-sync

All Topics (overview):

kafka-topics --describe --bootstrap-server localhost:9092

Output:

Topic: orders   PartitionCount: 10  ReplicationFactor: 3  Configs: ...
Topic: payments PartitionCount: 5   ReplicationFactor: 3  Configs: ...
Topic: logs     PartitionCount: 50  ReplicationFactor: 2  Configs: ...

4. Alter Topic (Add Partitions)

Add Partitions:

kafka-topics --alter --topic orders \
  --partitions 15 \
  --bootstrap-server localhost:9092

Warning:

WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected.

Key Point: You can only increase partition count (never decrease). Increasing partitions breaks key-to-partition mapping.

5. Delete Topic

kafka-topics --delete --topic logs --bootstrap-server localhost:9092

Result:

Topic logs is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

Broker Config Required:

delete.topic.enable=true  # Default: true (Kafka 1.0+)

If delete.topic.enable=false: Topic is only marked for deletion (not actually deleted).

6. Check if Topic Exists

kafka-topics --describe --topic orders --bootstrap-server localhost:9092 &> /dev/null && echo "Exists" || echo "Does not exist"

Alternative (list + grep):

kafka-topics --list --bootstrap-server localhost:9092 | grep -q "^orders$" && echo "Exists" || echo "Does not exist"

Broker Management Commands

1. List Brokers

Using kafka-broker-api-versions:

kafka-broker-api-versions --bootstrap-server localhost:9092

Output:

kafka1:9092 (id: 1 rack: us-east-1a) -> (
  ApiVersion(apiKey=PRODUCE, minVersion=0, maxVersion=9),
  ApiVersion(apiKey=FETCH, minVersion=0, maxVersion=13),
  ...
)
kafka2:9092 (id: 2 rack: us-east-1b) -> (
  ...
)
kafka3:9092 (id: 3 rack: us-east-1c) -> (
  ...
)

2. Describe Broker Configs

Specific Broker:

kafka-configs --describe --entity-type brokers --entity-name 1 \
  --bootstrap-server localhost:9092

Output:

Configs for broker 1 are:
  log.retention.ms=604800000 sensitive=false synonyms={DYNAMIC_BROKER_CONFIG:log.retention.ms=604800000}
  num.io.threads=8 sensitive=false synonyms={STATIC_BROKER_CONFIG:num.io.threads=8}

All Brokers (cluster-wide defaults):

kafka-configs --describe --entity-type brokers --entity-default \
  --bootstrap-server localhost:9092

Output:

Default configs for brokers in the cluster are:
  log.retention.ms=604800000 sensitive=false
  min.insync.replicas=2 sensitive=false

3. Alter Broker Configs

Single Broker:

kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

Cluster-Wide:

kafka-configs --alter --entity-type brokers --entity-default \
  --add-config log.retention.ms=1209600000 \
  --bootstrap-server localhost:9092

4. Check Broker Log Directories

View Disk Usage per Broker:

kafka-log-dirs --describe --bootstrap-server localhost:9092 --broker-list 1,2,3

Output (JSON):

{
  "version": 1,
  "brokers": [
    {
      "broker": 1,
      "logDirs": [
        {
          "logDir": "/mnt/disk1/kafka",
          "error": null,
          "partitions": [
            {"partition": "orders-0", "size": 1073741824, "offsetLag": 0, "isFuture": false},
            {"partition": "orders-1", "size": 536870912, "offsetLag": 0, "isFuture": false}
          ]
        }
      ]
    }
  ]
}

Use Case: Find which broker is running out of disk space.

Configuration Management Commands

1. View Topic Configs

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092

Output:

Configs for topic 'orders' are:
  compression.type=lz4 sensitive=false
  retention.ms=2592000000 sensitive=false

2. Set Topic Config

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Result:

Completed updating config for topic orders.

3. Remove Topic Config (Revert to Default)

kafka-configs --alter --entity-type topics --entity-name orders \
  --delete-config retention.ms \
  --bootstrap-server localhost:9092

Result:

Completed updating config for topic orders.

4. Set Multiple Configs at Once

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000,compression.type=lz4,segment.bytes=536870912 \
  --bootstrap-server localhost:9092

Partition Management Commands

1. List Under-Replicated Partitions

Filter topics with under-replicated partitions:

kafka-topics --describe --under-replicated-partitions --bootstrap-server localhost:9092

Output:

Topic: orders  Partition: 3  Leader: 1  Replicas: 1,2,3  Isr: 1,2

Interpretation: Partition 3 has 3 replicas, but only 2 are in-sync (broker 3 is lagging).

2. List Unavailable Partitions

Filter topics with no leader (partition is unavailable):

kafka-topics --describe --unavailable-partitions --bootstrap-server localhost:9092

Output:

Topic: payments  Partition: 0  Leader: none  Replicas: 1,2,3  Isr: 

Interpretation: Partition 0 has no leader (all replicas are down). This is a critical issue.

3. Reassign Partitions (Manual Rebalancing)

Step 1: Generate reassignment plan:

cat > topics-to-move.json <<EOF
{
  "topics": [{"topic": "orders"}],
  "version": 1
}
EOF

kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --broker-list 1,2,3,4 \
  --topics-to-move-json-file topics-to-move.json \
  --generate

Output:

Current partition replica assignment:
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},...]}

Proposed partition reassignment configuration:
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,4],"log_dirs":["any","any","any"]},...]}

Step 2: Save proposed assignment to file:

cat > reassignment.json <<EOF
{"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3,4],"log_dirs":["any","any","any"]},...]}
EOF

Step 3: Execute reassignment:

kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

Step 4: Verify reassignment:

kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Output:

Status of partition reassignment:
Reassignment of partition orders-0 is complete.
Reassignment of partition orders-1 is in progress.
...

Broker Scaling and Rebalancing

⚠️ Critical: Kafka Does NOT Auto-Rebalance

Important Operational Detail: When you add or remove brokers, Kafka does not automatically redistribute existing partitions.

What Happens:

When Adding Brokers:

Before (3 brokers):
  Broker 1: partitions 0, 3, 6, 9
  Broker 2: partitions 1, 4, 7
  Broker 3: partitions 2, 5, 8

After adding Broker 4:
  Broker 1: partitions 0, 3, 6, 9  ← Still overloaded
  Broker 2: partitions 1, 4, 7     ← Still overloaded
  Broker 3: partitions 2, 5, 8     ← Still overloaded
  Broker 4: (empty)                ← Idle!

Result: New broker sits idle. Existing brokers remain overloaded.

When Removing Brokers:

Before (4 brokers):
  Broker 1: partitions 0, 4, 8
  Broker 2: partitions 1, 5, 9
  Broker 3: partitions 2, 6
  Broker 4: partitions 3, 7

After removing Broker 4 (without rebalancing):
  Broker 1: partitions 0, 4, 8
  Broker 2: partitions 1, 5, 9
  Broker 3: partitions 2, 6
  (partitions 3, 7 become unavailable!)

Result: Partitions on removed broker become unavailable until manually reassigned.

Why Kafka Doesn't Auto-Rebalance

Design Decision: Auto-rebalancing would cause:

• Massive data movement (TBs of partition data)
• Network saturation
• Performance degradation during rebalance
• Risk of data inconsistency if done automatically

Trade-off: Kafka gives operators control over when and how to rebalance.

Manual Rebalancing Tools

1. Built-In: kafka-reassign-partitions.sh

Use Case: Small-scale manual rebalancing.

Example (move partitions from Broker 4 to other brokers):

# Step 1: Generate current assignment
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list 1,2,3 \
  --generate

# Step 2: Review proposed assignment (save to file)

# Step 3: Execute reassignment
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# Step 4: Monitor progress
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Limitation: Manual. Requires creating JSON files and monitoring.

2. Cruise Control (Recommended for Production)

What is Cruise Control?

• Open-source project by LinkedIn
• Automated cluster rebalancing
• Self-healing (auto-detects and fixes imbalances)
• Load-aware (balances based on disk, CPU, network, not just partition count)

Features:

• Auto-rebalance when brokers are added/removed
• Disk utilization balancing
• Network bandwidth balancing
• Anomaly detection (hot partitions, slow brokers)
• REST API for automation

Example:

# Add a new broker (Broker 4)
# Cruise Control automatically detects it

# Trigger rebalancing via REST API
curl -X POST "http://cruise-control:9090/kafkacruisecontrol/rebalance?dryrun=false"

# Cruise Control moves partitions to Broker 4 automatically

Installation: https://github.com/linkedin/cruise-control

When to Rebalance

Trigger rebalancing when:

Why Rebalance?

Benefits:

1. Even Disk Usage:

   Before:
     Broker 1: 90% disk (1.8 TB)
     Broker 2: 40% disk (800 GB)
   
   After rebalancing:
     Broker 1: 65% disk (1.3 TB)
     Broker 2: 65% disk (1.3 TB)
   

2. Even Partition Leadership Distribution:

   • Spread leaders across brokers (avoid single broker handling all produce requests)

3. Improved Resilience:

   • Losing one broker impacts fewer partitions

4. Better Throughput:

   • More brokers handling traffic = higher aggregate throughput

Rebalancing Best Practices

1. Use Throttling

Problem: Rebalancing can saturate network and impact production traffic.

Solution: Use replication throttle:

kafka-configs --alter --entity-type brokers --entity-name 1 \
  --add-config leader.replication.throttled.rate=10485760 \
  --bootstrap-server localhost:9092

kafka-configs --alter --entity-type brokers --entity-name 2 \
  --add-config follower.replication.throttled.rate=10485760 \
  --bootstrap-server localhost:9092

Result: Limits rebalancing to 10 MB/sec per broker (doesn't overwhelm network).

2. Rebalance During Low Traffic

Recommendation: Schedule rebalancing during off-peak hours (e.g., 2 AM).

Why: Rebalancing competes with production traffic for network/disk I/O.

3. Monitor Rebalancing Progress

Check partition reassignment status:

kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Monitor metrics:

• Network I/O per broker
• Replication lag
• Under-replicated partitions (should be 0 when complete)

4. Test in Staging First

Always test rebalancing in a staging environment before production.

Test:

• Does rebalancing complete?
• How long does it take?
• What's the impact on client throughput/latency?

Decommissioning a Broker (Step-by-Step)

Scenario: You need to remove Broker 4 from the cluster.

Step 1: Move all partitions off Broker 4:

# Generate reassignment to move partitions to brokers 1-3
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --topics-to-move-json-file all-topics.json \
  --broker-list 1,2,3 \
  --generate

# Execute reassignment
kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

Step 2: Verify all partitions moved:

kafka-reassign-partitions --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Step 3: Stop Broker 4:

kafka-server-stop.sh

Step 4: Remove Broker 4 from cluster metadata (Kafka 2.8+):

kafka-metadata --bootstrap-server localhost:9092 --broker 4 --remove

Summary: Broker Scaling Checklist

Key Takeaway: Always plan for rebalancing when scaling Kafka. It's not automatic, but it's essential for optimal cluster health.

Metadata Management Commands (KRaft Mode)

1. Inspect Cluster Metadata

KRaft Mode (Kafka 3.0+):

kafka-metadata --snapshot /var/lib/kafka-logs/__cluster_metadata-0/00000000000000000000.log \
  --print

Output: Cluster metadata (topics, partitions, brokers, configs).

2. Dump Metadata Quorum

KRaft Mode:

kafka-metadata --cluster-metadata-dir /var/lib/kafka-logs \
  --print-all

Use Case: Debug metadata inconsistencies (e.g., missing broker, stale partition assignment).

Safe CLI Workflows

Workflow 1: Creating a New Topic

Step 1: Check if topic exists:

kafka-topics --list --bootstrap-server localhost:9092 | grep -q "^orders$" && echo "Already exists" || echo "Does not exist"

Step 2: If it doesn't exist, create:

kafka-topics --create --topic orders \
  --partitions 10 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --bootstrap-server localhost:9092

Step 3: Verify creation:

kafka-topics --describe --topic orders --bootstrap-server localhost:9092

Workflow 2: Changing Topic Retention

Step 1: Check current retention:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092 | grep retention.ms

Step 2: Change retention:

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000 \
  --bootstrap-server localhost:9092

Step 3: Verify change:

kafka-configs --describe --entity-type topics --entity-name orders \
  --bootstrap-server localhost:9092 | grep retention.ms

Workflow 3: Deleting a Topic

Step 1: Verify topic name (avoid typos!):

kafka-topics --describe --topic logs --bootstrap-server localhost:9092

Step 2: Delete topic:

kafka-topics --delete --topic logs --bootstrap-server localhost:9092

Step 3: Wait for deletion (check periodically):

kafka-topics --list --bootstrap-server localhost:9092 | grep -q "^logs$" && echo "Still exists" || echo "Deleted"

Warning: Deletion is asynchronous. Topic may not disappear immediately (can take minutes).

Common CLI Pitfalls

Pitfall 1: Using --zookeeper (Deprecated)

Old Way (Kafka < 2.2):

kafka-topics --create --topic orders --zookeeper localhost:2181

New Way (Kafka 2.2+):

kafka-topics --create --topic orders --bootstrap-server localhost:9092

Why: Direct ZooKeeper access is deprecated (removed in Kafka 3.0+). Always use --bootstrap-server.

Pitfall 2: Forgetting --if-not-exists (Topic Creation)

Problem: Re-running create command fails if topic exists.

Solution: Use --if-not-exists:

kafka-topics --create --topic orders --if-not-exists \
  --partitions 10 --replication-factor 3 \
  --bootstrap-server localhost:9092

Result: Creates topic only if it doesn't exist (idempotent).

Pitfall 3: Deleting Topic Without Confirmation

Dangerous:

kafka-topics --delete --topic orders --bootstrap-server localhost:9092

Safer (with confirmation):

echo "Are you sure you want to delete topic 'orders'? Type 'yes' to confirm:"
read confirmation
if [ "$confirmation" = "yes" ]; then
  kafka-topics --delete --topic orders --bootstrap-server localhost:9092
else
  echo "Aborted"
fi

Pitfall 4: Adding Partitions Without Understanding Impact

Problem: Adding partitions breaks key-to-partition mapping.

Example:

# Original: 10 partitions
# hash("user-123") % 10 = 3  →  Partition 3

kafka-topics --alter --topic orders --partitions 15

# After: 15 partitions
# hash("user-123") % 15 = 8  →  Partition 8 (DIFFERENT!)

Impact: Per-key ordering is broken (old messages in partition 3, new messages in partition 8).

Recommendation: Plan partition count upfront. Avoid changing it.

CLI Automation Best Practices

1. Use Scripts for Repeatability

Example (bash script):

#!/bin/bash
set -e

TOPIC="orders"
PARTITIONS=10
REPLICATION_FACTOR=3
RETENTION_MS=604800000

# Create topic
kafka-topics --create --topic "$TOPIC" --if-not-exists \
  --partitions "$PARTITIONS" \
  --replication-factor "$REPLICATION_FACTOR" \
  --config retention.ms="$RETENTION_MS" \
  --bootstrap-server localhost:9092

# Verify
kafka-topics --describe --topic "$TOPIC" --bootstrap-server localhost:9092

2. Use Infrastructure-as-Code (Terraform)

Example:

resource "kafka_topic" "orders" {
  name               = "orders"
  partitions         = 10
  replication_factor = 3

  config = {
    "retention.ms" = "604800000"
  }
}

Benefit: Version control, declarative, audit trail.

3. Enable ACLs for CLI Access

Restrict who can create/delete topics:

kafka-acls --add --allow-principal User:admin \
  --operation Create --operation Delete \
  --topic '*' \
  --bootstrap-server localhost:9092

Result: Only User:admin can create/delete topics (prevents accidental deletions).

Key Takeaways

1. Core CLI Tools: kafka-topics (topic management), kafka-configs (config management), kafka-broker-api-versions (broker info).

2. Topic Operations: Create, list, describe, alter (add partitions), delete.

3. Broker Operations: List brokers, describe configs, alter configs, check log directories.

4. Config Operations: View configs, set configs, remove configs (revert to default).

5. Partition Operations: List under-replicated partitions, reassign partitions.

6. Safe Workflows: Check before create/delete, verify after changes, use --if-not-exists.

7. Common Pitfalls: Using --zookeeper (deprecated), forgetting --if-not-exists, adding partitions without understanding impact.

8. Automation: Use scripts for repeatability, Infrastructure-as-Code for audit trail.

9. Security: Enable ACLs to restrict CLI access.

10. Modern Approach: Always use --bootstrap-server (not --zookeeper).

What's Next?

Congratulations! You've Completed the Brokers & Topics Module.

What We Covered:

• ✅ CLI tools overview
• ✅ Topic management (create, list, describe, alter, delete)
• ✅ Broker management (list, describe, alter configs, check disk usage)
• ✅ Configuration management (view, set, remove configs)
• ✅ Partition management (under-replicated, unavailable, reassignment)
• ✅ Safe workflows and best practices
• ✅ Common pitfalls and automation

What You've Learned Across All Modules:

• ✅ Module 1: Broker fundamentals and cluster role
• ✅ Module 2: Topics, partitions, log structure, segments
• ✅ Module 3: Replication and fault tolerance
• ✅ Module 4: Log management (retention, compaction, cleanup policies)
• ✅ Module 5: Designing topics and partition strategy
• ✅ Module 6: Configuration model (broker vs topic, precedence)
• ✅ Module 7: Dynamic configuration management
• ✅ Module 8: CLI management

Next Steps:

• M3: Kafka Producers (how to write data to Kafka)
• M4: Kafka Consumers (how to read data from Kafka)
• M6: Operations (monitoring, performance tuning, capacity planning)
• M7: Internals (deep replication protocol, controller internals)

CLI Management for Brokers & Topics: Complete ✅

Brokers & Topics Course: Complete 🎉