KafkaIQ5.3 KRaft Metadata Quorum
Module Overview
Goal: Understand KRaft (Kafka Raft) — Kafka's modern self-managed coordination system.
What You'll Learn:
- KRaft quorum model and architecture
- Metadata log structure
- Controller quorum voters
- Raft consensus basics (Kafka-specific)
- Differences from ZooKeeper
- Migration from ZK to KRaft
What's NOT Covered Here:
- ZooKeeper details (covered in Module 5.2)
- General Raft algorithm (only Kafka's implementation)
- Controller operations (covered in Module 6.1)
Key Insight: KRaft eliminates Kafka's dependency on ZooKeeper by implementing self-managed metadata using the Raft consensus algorithm. This improves scalability, simplifies operations, and enables faster controller failover.
1. What is KRaft?
Overview
KRaft (Kafka Raft):
└─ Kafka's self-managed metadata quorum
Replaces ZooKeeper:
├─ No external coordination system needed
├─ Kafka manages its own metadata
└─ Uses Raft consensus algorithm
Key benefits:
├─ Simplified operations (one system, not two)
├─ Better scalability (millions of partitions)
├─ Faster controller failover
└─ Lower metadata latency
Architecture
KRAFT CLUSTER:
Brokers:
├─ Data brokers (handle client traffic)
└─ Store partition data
Controller Quorum:
├─ 3-5 dedicated controller servers (recommended)
├─ OR combined mode (brokers also act as controllers)
└─ Maintain metadata log
Metadata Log (__cluster_metadata topic):
├─ Special Kafka topic
├─ Stores all cluster metadata
├─ Replicated via Raft consensus
└─ Controllers read/write metadata log
2. Metadata Log
Structure
__cluster_metadata TOPIC:
Special internal topic:
├─ Single partition (partition 0)
├─ Replicated across controller quorum
├─ Ordered log of metadata changes
└─ Compact cleanup policy (latest state per key)
Log entries:
├─ RegisterBrokerRecord
├─ TopicRecord
├─ PartitionRecord
├─ PartitionChangeRecord
├─ ConfigRecord
└─ ... (all metadata changes)
Example entries:
1. Offset 0: RegisterBrokerRecord(id=1, host=broker1, ...)
2. Offset 1: TopicRecord(name=orders, id=UUID)
3. Offset 2: PartitionRecord(topic=orders, partition=0, replicas=[1,2,3])
4. Offset 3: PartitionChangeRecord(topic=orders, partition=0, leader=1, ISR=[1,2])
Metadata Snapshots
SNAPSHOTS FOR FAST RECOVERY:
Problem:
└─ Replaying entire metadata log is slow for large clusters
Solution:
└─ Periodic snapshots of metadata state
Snapshot process:
1. Controller captures current metadata state
2. Writes snapshot to disk
3. Older log segments can be deleted (before snapshot)
4. On restart: Load snapshot + replay recent log
Benefit:
└─ Fast controller startup (seconds, not minutes)
3. Controller Quorum
Quorum Model
KRAFT QUORUM:
Quorum size: 3 or 5 controllers (typical)
├─ Odd number for majority quorum
└─ Tolerates (N-1)/2 failures
Example (3 controllers):
├─ Quorum: [1, 2, 3]
├─ Majority: 2 controllers
├─ Tolerates: 1 failure
Example (5 controllers):
├─ Quorum: [1, 2, 3, 4, 5]
├─ Majority: 3 controllers
└─ Tolerates: 2 failures
Trade-off:
├─ More controllers: Higher fault tolerance
└─ More controllers: More coordination overhead
Controller Roles
RAFT ROLES IN KRAFT:
Leader (Active Controller):
├─ Handles all metadata writes
├─ Replicates to followers
├─ Serves metadata reads
└─ One leader at a time
Followers:
├─ Replicate metadata log from leader
├─ Can serve metadata reads (slightly stale)
├─ Participate in leader election
└─ Stand by to become leader
Observers (Brokers):
├─ Replicate metadata log (read-only)
├─ Do not participate in leader election
└─ Apply metadata changes to serve clients
4. Raft Consensus (Kafka's Implementation)
Leader Election
KRAFT LEADER ELECTION:
Trigger:
└─ Current leader fails or loses quorum
Process:
1. Followers detect leader failure (heartbeat timeout)
2. Follower transitions to candidate
3. Candidate increments epoch (term in Raft)
4. Candidate sends RequestVote to other controllers
5. Controllers vote for candidate (if eligible)
6. Candidate receives majority votes → Becomes leader
7. New leader sends heartbeats to establish authority
Election term (epoch):
└─ Incremented on each election
└─ Prevents split elections
Log Replication
LOG REPLICATION:
Write path:
1. Client (broker) sends metadata change to leader
2. Leader appends entry to local log
3. Leader replicates to followers (AppendEntries RPC)
4. Followers append to local log and acknowledge
5. Leader waits for majority acknowledgment
6. Leader commits entry (marks as committed)
7. Leader responds to client: Success
8. Committed entries applied to metadata state
Guarantees:
├─ Entries committed only after majority replication
└─ Committed entries durable (survived failures)
High Watermark (HW)
KRAFT HIGH WATERMARK:
Similar to partition HW:
└─ Last offset replicated to majority of quorum
Example:
├─ Leader log: offsets 0-1000
├─ Follower 1 log: offsets 0-998
├─ Follower 2 log: offsets 0-997
└─ HW: 997 (replicated to majority)
Committed metadata:
└─ Entries up to HW are committed
└─ Guaranteed durable (majority replication)
5. Broker Integration with KRaft
Broker Registration
BROKER STARTUP WITH KRAFT:
1. Broker starts and generates unique broker UUID
2. Broker sends BrokerRegistrationRequest to controller:
└─ Broker ID, host, port, rack, listeners, etc.
3. Controller appends RegisterBrokerRecord to metadata log
4. Record replicated to quorum majority
5. Controller responds: Registration successful
6. Broker periodically sends BrokerHeartbeatRequest:
└─ Proves liveness
7. If heartbeat missed (timeout):
└─ Controller marks broker offline
└─ Triggers partition leader elections
Broker Metadata Replication
BROKERS REPLICATE METADATA LOG:
1. Broker subscribes to metadata log
└─ Acts as follower (observer) of __cluster_metadata
2. Controller (leader) sends metadata updates:
└─ Via MetadataFetch or direct notification
3. Broker appends metadata changes to local log
4. Broker applies changes:
├─ Topic creation → Initialize partitions
├─ Partition leader change → Become leader/follower
└─ ISR change → Update local state
5. Broker serves clients based on local metadata
└─ Low latency (local cache, no remote call to controller)
6. Combined vs Dedicated Controllers
Deployment Modes
DEDICATED CONTROLLER MODE (Recommended):
Separate controllers:
├─ 3-5 dedicated controller servers
├─ Do NOT handle client traffic (no data partitions)
└─ Focus only on metadata management
Brokers:
├─ Handle client produce/fetch requests
├─ Replicate metadata log as observers
└─ Do not participate in controller quorum
Benefits:
├─ Resource isolation (controllers not impacted by client load)
├─ Simpler scaling (add brokers without affecting quorum)
└─ Recommended for production
COMBINED MODE:
Brokers also act as controllers:
├─ Some brokers are quorum voters
└─ Handle both client traffic and metadata
Use case:
└─ Small clusters (dev/test)
└─ Not recommended for production
7. KRaft vs ZooKeeper
Key Differences
KRAFT ADVANTAGES:
1. No External Dependency:
├─ ZK: Separate ZooKeeper cluster
└─ KRaft: Self-contained Kafka
2. Scalability:
├─ ZK: Struggles with millions of partitions
└─ KRaft: Designed for high partition count
3. Controller Failover:
├─ ZK: Minutes (load all znodes)
└─ KRaft: Seconds (snapshot + recent log)
4. Metadata Updates:
├─ ZK: Sequential writes to ZK (latency)
└─ KRaft: Log append (batched, faster)
5. Operations:
├─ ZK: Two systems to manage
└─ KRaft: One system
6. Quorum Model:
├─ ZK: ZK ensemble
└─ KRaft: Kafka-native Raft quorum
8. Migration from ZooKeeper to KRaft
Migration Strategies
TWO MIGRATION PATHS:
1. Offline Migration (Downtime):
├─ Stop ZK-based cluster
├─ Export metadata
├─ Start new KRaft cluster
├─ Import metadata
└─ Switch clients to new cluster
2. Live Migration (Zero Downtime, KIP-866):
├─ Dual-write metadata (ZK + KRaft)
├─ Gradually shift traffic to KRaft
├─ Verify consistency
├─ Decommission ZK
└─ Complex but no downtime
Recommendation:
└─ New clusters: Start with KRaft
└─ Existing clusters: Plan migration carefully
Migration Edge Cases
HYBRID MODE CONSIDERATIONS:
During ZK-to-KRaft migration (dual-write phase):
Issue: Controller and ZooKeeper disagree on leader assignments
Cause:
├─ ZK might have stale leader metadata
├─ KRaft metadata log might have newer state
└─ Network partition during transition
Resolution:
├─ Controller prefers KRaft metadata (source of truth in migration)
├─ Brief inconsistency possible (resolved on next metadata refresh)
└─ Clients retry with updated metadata
Mitigation:
├─ Plan migration during maintenance window
├─ Monitor both ZK and KRaft state during transition
├─ Validate consistency before finalizing migration
└─ Have rollback plan ready
9. Common KRaft Quorum Pitfalls
Critical Operational Mistakes to Avoid
COMMON PITFALLS:
1. Adding controller without updating voters list ⚠️ HIGH IMPACT
2. Mismatched cluster UUIDs
3. Insufficient quorum size
4. Network partition without majority
5. Forgetting to format storage
Pitfall #1: Adding Controller Without Updating Voters List
THE PROBLEM:
Admin adds a 4th broker to 3-broker cluster:
New broker config (broker-4):
```properties
process.roles=broker,controller
node.id=4
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
But forgets to update controller.quorum.voters on ALL existing brokers:
Existing brokers still have:
# ❌ WRONG: Missing node 4!
controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093
Should be:
# ✅ CORRECT: Include all controllers
controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093,4@broker4:9093
WHAT HAPPENS:
Broker 4 starts successfully (no error!)
Broker 4 attempts to join controller quorum: ├─ Sends VoteRequest to existing controllers └─ Existing controllers reject (not in their voters list)
Broker 4 cannot participate in elections: ├─ Not counted as voter ├─ Cannot become active controller └─ Does NOT increase fault tolerance
Admin's false assumption: ├─ Admin thinks: "4 nodes, quorum = 3, can tolerate 2 failures" └─ Reality: 3 voters, quorum = 2, can only tolerate 1 failure
Disaster scenario (2 brokers fail): ├─ Brokers 1 and 2 die ├─ Only broker 3 remains as voter (1/3 < majority) ├─ Broker 4 doesn't count (not in voters list!) ├─ Cluster loses quorum └─ No metadata operations possible: ├─ Cannot elect new leaders ├─ Cannot create topics └─ Cannot handle broker failures
OBSERVABILITY CLUES:
Check quorum status:
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
# Output shows:
CurrentVoters: [1,2,3] # ❌ Missing node 4!
CurrentObservers: [4,5,6] # Node 4 is observer, not voter
Broker 4 logs show:
WARN Not in the latest known voters configuration [1,2,3]
INFO Registered as observer, not as voter
HOW TO FIX:
- Update controller.quorum.voters on ALL brokers:
# On brokers 1, 2, 3, 4:
controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093,4@broker4:9093
- Perform rolling restart:
# Restart each broker one at a time
# Wait for broker to rejoin cluster between restarts
kafka-server-stop.sh
kafka-server-start.sh config/server.properties
- Verify quorum status:
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
# Should now show: CurrentVoters: [1,2,3,4]
PREVENTION:
✅ Use configuration management (Ansible, Terraform, Chef) └─ Ensures consistency across all nodes
✅ Validate before adding nodes:
# Check current voters
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
# Ensure new node ID is in voters list
grep controller.quorum.voters /etc/kafka/server.properties
✅ Document procedure: ├─ Step 1: Update voters config on all nodes ├─ Step 2: Rolling restart ├─ Step 3: Add new broker └─ Step 4: Verify quorum status
---
### Pitfall #2: process.roles vs Quorum Voters
SCENARIO COMPARISON:
Case A: New broker with process.roles=broker,controller (not in voters) ├─ Broker joins as observer (not voter) ├─ Does NOT increase fault tolerance ├─ Creates operational illusion └─ ❌ BAD: Admin expects higher fault tolerance
Case B: New broker with process.roles=broker (data broker only) ├─ Broker joins as data broker ├─ Does not affect controller quorum ├─ No expectation of increased fault tolerance └─ ✅ OK: Clear role separation
The problem with Case A: └─ process.roles=controller creates expectation that broker participates in quorum └─ But without updating voters list, it doesn't └─ Admin may assume cluster can tolerate more failures than it actually can
BEST PRACTICE:
If adding data broker only:
process.roles=broker # Clear: Not a controller
If adding controller:
- Update controller.quorum.voters on ALL nodes first
- Then start new controller broker
process.roles=broker,controller
---
### Pitfall #3: Mismatched Cluster UUIDs
THE PROBLEM:
Broker formatted with different cluster UUID than existing cluster
Example: Existing cluster UUID: xB3fK8fFRj-8z3yZq7X9wA New broker UUID: yC4gL9gGSk-9a4zZr8Y0xB # ❌ Different!
WHAT HAPPENS:
New broker cannot join cluster: ├─ Controller rejects registration ├─ UUID mismatch detected └─ Broker logs show:
ERROR Cluster ID yC4gL9gGSk-9a4zZr8Y0xB doesn't match stored ID xB3fK8fFRj-8z3yZq7X9wA
FATAL Exiting Kafka due to cluster ID mismatch
HOW TO FIX:
- Check existing cluster UUID:
kafka-metadata.sh --snapshot /var/lib/kafka/data/meta.log | grep clusterId
- Format new broker with correct UUID:
kafka-storage.sh format \
-t xB3fK8fFRj-8z3yZq7X9wA \ # Use existing cluster UUID!
-c server.properties
PREVENTION:
✅ Store cluster UUID in configuration management ✅ Automate formatting with correct UUID ✅ Validate UUID before starting broker
---
### Pitfall #4: Insufficient Quorum Size
THE PROBLEM:
Running KRaft with only 1 or 2 controller nodes
1-node quorum: ├─ Cannot tolerate any failures ├─ Single point of failure └─ ❌ NOT production-ready
2-node quorum: ├─ Quorum requires majority (2/2 = 100%) ├─ Any failure loses quorum └─ ❌ NO fault tolerance
MINIMUM PRODUCTION CONFIG:
3-node quorum: ├─ Quorum requires 2 nodes (majority of 3) ├─ Tolerates 1 failure └─ ✅ Minimum for production
5-node quorum: ├─ Quorum requires 3 nodes (majority of 5) ├─ Tolerates 2 failures └─ ✅ Recommended for critical systems
BEST PRACTICE:
Always use odd number of controllers: ├─ 3 controllers: Tolerates 1 failure ├─ 5 controllers: Tolerates 2 failures └─ 7+ controllers: Rarely needed (diminishing returns)
---
### Pitfall #5: Network Partition Without Majority
THE PROBLEM:
Network partition splits 3-node quorum: [1,2] vs [3]
Partition A (nodes 1, 2): ├─ Has majority (2/3) ├─ Can elect leader └─ ✅ Continues operating
Partition B (node 3): ├─ Minority (1/3) ├─ Cannot form quorum └─ ❌ Cannot elect leader, becomes read-only
If partition is [1] vs [2,3]: └─ Partition [2,3] has majority, continues └─ Partition [1] isolated
PREVENTION:
✅ Deploy controllers across failure domains (racks, AZs) ✅ Use 5 controllers for higher availability ✅ Monitor network connectivity between controllers ✅ Alert on quorum loss immediately
---
## 10. KRaft Configuration
### Key Configs
KRAFT-SPECIFIC CONFIGURATIONS:
process.roles: ├─ "broker" → Data broker only ├─ "controller" → Controller only └─ "broker,controller" → Combined mode
node.id: └─ Unique ID for this server (replaces broker.id in combined mode)
controller.quorum.voters: └─ List of controller voters: "1@controller1:9093,2@controller2:9093,3@controller3:9093"
metadata.log.dir: └─ Directory for metadata log (__cluster_metadata)
controller.listener.names: └─ Listener name for controller communication
metadata.log.max.record.bytes.between.snapshots: └─ Trigger snapshot creation (default: 20 GB)
---
### Advanced Controller Quorum Configurations
CONTROLLER ELECTION AND REPLICATION CONFIGS:
These configurations fine-tune controller quorum behavior, failover speed, and replication characteristics.
---
### controller.quorum.election.timeout.ms
CONFIGURATION:
controller.quorum.election.timeout.ms = 1000 # Default: 1000ms
Purpose: └─ How long a follower waits without hearing from leader before starting election
Behavior: ├─ Follower receives no heartbeat for this duration ├─ Follower transitions to candidate state ├─ Triggers leader election └─ New leader elected via Raft voting
Tuning trade-offs:
Lower values (e.g., 500ms): ├─ ✅ Faster failover detection ├─ ✅ Quicker recovery from leader failure ├─ ❌ Risk of false positive elections (network blip) └─ ❌ More election churn under transient issues
Higher values (e.g., 2000ms): ├─ ✅ More stable (fewer unnecessary elections) ├─ ✅ Tolerates brief network hiccups ├─ ❌ Slower failover (longer downtime) └─ ❌ Delayed detection of real failures
Recommended: └─ 1000-2000ms for production └─ Balance between speed and stability
---
### controller.quorum.fetch.timeout.ms
CONFIGURATION:
controller.quorum.fetch.timeout.ms = 2000 # Default: 2000ms
Purpose: └─ Timeout for metadata fetch requests during replication
Behavior: ├─ Followers send FetchRequest to leader ├─ If no response within timeout → retry └─ Used for metadata log replication
Tuning guidelines:
Should be higher than election timeout: └─ Typically: 2x election.timeout.ms └─ Prevents premature election during slow fetch
Example: ├─ election.timeout.ms = 1000ms └─ fetch.timeout.ms = 2000ms (2x)
If fetch.timeout < election.timeout: └─ Risk: Fetch timeout triggers unnecessary elections
Recommended: └─ 2000-4000ms └─ 2x election timeout as baseline
---
### controller.quorum.request.timeout.ms
CONFIGURATION:
controller.quorum.request.timeout.ms = 2000 # Default: 2000ms
Purpose: └─ Generic timeout for controller-to-controller RPC calls
Applies to: ├─ Vote requests (during election) ├─ Append requests (log replication) ├─ Metadata sync requests └─ All controller quorum communication
Tuning: ├─ Higher values: More tolerance for slow network ├─ Lower values: Faster detection of unreachable nodes └─ Should align with fetch.timeout.ms
Recommended: └─ 2000-3000ms └─ Match or slightly exceed fetch timeout
---
### controller.quorum.election.backoff.max.ms
CONFIGURATION:
controller.quorum.election.backoff.max.ms = 1000 # Default: 1000ms
Purpose: └─ Maximum backoff time between election retry attempts
Prevents election storms:
Without backoff: ├─ Failed election → immediate retry ├─ Network congestion worsens └─ Continuous election attempts
With backoff: ├─ Failed election → wait (randomized up to max) ├─ Gradually increasing delays ├─ Gives network time to stabilize └─ Reduces election storm impact
Example backoff sequence: ├─ 1st retry: ~100ms delay ├─ 2nd retry: ~200ms delay ├─ 3rd retry: ~400ms delay ├─ 4th retry: ~800ms delay └─ 5th+ retry: ~1000ms delay (max)
Recommended: └─ 1000-2000ms └─ Higher for unstable networks
---
### metadata.max.idle.interval.ms
CONFIGURATION:
metadata.max.idle.interval.ms = 500 # Default: 500ms
Purpose: └─ Maximum time controller waits idle before writing keepalive record
Behavior: ├─ If no metadata changes for this duration ├─ Controller writes NoOpRecord to metadata log └─ Ensures continuous log activity
Why it matters:
Prevents long gaps in metadata log: └─ Easier recovery (smaller replay window)
Keeps replication active: └─ Followers stay engaged
Helps with monitoring: └─ Continuous log writes = healthy controller
Trade-offs:
Lower values (e.g., 100ms): ├─ More frequent writes ├─ Higher overhead └─ Minimal log gaps
Higher values (e.g., 5000ms): ├─ Less overhead ├─ Larger log gaps └─ Slower follower detection
Recommended: └─ 500-1000ms (default is good) └─ Lower for high-availability clusters
---
### Configuration Matrix
RECOMMENDED CONFIGURATIONS BY CLUSTER SIZE:
Small Cluster (3 nodes): ├─ controller.quorum.election.timeout.ms = 1000 ├─ controller.quorum.fetch.timeout.ms = 2000 ├─ controller.quorum.request.timeout.ms = 2000 ├─ controller.quorum.election.backoff.max.ms = 1000 └─ metadata.max.idle.interval.ms = 500
Medium Cluster (5-10 nodes): ├─ controller.quorum.election.timeout.ms = 1500 ├─ controller.quorum.fetch.timeout.ms = 3000 ├─ controller.quorum.request.timeout.ms = 3000 ├─ controller.quorum.election.backoff.max.ms = 1500 └─ metadata.max.idle.interval.ms = 500
Large Cluster (10+ nodes): ├─ controller.quorum.election.timeout.ms = 2000 ├─ controller.quorum.fetch.timeout.ms = 4000 ├─ controller.quorum.request.timeout.ms = 4000 ├─ controller.quorum.election.backoff.max.ms = 2000 └─ metadata.max.idle.interval.ms = 1000
Unstable Network: ├─ Increase all timeouts by 50-100% └─ Higher backoff to reduce election storms
---
### Monitoring Configuration Impact
METRICS TO WATCH:
After tuning, monitor:
kafka.controller:type=KafkaController,name=ActiveControllerCount └─ Should be 1 (frequent changes = election issues)
kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs └─ Low rate = stable (high rate = configuration issue)
kafka.server:type=KafkaServer,name=MetadataLogOffset └─ Should advance continuously
JMX query example:
echo "get kafka.controller:type=KafkaController,name=ActiveControllerCount" | \
java -jar jmxterm.jar -l localhost:9999 -n
Alert conditions: ├─ Active controller count != 1 for >10 seconds ├─ Leader election rate > 1/minute └─ Metadata log offset not advancing
---
## 10. KRaft Operational Tools
### Essential Tools for KRaft Management
KRAFT-SPECIFIC COMMAND-LINE TOOLS:
- kafka-storage → Initialize and format storage
- kafka-metadata-shell → Inspect metadata log snapshots
- kafka-metadata-quorum → Monitor quorum status
- kafka-dump-log → Low-level log inspection
These tools are critical for operators managing KRaft clusters
---
### kafka-storage: Cluster Initialization
PURPOSE:
Format storage directories and generate cluster UUID └─ Required before first KRaft cluster start
COMMANDS:
- Generate cluster UUID:
kafka-storage.sh random-uuid
# Output: xB3fK8fFRj-8z3yZq7X9wA
- Format storage with UUID:
kafka-storage.sh format \
-t xB3fK8fFRj-8z3yZq7X9wA \
-c /etc/kafka/server.properties
# Output:
# Formatting /var/lib/kafka/data
# Formatting /var/lib/kafka/data/__cluster_metadata-0
- Format specific directories:
kafka-storage.sh format \
-t xB3fK8fFRj-8z3yZq7X9wA \
-c server.properties \
--add-scram SCRAM-SHA-256=[name=admin,password=admin-secret]
WHEN TO USE: ├─ Initial cluster setup (before first start) ├─ Adding new controller nodes └─ Rebuilding cluster metadata (disaster recovery)
⚠️ WARNING: └─ Formatting destroys existing metadata └─ Use same cluster UUID across all nodes └─ Never format a running cluster
---
### kafka-metadata-shell: Metadata Inspection
PURPOSE:
Interactive shell to inspect metadata log snapshots └─ Browse cluster state without starting broker
USAGE:
# Open metadata shell
kafka-metadata-shell.sh \
--snapshot /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log
# Interactive shell starts:
>> ls /
brokers topics configs producerIds acls
>> ls /brokers
1 2 3
>> cat /brokers/1
{
"id": 1,
"endpoints": ["PLAINTEXT://broker1:9092"],
"rack": "us-east-1a",
...
}
>> ls /topics
orders users payments
>> cat /topics/orders
{
"name": "orders",
"partitions": {
"0": {"replicas": [1,2,3], "leader": 1, "isr": [1,2]},
"1": {"replicas": [2,3,1], "leader": 2, "isr": [2,3]},
"2": {"replicas": [3,1,2], "leader": 3, "isr": [3,1,2]}
},
"configs": {"retention.ms": "604800000", ...}
}
>> ls /configs
brokers/ topics/
>> exit
WHEN TO USE: ├─ Debug metadata issues (offline inspection) ├─ Verify topic configurations ├─ Investigate broker registration problems ├─ Audit cluster state during incidents └─ Validate metadata log after recovery
BENEFITS: └─ No need to start broker └─ Read-only (safe) └─ Fast navigation
---
### kafka-metadata-quorum: Quorum Status Monitoring
PURPOSE:
Monitor controller quorum health and replication status └─ Essential for operational monitoring
USAGE:
# Describe quorum status
kafka-metadata-quorum.sh \
--bootstrap-server localhost:9092 \
describe --status
# Output:
ClusterId: xB3fK8fFRj-8z3yZq7X9wA
LeaderId: 1
LeaderEpoch: 5
HighWatermark: 12345
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 50
CurrentVoters: [1,2,3]
CurrentObservers: [4,5,6]
# Describe replication status
kafka-metadata-quorum.sh \
--bootstrap-server localhost:9092 \
describe --replication
# Output:
NodeId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp
1 12345 0 1640000000000 1640000000000
2 12345 0 1640000000050 1640000000050
3 12340 5 1640000000100 1639999999000
# Check if node is observer or voter
kafka-metadata-quorum.sh \
--bootstrap-server localhost:9092 \
describe --status --replication
METRICS TO MONITOR:
LeaderEpoch: └─ Increments on each leader election └─ Frequent changes = instability
HighWatermark: └─ Last offset replicated to majority └─ Should advance continuously
MaxFollowerLag: └─ Largest lag among followers └─ High lag = replication issues
MaxFollowerLagTimeMs: └─ Time since lagging follower last fetched └─ >1000ms = potential issue
CurrentVoters: └─ List of voting nodes └─ Should match configured voters
WHEN TO USE: ├─ Monitor quorum health (continuous) ├─ Detect controller failover issues ├─ Identify slow replication ├─ Verify voter configuration └─ Troubleshoot split quorum scenarios
ALERT ON: ├─ MaxFollowerLag > 1000 records ├─ MaxFollowerLagTimeMs > 2000ms └─ CurrentVoters count != expected
---
### kafka-dump-log: Low-Level Log Inspection
PURPOSE:
Dump contents of metadata log or data log for debugging └─ Last resort for deep troubleshooting
USAGE:
- Dump metadata log with decoder:
kafka-dump-log.sh \
--cluster-metadata-decoder \
--files /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log
# Output:
Dumping /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 0 CreateTime: 1640000000000 size: 89 magic: 2 compresscodec: none crc: 123456789
| offset: 0 CreateTime: 1640000000000 RegisterBrokerRecord(brokerId=1, brokerEpoch=0, ...)
baseOffset: 1 lastOffset: 1 count: 1 ...
| offset: 1 CreateTime: 1640000000010 TopicRecord(name='orders', topicId=abc-123, ...)
baseOffset: 2 lastOffset: 2 count: 1 ...
| offset: 2 CreateTime: 1640000000020 PartitionRecord(topicId=abc-123, partitionId=0, replicas=[1,2,3], ...)
- Dump data log (partition log):
kafka-dump-log.sh \
--files /var/lib/kafka/data/orders-0/00000000000000000000.log \
--print-data-log
# Shows actual records in partition
- Dump with offset/timestamp filters:
kafka-dump-log.sh \
--cluster-metadata-decoder \
--files /var/lib/kafka/data/__cluster_metadata-0/00000000000000000000.log \
--offsets-decoder \
--max-message-size 5242880
WHEN TO USE: ├─ Investigate log corruption ├─ Debug replication issues ├─ Understand metadata changes over time ├─ Recover from disaster scenarios └─ Analyze performance issues (batching, compression)
⚠️ WARNING: └─ Produces large output (pipe to file) └─ Read-only (safe) └─ Use on stopped brokers for consistency
---
### Default Metadata Log Location
METADATA LOG PATH:
Default location: └─ {log.dir}/__cluster_metadata-0/
Example paths: ├─ /var/lib/kafka/data/__cluster_metadata-0/ ├─ /tmp/kraft-combined-logs/__cluster_metadata-0/ └─ {metadata.log.dir}/__cluster_metadata-0/ (if explicitly set)
Files in metadata directory: ├─ 00000000000000000000.log → Metadata log segment ├─ 00000000000000000000.index → Offset index ├─ 00000000000000000000.timeindex → Time index ├─ 00000000000005000000.snapshot → Metadata snapshot └─ quorum-state → Raft quorum state
Configuration:
Separate metadata directory (recommended for dedicated controllers)
metadata.log.dir=/var/lib/kafka/metadata
Or use general log directory
log.dirs=/var/lib/kafka/data
---
### Tool Usage Best Practices
OPERATIONAL GUIDELINES:
kafka-storage: ├─ Always save cluster UUID ├─ Use same UUID across all nodes └─ Store UUID securely (disaster recovery)
kafka-metadata-shell: ├─ Use on snapshots (not active logs) ├─ Create snapshot copy before inspection └─ Automate health checks with scripting
kafka-metadata-quorum: ├─ Monitor continuously (every 30s) ├─ Alert on lag > threshold └─ Integrate with Prometheus/Grafana
kafka-dump-log: ├─ Last resort (very verbose) ├─ Pipe output to file for analysis └─ Use grep to filter relevant entries
MONITORING SCRIPT EXAMPLE:
#!/bin/bash
# Check KRaft quorum health
MAX_LAG=1000
MAX_LAG_TIME_MS=2000
STATUS=$(kafka-metadata-quorum.sh \
--bootstrap-server localhost:9092 \
describe --status)
FOLLOWER_LAG=$(echo "$STATUS" | grep MaxFollowerLag | awk '{print $2}')
LAG_TIME=$(echo "$STATUS" | grep MaxFollowerLagTimeMs | awk '{print $2}')
if [ "$FOLLOWER_LAG" -gt "$MAX_LAG" ]; then
echo "ALERT: Follower lag too high: $FOLLOWER_LAG"
exit 1
fi
if [ "$LAG_TIME" -gt "$MAX_LAG_TIME_MS" ]; then
echo "ALERT: Follower lag time too high: $LAG_TIME ms"
exit 1
fi
echo "KRaft quorum healthy"
exit 0
---
## 11. Key Takeaways
**You've learned:**
✅ **KRaft = self-managed metadata** using Raft consensus
✅ **Metadata log (__cluster_metadata)** stores all cluster state
✅ **Controller quorum** (3-5 servers) maintains metadata log
✅ **Raft consensus** ensures durability (majority replication)
✅ **Snapshots** enable fast controller recovery
✅ **Broker registration** via BrokerRegistrationRequest + heartbeats
✅ **Dedicated controllers recommended** for production
✅ **Combined mode** for small/test clusters
✅ **KRaft advantages** — scalability, speed, simplicity
✅ **Migration from ZK** — offline or live (zero downtime)
---
## Summary
**KRaft is Kafka's modern coordination system:**
Architecture = ├─ Controller quorum (3-5 servers) ├─ Metadata log (__cluster_metadata topic) ├─ Brokers as observers (replicate metadata) └─ Raft consensus (majority writes)
Metadata Log = ├─ Ordered log of metadata changes ├─ Single partition, replicated via Raft ├─ Snapshots for fast recovery └─ Compacted (latest state per key)
Raft Roles = ├─ Leader: Handles writes, replicates log ├─ Followers: Replicate log, standby for leadership └─ Observers (Brokers): Read-only replication
Advantages = ├─ No ZooKeeper dependency ├─ Better scalability (millions of partitions) ├─ Faster controller failover (seconds) ├─ Simpler operations (one system) └─ Modern consensus algorithm
Deployment = ├─ Dedicated controllers (recommended) ├─ Combined mode (dev/test) └─ New clusters should use KRaft
**Remember:** KRaft is the future of Kafka coordination. It eliminates ZooKeeper, improves scalability, and simplifies operations. All new Kafka deployments should use KRaft mode.
**Next up:** Module 6.1 — The Kafka Controller Internals — Deep dive into controller behavior.