7.1 Coordination System Operations (ZooKeeper & KRaft)

Module Overview

Goal: Explain Kafka's coordination layer once — this is the authoritative source.

What You'll Learn:

• Why coordination exists in Kafka
• ZooKeeper responsibilities (legacy architecture)
• KRaft architecture (modern, ZooKeeper-free)
• Controller quorum and Raft consensus
• Migration from ZooKeeper to KRaft
• Operational differences between ZK and KRaft
• Troubleshooting coordination issues

What's NOT Covered Here:

• Consumer group coordination (covered in Consumer modules)
• Stream processing coordination (covered in Streams modules)
• General distributed systems theory

Key Insight: Kafka's coordination system is being fundamentally reimagined. ZooKeeper (legacy) is being replaced by KRaft (built-in Raft consensus), eliminating external dependencies and improving scalability.

1. Why Coordination Exists

The Coordination Problem

DISTRIBUTED SYSTEM CHALLENGES:

1. Leader Election
   └─ Who is the controller?
   └─ Who is leader for partition X?

2. Cluster Membership
   └─ Which brokers are alive?
   └─ When did broker Y join/leave?

3. Metadata Management
   └─ What topics exist?
   └─ What are partition assignments?

4. Configuration Management
   └─ What are cluster configs?
   └─ What are topic configs?

5. Coordination Primitives
   └─ Distributed locks
   └─ Sequential IDs
   └─ Watches/notifications

NEED: Single source of truth for cluster state

What Coordination System Provides

CORE SERVICES:

1. Consensus
   └─ Agreement on cluster state across nodes

2. Persistence
   └─ Durable storage of metadata

3. Consistency
   └─ All nodes see same state (eventually)

4. Failure Detection
   └─ Detect when nodes fail

5. Notifications
   └─ Alert nodes when state changes

2. ZooKeeper Mode (Legacy)

ZooKeeper Overview

Apache ZooKeeper:
├─ External coordination service
├─ Runs as separate ensemble (3-5 nodes)
├─ Based on ZAB (ZooKeeper Atomic Broadcast) protocol
├─ Used by Kafka since inception
└─ Being phased out in favor of KRaft

Architecture:
┌──────────────────────────────────────┐
│  Kafka Brokers                       │
│  ├─ Broker 1                         │
│  ├─ Broker 2                         │
│  └─ Broker 3                         │
└──────────────────────────────────────┘
         ↕ (read/write metadata)
┌──────────────────────────────────────┐
│  ZooKeeper Ensemble                  │
│  ├─ ZK Node 1 (leader)               │
│  ├─ ZK Node 2 (follower)             │
│  └─ ZK Node 3 (follower)             │
└──────────────────────────────────────┘

ZooKeeper Responsibilities

WHAT ZOOKEEPER STORES FOR KAFKA:

/brokers/ids/
└─ List of live brokers (ephemeral nodes)

/brokers/topics/
└─ Topic configurations and partition assignments

/controller
└─ Current controller broker ID (ephemeral)

/controller_epoch
└─ Controller election counter

/config/
├─ Broker configs
├─ Topic configs
└─ Client configs

/admin/
└─ Administrative operations (topic creation, deletion, etc.)

/isr_change_notification/
└─ Notifications of ISR changes

/cluster/id
└─ Unique cluster identifier

ZooKeeper Operations

Starting ZooKeeper

# Start ZooKeeper (standalone)
zookeeper-server-start.sh config/zookeeper.properties

# ZooKeeper ensemble requires:
# - At least 3 nodes (recommended: 5)
# - Odd number of nodes (for quorum)
# - Fast, low-latency network between nodes

# ZooKeeper configuration (zoo.cfg):
tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888

Checking ZooKeeper Health

# Check ZooKeeper status
echo ruok | nc localhost 2181
# Should return: imok

# Get ZooKeeper mode (leader/follower)
echo stat | nc localhost 2181

# Check if ZooKeeper is in quorum
echo mntr | nc localhost 2181 | grep zk_server_state

Browsing ZooKeeper Data

# Start ZooKeeper CLI
zkCli.sh -server localhost:2181

# List brokers
ls /brokers/ids
# Output: [0, 1, 2]

# Get broker info
get /brokers/ids/0

# List topics
ls /brokers/topics

# Get controller
get /controller

# Check cluster ID
get /cluster/id

ZooKeeper Operational Challenges

ISSUES WITH ZOOKEEPER:

1. External Dependency
   └─ Separate system to deploy, manage, upgrade

2. Operational Complexity
   └─ Need expertise in both Kafka and ZooKeeper

3. Scaling Limitations
   └─ ZooKeeper struggles with > 200,000 partitions
   └─ Metadata overhead grows with partition count

4. Availability Risk
   └─ ZooKeeper outage impacts Kafka control plane
   └─ Cannot create topics, elect leaders, etc.

5. Slow Metadata Operations
   └─ Controller operations limited by ZooKeeper throughput
   └─ Large clusters experience slow leader elections

6. Split-Brain Risk
   └─ Network partitions can cause consistency issues

SOLUTION: KRaft (Kafka Raft)
└─ Removes ZooKeeper dependency entirely

3. KRaft Mode (Modern Architecture)

What is KRaft?

KRaft = Kafka Raft

Key Properties:
├─ Built-in consensus protocol (Raft)
├─ No external dependencies (ZooKeeper-free)
├─ Metadata stored in internal Kafka topic
├─ Dedicated controller quorum
└─ Production-ready since Kafka 3.3

Architecture:
┌──────────────────────────────────────┐
│  Kafka Controllers (KRaft Quorum)    │
│  ├─ Controller 1 (leader)            │
│  ├─ Controller 2 (follower)          │
│  └─ Controller 3 (follower)          │
│  Stores: __cluster_metadata topic    │
└──────────────────────────────────────┘
         ↕ (metadata replication)
┌──────────────────────────────────────┐
│  Kafka Brokers                       │
│  ├─ Broker 1                         │
│  ├─ Broker 2                         │
│  └─ Broker 3                         │
└──────────────────────────────────────┘

KRaft Controller Quorum

CONTROLLER MODES:

1. Dedicated Controllers (Recommended)
   └─ Separate controller nodes (don't serve data)
   └─ Better isolation, easier to scale

2. Combined Mode
   └─ Brokers also act as controllers
   └─ Simpler deployment (fewer nodes)

QUORUM SIZE:
├─ Minimum: 3 controllers (tolerate 1 failure)
├─ Recommended: 3-5 controllers
└─ Odd number (for Raft majority)

Example:
├─ 3 controllers: Tolerate 1 failure (2 needed for quorum)
├─ 5 controllers: Tolerate 2 failures (3 needed for quorum)
└─ 7 controllers: Tolerate 3 failures (4 needed for quorum)

KRaft Configuration

Controller Configuration

# server.properties for dedicated controller

# Unique node ID (must be unique across cluster)
node.id=1

# Controller role
process.roles=controller

# Controller quorum voters (all controller nodes)
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093

# Listener for controller communication
listeners=CONTROLLER://controller1:9093

# Metadata log directory
metadata.log.dir=/var/kafka/metadata

# Cluster ID (generated once, shared by all nodes)
cluster.id=YourClusterIdHere

Broker Configuration (KRaft)

# server.properties for broker

# Unique node ID
node.id=101

# Broker role
process.roles=broker

# Controller quorum voters (same as controller config)
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093

# Broker listeners
listeners=PLAINTEXT://broker1:9092

# Data log directory
log.dirs=/var/kafka/data

# Cluster ID (same as controllers)
cluster.id=YourClusterIdHere

Combined Mode Configuration

# server.properties for combined broker+controller

node.id=1

# Combined role
process.roles=broker,controller

controller.quorum.voters=1@node1:9093,2@node2:9093,3@node3:9093

# Both broker and controller listeners
listeners=PLAINTEXT://node1:9092,CONTROLLER://node1:9093

log.dirs=/var/kafka/data
metadata.log.dir=/var/kafka/metadata

cluster.id=YourClusterIdHere

Starting KRaft Cluster

Step 1: Generate Cluster ID

# Generate unique cluster ID (once per cluster)
kafka-storage.sh random-uuid

# Output: YourClusterIdHere (use this in all configs)

Step 2: Format Storage

# Format metadata directory (on each node)
kafka-storage.sh format \
  -t YourClusterIdHere \
  -c /path/to/server.properties

# This creates metadata.properties file

Step 3: Start Controllers

# Start controller nodes
kafka-server-start.sh /path/to/controller.properties

# Repeat for all controller nodes (3-5 nodes)

Step 4: Start Brokers

# Start broker nodes
kafka-server-start.sh /path/to/broker.properties

# Brokers will connect to controller quorum

Step 5: Verify Cluster

# Check controller status
kafka-metadata.sh --bootstrap-server localhost:9092 describe --status

# List brokers
kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Create test topic
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic test --partitions 3 --replication-factor 3

4. ZooKeeper vs KRaft Comparison

Architectural Differences

┌───────────────────┬─────────────────────┬──────────────────────┐
│ Feature           │ ZooKeeper Mode      │ KRaft Mode           │
├───────────────────┼─────────────────────┼──────────────────────┤
│ External Deps     │ Yes (ZooKeeper)     │ No                   │
│ Metadata Storage  │ ZooKeeper znodes    │ __cluster_metadata   │
│ Consensus         │ ZAB                 │ Raft                 │
│ Controller        │ Single controller   │ Controller quorum    │
│ Failover Time     │ Seconds             │ Milliseconds         │
│ Partition Limit   │ ~200K               │ Millions             │
│ Startup Time      │ Minutes (large)     │ Seconds              │
│ Operational Comp  │ High (2 systems)    │ Lower (1 system)     │
│ Production Ready  │ Yes (mature)        │ Yes (since 3.3)      │
│ Future Support    │ Deprecated (4.0+)   │ Long-term            │
└───────────────────┴─────────────────────┴──────────────────────┘

Performance Comparison

CONTROLLER FAILOVER:

ZooKeeper Mode:
├─ Detection: 6-30 seconds (ZK session timeout)
├─ Election: 1-10 seconds
├─ Metadata reload: 1-60 seconds (depends on partition count)
└─ Total: Seconds to minutes

KRaft Mode:
├─ Detection: Immediate (Raft heartbeats)
├─ Election: Milliseconds (Raft leader election)
├─ Metadata reload: Not needed (replicated state)
└─ Total: < 1 second typically

METADATA OPERATIONS:

ZooKeeper Mode:
└─ Limited by ZooKeeper write throughput
└─ Large clusters experience slowness

KRaft Mode:
└─ Limited by Kafka throughput (much higher)
└─ Scales to millions of partitions

5. Migration from ZooKeeper to KRaft

Migration Strategies

OPTION 1: Blue-Green Migration
├─ Build new KRaft cluster
├─ Migrate applications to new cluster
├─ Decommission old ZooKeeper cluster
└─ Use case: Can tolerate downtime, simplest approach

OPTION 2: In-Place Migration (Planned)
├─ Kafka will support ZK → KRaft migration
├─ Zero-downtime migration path
├─ Planned for Kafka 3.x (check current status)
└─ Use case: Cannot tolerate downtime

CURRENT STATUS (as of Kafka 3.7):
└─ KRaft is production-ready
└─ ZooKeeper deprecated (removal in Kafka 4.0)
└─ In-place migration tool under development

RECOMMENDATION:
└─ New clusters: Use KRaft
└─ Existing clusters: Plan migration to KRaft before 4.0

Blue-Green Migration Process

STEP 1: Build KRaft Cluster

1. Deploy new Kafka cluster with KRaft
2. Configure with same capacity as old cluster
3. Create topics (match old cluster)
4. Validate cluster health

STEP 2: Migrate Producers

1. Deploy producer applications with new bootstrap servers
2. Dual-write (both clusters) temporarily
3. Validate data in new cluster
4. Stop writing to old cluster

STEP 3: Migrate Consumers

1. Deploy consumer applications with new bootstrap servers
2. Reset offsets or let consumers catch up
3. Validate consumption from new cluster
4. Stop old consumers

STEP 4: Decommission Old Cluster

1. Verify no traffic to old cluster
2. Backup data if needed
3. Stop old Kafka brokers
4. Stop ZooKeeper ensemble
5. Decommission infrastructure

DURATION: Days to weeks (depends on application count)

6. Operational Differences

Administrative Operations

CREATING TOPICS:

ZooKeeper Mode:
kafka-topics.sh --zookeeper localhost:2181 --create ...
(deprecated)

KRaft Mode:
kafka-topics.sh --bootstrap-server localhost:9092 --create ...
(same for all operations)

KEY DIFFERENCE:
└─ KRaft: All operations via bootstrap-server
└─ No need for ZooKeeper connection string

Monitoring

ZOOKEEPER MODE MONITORING:

Must monitor:
├─ Kafka brokers (JMX)
├─ ZooKeeper ensemble (JMX + mntr command)
└─ Kafka-ZooKeeper connectivity

KRAFT MODE MONITORING:

Monitor:
├─ Kafka brokers (JMX)
├─ Controller quorum (JMX)
├─ __cluster_metadata topic (replication lag)
└─ Raft metrics (leader election rate)

BENEFIT:
└─ Fewer systems to monitor

7. Troubleshooting Coordination Issues

ZooKeeper Mode Issues

Issue: ZooKeeper Unavailable

SYMPTOM:
├─ Cannot create/delete topics
├─ Controller fails to elect
└─ Brokers log ZK connection errors

INVESTIGATION:
1. Check ZooKeeper ensemble health
   └─ echo ruok | nc localhost 2181

2. Check ZooKeeper quorum
   └─ At least (n/2 + 1) nodes must be alive

3. Check network connectivity
   └─ Brokers can reach ZooKeeper?

4. Check ZooKeeper logs
   └─ Disk full? OOM? Crash?

RESOLUTION:
├─ Restart failed ZK nodes
├─ Restore ZK quorum
└─ Fix underlying issue (disk, network, etc.)

Issue: Metadata Out of Sync

SYMPTOM:
└─ Brokers have different view of cluster state

INVESTIGATION:
1. Check controller status
   └─ Is there exactly 1 controller?

2. Check ZooKeeper data
   └─ Browse /brokers/topics, /controller

3. Check broker logs
   └─ Metadata update failures?

RESOLUTION:
└─ Restart brokers (metadata resync from ZK)

KRaft Mode Issues

Issue: Controller Quorum Lost

SYMPTOM:
└─ No active controller, cannot perform admin operations

INVESTIGATION:
1. Check controller status
   kafka-metadata.sh --bootstrap-server localhost:9092 describe --status

2. Check quorum health
   └─ At least (n/2 + 1) controllers must be alive

3. Check controller logs
   └─ Raft election failures?

RESOLUTION:
├─ Restart failed controller nodes
├─ Restore controller quorum (need majority)
└─ If majority lost, may need manual recovery

Issue: Metadata Replication Lag

SYMPTOM:
└─ Slow admin operations, stale metadata on brokers

INVESTIGATION:
1. Check __cluster_metadata topic lag
   kafka-topics.sh --bootstrap-server localhost:9092 \
     --describe --topic __cluster_metadata

2. Check controller performance
   └─ CPU, disk I/O on controller nodes

3. Check Raft metrics
   └─ High commit latency?

RESOLUTION:
├─ Scale controller quorum (add nodes)
├─ Increase controller resources
└─ Check network between controllers

8. Best Practices

ZooKeeper Mode (Legacy)

✅ DO:
├─ Use 3-5 node ZooKeeper ensemble (odd number)
├─ Dedicate ZooKeeper nodes (don't co-locate with Kafka)
├─ Use fast disks for ZooKeeper (SSD)
├─ Monitor ZooKeeper health closely
├─ Keep ZooKeeper and Kafka versions compatible
└─ Plan migration to KRaft

❌ DON'T:
├─ Run single-node ZooKeeper in production
├─ Co-locate ZooKeeper with Kafka brokers
├─ Ignore ZooKeeper performance issues
└─ Exceed 200,000 partitions per cluster

KRaft Mode (Modern)

✅ DO:
├─ Use 3-5 controller nodes (odd number)
├─ Prefer dedicated controllers (not combined mode)
├─ Use fast disks for metadata.log.dir (SSD)
├─ Monitor controller quorum health
├─ Monitor __cluster_metadata replication lag
└─ Test failover scenarios

❌ DON'T:
├─ Run single controller in production
├─ Mix controller and broker roles without understanding trade-offs
├─ Ignore Raft metrics
└─ Manually modify __cluster_metadata topic

9. Key Takeaways

You've learned:

✅ Coordination provides consensus, metadata storage, and failure detection
✅ ZooKeeper is the legacy coordination system (being phased out)
✅ KRaft is the modern, built-in Raft consensus (ZooKeeper-free)
✅ Controller quorum (KRaft) provides faster failover than ZooKeeper
✅ KRaft scales better: millions of partitions vs. ~200K with ZooKeeper
✅ Migration: Blue-green or in-place (when available)
✅ Operational differences: KRaft is simpler (one system instead of two)
✅ ZooKeeper deprecated: Removal planned for Kafka 4.0
✅ Recommendation: Use KRaft for new clusters

10. Coordination System Checklist

ZOOKEEPER MODE:

☑ 3-5 ZooKeeper nodes (odd number)
☑ Fast disks (SSD) for ZooKeeper
☑ Monitor ZooKeeper health
☑ Monitor Kafka-ZK connectivity
☑ Plan migration to KRaft

KRAFT MODE:

☑ 3-5 controller nodes (odd number)
☑ Prefer dedicated controllers
☑ Fast disks for metadata.log.dir
☑ Monitor controller quorum
☑ Monitor __cluster_metadata replication
☑ Test controller failover

BOTH:

☑ Low-latency network between coordinator nodes
☑ Monitor leader election rate
☑ Document cluster ID and configuration
☑ Backup configuration files

Summary

Kafka's coordination is evolving:

ZooKeeper (Past):
├─ External dependency
├─ Operational complexity
├─ Scaling limitations
└─ Being phased out (removed in Kafka 4.0)

KRaft (Future):
├─ Built-in consensus (Raft)
├─ No external dependencies
├─ Better scalability
├─ Faster failover
├─ Simpler operations
└─ Production-ready now

Migration Path:
├─ New clusters: Use KRaft
└─ Existing clusters: Plan migration before Kafka 4.0

Remember: KRaft represents the future of Kafka. It simplifies operations, improves performance, and scales better. Start using it for new deployments today.

Congratulations!

You've completed the Kafka Operations & Administration course!

You now understand:

• ✅ Cluster architecture and administration fundamentals
• ✅ Broker lifecycle and topic management
• ✅ Partitions, replication, and ISR health
• ✅ Dynamic configuration and runtime changes
• ✅ Comprehensive monitoring, metrics, and alerting
• ✅ Performance tuning and capacity planning
• ✅ Quotas and resource management
• ✅ Coordination systems (ZooKeeper and KRaft)

Next steps:

• Practice in non-production environments
• Build runbooks for your specific use cases
• Set up monitoring and alerting
• Plan capacity for your workloads
• Stay current with Kafka releases
• Join the Kafka community

You're now equipped to operate production Kafka clusters with confidence!