5.2 ZooKeeper Coordination (Legacy)

Module Overview

Goal: Understand ZooKeeper's role in Kafka coordination (legacy, pre-KRaft).

What You'll Learn:

• ZooKeeper responsibilities in Kafka
• Znode layout and structure
• Controller election (ZK-based)
• Metadata storage in ZK
• Watches and notifications
• ZooKeeper limitations

What's NOT Covered Here:

• KRaft coordination (covered in Module 5.3)
• Controller internals (covered in Module 6.1)
• General ZooKeeper administration (out of scope)

Key Insight: ZooKeeper was Kafka's original coordination layer, providing consensus, metadata storage, and failure detection. Understanding ZK's role explains legacy Kafka behavior and motivates the move to KRaft.

Note: ZooKeeper mode is being deprecated. Kafka 3.3+ supports KRaft as the recommended coordination system.

1. ZooKeeper's Role in Kafka

Core Responsibilities

ZOOKEEPER PROVIDES:

1. Controller Election:
   └─ Leader election among brokers to determine controller

2. Broker Registration:
   └─ Ephemeral znodes track live brokers

3. Metadata Storage:
   └─ Topic configs, partition assignments, ISR state

4. Configuration Management:
   └─ Broker and topic configuration overrides

5. Failure Detection:
   └─ Session timeouts for broker liveness

6. Distributed Coordination:
   └─ Watches notify brokers of changes

2. Znode Layout

ZooKeeper Path Structure

KAFKA'S ZNODE HIERARCHY:

/
├─ admin/
│   ├─ delete_topics/
│   │   └─ {topic}  # Topics pending deletion
│   └─ reassign_partitions  # Partition reassignment state
│
├─ brokers/
│   ├─ ids/
│   │   ├─ 0  # Ephemeral node for Broker 0
│   │   ├─ 1  # Ephemeral node for Broker 1
│   │   └─ ...
│   ├─ topics/
│   │   └─ {topic}/
│   │       └─ partitions/
│   │           └─ {partition}/
│   │               └─ state  # Partition leader, ISR
│   └─ seqid  # Broker ID sequence generator
│
├─ config/
│   ├─ topics/
│   │   └─ {topic}  # Topic-level config overrides
│   ├─ brokers/
│   │   └─ {broker_id}  # Broker-level config overrides
│   └─ changes/  # Config change notifications
│
├─ controller  # Current controller broker ID
│
├─ controller_epoch  # Controller epoch (version)
│
├─ isr_change_notification/  # ISR change events
│
└─ log_dir_event_notification/  # Log directory failure events

3. Broker Registration

Ephemeral Znode for Liveness

BROKER STARTUP:

1. Broker connects to ZooKeeper
   └─ Establishes session

2. Broker creates ephemeral znode:
   └─ Path: /brokers/ids/{broker_id}
   └─ Data: {
         "version": 4,
         "host": "broker1.example.com",
         "port": 9092,
         "endpoints": ["PLAINTEXT://broker1:9092"],
         "rack": "us-east-1a",
         "timestamp": "1609459200000"
       }

3. Controller watches /brokers/ids/
   └─ Notified when brokers join/leave

4. If broker crashes:
   └─ ZK session expires
   └─ Ephemeral znode deleted automatically
   └─ Controller detects broker failure

4. Controller Election

Leader Election Using ZK

CONTROLLER ELECTION PROCESS:

1. All brokers watch /controller znode

2. Current controller:
   └─ Holds ephemeral znode at /controller
   └─ Data: {"brokerid": 0, "timestamp": "..."}

3. If controller crashes:
   ├─ ZK session expires
   ├─ /controller znode deleted
   └─ All brokers notified (via watch)

4. Brokers race to create /controller znode:
   ├─ First broker to successfully create becomes controller
   ├─ Others fail (znode already exists)
   └─ Failed brokers continue watching

5. New controller:
   ├─ Increments controller epoch
   ├─ Loads cluster state from ZK
   └─ Starts managing partitions

Controller Epoch

CONTROLLER VERSIONING:

/controller_epoch znode:
└─ Contains integer epoch value

Incremented on each controller election:
├─ Controller 0 (epoch 1)
├─ Controller 1 (epoch 2)
└─ Controller 0 again (epoch 3)

Purpose:
├─ Prevents stale controllers from making decisions
└─ Brokers reject requests from old controller epochs

5. Topic and Partition Metadata

Topic Configuration

TOPIC METADATA IN ZK:

/brokers/topics/{topic}:
└─ Data: {
     "version": 1,
     "partitions": {
       "0": [1, 2, 3],  # Partition 0, replicas on brokers 1,2,3
       "1": [2, 3, 1],
       "2": [3, 1, 2]
     }
   }

/config/topics/{topic}:
└─ Data: {
     "version": 1,
     "config": {
       "retention.ms": "604800000",
       "cleanup.policy": "delete",
       "min.insync.replicas": "2"
     }
   }

Partition State

PARTITION STATE IN ZK:

/brokers/topics/{topic}/partitions/{partition}/state:
└─ Data: {
     "version": 1,
     "leader": 1,
     "leader_epoch": 5,
     "controller_epoch": 2,
     "isr": [1, 2],
     "version": 1
   }

Updated by controller:
├─ On leader election
├─ On ISR changes
└─ Brokers read this for leadership info

6. Watches and Notifications

ZooKeeper Watches

KAFKA USES WATCHES FOR CHANGE DETECTION:

Controller watches:
├─ /brokers/ids/ → Broker join/leave
├─ /brokers/topics/{topic} → Topic changes
├─ /admin/delete_topics/ → Topic deletion requests
└─ /admin/reassign_partitions → Reassignment requests

Brokers watch:
├─ /controller → Controller changes
└─ Topic/partition state changes (via controller notifications)

When znode changes:
├─ ZK sends one-time notification
├─ Watcher must re-register watch
└─ Kafka processes change and re-watches

ISR Change Notifications

ISR CHANGES:

When leader updates ISR:
1. Leader writes new ISR to /brokers/topics/{topic}/partitions/{partition}/state
2. Leader creates child znode in /isr_change_notification/
3. Controller watches /isr_change_notification/
4. Controller notified of ISR change
5. Controller processes change
6. Controller deletes notification znode (acknowledgment)

Benefit:
└─ Controller learns of ISR changes immediately
└─ Updates cluster metadata accordingly

7. ZooKeeper Limitations

Scalability Issues

ZK LIMITATIONS IN LARGE KAFKA CLUSTERS:

1. Write Throughput:
   └─ ZK is designed for low-volume metadata
   └─ Frequent ISR changes → high ZK write load
   └─ Becomes bottleneck in large clusters

2. Partition Scale:
   └─ Millions of partitions → millions of znodes
   └─ ZK metadata size grows large
   └─ Slower controller failover (must load all znodes)

3. Controller Bottleneck:
   └─ Single controller reads/writes ZK
   └─ ZK latency impacts controller throughput

4. Operational Complexity:
   └─ Separate ZK cluster to manage
   └─ Different failure modes
   └─ Additional operational overhead

Session Management Complexity

ZK SESSION ISSUES:

Long GC pauses:
├─ Broker JVM pauses
├─ ZK heartbeats missed
├─ Session expires (false positive)
└─ Broker considered dead (even though alive)

Split-brain risk:
├─ Broker loses ZK connectivity
├─ Controller thinks broker dead
├─ Broker thinks it's still alive
└─ Must fence operations

Session timeout trade-off:
├─ Low timeout: Fast failure detection, false positives
└─ High timeout: Slower failure detection, fewer false positives

8. ZooKeeper vs KRaft

Why Move to KRaft?

KRAFT ADVANTAGES:

1. No External Dependency:
   └─ Kafka manages its own metadata (self-sufficient)

2. Better Scalability:
   └─ Metadata log designed for high partition count
   └─ Faster controller failover

3. Simplified Operations:
   └─ No separate ZK cluster to manage
   └─ One system to operate and monitor

4. Improved Performance:
   └─ Lower latency for metadata updates
   └─ Higher throughput for partition operations

5. Modern Design:
   └─ Raft-based consensus (proven algorithm)
   └─ Built for Kafka's workload

Deprecation timeline:
├─ Kafka 2.8: KRaft preview
├─ Kafka 3.3: KRaft production-ready
├─ Kafka 4.0: ZK mode removal (planned)
└─ New deployments should use KRaft

9. Key Takeaways

You've learned:

✅ ZooKeeper provides controller election, broker registration, metadata storage
✅ Ephemeral znodes track broker liveness (deleted on session expiry)
✅ Controller election via race to create /controller znode
✅ Znode layout organizes brokers, topics, configs, partitions
✅ Watches notify controller and brokers of changes
✅ ISR changes propagated via /isr_change_notification/
✅ ZK limitations — scalability, latency, operational complexity
✅ KRaft replaces ZK — self-managed metadata, better performance
✅ Deprecation timeline — ZK mode being phased out
✅ New deployments should use KRaft

Summary

ZooKeeper was Kafka's original coordination layer:

ZooKeeper Role =
├─ Controller election (leader election)
├─ Broker registration (ephemeral znodes)
├─ Metadata storage (topics, partitions, ISR)
├─ Failure detection (session timeouts)
└─ Change notifications (watches)

Znode Structure =
├─ /brokers/ids/{id} → Broker registration
├─ /brokers/topics/{topic} → Partition assignments
├─ /controller → Current controller
├─ /controller_epoch → Controller version
└─ /config/topics/{topic} → Topic configs

Limitations =
├─ Scalability bottleneck (large clusters)
├─ Operational complexity (separate system)
├─ Session management issues (false positives)
└─ Motivates KRaft migration

Migration =
├─ KRaft is the future (self-managed metadata)
├─ ZK mode deprecated (Kafka 4.0 removal planned)
└─ New deployments should use KRaft

Remember: ZooKeeper served Kafka well for many years but has scalability and operational limitations. KRaft (next module) is Kafka's modern, self-managed coordination system.

Next up: Module 5.3 — KRaft Metadata Quorum — Modern Kafka coordination.