KafkaIQ5.1 Metadata Model & Cluster State
Module Overview
Goal: Explain how Kafka represents and manages cluster state.
What You'll Learn:
- Cluster metadata structure
- Topic and partition metadata
- Broker registration and discovery
- Metadata versioning and propagation
- Client metadata consumption
- Metadata cache consistency
What's NOT Covered Here:
- ZooKeeper-specific implementation (covered in Module 5.2)
- KRaft-specific implementation (covered in Module 5.3)
- Controller logic (covered in Module 6.1)
Key Insight: Kafka's metadata model is the source of truth for cluster topology. Understanding metadata structure explains how brokers, topics, and partitions are discovered and coordinated.
1. What is Cluster Metadata?
Definition
CLUSTER METADATA = Complete state of Kafka cluster
Includes:
├─ Broker information (id, host, port, rack)
├─ Topic configuration (partition count, RF, configs)
├─ Partition assignments (which brokers host which partitions)
├─ Leadership information (leader broker per partition)
├─ ISR membership (per partition)
├─ Controller information (which broker is controller)
└─ Configuration overrides (topic/broker-specific configs)
Purpose:
└─ Single source of truth for cluster topology
└─ Enables brokers and clients to discover and coordinate
2. Broker Metadata
Broker Registration
BROKER METADATA STRUCTURE:
Broker ID: 1
├─ Host: broker1.example.com
├─ Port: 9092
├─ Rack: us-east-1a (optional, for rack awareness)
├─ Endpoints: [PLAINTEXT://broker1:9092, SSL://broker1:9093]
├─ JMX port: 9999 (optional)
└─ Broker epoch: Used in KRaft for fencing
Broker registration:
├─ On startup: Broker registers with coordination system (ZK/KRaft)
├─ Ephemeral (ZK) or heartbeat-based (KRaft)
└─ Controller discovers new brokers
Broker Liveness
LIVENESS TRACKING:
ZooKeeper mode:
└─ Ephemeral znodes (disappear if session expires)
└─ Controller watches for broker additions/removals
KRaft mode:
└─ Brokers send periodic heartbeats to controller
└─ Controller tracks last heartbeat time
└─ Timeout → broker considered dead
Liveness states:
├─ Online: Broker responsive
├─ Offline: Broker unresponsive (failure or network issue)
└─ Fenced: Broker explicitly excluded (KRaft)
3. Topic Metadata
Topic Structure
TOPIC METADATA:
Topic: "orders"
├─ Partition count: 3
├─ Replication factor: 3
├─ Configuration:
│ ├─ retention.ms: 604800000 (7 days)
│ ├─ cleanup.policy: delete
│ ├─ compression.type: producer
│ ├─ min.insync.replicas: 2
│ └─ ... (other configs)
├─ Partitions:
│ ├─ Partition 0:
│ │ ├─ Replicas: [1, 2, 3]
│ │ ├─ Leader: 1
│ │ ├─ ISR: [1, 2]
│ │ └─ Offline replicas: [3]
│ ├─ Partition 1:
│ │ └─ ... (similar structure)
│ └─ Partition 2:
│ └─ ...
└─ Topic epoch: Incremented on metadata changes (KRaft)
Partition Assignment
PARTITION METADATA:
Partition: orders-0
├─ Topic: orders
├─ Partition ID: 0
├─ Replicas: [1, 2, 3]
│ └─ Ordered list of broker IDs hosting this partition
├─ Leader: 1
│ └─ Current leader broker
├─ ISR: [1, 2]
│ └─ In-sync replica set
├─ Leader epoch: 5
│ └─ Incremented on each leader change
├─ Offline replicas: [3]
│ └─ Replicas not currently available
└─ Adding replicas: [] (during reassignment)
4. Controller Metadata
Controller Registration
CONTROLLER IDENTIFICATION:
At any time, exactly one broker is the controller:
Controller metadata:
├─ Controller ID: Broker ID of current controller
├─ Controller epoch: Incremented on each controller election
└─ Timestamp: When controller was elected
Purpose:
├─ Manage partition leadership
├─ Handle broker failures
├─ Coordinate partition reassignments
└─ Update cluster metadata
All brokers know:
└─ Which broker is currently the controller
└─ Can route admin requests accordingly
5. Metadata Versioning
Epochs for Consistency
EPOCHS TRACK CHANGES:
Controller epoch:
└─ Incremented each time controller changes
└─ Prevents stale controllers from making decisions
Leader epoch (per partition):
└─ Incremented each time partition leader changes
└─ Enables divergence detection
Topic epoch (KRaft):
└─ Incremented on topic metadata changes
└─ Version tracking for topic configs
Broker epoch (KRaft):
└─ Incremented when broker registers
└─ Used for broker fencing (prevent zombie brokers)
Version Example
TIMELINE:
T=0:
├─ Controller: Broker 0 (epoch 1)
├─ Partition orders-0 leader: Broker 1 (epoch 0)
T=1: Controller fails, Broker 1 becomes controller
├─ Controller: Broker 1 (epoch 2)
└─ Controller epoch incremented
T=2: Partition orders-0 leader fails, Broker 2 elected
├─ Partition orders-0 leader: Broker 2 (epoch 1)
└─ Leader epoch incremented
Result:
└─ Each change tracked by epoch
└─ Stale messages rejected (lower epoch)
6. Metadata Propagation
How Metadata Spreads
METADATA DISTRIBUTION:
1. Controller makes changes:
└─ Leader election, ISR updates, etc.
2. Controller updates metadata:
└─ ZooKeeper (ZK mode) or Metadata Log (KRaft mode)
3. Controller sends UpdateMetadataRequest to all brokers:
└─ Contains updated metadata for affected partitions
4. Brokers receive and apply updates:
├─ Update local metadata cache
└─ Ready to serve updated metadata to clients
5. Clients fetch metadata:
└─ MetadataRequest to any broker
└─ Broker responds with cached metadata
UpdateMetadataRequest
UPDATE METADATA REQUEST STRUCTURE:
UpdateMetadataRequest:
├─ Controller ID: 0
├─ Controller epoch: 5
├─ Broker metadata: [(id=1, host=...), (id=2, host=...), ...]
├─ Partition states:
│ └─ For each updated partition:
│ ├─ Topic: orders
│ ├─ Partition: 0
│ ├─ Leader: 2
│ ├─ Leader epoch: 3
│ ├─ ISR: [2, 3]
│ └─ Replicas: [1, 2, 3]
└─ Live brokers: [0, 1, 2, 3]
Brokers apply updates:
└─ Merge into local metadata cache
└─ Now serve updated metadata to clients
7. Client Metadata Requests
MetadataRequest
CLIENT METADATA REQUEST:
Client sends MetadataRequest:
├─ Topics: ["orders", "users"] (or empty for all topics)
└─ Include internal topics: false
Broker responds with MetadataResponse:
├─ Brokers: [(id=1, host=broker1, port=9092), ...]
├─ Controller ID: 0
└─ Topic metadata:
└─ For each topic:
├─ Topic: orders
├─ Error: NONE
├─ Partitions:
│ └─ For each partition:
│ ├─ Partition: 0
│ ├─ Leader: 1
│ ├─ Replicas: [1, 2, 3]
│ ├─ ISR: [1, 2]
│ └─ Offline replicas: []
└─ Is internal: false
Client Metadata Caching
CLIENT-SIDE BEHAVIOR:
1. Initial connection:
└─ Connect to bootstrap servers
└─ Fetch full metadata
2. Cache metadata locally:
├─ Topic/partition mappings
├─ Leader brokers
└─ Broker endpoints
3. Use cached metadata for routing:
├─ Producer: Route to partition leader
└─ Consumer: Route to partition leader (or follower if enabled)
4. Refresh metadata on:
├─ Errors (NOT_LEADER_FOR_PARTITION, etc.)
├─ Periodic refresh (metadata.max.age.ms, default 5 min)
└─ Topic list changes
5. Incremental updates (KIP-227):
└─ Fetch only changed metadata (reduces overhead)
8. Metadata Consistency
Consistency Model
EVENTUAL CONSISTENCY:
Metadata updates are eventually consistent:
├─ Controller updates metadata
├─ Sends UpdateMetadataRequest to brokers
├─ Brokers apply updates asynchronously
└─ Brief window where brokers have different views
Client impact:
├─ Client metadata may be stale
├─ Clients receive errors (NOT_LEADER_FOR_PARTITION)
└─ Clients refresh and retry
Trade-off:
├─ High availability (no strong consensus per update)
└─ Occasional client retries (acceptable cost)
Handling Stale Metadata
CLIENT ERROR HANDLING:
1. Client sends request to broker based on cached metadata
2. Broker responds with error:
└─ NOT_LEADER_FOR_PARTITION (leadership changed)
└─ UNKNOWN_TOPIC_OR_PARTITION (topic deleted)
3. Client actions:
├─ Fetch fresh metadata
├─ Update local cache
└─ Retry request to correct broker
4. Exponential backoff (if repeated errors)
Typical retry loop:
└─ Send → Error → Refresh → Retry → Success
└─ Usually resolves in 1-2 retries
9. Metadata Storage
Storage Location
WHERE METADATA IS STORED:
ZooKeeper mode:
└─ ZooKeeper stores authoritative metadata
└─ Controller reads/writes to ZK
└─ Brokers cache metadata locally
KRaft mode:
└─ Metadata log (Kafka topic: __cluster_metadata)
└─ Controller quorum maintains metadata log
└─ Brokers replicate metadata log
Key difference:
├─ ZK: External coordination system
└─ KRaft: Self-managed metadata (no external dependency)
10. Key Takeaways
You've learned:
✅ Cluster metadata includes brokers, topics, partitions, leadership, ISR
✅ Broker registration — brokers register on startup, tracked by controller
✅ Topic metadata — partition count, RF, configs, assignments
✅ Partition metadata — replicas, leader, ISR, leader epoch
✅ Controller metadata — which broker is controller, controller epoch
✅ Epochs track changes — controller, leader, topic, broker epochs
✅ Metadata propagation — UpdateMetadataRequest from controller to brokers
✅ Client metadata caching — fetch, cache, refresh on errors
✅ Eventual consistency — brief staleness, clients retry
✅ Storage — ZooKeeper (legacy) or metadata log (KRaft)
Summary
Kafka metadata is the cluster's single source of truth:
Metadata Components =
├─ Brokers: id, host, port, rack, liveness
├─ Topics: partition count, RF, configs
├─ Partitions: replicas, leader, ISR, epochs
├─ Controller: current controller, epoch
└─ Versioning: Epochs track changes
Propagation =
├─ Controller updates metadata
├─ UpdateMetadataRequest to brokers
├─ Brokers cache and serve to clients
└─ Eventual consistency (brief staleness)
Client Interaction =
├─ MetadataRequest to any broker
├─ Cache metadata locally
├─ Refresh on errors or periodic timeout
└─ Retry with fresh metadata
Storage =
├─ ZooKeeper (legacy, external system)
└─ KRaft (modern, self-managed metadata log)
Remember: Metadata is the nervous system of Kafka. Understanding how metadata is structured, propagated, and cached explains how the cluster coordinates and how clients discover brokers and partitions.
Next up: Module 5.2 — ZooKeeper Coordination (Legacy) — Historical coordination system.