1.1 – Broker Fundamentals & Cluster Role

What is a Kafka Broker?

A Kafka broker is a server process that:

1. Stores messages on disk
2. Serves read/write requests from producers and consumers
3. Participates in replication to ensure fault tolerance
4. Manages partitions for topics assigned to it

Key Point: A broker is NOT just a "message queue server." It's a distributed log storage system with specific responsibilities in a cluster.

Broker vs Cluster

Example:

• A 3-broker cluster has 3 separate machines (or containers), each running kafka-server-start.
• Clients connect to any broker in the cluster (all brokers can serve requests).
• Data is distributed across brokers via partition assignment.

┌─────────────────────────────────────────────────┐
│              Kafka Cluster                      │
│                                                 │
│   ┌─────────┐   ┌─────────┐   ┌─────────┐    │
│   │ Broker 1│   │ Broker 2│   │ Broker 3│    │
│   │ ID=1    │   │ ID=2    │   │ ID=3    │    │
│   │         │   │         │   │         │    │
│   │ Stores: │   │ Stores: │   │ Stores: │    │
│   │ topic-A │   │ topic-A │   │ topic-B │    │
│   │  p0,p1  │   │  p2     │   │  p0     │    │
│   └─────────┘   └─────────┘   └─────────┘    │
│                                                 │
└─────────────────────────────────────────────────┘

Broker Responsibilities

1. Data Storage & Persistence

What It Does:

• Writes incoming messages to append-only log files on disk
• Stores messages for a configurable retention period (e.g., 7 days)
• Manages disk I/O for high-throughput sequential writes

Example:

• Producer sends message to topic orders, partition 0
• Broker 1 (leader for partition 0) appends message to /var/lib/kafka-logs/orders-0/00000000000000000000.log

Key Point: Brokers are stateful — they store data on disk. If a broker crashes, its disk holds the partition data.

2. Request Handling (Produce & Fetch)

What It Does:

• Accepts Produce requests from producers (writes data)
• Accepts Fetch requests from consumers (reads data)
• Handles Metadata requests (which broker hosts which partition?)

Request Flow:

Producer                    Broker (Leader)
   │                           │
   ├──────Produce Request──────▶│
   │  (topic: orders, key: 123) │
   │                            │
   │                   ┌────────▼───────┐
   │                   │ Append to log  │
   │                   │ Replicate to   │
   │                   │ followers      │
   │                   └────────┬───────┘
   │                            │
   │◀────Produce Response────────┤
   │   (offset: 45678, ack)     │

Key Point: Only the partition leader handles produce/fetch requests. Followers replicate data but don't serve client requests (covered in Module 3.1).

3. Partition Leadership

What It Does:

• Each partition has one leader broker and N follower brokers
• Leader accepts writes and reads
• Followers replicate data from the leader

Example:

• Topic orders has 3 partitions, replication factor 2
• Broker 1 is leader for partition 0, follower for partition 1
• Broker 2 is leader for partition 1, follower for partition 0
• Broker 3 is leader for partition 2

Partition Assignment:
┌──────────┬──────────┬──────────┐
│ Broker 1 │ Broker 2 │ Broker 3 │
├──────────┼──────────┼──────────┤
│ p0 (L)   │ p0 (F)   │ p2 (L)   │
│ p1 (F)   │ p1 (L)   │          │
└──────────┴──────────┴──────────┘

L = Leader, F = Follower

Key Point: Leadership is partition-specific. A broker can be a leader for some partitions and a follower for others.

4. Replication (Follower Responsibilities)

What It Does:

• Follower brokers continuously fetch data from the leader
• Followers stay in-sync by replicating new messages
• If leader fails, a follower becomes the new leader

Replication Flow:

Leader (Broker 1)          Follower (Broker 2)
   │                           │
   │◀────Fetch Request──────────┤
   │  (fetch from offset 100)   │
   │                            │
   ├────Fetch Response────────▶│
   │  (messages 100-150)        │
   │                            │
   │                   ┌────────▼───────┐
   │                   │ Write to local │
   │                   │ log replica    │
   │                   └────────────────┘

Key Point: Replication is leader-driven in terms of serving data, but follower-driven in terms of pulling data. Followers actively fetch; leaders don't push.

5. Cluster Coordination (Controller vs Broker)

What It Does:

• One broker in the cluster is elected as the controller
• Controller manages:
  • Partition leader election
  • Topic creation/deletion
  • Replica assignment
  • ISR management (tracking which replicas are in-sync)
  • Config propagation (distributing dynamic config changes)
• Non-controller brokers follow controller instructions

Important Distinction:

Key Point: Every broker can handle data requests. Only one broker is the controller (for coordination).

Example:

• Broker 2 is the controller
• Broker 1 (leader for partition 0) crashes
• Controller (Broker 2) elects Broker 3 as the new leader for partition 0
• All brokers update their metadata to reflect the new leader

How Controller Election Works

ZooKeeper Mode (Kafka < 3.0):

1. All brokers try to create an ephemeral node /controller in ZooKeeper
2. The first broker to create the node becomes the controller
3. Other brokers watch the /controller node
4. If the controller fails (node disappears), brokers race to create the node again
5. The first broker to recreate the node becomes the new controller

KRaft Mode (Kafka 3.0+):

1. Controller is a dedicated role (not a regular broker)
2. Controllers form a Raft quorum (e.g., 3 controllers)
3. One controller is elected leader via Raft consensus
4. If the leader fails, remaining controllers elect a new leader (typically < 1 second)

Typical failover time:

• ZooKeeper mode: 5-10 seconds
• KRaft mode: < 1 second

Controller Responsibilities in Detail

1. Leader Election

When a broker fails:

Before:
  orders-0: Leader=Broker 1, Replicas=[1,2,3], ISR=[1,2,3]

Broker 1 fails:
  orders-0: Leader=none, Replicas=[1,2,3], ISR=[2,3]

Controller detects failure:
  1. Removes Broker 1 from ISR
  2. Elects new leader from ISR (e.g., Broker 2)
  3. Sends LeaderAndIsr request to Broker 2 (you are now leader)
  4. Sends UpdateMetadata request to all brokers (Broker 2 is leader)

After:
  orders-0: Leader=Broker 2, Replicas=[1,2,3], ISR=[2,3]

2. ISR Management

Controller tracks which replicas are in-sync:

orders-0: Leader=Broker 1, Replicas=[1,2,3], ISR=[1,2,3]

Broker 3 falls behind (replica.lag.time.max.ms exceeded):
  Controller removes Broker 3 from ISR
  orders-0: Leader=Broker 1, Replicas=[1,2,3], ISR=[1,2]

Broker 3 catches up:
  Controller adds Broker 3 back to ISR
  orders-0: Leader=Broker 1, Replicas=[1,2,3], ISR=[1,2,3]

Why ISR matters:

• Only ISR replicas can become leader
• Producers with acks=all wait for ISR replicas to acknowledge

3. Partition Reassignment

When you move partitions between brokers (e.g., rebalancing):

Before:
  orders-0: Replicas=[1,2,3]

Reassignment request: Move to [2,3,4]

Controller orchestrates:
  1. Add Broker 4 as a follower (Replicas=[1,2,3,4])
  2. Wait for Broker 4 to catch up (join ISR)
  3. Remove Broker 1 from replicas (Replicas=[2,3,4])
  4. If Broker 1 was leader, elect new leader from [2,3,4]

After:
  orders-0: Replicas=[2,3,4]

4. Topic Creation/Deletion

When you create a topic:

kafka-topics --create --topic orders --partitions 10 --replication-factor 3

Controller:

1. Assigns partitions to brokers (e.g., partition 0 → [1,2,3], partition 1 → [2,3,4], ...)
2. Sends CreatePartitions request to brokers
3. Updates cluster metadata (all brokers learn about new topic)

5. Config Propagation

When you change a dynamic config:

kafka-configs --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=86400000

Controller:

1. Updates config in metadata store (ZooKeeper or KRaft log)
2. Sends UpdateMetadata request to all brokers
3. Brokers apply new config (no restart needed)

Controller Failure and Recovery

What happens if the controller fails?

ZooKeeper Mode:

1. Controller's ephemeral node /controller disappears
2. All brokers detect the failure (ZooKeeper watch triggers)
3. Brokers race to create /controller node again
4. First broker to create node becomes new controller
5. New controller reloads cluster state from ZooKeeper
6. New controller resumes leader election, ISR management, etc.

KRaft Mode:

1. Remaining controllers detect leader failure (Raft heartbeat timeout)
2. Controllers run Raft leader election (typically < 1 second)
3. New controller leader resumes operations

Controller Performance Considerations

Controller is a bottleneck in large clusters:

Why?

• Single controller handles all metadata changes (leader elections, ISR updates, etc.)
• In a cluster with 10,000 partitions, a broker failure triggers 10,000 leader elections
• Controller must send UpdateMetadata requests to all brokers (N² communication)

Symptoms of controller overload:

• Slow leader elections (> 30 seconds)
• Metadata propagation delays (brokers have stale metadata)
• Controller log shows "Controller event queue is full"

Solutions:

• Upgrade to KRaft mode (much faster controller, no ZooKeeper overhead)
• Reduce partition count (fewer partitions = less controller work)
• Increase controller resources (more CPU, faster disk for metadata log)

Monitoring the Controller

Check which broker is the controller:

ZooKeeper Mode:

zookeeper-shell localhost:2181 get /controller

Output:

{"version":1,"brokerid":1,"timestamp":"1609459200000"}

Interpretation: Broker 1 is the controller.

Metrics to monitor:

• kafka.controller:type=KafkaController,name=ActiveControllerCount (should be 1)
• kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs (leader election latency)
• kafka.controller:type=KafkaController,name=OfflinePartitionsCount (should be 0)

Deep internals of controller behavior are covered in M7: Kafka Internals.

Broker IDs & Identity

Broker ID Configuration

Every broker must have a unique ID within the cluster.

Configuration (server.properties):

# Broker ID (must be unique per broker)
broker.id=1

# Alternative: Auto-generate broker ID
broker.id.generation.enable=true

Manual ID Assignment (recommended for production):

• Broker 1: broker.id=1
• Broker 2: broker.id=2
• Broker 3: broker.id=3

Auto-Generated IDs (useful for cloud/containerized deployments):

• Kafka generates unique IDs automatically
• IDs are persisted in meta.properties file
• Useful for dynamic scaling (e.g., Kubernetes)

Broker Registration

How Brokers Join the Cluster:

1. Broker starts and reads server.properties
2. Broker connects to ZooKeeper (or KRaft quorum in KRaft mode)
3. Broker registers itself with metadata:
   • Broker ID
   • Host/port
   • Rack ID (for rack-aware replication)
4. Controller detects new broker and assigns partitions

ZooKeeper Mode (legacy):

• Broker creates ephemeral node in /brokers/ids/<broker.id>
• If broker crashes, ephemeral node disappears (controller knows broker is down)

KRaft Mode (Kafka 3.0+):

• Broker sends BrokerRegistration request to controller
• No ZooKeeper dependency

Key Point: Broker identity is tied to broker.id. If a broker restarts with the same ID, it rejoins with the same partition assignments.

Data vs Control Plane Responsibilities

Brokers have two types of responsibilities:

1. Data Plane (Client-Facing)

What It Handles:

• Produce requests (write messages)
• Fetch requests (read messages)
• ListOffsets requests (find offset for timestamp)
• Replication fetches (follower-to-leader communication)

Characteristics:

• High throughput (100K+ requests/sec per broker)
• Latency-sensitive (p99 should be < 50ms)
• Handles large payloads (messages up to 1MB default, 100MB max)

2. Control Plane (Cluster Coordination)

What It Handles:

• Leader election
• Partition assignment
• Topic metadata updates
• ISR (In-Sync Replica) updates
• Controller failover

Characteristics:

• Low throughput (100s of requests/sec)
• Handled by controller broker
• Metadata changes propagate to all brokers

Separate Control Plane (Kafka 2.4+)

Problem: Before Kafka 2.4, control plane and data plane shared the same network thread.

• During high client load, controller requests could be delayed
• Delayed controller requests → delayed leader elections → longer outages

Solution: Kafka 2.4+ allows a dedicated listener for controller communication.

Configuration:

# Separate control plane listener
control.plane.listener.name=CONTROLLER
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

# Regular listeners
listeners=PLAINTEXT://:9092,CONTROLLER://:9093

Result: Controller traffic is isolated from client traffic, improving cluster stability.

Broker Request Handling (High-Level)

Network Thread Model

Kafka brokers use a multi-threaded request handling model:

┌─────────────────────────────────────────────┐
│             Kafka Broker                    │
│                                             │
│  ┌────────────────────────────────────┐   │
│  │  Network Threads (Acceptor + N IO) │   │
│  │  - Accept connections               │   │
│  │  - Read requests from sockets       │   │
│  │  - Write responses to sockets       │   │
│  └──────────┬─────────────────────────┘   │
│             │                               │
│             ▼                               │
│  ┌────────────────────────────────────┐   │
│  │  Request Queue                      │   │
│  └──────────┬─────────────────────────┘   │
│             │                               │
│             ▼                               │
│  ┌────────────────────────────────────┐   │
│  │  Request Handler Threads (Pool)    │   │
│  │  - Process Produce/Fetch requests   │   │
│  │  - Write to log                      │   │
│  │  - Read from log                     │   │
│  └──────────┬─────────────────────────┘   │
│             │                               │
│             ▼                               │
│  ┌────────────────────────────────────┐   │
│  │  Response Queue                     │   │
│  └────────────────────────────────────┘   │
│                                             │
└─────────────────────────────────────────────┘

Key Parameters:

Flow:

1. Network thread receives request from socket
2. Request placed in queue
3. I/O thread picks up request, processes it (read/write log)
4. I/O thread generates response, places in response queue
5. Network thread sends response to client

Key Point: This model allows Kafka to handle thousands of concurrent connections efficiently without blocking.

Broker Lifecycle States

A broker transitions through several states:

Controlled Shutdown (graceful stop):

controlled.shutdown.enable=true

What Happens:

1. Broker receives shutdown signal (SIGTERM)
2. Broker stops accepting new requests
3. Broker transfers partition leadership to other brokers
4. Broker flushes all logs to disk
5. Broker exits

Benefit: Minimizes unavailability. Leadership transfer happens before broker stops, so clients experience no downtime.

Uncontrolled Shutdown (crash):

• Broker dies immediately (e.g., kill -9, out-of-memory)
• Controller detects broker is down (ZooKeeper session expires or KRaft heartbeat timeout)
• Controller elects new leaders for partitions hosted on dead broker
• Clients experience temporary errors until new leaders are elected (~5-10 seconds)

Key Takeaways

1. Broker = Storage + Request Handler: A broker stores partition data on disk and serves produce/fetch requests from clients.

2. Partition Leadership: Each partition has one leader (accepts writes) and N followers (replicate data).

3. Cluster Role: One broker is the controller (manages leader elections and metadata). All brokers can serve data requests.

4. Broker ID: Every broker has a unique ID. ID persists across restarts (same broker, same ID).

5. Data Plane vs Control Plane: Brokers handle both client requests (data plane) and cluster coordination (control plane). Modern Kafka can separate these for better stability.

6. Controlled Shutdown: Graceful shutdown transfers leadership before stopping, minimizing client impact.

What's Next?

Next Module: 2.1 – Topics & Partitions Explained

What We Covered:

• ✅ What a broker is
• ✅ Broker responsibilities (storage, requests, leadership, replication, coordination)
• ✅ Broker IDs and registration
• ✅ Request handling model
• ✅ Broker lifecycle

What We Didn't Cover (Explicitly Deferred):

• ❌ Deep replication protocol internals → M7: Kafka Internals
• ❌ Performance tuning and capacity planning → M6: Operations
• ❌ Security hardening → M9: Security
• ❌ Topic and partition design → Module 5.1

Broker Fundamentals: Complete ✅