KafkaIQ

4.1 Partition Leadership

Module Overview

Goal: Define partition leadership responsibilities and client routing behavior.

What You'll Learn:

  • Leader duties and responsibilities
  • Follower duties
  • Client routing to leaders
  • Leadership metadata
  • Preferred leader concept
  • Read-from-follower (optional feature)

What's NOT Covered Here:

  • Leader election algorithms (covered in Module 4.2)
  • Failover scenarios (covered in Module 4.3)
  • Controller mechanics (covered in Module 6.1)

Key Insight: Partition leadership is Kafka's mechanism for coordinating writes and ensuring consistency. Understanding leader responsibilities is key to understanding Kafka's performance and durability characteristics.

1. Leader Responsibilities

Core Duties

PARTITION LEADER MUST:

1. Accept client writes
   └─ Only leader can append to log (assign offsets)

2. Serve client reads (default)
   └─ Consumers fetch from leader

3. Coordinate replication
   ├─ Accept follower fetch requests
   ├─ Track follower LEOs
   └─ Manage ISR membership

4. Advance High Watermark
   └─ Calculate HW from ISR member LEOs

5. Report to controller
   ├─ ISR changes (AlterPartitionRequest)
   └─ Partition health metrics

Leader-Only Operations

OPERATIONS ONLY LEADER CAN PERFORM:

Write operations:
├─ Assign offsets to new records
├─ Append records to log
├─ Validate producer requests
└─ Respond with acknowledgments

Coordination:
├─ Track ISR membership
├─ Advance HW
├─ Handle fetch requests (from consumers and followers)
└─ Manage producer epochs (for idempotence/transactions)

Read operations (default):
└─ Serve consumer fetch requests
└─ (Followers can also serve with KIP-392)

2. Follower Responsibilities

Core Duties

PARTITION FOLLOWER MUST:

1. Replicate from leader
   └─ Continuous fetch loop

2. Maintain exact log copy
   └─ Same offsets, same data as leader

3. Stand by for leadership
   └─ Can become leader if current leader fails

4. Report readiness
   └─ Fetch frequency determines ISR membership

Optional (KIP-392):
└─ Serve client reads (if configured)

Follower Restrictions

FOLLOWERS CANNOT:

✗ Accept producer writes
  └─ Producers must write to leader

✗ Assign offsets
  └─ Only leader assigns offsets

✗ Modify ISR
  └─ Only leader manages ISR

✗ Coordinate replication
  └─ Followers are passive (just replicate)

✗ Serve consumer reads (default)
  └─ Consumers read from leader
  └─ (Can be enabled with rack-aware consumer fetching)

3. Client Routing to Leaders

Metadata Discovery

HOW CLIENTS FIND LEADERS:

1. Client connects to any broker (bootstrap server)
   └─ MetadataRequest

2. Broker responds with cluster metadata:
   ├─ All brokers: [(id=1, host=broker1, port=9092), ...]
   ├─ All topics and partitions
   └─ Leader for each partition: (topic=orders, partition=0, leader=1)

3. Client caches metadata locally
   └─ Knows which broker is leader for each partition

4. Client sends requests directly to leader
   └─ ProduceRequest → Leader broker
   └─ FetchRequest → Leader broker (or follower if KIP-392)

5. Metadata refresh on errors
   └─ NOT_LEADER_FOR_PARTITION → Refresh metadata

Metadata Refresh Triggers

CLIENTS REFRESH METADATA WHEN:

1. NOT_LEADER_FOR_PARTITION error
   └─ Leader changed, client's metadata stale

2. UNKNOWN_TOPIC_OR_PARTITION error
   └─ Topic created/deleted

3. Periodic refresh
   └─ metadata.max.age.ms (default: 5 minutes)

4. Topic list change
   └─ Subscribe to new topics

5. Broker connection failure
   └─ Broker down, need to find new leader

Process:
├─ Send MetadataRequest
├─ Receive updated metadata
├─ Update local cache
└─ Retry operation with correct leader

4. Leadership Metadata

Metadata Structure

PARTITION METADATA:

Partition: orders-0
├─ Leader: Broker 1
├─ Replicas: [1, 2, 3]
├─ ISR: [1, 2]
├─ Offline replicas: [3]
└─ Leader epoch: 5

Client receives:
├─ Leader ID → Connect to Broker 1
├─ Replicas → Awareness of replication factor
└─ ISR → (Optional) Client-side durability check

5. Preferred Leader

Concept

PREFERRED LEADER:

Definition:
└─ First replica in replica list is "preferred leader"

Example:
├─ Replicas: [1, 2, 3]
└─ Preferred leader: Broker 1

Purpose:
└─ Balance leadership across brokers
└─ Avoid leadership concentration on few brokers

Preferred Leader Election

AUTOMATIC REBALANCING:

Configuration:
├─ auto.leader.rebalance.enable = true (default)
├─ leader.imbalance.per.broker.percentage = 10 (default)
└─ leader.imbalance.check.interval.ms = 300000 (5 minutes)

Behavior:
1. Controller periodically checks leadership balance
2. If broker has > 10% leadership imbalance:
   ├─ Controller triggers preferred leader elections
   └─ Move leadership to preferred leaders

3. Result: Leadership distributed evenly

Example:
├─ Broker 1: 50 partitions as leader (should be 30)
├─ Imbalance: (50-30)/30 = 66% (> 10% threshold)
└─ Controller elects preferred leaders to rebalance

6. Leader Epoch in Leadership

Epoch Increment

EACH LEADER CHANGE INCREMENTS EPOCH:

Initial:
├─ Leader: Broker 1
└─ Epoch: 0

Broker 1 fails:
├─ New leader: Broker 2
├─ Epoch: 1
└─ Epoch incremented

Broker 1 returns as leader (preferred leader election):
├─ Leader: Broker 1
├─ Epoch: 2
└─ Epoch incremented again

Purpose:
└─ Detect stale leaders
└─ Enable divergence detection (as covered in Module 3.3)

7. Leadership and Performance

Why Leaders Matter for Performance

LEADER AS BOTTLENECK:

All writes for partition go through leader:
├─ Leader handles ProduceRequests
├─ Leader appends to log
├─ Leader coordinates replication
└─ Leader serves FetchRequests

Implications:

Skewed leadership:
├─ Few brokers handle most partitions
├─ Hot brokers (CPU, disk, network saturated)
└─ Reduced cluster throughput

Balanced leadership:
├─ Even load distribution
├─ All brokers utilized
└─ Maximum cluster throughput

Leadership Distribution

BEST PRACTICES:

1. Use preferred leader election
   └─ auto.leader.rebalance.enable = true

2. Design topic replication carefully
   ├─ Partition count = multiple of broker count
   └─ Even leadership distribution

3. Monitor leadership per broker
   └─ Alert if imbalance detected

4. Manual reassignment if needed
   └─ kafka-reassign-partitions.sh
   └─ Rebalance leadership manually

Goal:
└─ Each broker leads ~equal number of partitions

8. Read-From-Follower (KIP-392)

Optional Feature

RACK-AWARE CONSUMER FETCHING:

Traditional (default):
└─ Consumers read from leader

KIP-392 (optional):
└─ Consumers can read from nearest follower
└─ Based on rack awareness

Benefits:
├─ Reduced cross-rack network traffic
├─ Lower latency (geographically closer)
└─ Better bandwidth utilization

Trade-offs:
├─ Slightly stale reads (follower HW may lag)
├─ More complex routing logic
└─ Only useful in multi-rack/multi-AZ deployments

How It Works

CONSUMER READS FROM FOLLOWER:

1. Consumer configured with:
   └─ client.rack = "us-east-1a"

2. Broker provides metadata:
   ├─ Leader: Broker 1 (rack: us-east-1b)
   ├─ Followers: Broker 2 (rack: us-east-1a), Broker 3 (us-east-1c)
   └─ Preferred read replica: Broker 2 (same rack as consumer)

3. Consumer sends FetchRequest to Broker 2 (follower)

4. Follower serves read from local log
   └─ Up to follower's HW (may be slightly behind leader's HW)

Result:
└─ Reduced cross-AZ traffic
└─ Acceptable for latency-tolerant consumers

9. Key Takeaways

You've learned:

✅ Leaders accept writes and coordinate replication — followers replicate
✅ Only leaders assign offsets — single source of truth
✅ Clients route to leaders via metadata discovery
✅ Preferred leader = first replica in replica list
✅ Preferred leader election balances leadership across brokers
✅ Leader epochs increment on each leadership change
✅ Leadership distribution impacts performance — avoid hot brokers
✅ Read-from-follower (KIP-392) reduces cross-rack traffic (optional)
✅ Metadata refresh triggered by errors or timeouts
✅ Balanced leadership = higher cluster throughput

Summary

Partition leadership defines Kafka's coordination model:

Leader Duties =
├─ Accept writes (assign offsets)
├─ Coordinate replication (track ISR)
├─ Advance High Watermark
├─ Serve reads (default)
└─ Report to controller

Follower Duties =
├─ Replicate from leader
├─ Stand by for leadership
└─ Optionally serve reads (KIP-392)

Client Routing =
├─ Metadata discovery (which broker is leader)
├─ Direct connection to leader
├─ Refresh on errors
└─ Periodic metadata updates

Leadership Balance =
├─ Preferred leader = first in replica list
├─ auto.leader.rebalance.enable (automatic rebalancing)
└─ Even distribution = better performance

Remember: Leadership is Kafka's mechanism for coordinating writes and ensuring consistency. Balanced leadership is critical for performance. Understanding leader responsibilities explains Kafka's behavior during normal operations and failures.

Next up: Module 4.2 — Leader Election Algorithms — How Kafka chooses leaders.