KafkaIQ4.2 Leader Election Algorithms
Module Overview
Goal: Centralize all leader election logic — the ONE place where election algorithms are explained.
What You'll Learn:
- Preferred leader election algorithm
- Controlled leader election (failover)
- Unclean leader election
- Election triggers and conditions
- Controller-driven election process
- ISR-based leader selection
What's NOT Covered Here:
- Partition leadership responsibilities (covered in Module 4.1)
- Failover edge cases (covered in Module 4.3)
- Controller internals (covered in Module 6.1)
Key Insight: Kafka leader election prioritizes ISR members and preferred leaders. The controller orchestrates elections to maintain availability and durability.
1. Overview of Leader Election
When Elections Occur
LEADER ELECTION TRIGGERS:
1. Broker failure
└─ Current leader dies, need new leader
2. Controlled shutdown
└─ Broker gracefully stopped, leadership transferred
3. Preferred leader election
└─ Rebalance leadership to preferred leader
4. Partition creation
└─ New partition needs initial leader
5. Partition reassignment
└─ Changing replica set, may need new leader
6. Controller failover
└─ New controller may trigger elections (rare)
Election Authority
WHO DECIDES:
Controller (designated broker):
├─ Monitors broker liveness
├─ Detects leader failures
├─ Selects new leaders
├─ Sends LeaderAndIsrRequest to brokers
└─ Updates cluster metadata
Brokers:
└─ Execute controller's decisions (not autonomous)
Election is centralized:
└─ Controller has global view
└─ Makes informed decisions
2. Preferred Leader Election
Algorithm
PREFERRED LEADER ELECTION:
Goal:
└─ Move leadership to first replica in replica list
Condition:
└─ Preferred leader must be in ISR
Process:
1. Controller identifies partition:
├─ Current leader: Broker 2
├─ Replicas: [1, 2, 3]
├─ Preferred leader: Broker 1 (first in list)
└─ ISR: [1, 2]
2. Controller checks: Is preferred leader (1) in ISR?
└─ Yes → Proceed with election
3. Controller sends LeaderAndIsrRequest to brokers:
├─ New leader: Broker 1
├─ New ISR: [1, 2]
└─ Leader epoch: Incremented
4. Brokers execute transition:
├─ Broker 1: Become leader
├─ Broker 2: Become follower
└─ Start replication with new leader
5. Controller updates metadata
└─ Partition leader = Broker 1
Triggering Preferred Leader Election
AUTOMATIC TRIGGER:
Configuration:
├─ auto.leader.rebalance.enable = true
├─ leader.imbalance.per.broker.percentage = 10
└─ leader.imbalance.check.interval.ms = 300000 (5 min)
Controller logic (every 5 minutes):
1. Calculate leadership imbalance per broker
2. If imbalance > 10%:
└─ Trigger preferred leader elections for affected partitions
MANUAL TRIGGER:
Command:
└─ kafka-leader-election.sh --election-type preferred --all-topic-partitions
Use case:
└─ After broker restart (rebalance leadership)
3. Controlled Leader Election (Failover)
Scenario: Leader Failure
WHEN LEADER DIES:
Initial state:
├─ Leader: Broker 1
├─ Replicas: [1, 2, 3]
└─ ISR: [1, 2, 3]
Event: Broker 1 dies
Controller detects:
└─ Broker 1 session expired (ZooKeeper) or heartbeat lost (KRaft)
└─ Triggers leader election for all partitions led by Broker 1
Election Algorithm
CONTROLLED ELECTION (ISR-based):
1. Controller identifies affected partitions
└─ All partitions where Broker 1 is leader
2. For each partition:
a. Get current ISR
└─ ISR: [1, 2, 3]
b. Remove dead leader from ISR
└─ New ISR: [2, 3]
c. Select new leader:
└─ First alive replica in ISR
└─ New leader: Broker 2 (first in remaining ISR)
d. Increment leader epoch
└─ Epoch: N → N+1
3. Controller sends LeaderAndIsrRequest:
├─ Target brokers: 2 and 3
├─ New leader: Broker 2
├─ New ISR: [2, 3]
└─ New epoch: N+1
4. Brokers execute:
├─ Broker 2: Become leader (start accepting writes)
├─ Broker 3: Continue as follower (replicate from new leader)
└─ Both: Update local metadata
5. Controller updates metadata
└─ Partition leader = Broker 2
ISR-Based Selection Logic
LEADER SELECTION FROM ISR:
Priority:
1. Prefer live replicas in ISR
2. Within ISR, prefer earlier position in replica list
3. If no ISR member available → Unclean election (if enabled)
Example:
Replicas: [1, 2, 3]
ISR: [1, 2, 3]
Leader: 1 (dies)
Selection:
├─ Remove 1 from ISR → ISR: [2, 3]
├─ Choose first in ISR → Broker 2
└─ Broker 2 becomes leader
Why this order?
└─ Earlier replicas in list are often preferred leaders
└─ Maintains replica ordering preference
4. Unclean Leader Election
What is Unclean Election?
UNCLEAN ELECTION:
Definition:
└─ Elect a leader from outside ISR (out-of-sync replica)
When it happens:
└─ All ISR members are dead
└─ AND unclean.leader.election.enable = true
Trade-off:
├─ Availability: Partition becomes writable again
└─ Durability: Data loss (uncommitted data lost)
Unclean Election Algorithm
SCENARIO:
Replicas: [1, 2, 3]
ISR: [1, 2] (Broker 3 out-of-sync)
Leader: 1 (dies)
Broker 2 also dies
State:
├─ No ISR members available
├─ Only Broker 3 alive (out-of-sync)
└─ Partition offline (no leader)
If unclean.leader.election.enable = true:
1. Controller detects: No ISR members available
2. Controller checks: Unclean election enabled?
└─ Yes → Proceed
3. Controller selects: First alive replica (Broker 3)
└─ Even though not in ISR
4. Controller sends LeaderAndIsrRequest:
├─ New leader: Broker 3
├─ New ISR: [3]
└─ Epoch incremented
5. Broker 3 becomes leader
├─ Accepts writes
├─ Data loss: Records in old ISR not in Broker 3
└─ Partition available again
Data loss example:
├─ Old leader (Broker 1) had: offsets 0-1000
├─ New leader (Broker 3) has: offsets 0-980
└─ Offsets 981-1000 lost forever
Unclean Election Configuration
unclean.leader.election.enable:
false (default, recommended):
├─ Never elect out-of-sync replica
├─ Partition stays offline until ISR member returns
└─ Prioritizes durability over availability
true (dangerous):
├─ Elect out-of-sync replica if no ISR available
├─ Partition becomes available (data loss risk)
└─ Prioritizes availability over durability
Best practice:
└─ Leave disabled (false)
└─ Only enable for non-critical topics where availability > durability
5. Partition Creation Election
Initial Leader Selection
WHEN CREATING PARTITION:
1. Controller assigns replicas:
└─ Replicas: [1, 2, 3]
2. Controller selects initial leader:
└─ First replica in list (Broker 1)
3. Controller initializes ISR:
└─ ISR: [1] (only leader initially)
4. Controller sends LeaderAndIsrRequest:
├─ Leader: Broker 1
├─ ISR: [1]
└─ Epoch: 0 (initial epoch)
5. Followers start replicating:
├─ Brokers 2 and 3 fetch from Broker 1
└─ Eventually join ISR
6. Final state (after replication):
├─ Leader: 1
└─ ISR: [1, 2, 3]
6. Election During Partition Reassignment
Reassignment Scenario
MOVING PARTITION TO NEW BROKERS:
Current:
├─ Replicas: [1, 2, 3]
└─ Leader: 1
Target:
└─ Replicas: [2, 3, 4] (add 4, remove 1)
Process:
1. Add new replica (Broker 4):
├─ Replicas: [1, 2, 3, 4]
├─ Leader: 1 (unchanged)
└─ Broker 4 starts replicating
2. Wait for Broker 4 to catch up:
└─ ISR: [1, 2, 3, 4]
3. Elect new leader (if needed):
├─ If removing current leader (Broker 1)
├─ Elect new leader from [2, 3, 4]
└─ New leader: Broker 2
4. Remove old replica:
├─ Replicas: [2, 3, 4]
└─ ISR: [2, 3, 4]
5. Final state:
├─ Leader: 2
└─ Replicas: [2, 3, 4]
7. Controlled Shutdown Election
Graceful Leadership Transfer
WHEN BROKER SHUTS DOWN GRACEFULLY:
1. Broker sends ControlledShutdownRequest to controller
└─ "I'm shutting down, please move my partitions"
2. Controller identifies partitions led by this broker
3. For each partition:
a. Select new leader from ISR (excluding shutting-down broker)
b. Send LeaderAndIsrRequest
c. New leader starts accepting writes
4. Controller responds to broker: "Leadership transferred"
5. Broker completes shutdown:
├─ Flush logs
├─ Close connections
└─ Exit
Benefit:
└─ Zero downtime (leadership transferred before shutdown)
vs Uncontrolled Shutdown
COMPARISON:
Controlled Shutdown:
├─ Broker notifies controller
├─ Leadership transferred proactively
├─ No client interruption
└─ Clean failover
Uncontrolled Shutdown (crash):
├─ Controller detects after timeout
├─ Leadership transferred reactively
├─ Brief client unavailability
└─ Potential ISR shrink
8. Key Takeaways
You've learned:
✅ Controller orchestrates elections — centralized decision-making
✅ Preferred leader election rebalances leadership to first replica
✅ Controlled election selects new leader from ISR on failure
✅ ISR-based selection prioritizes durability (caught-up replicas)
✅ Unclean election sacrifices durability for availability (disabled by default)
✅ Controlled shutdown proactively transfers leadership (zero downtime)
✅ Leader epoch incremented on each election
✅ LeaderAndIsrRequest sent by controller to brokers
✅ Election triggers: failure, shutdown, rebalance, creation, reassignment
✅ Election prioritizes ISR members — out-of-sync replicas excluded
Summary
Kafka leader election balances availability and durability:
Election Types =
├─ Preferred leader: Rebalance to first replica
├─ Controlled failover: ISR-based on failure
├─ Unclean failover: Out-of-sync if no ISR (dangerous)
├─ Controlled shutdown: Proactive transfer
└─ Partition creation: Initial leader = first replica
Selection Algorithm =
├─ Prioritize ISR members (durability)
├─ Prefer earlier replicas (load balance)
├─ Increment leader epoch (detect changes)
└─ Controller decides, brokers execute
Configuration =
├─ auto.leader.rebalance.enable (automatic rebalancing)
├─ unclean.leader.election.enable (availability vs durability)
└─ controlled.shutdown.enable (graceful failover)
Remember: Leader election is controller-driven and ISR-based. Understanding election algorithms explains Kafka's behavior during failures and helps configure the right availability-durability trade-offs.
Next up: Module 4.3 — Failover Scenarios & Edge Cases — Real failure behavior and data loss boundaries.