KafkaIQ

4.3 Failover Scenarios & Edge Cases

Module Overview

Goal: Explain real failure behavior, edge cases, and data loss boundaries.

What You'll Learn:

  • Broker crash scenarios and recovery
  • Network partition behavior
  • Split-brain prevention
  • Data loss scenarios and boundaries
  • Cascading failures
  • Recovery patterns

What's NOT Covered Here:

  • Basic leader election (covered in Module 4.2)
  • ISR mechanics (covered in Module 3.2)
  • Broker lifecycle (covered in Module 7.1)

Key Insight: Kafka's failover behavior is well-defined but has edge cases. Understanding real failure scenarios helps you design resilient systems and set realistic expectations.

1. Single Broker Failure

Clean Failover (ISR Intact)

SCENARIO:

Initial state:
├─ Replicas: [1, 2, 3]
├─ Leader: 1
└─ ISR: [1, 2, 3]

Event: Broker 1 dies

Failover:
1. Controller detects Broker 1 failure
   └─ Session timeout (ZK) or heartbeat miss (KRaft)

2. Controller triggers election:
   ├─ Remove Broker 1 from ISR → ISR: [2, 3]
   ├─ Select new leader → Broker 2
   └─ Increment epoch

3. Controller sends LeaderAndIsrRequest:
   └─ Brokers 2 and 3 updated

4. Broker 2 becomes leader:
   ├─ Starts accepting writes
   └─ HW advances with new ISR

5. Clients:
   ├─ Receive NOT_LEADER_FOR_PARTITION error
   ├─ Refresh metadata
   └─ Retry to Broker 2 (new leader)

Result:
├─ Brief unavailability (~seconds)
├─ No data loss (ISR had all committed data)
└─ Service restored

Degraded Failover (ISR Shrinks)

SCENARIO:

Initial state:
├─ Replicas: [1, 2, 3]
├─ Leader: 1
└─ ISR: [1, 2] (Broker 3 already lagging)

Event: Broker 1 dies

Failover:
1. Controller removes Broker 1 from ISR:
   └─ ISR: [2] (only one ISR member left!)

2. Controller elects Broker 2 as leader

3. Problem: ISR size = 1
   ├─ If min.insync.replicas = 2:
   │   └─ Writes fail with NOT_ENOUGH_REPLICAS
   └─ Partition degraded (no write availability)

4. Recovery:
   ├─ Broker 3 catches up
   ├─ Rejoins ISR → ISR: [2, 3]
   └─ Writes resume

Result:
├─ Write unavailability until ISR recovers
└─ Read availability maintained

2. Multiple Broker Failures

Majority Failure

SCENARIO:

Initial state:
├─ Replicas: [1, 2, 3]
├─ Leader: 1
└─ ISR: [1, 2, 3]

Event: Brokers 1 and 2 die simultaneously

State:
├─ Only Broker 3 alive
├─ ISR: [3] (if Broker 3 was in ISR)
└─ Or ISR: [] (if Broker 3 was out-of-sync)

Scenario A: Broker 3 in ISR
├─ Broker 3 becomes leader
├─ ISR: [3]
├─ Writes succeed (if min.insync.replicas=1)
└─ No data loss

Scenario B: Broker 3 out-of-sync
├─ No ISR members available
├─ If unclean.leader.election.enable=false:
│   └─ Partition offline (no leader)
├─ If unclean.leader.election.enable=true:
│   ├─ Broker 3 elected (unclean)
│   └─ Data loss (uncommitted records on Brokers 1,2 lost)
└─ Trade-off: Availability vs Durability

All Replicas Down

SCENARIO:

Event: All brokers hosting partition replicas die

Result:
├─ Partition completely offline
├─ No leader available
├─ Clients receive LEADER_NOT_AVAILABLE error
└─ Partition unavailable until at least one replica recovers

Recovery:
1. First replica to restart joins
2. Controller elects it as leader
3. ISR: [first replica]
4. Partition available again

5. Other replicas restart:
   ├─ Join as followers
   ├─ Catch up via replication
   └─ ISR expands

3. Network Partitions

Split-Brain Prevention

KAFKA USES CENTRALIZED CONTROLLER:

Controller-based coordination:
├─ Single controller makes leadership decisions
├─ Only controller can elect leaders
└─ Brokers cannot self-elect

Prevents split-brain:
├─ Network partition isolates brokers
├─ Isolated brokers cannot elect new leaders
├─ Only controller-side partition makes decisions
└─ No dual leadership possible

Network Partition Scenario

SCENARIO: Network Split

Topology:
├─ Controller: Broker 0
├─ Partition replicas: [1, 2, 3]
└─ Leader: Broker 1

Event: Network partitions cluster:
├─ Partition A: Brokers 0, 1 (controller + leader)
└─ Partition B: Brokers 2, 3 (isolated)

Behavior:

Partition A (with controller):
├─ Controller still functional
├─ Detects Brokers 2, 3 as failed (timeout)
├─ Shrinks ISR: [1]
└─ Broker 1 continues as leader

Partition B (without controller):
├─ Brokers 2, 3 isolated
├─ Cannot elect new leader (no controller)
├─ If they were followers: Stop serving partition
└─ Wait for network to heal

Result:
└─ Partition with controller maintains service
└─ Isolated partition becomes unavailable

Controller Itself Isolated

SCENARIO: Controller Isolated

Event: Network partitions controller from rest of cluster

Behavior:
1. Controller loses contact with brokers
2. Brokers detect controller failure
3. Brokers elect new controller (from majority side)
4. New controller takes over leadership decisions

Old controller:
├─ Eventually detects isolation (via ZK/KRaft)
├─ Steps down as controller
└─ Stops making decisions

Result:
└─ Controller failover (new controller elected)
└─ Service continues on majority side

4. Data Loss Scenarios

Uncommitted Data Loss (Expected)

SCENARIO:

1. Producer writes with acks=1:
   ├─ Leader appends: offset 1000
   ├─ Leader responds: Success
   └─ Followers haven't replicated yet

2. Leader dies before replication

3. New leader elected (doesn't have offset 1000)

Result:
└─ Offset 1000 lost (never replicated)

Why it happens:
└─ acks=1 doesn't wait for replication
└─ Data was not committed (HW didn't advance)
└─ Expected behavior for acks=1

Mitigation:
└─ Use acks=all + min.insync.replicas=2

Committed Data Loss (Unclean Election)

SCENARIO:

1. Producer writes with acks=all:
   ├─ Leader: offsets 0-1000 (committed)
   ├─ ISR: [1, 2]
   └─ Follower 3: offsets 0-980 (out-of-sync)

2. Brokers 1 and 2 die (all ISR members down)

3. Unclean election (if enabled):
   └─ Broker 3 becomes leader

Result:
├─ Offsets 981-1000 lost
├─ Even though they were committed!
└─ Data loss of committed data

Why it happens:
└─ unclean.leader.election.enable=true
└─ Availability prioritized over durability

Mitigation:
└─ unclean.leader.election.enable=false (default)
└─ Accept partition offline rather than data loss

Correlated Failures

SCENARIO: Disk Failure on All Replicas

Event:
├─ Shared storage backend fails (e.g., SAN)
├─ All replicas lose same data
└─ Even ISR members affected

Result:
└─ Data loss even with RF=3, acks=all
└─ Infrastructure failure (beyond Kafka's control)

Mitigation:
├─ Use independent disks (not shared SAN)
├─ Rack awareness (spread replicas across racks)
└─ Multi-DC replication (for disaster recovery)

5. Cascading Failures

ISR Shrink Cascade

SCENARIO:

1. Broker 1 dies:
   └─ ISR: [1, 2, 3] → [2, 3]

2. Increased load on Brokers 2, 3:
   ├─ More partitions to lead
   ├─ Higher CPU, disk, network usage
   └─ Brokers 2, 3 slow down

3. Some followers lag due to overload:
   └─ ISR shrinks further: [2, 3] → [2]

4. If Broker 2 also dies:
   └─ ISR: [2] → []
   └─ Partition offline (if unclean election disabled)

Mitigation:
├─ Provision capacity for failure scenarios
├─ Monitor ISR shrinks (early warning)
└─ Scale cluster before cascading failures

6. Recovery Patterns

Broker Restarts

SCENARIO: Broker Restarts After Failure

1. Broker starts up:
   ├─ Loads partition logs
   ├─ Registers with controller
   └─ Marks itself available

2. For partitions where broker is follower:
   ├─ Starts replication from leader
   ├─ Catches up (fetches missing data)
   └─ Rejoins ISR

3. For partitions where broker was leader:
   ├─ Now follower (new leader elected during downtime)
   ├─ Fetches from new leader
   ├─ Detects epoch change (if log diverged)
   ├─ Truncates divergent data
   └─ Replicates from new leader

Result:
└─ Broker rejoins cluster
└─ ISR expands
└─ Leadership may rebalance (preferred leader election)

Disk Recovery

SCENARIO: Disk Failure and Replacement

1. Disk fails (data lost)
2. Replace disk (empty)
3. Broker restarts

Kafka behavior:
├─ Detects missing partition data
├─ Starts replication from scratch
└─ Fetches all data from leader (full sync)

Recovery time:
└─ Depends on partition size and network speed
└─ Large partitions (100s of GB) may take hours

Impact:
└─ Partition under-replicated until recovery complete
└─ Higher risk (fewer replicas available)

Mitigation:
└─ RAID for disk redundancy
└─ Monitor under-replicated partitions

7. Edge Cases

Leader Epoch Wrap-Around (Theoretical)

EDGE CASE:

Leader epoch is 32-bit integer:
└─ Max value: 2,147,483,647

If leader changes 2 billion times:
└─ Epoch wraps around (overflows)

Reality:
└─ Never happens in practice
└─ Would require leader change every second for 68 years

Kafka handles wrap-around gracefully (if it ever occurs)

Simultaneous Controller and Leader Failure

SCENARIO:

Event: Controller broker dies, and it was also leader for partition

Behavior:
1. Brokers detect controller failure
2. Elect new controller
3. New controller detects partition leader failure
4. New controller elects new partition leader

Result:
└─ Both controller and partition leadership recovered
└─ Two failovers happen (controller, then partition)
└─ Brief service interruption (seconds)

8. Broker Reintegration After Failure

When Failed Broker Returns

SCENARIO:

Timeline:
1. Broker 1 crashes (was leading partitions)
2. Controller elects new leaders (Broker 2 takes over)
3. Partitions continue operating with new leader
4. Broker 1 restarts and rejoins cluster

What happens during reintegration?

Step 1: Log Reconciliation

BROKER 1 STARTUP AND RECONCILIATION:

1. Broker 1 loads local logs from disk:
   ├─ Partition orders-0: LEO 1050, Epoch 1
   ├─ Last known leader: Self (Broker 1)
   └─ Log contains offsets 0-1049

2. Broker 1 registers with controller:
   └─ Controller sends LeaderAndIsrRequest:
       ├─ New leader: Broker 2
       ├─ New epoch: 2
       └─ ISR: [2, 3] (Broker 1 not in ISR yet)

3. Broker 1 realizes it's now a follower:
   └─ Connects to new leader (Broker 2)

4. Broker 1 sends OffsetsForLeaderEpochRequest:
   └─ "What offset did Epoch 2 start at?"

5. Leader (Broker 2) responds:
   └─ "Epoch 2 started at offset 1009"

6. Broker 1 checks local log:
   ├─ Has: Epochs 0 and 1, up to offset 1049
   ├─ Realizes: Offsets 1009-1049 from Epoch 1 are divergent
   └─ These offsets conflict with Epoch 2 (new leader's writes)

7. Broker 1 truncates divergent data:
   ├─ Truncates log to offset 1009
   ├─ Discards offsets 1009-1049 (41 records lost locally)
   └─ Updates LEO to 1009

Result:
└─ Broker 1 log now consistent with current leader
└─ Ready to replicate from offset 1009 onward

Step 2: Role Assignment

BROKER BECOMES FOLLOWER:

Controller actions:
1. Sends LeaderAndIsrRequest to Broker 1:
   └─ Role: Follower for all previously led partitions
   └─ Leader: Broker 2
   └─ ISR: [2, 3] (Broker 1 excluded)

2. Broker 1 transitions:
   ├─ Replica state: OfflineReplica → TruncatingLog → OnlineFetchingReplica
   └─ Starts follower fetch loop

Broker 1 responsibilities:
├─ Fetch from new leader (Broker 2)
├─ Replicate new records
├─ Do NOT accept client writes
└─ Stand by for potential future leadership

Step 3: Catch-Up Replication

BROKER 1 CATCHES UP:

Replication loop:
1. Broker 1 sends FetchRequest to Broker 2:
   └─ Fetch offset: 1009 (current LEO)

2. Broker 2 (leader) responds:
   └─ Records: offsets 1009-2000 (current leader LEO)
   └─ High Watermark: 1995

3. Broker 1 appends to local log:
   ├─ Writes offsets 1009-2000
   ├─ Updates LEO to 2001
   └─ Updates local HW to 1995

4. Repeat:
   └─ Broker 1 continues fetching
   └─ Gap between Broker 1 LEO and leader LEO narrows

Catch-up speed:
├─ Depends on network bandwidth
├─ Depends on replication rate (replica.fetch.max.bytes)
└─ Typically: Minutes for GBs of data

Step 4: ISR Reintegration

ISR MEMBERSHIP RESTORED:

Condition for rejoining ISR:
└─ Broker 1 LEO caught up AND lag < replica.lag.time.max.ms

Timeline:
1. Broker 1 LEO: 2000, Leader LEO: 2001 (lag: 1 record)
2. Broker 1 last fetch: 1 second ago
3. replica.lag.time.max.ms: 10 seconds
4. Leader determines: Broker 1 is caught up

5. Leader (Broker 2) adds Broker 1 to ISR:
   └─ ISR: [2, 3] → [1, 2, 3]

6. Leader sends AlterPartitionRequest to controller:
   └─ New ISR: [1, 2, 3]

7. Controller updates metadata:
   └─ Partition ISR now includes Broker 1

8. Broker 1 now participates in durability:
   └─ acks=all waits for Broker 1 to replicate

Full circle:
└─ Broker 1 rejoined cluster and ISR
└─ Fault tolerance restored

Data Loss Analysis

WHAT WAS LOST:

Unreplicated writes on Broker 1 are permanently lost:

Before crash:
├─ Leader: Broker 1
├─ HW: 1008 (last committed offset)
├─ Broker 1 LEO: 1050 (42 unreplicated records: 1009-1050)
├─ Broker 2 LEO: 1008
└─ Broker 3 LEO: 1007

After crash:
├─ New leader: Broker 2
├─ New HW: 1007 (min of new ISR LEOs: [2,3])
├─ Broker 2 LEO: 1008
└─ Records 1008-1050 lost forever (43 records)

When Broker 1 restarts:
├─ Truncates to offset 1009 (new epoch start)
├─ Offsets 1009-1050 discarded
└─ Never recoverable

WHY THIS IS ACCEPTABLE:

Records 1008-1050 were NOT committed:
├─ HW was 1008 (last committed)
├─ Records 1009-1050 beyond HW (uncommitted)
└─ Never visible to consumers (read_committed)

Producer behavior (with acks=all):
├─ Producers writing 1009-1050 did NOT receive acks
├─ Producer times out or retries
├─ Eventually writes successfully to new leader
└─ Consistent with Kafka's guarantees

Conclusion:
└─ Data loss limited to uncommitted records
└─ No consumer-visible data lost
└─ Durable data (up to HW) preserved

Monitoring Reintegration

TRACKING RECOVERY PROGRESS:

Check partition status:
```bash
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

# Look for:
# Leader: 2 (new leader)
# Replicas: 1,2,3
# Isr: 2,3 (Broker 1 not yet in ISR)

# After catch-up:
# Isr: 1,2,3 (Broker 1 rejoined)

Check under-replicated partitions:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

# Should show partitions where ISR < Replicas
# Decreases to 0 as Broker 1 catches up

JMX metrics:

# Under-replicated partitions
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
└─ Should decrease to 0

# Replica lag (on Broker 1)
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=ReplicaFetcherThread-0-2
└─ Lag decreases as replication progresses

# Fetch rate (on Broker 1)
kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec
└─ High rate indicates active replication

Controller logs:

INFO [Controller] Partition [orders,0] ISR updated to [2,3]
INFO [Controller] Partition [orders,0] ISR expanded to [1,2,3]


---

### Best Practices for Broker Recovery

OPERATIONAL RECOMMENDATIONS:

  1. Monitor under-replicated partitions:
# Continuous monitoring
watch -n 5 'kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions | wc -l'
  1. Increase replication speed (if needed):
# Broker config
num.replica.fetchers=4  # More threads for replication (default: 1)
replica.fetch.max.bytes=1048576  # Larger fetch batches (default: 1 MB)
replica.fetch.min.bytes=1  # Minimum bytes before response
  1. Prevent data loss:
# Topic/broker config
min.insync.replicas=2  # Require 2 ISR members for writes
unclean.leader.election.enable=false  # Never elect out-of-sync leader
acks=all  # Producer waits for all ISR (producer config)

  1. Plan for broker restarts: ├─ Use controlled shutdown (graceful) ├─ Monitor ISR after restart ├─ Verify partitions caught up before next maintenance └─ Avoid multiple broker restarts simultaneously

  2. Alert on prolonged under-replication:

# Alert if under-replicated for >10 minutes
if [ $(kafka-topics.sh --describe --under-replicated-partitions | wc -l) -gt 0 ]; then
  echo "ALERT: Under-replicated partitions detected"
fi


---

### Recovery Time Estimation

FACTORS AFFECTING RECOVERY TIME:

  1. Data size to replicate: └─ Lag: Leader LEO - Follower LEO

  2. Network bandwidth: └─ Example: 1 Gbps network = ~120 MB/s

  3. Disk I/O speed: └─ SSDs: Faster replication than HDDs

  4. Replication configuration: └─ num.replica.fetchers: More threads = faster └─ replica.fetch.max.bytes: Larger batches = faster

EXAMPLE CALCULATION:

Lag: 10 GB Network: 1 Gbps (120 MB/s) Disk: SSD (500 MB/s write) Bottleneck: Network (120 MB/s)

Recovery time: └─ 10 GB / 120 MB/s = ~85 seconds

Plus overhead (metadata, index rebuild): └─ Total: ~2 minutes

REALISTIC SCENARIOS:

Small lag (<1 GB): └─ Recovery: <30 seconds

Medium lag (1-10 GB): └─ Recovery: 1-5 minutes

Large lag (>10 GB): └─ Recovery: 10+ minutes └─ Consider: Is data needed? Maybe remove and add as new replica


---

## 9. Key Takeaways

**You've learned:**

✅ **Single broker failure** — ISR-based failover, brief unavailability  
✅ **Multiple broker failures** — may lead to offline partitions or unclean election  
✅ **Network partitions** — controller-side continues, isolated side unavailable  
✅ **Split-brain prevented** — centralized controller coordination  
✅ **Uncommitted data loss** — expected with acks < all  
✅ **Committed data loss** — only with unclean election  
✅ **Cascading failures** — ISR shrinks under load, capacity planning critical  
✅ **Recovery patterns** — truncation, replication, ISR expansion  
✅ **Correlated failures** — shared infrastructure risks  
✅ **Controller + leader failure** — both recover independently

---

## Summary

**Kafka failover is robust but has trade-offs:**

Failure Scenarios = ├─ Single broker: Clean failover (ISR-based) ├─ Multiple brokers: May go offline (if ISR exhausted) ├─ Network partition: Controller-side continues └─ All replicas down: Complete unavailability

Data Loss Boundaries = ├─ acks=0/1: Data loss expected (no replication guarantee) ├─ acks=all: No loss (if unclean election disabled) └─ Unclean election: Committed data may be lost

Recovery = ├─ Broker restart: Truncate divergent, replicate, rejoin ISR ├─ Disk failure: Full replication from leader ├─ Cascading failures: Capacity planning critical └─ Split-brain: Prevented by centralized controller

Configuration = ├─ unclean.leader.election.enable=false (prevent committed data loss) ├─ min.insync.replicas=2 (ensure durability) └─ RF=3 (tolerate up to 2 failures)


**Remember:** Kafka's failover behavior is well-defined. Understanding failure scenarios helps you configure appropriate durability settings and plan for disaster recovery.

**Next up:** Module 5.1 — Metadata Model & Cluster State — How Kafka tracks cluster topology.