KafkaIQ

10.3.2 – Broker Failure Recovery Playbooks

Module: M10 – Advanced Topics & Architecture
Section: 10.3 – High Availability & Disaster Recovery
Goal: Operational playbooks for broker failure scenarios

Playbook 1: Single Broker Crash

Detection: Monitoring alerts (broker offline, under-replicated partitions)

Impact: Partitions with leader on failed broker unavailable for writes/reads until failover completes (5-30 seconds)

Recovery Steps:

  1. Confirm broker is offline (check monitoring, try SSH)
  2. Wait for automatic leader election (controller handles this)
  3. Verify no offline partitions: kafka-topics.sh --describe --under-replicated-partitions
  4. Investigate broker crash (check logs: /var/log/kafka/server.log)
  5. Fix root cause (OOM, disk full, network issue)
  6. Restart broker: systemctl start kafka
  7. Monitor ISR rejoin: Watch IsrExpandsPerSec metric
  8. Re-balance leaders (optional): kafka-leader-election.sh --election-type preferred

RTO: 10-30 minutes (broker restart + ISR sync)
RPO: 0 (no data loss if RF≥2)

Playbook 2: Multiple Broker Failures

Detection: Multiple brokers offline simultaneously

Impact: Some partitions may lose quorum (no ISR members available)

Recovery Steps:

  1. Immediate: Check if any partitions offline: kafka-topics.sh --unavailable-partitions
  2. Triage: Prioritize bringing up brokers hosting critical topic partitions
  3. Recover Brokers: Restart brokers one by one (allow ISR sync between restarts)
  4. Check ISR: Verify all partitions have min.insync.replicas
  5. Unclean Election (last resort): If all ISR offline, consider unclean.leader.election.enable=true (DATA LOSS WARNING)

RTO: 30-60 minutes
RPO: 0 (if ISR maintained), potential data loss if unclean election used

Playbook 3: Disk Failure

Detection: Broker logs show I/O errors, disk full, or RAID failure

Recovery Steps:

  1. Identify failed disk: df -h, smartctl -a /dev/sda
  2. Stop broker gracefully: kafka-server-stop.sh
  3. Replace/repair disk
  4. Restore partition data:
    • Option A: Let broker resync from leaders (automatic, slow)
    • Option B: Copy from replica if available (faster)
  5. Restart broker
  6. Monitor replication lag until caught up

RTO: 2-8 hours (disk replacement + data sync)
RPO: 0 (replicas on other brokers intact)

Playbook 4: Controller Failure

Detection: Logs show "No controller found", metadata operations slow/failing

Recovery Steps:

  1. Verify controller offline: zkCli.sh get /controller (ZK) or check KRaft logs
  2. Wait for new controller election (automatic, <10 seconds)
  3. Verify new controller: Check ActiveControllerCount=1 metric
  4. If controller not elected (ZK issues):
    • Check ZooKeeper quorum health: echo ruok | nc localhost 2181
    • Restart ZooKeeper if quorum lost

RTO: 10-30 seconds (auto-election)
RPO: 0

Playbook 5: ZooKeeper Quorum Loss

Detection: Kafka brokers log "Unable to connect to ZooKeeper"

Impact: Cluster metadata frozen, no leader elections, cluster effectively down

Recovery Steps:

  1. Check ZK quorum: echo ruok | nc localhost 2181 on all ZK nodes
  2. Identify issue: Network partition? ZK disk full? Config error?
  3. Restore quorum:
    • If 1 of 3 down: Restart failed ZK node
    • If 2 of 3 down: Fix both to restore quorum
  4. Verify Kafka reconnects: Check broker logs for "Connected to ZooKeeper"
  5. Trigger metadata refresh: kafka-metadata.sh --snapshot

RTO: 10-30 minutes
RPO: 0 (no data loss, but writes blocked during outage)

Playbook 6: Network Partition

Detection: Brokers can't reach each other, split brain warnings

Recovery Steps:

  1. Identify partition: ping, traceroute between brokers
  2. Don't restart brokers (will make situation worse)
  3. Fix network issue (firewall, switch, routing)
  4. Wait for network heal
  5. Brokers auto-reconnect and sync
  6. Check for under-replicated partitions after heal

RTO: Until network fixed
RPO: 0

Playbook 7: Full Cluster Crash

Detection: All brokers offline

Recovery Steps:

  1. Start ZooKeeper ensemble first (if using ZK)
  2. Start brokers one by one (don't start all simultaneously)
  3. Wait for each broker to become healthy before starting next
  4. Verify controller election: ActiveControllerCount=1
  5. Check all partitions have leaders: kafka-topics.sh --describe
  6. Resume traffic

RTO: 30-60 minutes
RPO: 0 (data on disk, if not corrupted)

Key Takeaways

  1. Single Broker: Auto-recovers in 10-30 seconds (leader election).
  2. Multiple Brokers: Restart one by one, prioritize critical partitions.
  3. Disk Failure: Replace disk, resync data (2-8 hours).
  4. Controller: Auto-elects new controller (<30 seconds).
  5. ZK Quorum Loss: Restore quorum ASAP (cluster frozen otherwise).
  6. Network Partition: Fix network, don't restart brokers.
  7. Full Cluster: Start ZK first, then brokers one by one.

What's Next?

Next Module: 10.3.3 – KRaft vs ZooKeeper Resilience

References:

  • Broker configs → M6: Operations
  • Replication internals → M7: Kafka Internals