KafkaIQ

10.3.1 – Fault Tolerance Mechanisms Review

Module: M10 – Advanced Topics & Architecture
Section: 10.3 – High Availability & Disaster Recovery
Goal: Review Kafka's fault tolerance mechanisms at the architecture level

Introduction

This module reviews Kafka's built-in fault tolerance mechanisms at a high level, focusing on how they work together to ensure high availability.

Note: Deep replication mechanics covered in M7: Kafka Internals. This is an architectural overview.

Fault Tolerance Layers

Layer 1: Replication (Partition-Level)

Mechanism: Each partition has multiple replicas (leader + followers).

Config:

replication.factor = 3  # 3 copies of each partition
min.insync.replicas = 2 # At least 2 replicas must acknowledge

Failure Scenario: Leader broker fails
Recovery: ISR follower promoted to leader (automatic)
Data Loss: None (if min.insync.replicas satisfied)

Tolerance: Can lose RF-1 brokers per partition

Layer 2: ISR (In-Sync Replicas)

Purpose: Track which replicas are caught up with leader.

How It Works:

  • Leader tracks follower lag
  • If follower falls behind (replica.lag.time.max.ms), removed from ISR
  • Only ISR members eligible for leadership

Failure Scenario: Slow follower
Recovery: Remove from ISR, continue with remaining ISR members
Data Loss: None (ISR guarantees consistency)

Layer 3: Controller Failover

Purpose: Manage cluster metadata and leader election.

What Controller Does:

  • Elects partition leaders
  • Tracks broker liveness
  • Propagates metadata updates

Failure Scenario: Controller broker crashes
Recovery: New controller elected from remaining brokers (automatic)
Downtime: Seconds (metadata operations pause briefly)

Layer 4: ZooKeeper Quorum / KRaft Quorum

Purpose: Store cluster metadata and coordinate brokers.

ZooKeeper:

  • 3-5 node quorum
  • Tolerates (N-1)/2 failures (e.g., 3 nodes → tolerate 1 failure)

KRaft:

  • Raft-based consensus (no ZooKeeper)
  • 3-5 controller nodes
  • Same quorum math

Failure Scenario: Quorum member fails
Recovery: Remaining quorum continues (automatic)
Downtime: None (if quorum maintained)

Failure Modes and Recovery

Broker Failure

Detection: Controller detects missed heartbeats
Recovery Steps:

  1. Controller marks broker offline
  2. For each partition where broker was leader: Elect new leader from ISR
  3. Clients redirect to new leaders (automatically, via metadata refresh)

Time to Recover: 5-30 seconds
Data Loss: None (if ISR > 1)

Network Partition

Scenario: Broker isolated from cluster (network split).

Detection: ZooKeeper/KRaft marks broker as unavailable
Recovery:

  1. Broker loses leadership for all partitions
  2. ISR followers become leaders
  3. When network heals, broker rejoins as follower

Time to Recover: Until network heals + replication catchup
Data Loss: None (rejoins as follower)

Disk Failure

Scenario: Disk hosting partition data fails.

Recovery:

  1. Broker fails health check
  2. Controller fails over partitions to replicas on other brokers
  3. Replace disk, restart broker
  4. Broker syncs data from leaders (rejoins as follower)

Time to Recover: Hours (disk replacement + data sync)
Data Loss: None (replicas on other brokers)

Unclean Leader Election

Scenario: All ISR members offline (catastrophic failure).

Config:

unclean.leader.election.enable = false  # Recommended: no data loss

If Disabled: Partition offline until ISR member recovers (data safe)
If Enabled: Non-ISR follower becomes leader (data loss)

Recommendation: Keep disabled for critical topics (accept downtime over data loss).

Availability Guarantees

SLA Calculations

Formula:

Availability = (Total Time - Downtime) / Total Time × 100%

Example:
  99.9% ("three nines") = 43 minutes downtime/month
  99.99% ("four nines") = 4.3 minutes downtime/month

Kafka Availability Factors:

  • RF=3, min.insync.replicas=2: Can lose 1 broker per partition (99.9%+)
  • Multi-AZ deployment: Tolerate full AZ failure (99.95%+)
  • Multi-region active-active: Tolerate full region failure (99.99%+)

Monitoring Fault Tolerance

Key Metrics

1. Under-Replicated Partitions

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

Healthy: 0
Alert: > 0 (some partitions lack full replication)

2. Offline Partitions

kafka.controller:type=KafkaController,name=OfflinePartitionsCount

Healthy: 0
Alert: > 0 (partitions with no leader)

3. Active Controller Count

kafka.controller:type=KafkaController,name=ActiveControllerCount

Healthy: Exactly 1
Alert: 0 (no controller) or > 1 (split brain)

Key Takeaways

  1. Replication: RF=3, min.insync.replicas=2 provides strong fault tolerance.

  2. ISR: Only in-sync replicas eligible for leadership (guarantees consistency).

  3. Controller: Manages leader election, automatically fails over.

  4. Quorum: ZooKeeper/KRaft quorum tolerates (N-1)/2 failures.

  5. Broker Failure: Auto-recovers in 5-30 seconds, no data loss (if ISR > 1).

  6. Unclean Leader Election: Disable for critical topics (prefer downtime over data loss).

  7. Monitoring: Track under-replicated partitions, offline partitions, controller count.

What's Next?

Next Module: 10.3.2 – Broker Failure Recovery Playbooks

References:

  • Replication internals → M7: Kafka Internals
  • Broker configs → M6: Operations