3.1 – Replication & Fault Tolerance (Conceptual Model)

Why Replication?

Problem: If a broker crashes, all partition data on that broker becomes unavailable.

Without Replication:

Broker 1 crashes
   ↓
All partitions on Broker 1 are lost
   ↓
Producers and consumers can't access those partitions
   ↓
Data loss and downtime

Solution: Replicate each partition across multiple brokers.

What is Replication?

Replication means storing multiple copies of each partition on different brokers.

Key Terms:

• Replication Factor (RF): Number of copies of each partition

  • RF=1: No replication (single copy)
  • RF=2: One original + one replica
  • RF=3: One original + two replicas (most common)

• Leader: The broker that handles all read/write requests for a partition

• Follower: Brokers that replicate data from the leader

• In-Sync Replica (ISR): Followers that are fully caught up with the leader

Example: Replication Factor 3

Topic Configuration:

Topic: orders
Partitions: 3
Replication Factor: 3

Partition Distribution:

┌──────────────────────────────────────────────────┐
│               Kafka Cluster                      │
│                                                  │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐│
│  │  Broker 1   │ │  Broker 2   │ │  Broker 3   ││
│  ├─────────────┤ ├─────────────┤ ├─────────────┤│
│  │ p0 (Leader) │ │ p0 (Foll.)  │ │ p0 (Foll.)  ││
│  │ p1 (Foll.)  │ │ p1 (Leader) │ │ p1 (Foll.)  ││
│  │ p2 (Foll.)  │ │ p2 (Foll.)  │ │ p2 (Leader) ││
│  └─────────────┘ └─────────────┘ └─────────────┘│
└──────────────────────────────────────────────────┘

Key Points:

• Each partition has 3 replicas (one on each broker)
• Each partition has 1 leader (handles client requests)
• Each partition has 2 followers (replicate data from leader)
• Leadership is distributed (no single broker is a bottleneck)

Leader and Followers (Conceptual)

Leader Replica

Responsibilities:

1. Accept produce requests (write operations)
2. Accept fetch requests (read operations from consumers)
3. Accept replication fetch requests (from followers)
4. Maintain high watermark (the offset up to which all ISR replicas are in-sync)

Key Point: Only the leader handles client traffic. Followers don't serve client read/write requests.

Follower Replicas

Responsibilities:

1. Fetch data from the leader continuously
2. Replicate leader's log to their own local disk
3. Stay in-sync (keep up with leader's latest offset)

Key Point: Followers are active replicators (they pull data from the leader, not passive backups).

Leader-Follower Communication

Leader (Broker 1)          Follower (Broker 2)
   │                           │
   │◀────Fetch Request──────────┤
   │  (fetch from offset 100)   │
   │                            │
   ├────Fetch Response────────▶│
   │  (records 100-150)         │
   │                            │
   │                   ┌────────▼─────────┐
   │                   │ Write to local   │
   │                   │ log replica      │
   │                   └──────────────────┘
   │                            │
   │◀────Fetch Request──────────┤
   │  (fetch from offset 151)   │
   │                            │
   ├────Fetch Response────────▶│
   │  (records 151-200)         │

Key Point: Replication is follower-driven (followers actively fetch). Leaders don't push data to followers.

In-Sync Replicas (ISR)

What is ISR?

ISR (In-Sync Replica) is the set of replicas that are:

1. Alive (broker is reachable)
2. Caught up (within replica.lag.time.max.ms of the leader)

Example:

Partition 0:
  Leader: Broker 1
  ISR: [Broker 1, Broker 2, Broker 3]  ← All replicas are in-sync

If Broker 3 falls behind:
  Leader: Broker 1
  ISR: [Broker 1, Broker 2]  ← Broker 3 removed from ISR

Why ISR Matters

Durability Guarantee: A message is considered committed (durable) only when it's replicated to all ISR replicas.

Example:

Producer sends message to leader (Broker 1)
   ↓
Broker 1 writes to local log
   ↓
Followers (Broker 2, Broker 3) fetch and write to their logs
   ↓
Once all ISR replicas have written the message:
   ↓
Leader marks message as "committed"
   ↓
Producer receives ack (if acks=all)

Key Point: If ISR = [Broker 1, Broker 2], then the message is committed when both Broker 1 and Broker 2 have written it. Broker 3 (if lagging) doesn't participate.

Replica Lag

How does a follower fall out of ISR?

Criteria (replica.lag.time.max.ms, default: 10 seconds):

• If a follower hasn't fetched data from the leader for > 10 seconds, it's removed from ISR.

Causes of Replica Lag:

1. Network partition: Follower can't reach leader
2. Slow disk: Follower's disk is too slow to write replicated data
3. GC pause: Follower's JVM is in a long GC pause
4. Broker overload: Follower is processing too many client requests

Example:

Time 0:
  Leader: Broker 1 (offset 1000)
  Follower: Broker 2 (offset 1000, in ISR)
  Follower: Broker 3 (offset 1000, in ISR)

Time 10s:
  Leader: Broker 1 (offset 1500)
  Follower: Broker 2 (offset 1500, in ISR)  ← Still in-sync
  Follower: Broker 3 (offset 1020, out of ISR)  ← Fell behind

Controller removes Broker 3 from ISR

Recovery: Once Broker 3 catches up (fetches offsets 1020-1500), it's added back to ISR.

Replication Factor: Choosing the Right Value

Common Replication Factors

Why RF=3 is Standard

Trade-off:

• RF=1: No durability (broker crash = data loss)
• RF=2: Durability, but split-brain risk (if one broker is slow, ISR could shrink to 1)
• RF=3: Good durability (can lose 1 broker and still have 2 copies), safe minimum
• RF=4+: Higher durability, but 4x storage cost (usually overkill)

Recommendation: Use RF=3 for all production topics.

Fault Tolerance: Leader Failover

Scenario: Leader Crashes

Before Failure:

Partition 0:
  Leader: Broker 1 (offset 1000)
  ISR: [Broker 1, Broker 2, Broker 3]

Broker 1 Crashes:

Broker 1 is down
   ↓
Controller detects failure (via ZooKeeper session expiration or KRaft heartbeat)
   ↓
Controller elects new leader from ISR (Broker 2 or Broker 3)
   ↓
Controller updates metadata: new leader is Broker 2
   ↓
Producers/consumers fetch new metadata and connect to Broker 2

After Failover:

Partition 0:
  Leader: Broker 2 (offset 1000)
  ISR: [Broker 2, Broker 3]  ← Broker 1 removed from ISR

Key Point: Failover is automatic and typically takes 5-10 seconds.

Leader Election Rules

Rule: The controller elects a new leader from the current ISR.

Why ISR-only?

If the controller elected a follower that's not in ISR:

• That follower's log might be behind (e.g., offset 900 vs leader's 1000)
• Messages 900-1000 would be lost (data loss)

By restricting leader election to ISR:

• The new leader is guaranteed to have all committed messages
• No data loss (as long as at least one ISR replica survives)

What if All ISR Replicas Fail?

Scenario: RF=3, but all 3 brokers crash simultaneously.

Options (configured via unclean.leader.election.enable):

Production Recommendation: unclean.leader.election.enable=false (prefer consistency over availability).

Use unclean.leader.election.enable=true only if:

• Availability is more critical than consistency (e.g., metrics, logs)
• You can tolerate data loss

Durability Model

Producer Acknowledgment Modes

Producers control how many replicas must acknowledge a write before considering it successful.

Config: acks

Example (acks=all):

Producer sends message
   ↓
Leader (Broker 1) writes to local log
   ↓
Leader waits for followers (Broker 2, Broker 3) to fetch and write
   ↓
All ISR replicas confirm write
   ↓
Leader sends ack to producer

Key Point: acks=all ensures no data loss (as long as at least one ISR replica survives).

Producer Config: min.insync.replicas

Problem: What if ISR shrinks to 1 (just the leader) while using acks=all?

Example:

Partition 0:
  Leader: Broker 1
  ISR: [Broker 1]  ← Only leader in ISR (followers are lagging)

Producer sends message with acks=all
   ↓
Leader writes to local log
   ↓
Leader acks immediately (no followers to wait for)
   ↓
If leader crashes before followers catch up → data loss

Solution: min.insync.replicas

Config (broker or topic level):

min.insync.replicas=2

Behavior:

• If ISR size < min.insync.replicas, producer requests fail with NOT_ENOUGH_REPLICAS error.
• This forces producers to wait until enough replicas are in-sync before accepting writes.

Recommended Config for RF=3:

replication.factor=3
min.insync.replicas=2
acks=all

Interpretation:

• At least 2 replicas must be in-sync for writes to succeed
• If only 1 broker is alive, writes fail (prevents data loss)
• If 2+ brokers are alive, writes succeed (good availability)

High Watermark (Conceptual)

What is the High Watermark?

High Watermark (HW) is the offset up to which all ISR replicas have replicated.

Key Point: Consumers can only read up to the high watermark (not beyond).

Why?

If consumers could read beyond the high watermark:

• They might read messages that aren't yet replicated
• If the leader crashes before replication, those messages would be lost
• Consumer would see a message, then it disappears (data inconsistency)

Example: High Watermark in Action

Leader (Broker 1):
  Log End Offset (LEO): 105  ← Latest message written by leader
  High Watermark (HW): 100   ← Latest message replicated to all ISR

Follower (Broker 2):
  Log End Offset (LEO): 100  ← Latest message replicated from leader

Follower (Broker 3):
  Log End Offset (LEO): 100  ← Latest message replicated from leader

Interpretation:

• Leader has written messages 0-104 to its local log
• Followers have replicated messages 0-99
• High watermark is 100 (the minimum LEO across all ISR replicas)
• Consumers can read messages 0-99 (not 100-104 yet)

High Watermark Advancement

Step 1: Producer sends message (offset 105) to leader

Leader LEO: 105
Follower 1 LEO: 100
Follower 2 LEO: 100
HW: 100  ← Not advanced yet

Step 2: Followers fetch and replicate message 105

Leader LEO: 105
Follower 1 LEO: 105  ← Caught up
Follower 2 LEO: 105  ← Caught up
HW: 105  ← Advanced (all ISR replicas have offset 105)

Step 3: Consumers can now read message 105

Consumer fetches: offset 105 ✅ (HW = 105)

Key Point: There's a slight delay between when the leader writes a message and when consumers can read it (replication lag). Typically < 10ms.

Replication vs Durability (Summary)

Replication Guarantees

Production Recommendation:

replication.factor=3
min.insync.replicas=2
acks=all  # Producer config

This ensures:

• At least 2 copies of every message (even if 1 broker fails)
• Writes fail if only 1 broker is available (prevents silent data loss)

Key Takeaways

1. Replication = Fault Tolerance: Kafka replicates each partition across multiple brokers to survive broker failures.

2. Leader Handles Client Traffic: Only the leader replica accepts produce/fetch requests from clients. Followers replicate data from the leader.

3. ISR = Durability Boundary: A message is committed (durable) only when all ISR replicas have written it.

4. Leader Failover: If the leader crashes, the controller elects a new leader from the ISR (typically takes 5-10 seconds).

5. Replication Factor 3: Standard for production. Tolerates 1 broker failure without data loss.

6. acks=all + min.insync.replicas=2: Ensures no data loss (messages are replicated to at least 2 brokers before ack).

7. High Watermark: Consumers can only read messages up to the high watermark (the offset replicated to all ISR replicas).

8. Unclean Leader Election: Disabled by default. Prefer consistency (wait for ISR replica) over availability (elect non-ISR replica).

What's Next?

Next Module: 4.1 – Retention Policies

What We Covered:

• ✅ Replication factor and replica distribution
• ✅ Leader and followers (conceptual roles)
• ✅ In-Sync Replicas (ISR)
• ✅ Leader failover and leader election
• ✅ Durability model (acks, min.insync.replicas)
• ✅ High watermark (conceptual)

What We Didn't Cover (Explicitly Deferred):

• ❌ Deep replication protocol (leader epoch, replica fetch protocol) → M7: Kafka Internals
• ❌ ISR management algorithm → M7: Kafka Internals
• ❌ Controller internals → M7: Kafka Internals
• ❌ Performance impact of replication → M6: Operations

Replication & Fault Tolerance (Conceptual): Complete ✅