KafkaIQ

3.1 Replication Protocol Explained

Module Overview

Goal: The ONE authoritative explanation of Kafka's replication protocol.

What You'll Learn:

  • Leader-follower replication model
  • Write path replication internals
  • Acknowledgment flow
  • Replica fetch protocol details
  • Replication states and transitions

What's NOT Covered Here:

  • ISR membership rules (covered in Module 3.2)
  • High watermark calculation (covered in Module 3.2)
  • Detailed follower fetch behavior (covered in Module 3.3)

Key Insight: Kafka uses leader-based replication where the partition leader coordinates all writes and followers replicate asynchronously. This design prioritizes availability and throughput over strong consistency.

1. Why Replication?

The Durability Problem

WITHOUT REPLICATION:

Single broker stores partition:
├─ Broker crashes → Data lost
├─ Disk fails → Data lost
└─ No fault tolerance

WITH REPLICATION (RF=3):

Three brokers store partition:
├─ One leader, two followers
├─ Any broker fails → Data still available
└─ Fault tolerance = RF - 1 failures

2. Leader-Based Replication Model

Roles

PARTITION REPLICAS:

Leader:
├─ Accepts all client writes
├─ Assigns offsets
├─ Serves client reads (default)
├─ Coordinates replication
└─ Tracks follower progress

Followers (RF - 1):
├─ Fetch data from leader (pull-based)
├─ Replicate leader's log exactly
├─ Do NOT accept client writes
├─ Stand by for leader failure
└─ Can serve reads (if configured)

Why Leader-Based?

ALTERNATIVE: Multi-Writer (consensus per write)

Problems:
├─ Requires distributed consensus (Paxos, Raft)
├─ Slower writes (quorum for each write)
└─ Complex coordination

KAFKA'S APPROACH: Single Writer (leader)

Benefits:
├─ Fast writes (no consensus per write)
├─ Simple replication (just copy log)
├─ High throughput
└─ Leader handles all coordination

3. Write Path Replication

Producer Write Flow

STEP-BY-STEP:

1. Producer sends ProduceRequest to leader
   └─ Contains batch of records for partition

2. Leader validates request
   ├─ Authorization (if security enabled)
   └─ Request size within limits

3. Leader appends to local log
   ├─ Assigns sequential offsets
   ├─ Writes to active segment
   └─ Updates Log End Offset (LEO)

4. Leader waits for replication (if acks=all)
   ├─ Tracks follower LEOs
   ├─ Waits for all ISR members to fetch
   └─ Advances High Watermark

5. Leader responds to producer
   ├─ Success: Base offset of batch
   └─ Error: Error code (if failed)

Replication Flow

FOLLOWER REPLICATION:

While (true) {
    1. Follower sends FetchRequest to leader
       └─ Includes current offset (follower's LEO)

    2. Leader responds with new records
       └─ From follower's LEO to leader's LEO

    3. Follower appends to local log
       ├─ Validates offsets match
       ├─ Writes to disk
       └─ Updates follower's LEO

    4. Repeat (continuous fetching)
}

Key points:
├─ Pull-based (follower initiates)
├─ Async (follower fetches independently)
└─ Leader tracks progress

4. Acknowledgment Levels

acks=0 (No Acknowledgment)

BEHAVIOR:

Producer:
└─ Fire-and-forget
└─ Doesn't wait for broker response

Leader:
└─ Accepts request
└─ May not write before responding

Durability: None
├─ Data may be lost
└─ No confirmation of write

Throughput: Highest
└─ No waiting

acks=1 (Leader Acknowledgment)

BEHAVIOR:

Producer:
└─ Waits for leader acknowledgment

Leader:
├─ Appends to local log
├─ Responds immediately (doesn't wait for followers)
└─ Replication happens asynchronously

Durability: Moderate
├─ Data on leader
├─ But not yet replicated
└─ Leader failure may lose data

Throughput: High
└─ Minimal waiting

acks=all (ISR Acknowledgment)

BEHAVIOR:

Producer:
└─ Waits for all ISR members to acknowledge

Leader:
├─ Appends to local log
├─ Waits for all ISR followers to fetch
├─ Advances High Watermark
└─ Responds to producer

Durability: Highest
├─ Data on all ISR replicas
├─ Safe from leader failure (if min.insync.replicas configured)
└─ Best durability guarantee

Throughput: Moderate
└─ Waits for replication

5. Replica Fetch Request Details

FetchRequest Contents

FETCH REQUEST FROM FOLLOWER:

FetchRequest:
├─ replica id: Follower's broker ID
├─ max wait: Max time to wait if no data
├─ min bytes: Minimum bytes to return
└─ partitions:
    ├─ topic: partition's topic
    ├─ partition: partition ID
    ├─ fetch offset: Follower's current LEO
    └─ max bytes: Max bytes for this partition

Example:
├─ Follower (broker 2) fetching from leader (broker 1)
├─ Partition: orders-0
├─ Fetch offset: 1000 (follower's LEO)
└─ Leader responds with records 1000-1050

FetchResponse Contents

FETCH RESPONSE FROM LEADER:

FetchResponse:
├─ partition data:
│   ├─ topic: orders
│   ├─ partition: 0
│   ├─ high watermark: 950 (leader's HW)
│   ├─ log start offset: 0
│   ├─ records: [1000-1050] (requested data)
│   └─ aborted transactions: (if any)
└─ throttle time: 0 ms

Follower actions:
├─ Append records to local log
├─ Update follower LEO to 1051
└─ Note leader's HW (for consumer visibility)

6. Replication States

Replica State Machine

FOLLOWER REPLICA STATES:

NewReplica:
└─ Just created, not yet started replication

OnlineFetchingReplica:
└─ Actively fetching from leader (normal state)

OfflineReplica:
└─ Broker hosting replica is down

TruncatingLog:
└─ Truncating divergent log after leader change

DeletingReplica:
└─ Being deleted (partition reassignment or deletion)

TRANSITIONS:

NewReplica → OnlineFetchingReplica:
└─ Start replication

OnlineFetchingReplica → OfflineReplica:
└─ Broker failure

OfflineReplica → OnlineFetchingReplica:
└─ Broker recovery

OnlineFetchingReplica → TruncatingLog:
└─ Leader change detected, divergence found

7. Log End Offset (LEO) Tracking

Per-Replica LEO

LEADER MAINTAINS LEO FOR EACH REPLICA:

Partition: orders-0
Leader: Broker 1
Replicas: [1, 2, 3]

LEOs:
├─ Broker 1 (leader): 1010
├─ Broker 2 (follower): 1009
└─ Broker 3 (follower): 1008

Each fetch request updates leader's view of follower LEO:
└─ Follower sends: "My LEO is X"
└─ Leader records: "Follower Y is at offset X"

LEO Update Mechanism

WHEN FOLLOWER FETCHES:

1. Follower sends FetchRequest
   └─ fetch offset = follower's current LEO (e.g., 1000)

2. Leader interprets this as:
   └─ "Follower has replicated up to offset 999"
   └─ "Follower's LEO is 1000"

3. Leader records: Follower LEO = 1000

4. Leader responds with records 1000+

5. Follower appends, updates local LEO to 1010

6. Next FetchRequest: fetch offset = 1010
   └─ Leader updates: Follower LEO = 1010

Continuous loop keeps leader's view up-to-date

8. Replication Lag

Definition

REPLICATION LAG:

Lag = Leader LEO - Follower LEO

Example:
├─ Leader LEO: 1010
├─ Follower LEO: 1008
└─ Lag: 2 records

Units:
├─ Records (message count)
└─ Time (replica.lag.time.max.ms)

Lag Detection

LEADER TRACKS FOLLOWER LAG:

For each follower:
├─ Last fetch timestamp
├─ Last fetch offset
└─ Current lag

If (now - last fetch time) > replica.lag.time.max.ms:
└─ Follower considered "out of sync"
└─ Removed from ISR (covered in Module 3.2)

Example:
├─ replica.lag.time.max.ms: 10 seconds
├─ Follower hasn't fetched for 12 seconds
└─ Leader removes follower from ISR

9. Replication Throttling

Purpose

WHY THROTTLE?

Replication can saturate network:
├─ High write throughput → high replication traffic
├─ Replication shares network with client traffic
└─ May impact client latency

THROTTLING:

Limit replication bandwidth:
├─ leader.replication.throttled.rate
└─ follower.replication.throttled.rate

Applied during:
├─ Normal replication (if configured)
└─ Partition reassignment (to avoid overload)

Throttling Mechanism

LEADER-SIDE THROTTLING:

Leader tracks bytes sent to followers:
├─ If rate > throttle limit:
│   └─ Delay FetchResponse
└─ Else: Respond immediately

FOLLOWER-SIDE THROTTLING:

Follower tracks bytes fetched:
├─ If rate > throttle limit:
│   └─ Delay next FetchRequest
└─ Else: Fetch immediately

Result:
└─ Replication slowed to configured rate
└─ Protects network from saturation

10. Replication and Zero-Copy

sendfile() Optimization

WHEN FOLLOWER FETCHES:

Leader serving data to follower:
├─ Data read from disk (or page cache)
├─ Sent over network to follower

Zero-copy path:
└─ sendfile() syscall
└─ Disk → OS buffer → Network (kernel space)
└─ No copy to user space (Kafka process)

Benefits:
├─ Lower CPU usage
├─ Higher throughput
└─ Less memory pressure

Same optimization as consumer fetches

11. Replication During Reassignment

Partition Reassignment

WHEN ADDING NEW REPLICA:

Example: Add broker 4 as new replica for orders-0

1. Controller sends LeaderAndIsrRequest
   └─ New replica list: [1, 2, 3, 4]

2. Broker 4 starts as NewReplica
   └─ Fetches from beginning of log (offset 0)

3. Broker 4 catches up
   ├─ Replicates all existing data
   └─ Eventually reaches leader's LEO

4. Broker 4 added to ISR
   └─ Now in-sync, participates in durability

5. (Optional) Remove old replica
   └─ If reassigning away from old broker

12. Key Takeaways

You've learned:

✅ Leader-based replication: Single writer (leader) per partition
✅ Pull-based: Followers fetch from leader (not pushed)
✅ acks levels: 0 (none), 1 (leader), all (ISR)
✅ FetchRequest: Follower sends current LEO, leader responds with new records
✅ LEO tracking: Leader maintains view of each follower's LEO
✅ Replication lag: Monitored via time and record count
✅ Replica states: New → Fetching → Offline (and others)
✅ Throttling: Controls replication bandwidth
✅ Zero-copy: sendfile() optimization for replication
✅ Reassignment: New replicas catch up from beginning

Summary

Kafka's replication is designed for high throughput:

Replication Model =
├─ Leader-based (single writer)
├─ Pull-based (followers initiate)
├─ Asynchronous (followers catch up independently)
└─ No consensus per write (fast writes)

Write Flow =
├─ Producer → Leader
├─ Leader appends to log
├─ Followers fetch asynchronously
├─ Leader tracks follower progress
└─ Acknowledgment based on acks level

Durability =
├─ acks=0: None (fire-and-forget)
├─ acks=1: Leader only (fast, moderate durability)
└─ acks=all: All ISR members (slow, high durability)

Remember: Replication in Kafka prioritizes availability and throughput. The leader-based, pull model allows high write rates without waiting for distributed consensus on every write.

Next up: Module 3.2 — ISR & High Watermark Mechanics — The authoritative explanation of durability boundaries.