2.3 Partitions, Replication & ISR Management

Module Overview

Goal: Make replication operationally clear. This is the ONLY place ISR and replication health are explained deeply.

What You'll Learn:

• Partition leadership mechanics
• Replication factor and replica assignment
• In-Sync Replicas (ISR) mechanics
• Leader election process
• Unclean leader election
• Under-replicated partitions
• Monitoring and troubleshooting replication health

What's NOT Covered Here:

• Topic creation (covered in Module 2.2)
• Performance tuning (covered in Module 5.1)
• Metrics details (covered in Module 4.x)

Key Insight: Replication is what makes Kafka fault-tolerant. Understanding ISR and leader election is critical for operating a reliable Kafka cluster.

1. Partitions: Units of Parallelism

What is a Partition?

A Partition is:
├─ A totally ordered sequence of messages
├─ Unit of parallelism for reads and writes
├─ Assigned to exactly one leader broker
├─ Replicated to follower brokers
└─ The smallest unit of data movement

Key Properties:
├─ Messages within a partition are ordered
├─ Each message has a unique offset
├─ Partitions are independent of each other
└─ Partition count cannot be decreased (only increased)

Partition Distribution

Topic: "orders" with 6 partitions, RF=3

Broker 1:  Broker 2:  Broker 3:
P0 (L)     P0 (F)     P0 (F)
P1 (F)     P1 (L)     P1 (F)
P2 (F)     P2 (F)     P2 (L)
P3 (L)     P3 (F)     P3 (F)
P4 (F)     P4 (L)     P4 (F)
P5 (F)     P5 (F)     P5 (L)

L = Leader
F = Follower

Each partition has:
├─ 1 Leader (handles reads/writes)
└─ 2 Followers (replicate data)

2. Replication Factor

What is Replication Factor?

Replication Factor (RF) = Number of copies of each partition

RF=1:  No replication (1 copy)
RF=2:  Leader + 1 follower
RF=3:  Leader + 2 followers (RECOMMENDED)
RF=4+: Leader + 3+ followers

Each copy is called a REPLICA

Replication Factor Rules

CONSTRAINTS:

1. RF cannot exceed broker count
   ├─ 3 brokers → max RF = 3
   └─ If RF > brokers, topic creation fails

2. RF is set per topic
   ├─ Different topics can have different RF
   └─ Cannot change RF without reassignment

3. RF applies to all partitions in topic
   ├─ If topic has 12 partitions, RF=3
   └─ Total replicas = 12 × 3 = 36

Choosing Replication Factor

RF=1: ❌ NOT FOR PRODUCTION
├─ No fault tolerance
├─ Broker failure = data loss
└─ Use only for: Development, disposable data

RF=2: ⚠️ MINIMAL (Not Recommended)
├─ Survives 1 broker failure
├─ No room for maintenance
└─ Use only for: Non-critical data

RF=3: ✅ RECOMMENDED FOR PRODUCTION
├─ Survives 2 broker failures
├─ Allows rolling restarts
├─ Industry standard
└─ Use for: Most production workloads

RF=4+: ✅ HIGH AVAILABILITY
├─ Survives 3+ broker failures
├─ Higher cost (more storage)
└─ Use for: Mission-critical data

3. Leader and Follower Roles

Leader Replica

LEADER RESPONSIBILITIES:

1. Handle ALL writes
   └─ Producers send messages only to leader

2. Serve reads (default)
   └─ Consumers fetch from leader

3. Track ISR
   └─ Determine which followers are in-sync

4. Accept follower fetch requests
   └─ Followers replicate from leader

LEADER PROPERTIES:
├─ Only one leader per partition at a time
├─ Leader elected by controller
└─ Leader can fail over to follower

Follower Replicas

FOLLOWER RESPONSIBILITIES:

1. Replicate data from leader
   └─ Continuously fetch new messages

2. Stay in-sync with leader
   └─ Keep up to date to join ISR

3. Be ready to become leader
   └─ If current leader fails

FOLLOWER PROPERTIES:
├─ Do NOT serve writes (only leader)
├─ Do NOT serve reads (by default, see replica.fetch.lag.max.messages)
└─ Passive replication (pull-based)

Leadership Distribution

GOOD LEADERSHIP DISTRIBUTION:

Broker 1: Leaders for P0, P3, P6, P9  (4 leaders)
Broker 2: Leaders for P1, P4, P7, P10 (4 leaders)
Broker 3: Leaders for P2, P5, P8, P11 (4 leaders)

Balanced load across brokers

BAD LEADERSHIP DISTRIBUTION:

Broker 1: Leaders for P0-P9 (10 leaders)
Broker 2: Leaders for P10-P11 (2 leaders)
Broker 3: No leaders (0 leaders)

Broker 1 overloaded! This happens if:
├─ Automatic leader rebalancing disabled
├─ Manual reassignments not balanced
└─ Broker restarts caused leader concentration

4. In-Sync Replicas (ISR)

What is ISR?

ISR = In-Sync Replica Set

ISR is the set of replicas that are:
├─ Fully caught up with the leader
├─ Within replica.lag.time.max.ms of leader
└─ Eligible to become leader if current leader fails

For a partition with RF=3:
├─ ISR might be [1, 2, 3]  ← All replicas in-sync
├─ ISR might be [1, 2]     ← Replica 3 fell behind
└─ ISR might be [1]        ← Only leader (DANGER!)

ISR Membership Conditions

A replica is in ISR if:

1. Replica is alive
   └─ Broker is running and reachable

2. Replica is caught up
   └─ Last fetch timestamp within replica.lag.time.max.ms
   └─ Default: 30 seconds

3. Replica is not stuck
   └─ Actively fetching from leader

If ANY condition fails:
└─ Replica removed from ISR by leader

ISR Configuration

# Broker-level config (applies to all topics)
replica.lag.time.max.ms=30000  # 30 seconds (default)

# How it works:
# If follower hasn't fetched in 30s → removed from ISR
# If follower catches up → added back to ISR

Impact of replica.lag.time.max.ms:

LOWER VALUE (e.g., 10s):
├─ Faster detection of slow replicas
├─ More aggressive ISR shrinking
└─ Risk: Temporary network issues cause ISR thrash

HIGHER VALUE (e.g., 60s):
├─ More tolerant of temporary slowness
├─ Less ISR thrashing
└─ Risk: Slower detection of truly failed replicas

RECOMMENDED: 30 seconds (default)

ISR State Transitions

NORMAL OPERATION:

ISR = [1, 2, 3]  ← All replicas caught up
└─ Leader: Broker 1
└─ Followers: Brokers 2, 3 (both in-sync)

FOLLOWER FALLS BEHIND:

Broker 3 slows down (disk issue, GC pause)
├─ Leader notices: last fetch > 30s ago
├─ Leader removes replica 3 from ISR
└─ ISR = [1, 2]  ← Shrunken ISR

FOLLOWER CATCHES UP:

Broker 3 recovers and fetches all missing data
├─ Leader notices: replica caught up
├─ Leader adds replica 3 back to ISR
└─ ISR = [1, 2, 3]  ← ISR restored

Checking ISR Status

# Describe topic to see ISR
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

# Output:
# Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# Topic: orders  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2

ISR Interpretation:

Isr: 1,2,3  ← HEALTHY (ISR == Replicas)
Isr: 1,2    ← DEGRADED (replica 3 out of sync)
Isr: 1      ← DANGER (only leader in ISR, no redundancy!)

5. Under-Replicated Partitions

What are Under-Replicated Partitions?

Under-Replicated Partition (URP):
├─ A partition where ISR < Replicas
├─ Not all replicas are in-sync
└─ Indicates replication lag or failure

Example:
├─ Replicas: [1, 2, 3]
├─ ISR: [1, 2]
└─ Under-replicated! (replica 3 missing from ISR)

Causes of Under-Replication

COMMON CAUSES:

1. Slow Broker
   ├─ Disk I/O bottleneck
   ├─ Network saturation
   ├─ CPU overload
   └─ GC pauses

2. Broker Failure
   ├─ Broker crashed
   ├─ Broker stopped for maintenance
   └─ Network partition

3. Rebalance in Progress
   ├─ Partition reassignment ongoing
   ├─ New replica catching up
   └─ Temporary URP (normal)

4. Configuration Issues
   ├─ replica.lag.time.max.ms too aggressive
   ├─ Insufficient disk throughput
   └─ Under-provisioned cluster

Detecting Under-Replicated Partitions

# List under-replicated partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --under-replicated-partitions

# Output (if any URPs exist):
# Topic: orders  Partition: 5  Leader: 1  Replicas: 1,2,3  Isr: 1,2

# Count URPs
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --under-replicated-partitions | grep "Partition:" | wc -l

Impact of Under-Replication

IMMEDIATE IMPACT:

✅ Data plane UNAFFECTED
   └─ Producers and consumers continue normally
   └─ No data loss (yet)

⚠️ REDUCED FAULT TOLERANCE
   └─ Fewer replicas available
   └─ Higher risk of data loss if leader fails

Example:
├─ RF=3, ISR=[1,2,3]: Can survive 2 broker failures
└─ RF=3, ISR=[1]:     Cannot survive ANY broker failure

Resolving Under-Replication

TROUBLESHOOTING STEPS:

1. Identify the cause
   └─ Check broker health (CPU, disk, network)
   └─ Check logs for errors

2. Fix the underlying issue
   ├─ If disk slow: Add capacity or improve I/O
   ├─ If broker down: Restart broker
   ├─ If network issue: Fix network
   └─ If reassignment: Wait for completion

3. Monitor ISR recovery
   └─ Watch for replica to rejoin ISR
   └─ May take minutes to hours depending on lag

4. Validate cluster health
   └─ Ensure URP count returns to 0

6. Leader Election

When Does Leader Election Occur?

LEADER ELECTION TRIGGERED BY:

1. Broker Failure
   └─ Leader broker crashes or becomes unreachable

2. Controlled Shutdown
   └─ Leader broker shutting down gracefully

3. Partition Reassignment
   └─ Manual or automatic partition movement

4. Preferred Leader Election
   └─ Rebalancing leaders across brokers

Clean Leader Election

CLEAN LEADER ELECTION:

Conditions:
└─ At least one ISR replica available (besides leader)

Process:
1. Controller detects leader failure
2. Controller selects new leader from ISR
   └─ Typically the first replica in ISR list
3. Controller updates metadata
4. New leader starts accepting writes
5. Followers start fetching from new leader

Data Loss: NONE
└─ New leader is guaranteed caught up (was in ISR)

Unclean Leader Election

UNCLEAN LEADER ELECTION:

Conditions:
└─ NO ISR replicas available
└─ Only out-of-sync replicas exist

Process:
1. Controller detects leader failure
2. NO replicas in ISR!
3. If unclean.leader.election.enable=true:
   ├─ Controller elects non-ISR replica as leader
   ├─ Out-of-sync replica becomes leader
   └─ DATA LOSS: Messages not replicated to this follower are lost
4. If unclean.leader.election.enable=false:
   └─ Partition becomes OFFLINE (unavailable)

Unclean Leader Election Configuration

# Per-topic config (recommended)
unclean.leader.election.enable=false

# Broker-level default
unclean.leader.election.enable=false  # Default since Kafka 1.0

# Trade-off:
# false: Prefer consistency (avoid data loss)
# true:  Prefer availability (tolerate data loss)

When to enable unclean election:

ENABLE (true) when:
├─ Availability > Consistency
├─ Data loss is acceptable
└─ Must keep partition online

DISABLE (false) when:
├─ Consistency > Availability
├─ Data loss NOT acceptable
└─ Prefer downtime over data loss

RECOMMENDATION for most use cases: false (disabled)

7. Preferred Leader Election

What is Preferred Leader?

Preferred Leader = The first replica in replica assignment

Example:
├─ Partition 0: Replicas [1, 2, 3]
└─ Preferred leader: Broker 1

Why "preferred"?
└─ Initial assignment attempts to balance leaders
└─ If all replicas healthy, first replica should be leader

Why Leaders Become Unbalanced

CAUSES OF IMBALANCE:

1. Broker Restarts
   └─ Leader fails over to follower
   └─ When original broker returns, doesn't reclaim leadership

2. Broker Failures
   └─ Leaders migrate to remaining brokers
   └─ After recovery, leaders not rebalanced

3. Partition Reassignments
   └─ Manual reassignments change replica set
   └─ Leaders may concentrate on subset of brokers

RESULT:
├─ Some brokers have many leaders (overloaded)
├─ Some brokers have few leaders (underutilized)
└─ Uneven load distribution

Automatic Preferred Leader Election

# Enable automatic rebalancing
auto.leader.rebalance.enable=true  # Default: true

# Imbalance threshold (percentage)
leader.imbalance.per.broker.percentage=10  # Default: 10%

# Check interval
leader.imbalance.check.interval.seconds=300  # Default: 300s (5 min)

# How it works:
# Controller checks leader imbalance every 5 minutes
# If imbalance > 10%, triggers preferred leader election

Manual Preferred Leader Election

# Trigger preferred leader election for all topics
kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --election-type preferred \
  --all-topic-partitions

# Trigger for specific topic
kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --election-type preferred \
  --topic orders

# Trigger for specific partition
kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --election-type preferred \
  --topic orders \
  --partition 5

Preferred Leader Election Best Practices

✅ DO:
├─ Enable auto.leader.rebalance.enable in production
├─ Run manual election after broker restarts
├─ Monitor leader distribution
└─ Validate after partition reassignments

❌ DON'T:
├─ Trigger during high traffic (causes brief unavailability)
├─ Run too frequently (causes churn)
└─ Disable automatic rebalancing without good reason

8. Partition Reassignment

When to Reassign Partitions

REASONS FOR REASSIGNMENT:

1. Adding Brokers
   └─ Rebalance partitions to new brokers

2. Removing Brokers
   └─ Move partitions off broker before decommissioning

3. Rebalancing Load
   └─ Fix uneven partition distribution

4. Rack Awareness
   └─ Ensure replicas spread across racks

5. Disk Usage
   └─ Move partitions from full disk to empty disk

Reassignment Process

STEP 1: Generate Reassignment Plan

# Create topics-to-move.json
{
  "topics": [
    {"topic": "orders"},
    {"topic": "payments"}
  ],
  "version": 1
}

# Generate plan
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics-to-move.json \
  --broker-list "0,1,2,3" \
  --generate

# Outputs:
# - Current assignment (backup)
# - Proposed assignment (new plan)

STEP 2: Review and Save Plan

# Save proposed assignment to reassignment.json
# Review carefully before executing!

STEP 3: Execute Reassignment

kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# Kafka begins moving data in background

STEP 4: Monitor Progress

kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

# Output:
# Status of each partition (ongoing or complete)

Reassignment Impact

DURING REASSIGNMENT:

✅ Data plane: Operational
   └─ Producers and consumers continue normally

⚠️ Increased load:
   ├─ Network: Data being copied between brokers
   ├─ Disk: Additional I/O for replication
   └─ CPU: Compression, network processing

⏱ Duration:
   └─ Hours to days depending on data volume

AFTER REASSIGNMENT:

✅ Partitions moved to new brokers
✅ Old replicas deleted
└─ Cluster rebalanced

Throttling Reassignment

# Limit replication bandwidth to 50 MB/s
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers \
  --entity-name 0 \
  --alter \
  --add-config leader.replication.throttled.rate=52428800,follower.replication.throttled.rate=52428800

# Do this for all brokers involved

# After reassignment complete, remove throttle:
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers \
  --entity-name 0 \
  --alter \
  --delete-config leader.replication.throttled.rate,follower.replication.throttled.rate

9. Monitoring Replication Health

Key Metrics to Monitor

1. Under-Replicated Partitions
   └─ Target: 0

2. Offline Partitions
   └─ Target: 0

3. ISR Shrinks
   └─ Frequency of ISR changes

4. ISR Expands
   └─ Recovery rate

5. Leader Election Rate
   └─ Frequency of elections (should be low)

6. Preferred Leader Imbalance
   └─ Percentage of misplaced leaders

7. Replica Lag (Time)
   └─ How far behind followers are

Health Check Commands

# 1. Check under-replicated partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

# 2. Check unavailable (offline) partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --unavailable-partitions

# 3. Check specific topic health
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

# 4. Get cluster-wide partition count
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "PartitionCount" | \
  awk '{sum += $4} END {print sum}'

10. Common Replication Issues

Issue 1: Persistent Under-Replication

SYMPTOM:
└─ Under-replicated partitions never go to 0

CAUSES:
├─ Broker consistently slow (disk, CPU, network)
├─ Insufficient cluster capacity
├─ replica.lag.time.max.ms too aggressive
└─ Ongoing reassignment never completes

RESOLUTION:
├─ Identify slow broker (check metrics)
├─ Add capacity if cluster undersized
├─ Increase replica.lag.time.max.ms if appropriate
└─ Cancel stuck reassignments

Issue 2: ISR Thrashing

SYMPTOM:
└─ Replicas constantly joining/leaving ISR

CAUSES:
├─ Network instability
├─ GC pauses
├─ Borderline capacity (broker just barely keeping up)
└─ replica.lag.time.max.ms too low

RESOLUTION:
├─ Fix network issues
├─ Tune JVM GC
├─ Add capacity
└─ Increase replica.lag.time.max.ms

Issue 3: Offline Partitions

SYMPTOM:
└─ Partitions with no leader (unavailable)

CAUSES:
├─ All ISR replicas down
├─ Unclean leader election disabled
└─ No healthy replicas available

RESOLUTION:
├─ Restart failed brokers
├─ Wait for replicas to rejoin ISR
├─ Consider enabling unclean election (if data loss acceptable)
└─ Last resort: Manual intervention with kafka-leader-election.sh

11. Key Takeaways

You've learned:

✅ Partitions are units of parallelism with a single leader
✅ Replication factor determines number of copies (RF ≥ 3 recommended)
✅ ISR is the set of replicas caught up with leader
✅ Under-replicated partitions indicate replication lag or failures
✅ Clean leader election preserves data (new leader from ISR)
✅ Unclean leader election may cause data loss (non-ISR leader)
✅ Preferred leader election rebalances leaders across brokers
✅ Partition reassignment moves partitions between brokers
✅ Monitoring ISR and URP is critical for cluster health

12. Operational Checklist

DAILY REPLICATION HEALTH CHECKS:

☑ Under-replicated partitions = 0
☑ Offline partitions = 0
☑ ISR matches replica set for all partitions
☑ Leader distribution balanced
☑ No ongoing reassignments (unless planned)

WEEKLY MAINTENANCE:

☑ Review replica lag trends
☑ Check for ISR thrashing
☑ Validate preferred leaders
☑ Plan capacity if URPs increasing

AFTER BROKER RESTART:

☑ Verify all partitions back in ISR
☑ Trigger preferred leader election
☑ Monitor catch-up time

Summary

Replication health is the foundation of Kafka reliability:

Healthy Replication =
├─ ISR == Replicas (all caught up)
├─ Under-replicated partitions = 0
├─ Leaders balanced across brokers
├─ No offline partitions
└─ Quick recovery from failures

Remember: Monitor ISR closely, investigate under-replication immediately, and always maintain RF ≥ 3 in production.

Next up: Module 3.1 — Dynamic Configuration & Runtime Changes — Learn how to safely change configurations without restarts.