2.2 Broker-Side Partition & Leader Management

Module Overview

Goal: Understand how Kafka brokers manage partition leadership and distribution — a complementary perspective to consumer-side coordination.

What You'll Learn:

• How partition leaders are distributed across brokers
• Automatic leader rebalancing mechanisms
• Leader vs follower replica roles
• Manual partition reassignment tools
• Rack awareness for high availability
• Tuning considerations for large clusters

What's NOT Covered Here:

• Consumer group partition assignment (covered in Module 3.1)
• Consumer rebalancing (covered in Module 4.1)
• Producer partitioning strategy (covered in Producer modules)

Key Insight: While consumers assign partitions among themselves, brokers manage which broker leads each partition. This is orthogonal to consumer coordination but affects availability and performance.

1. Partition Leadership Basics

Leaders vs Followers

Every partition has multiple replicas for fault tolerance:

Topic: "orders", Partition 0, Replication Factor: 3

┌────────────────────────────────────────────┐
│ Broker 1: LEADER (Partition 0)            │
│  ├─ Handles all reads and writes          │
│  ├─ Coordinates follower replication      │
│  └─ Responds to producer/consumer requests │
└────────────────────────────────────────────┘
         │
         ├──── replication ────→ ┌────────────────────────┐
         │                        │ Broker 2: FOLLOWER     │
         │                        │  └─ Receives updates   │
         │                        └────────────────────────┘
         │
         └──── replication ────→ ┌────────────────────────┐
                                  │ Broker 3: FOLLOWER     │
                                  │  └─ Receives updates   │
                                  └────────────────────────┘

Key Roles:

Why This Matters:

• Only the leader serves client requests
• If leaders are unevenly distributed, some brokers become bottlenecks
• Automatic leader rebalancing prevents hotspots

Preferred Leader

Each partition has a preferred leader — the replica that Kafka prefers to be the leader.

Topic: "orders", Partition 0
Replicas: [1, 2, 3]  ← Broker 1 is the preferred leader
         ↑
         Preferred Leader (first in list)

Topic: "orders", Partition 1
Replicas: [2, 3, 1]  ← Broker 2 is the preferred leader
         ↑
         Preferred Leader

Preferred Leader Election:

• By default, the first replica in the replica list is the preferred leader
• Kafka tries to keep the preferred leader as the actual leader
• This distributes leadership evenly across brokers

2. Leader Rebalancing Configurations

The Problem: Leader Skew

Scenario: 3-broker cluster, 12 partitions

Initial (balanced):
Broker 1: Leader for P0, P3, P6, P9      (4 leaders)
Broker 2: Leader for P1, P4, P7, P10     (4 leaders)
Broker 3: Leader for P2, P5, P8, P11     (4 leaders)

After Broker 2 crashes and recovers:
Broker 1: Leader for P0, P1, P3, P4, P6, P7, P9, P10  (8 leaders!)
Broker 2: Leader for NONE                              (0 leaders)
Broker 3: Leader for P2, P5, P8, P11                   (4 leaders)

Result: Broker 1 is overloaded, Broker 2 is idle

Solution: Automatic leader rebalancing moves leadership back to preferred leaders.

Configuration: auto.leader.rebalance.enable

# Enable automatic leader rebalancing (default: true)
auto.leader.rebalance.enable=true

What It Does:

• Kafka periodically checks for leader imbalance
• If imbalance exceeds threshold, triggers preferred leader election
• Moves leadership back to preferred replicas

When to Disable:

• During maintenance windows
• If you prefer manual control
• In very large clusters with frequent changes

Configuration: leader.imbalance.check.interval.ms

# How often to check for leader imbalance (default: 5 minutes)
leader.imbalance.check.interval.ms=300000

What It Does:

• Controls frequency of leader balance checks
• Lower values = faster rebalancing but more overhead
• Higher values = slower rebalancing but less overhead

Tuning Guidance:

Configuration: leader.imbalance.per.broker.percentage

# Trigger rebalancing if imbalance exceeds 10% (default)
leader.imbalance.per.broker.percentage=10

How It's Calculated:

Leader Imbalance % = (Actual Leaders - Expected Leaders) / Expected Leaders × 100

Example:
Expected Leaders per Broker: 100
Broker 1 Actual Leaders: 115

Imbalance = (115 - 100) / 100 × 100 = 15%

Since 15% > 10% (threshold), rebalancing triggers

Tuning Guidance:

Configuration: unclean.leader.election.enable

# Allow out-of-sync replicas to become leader (default: false)
unclean.leader.election.enable=false

What It Controls:

Scenario: Leader dies, all ISR replicas also unavailable

Option 1: unclean.leader.election.enable=false
├─ Wait for ISR replica to come back
├─ Partition remains unavailable
└─ NO DATA LOSS

Option 2: unclean.leader.election.enable=true
├─ Elect out-of-sync replica as leader
├─ Partition becomes available immediately
└─ POTENTIAL DATA LOSS (replica may be behind)

Availability vs Durability Trade-off:

⚠️ Warning: Enabling unclean leader election can cause:

• Message loss
• Offset inconsistencies
• Consumer confusion (messages may "disappear")

3. Manual Partition Reassignment

When to Use Manual Reassignment

Common Scenarios:
1. Adding new brokers to cluster
   → Redistribute partitions to new brokers

2. Decommissioning brokers
   → Move partitions off broker before shutdown

3. Fixing imbalanced partition distribution
   → Manually rebalance storage or network load

4. Rack awareness configuration
   → Ensure replicas spread across racks/AZs

Tool: kafka-reassign-partitions.sh

Step 1: Generate Reassignment Plan

# Create topics.json with topics to reassign
cat > topics.json <<EOF
{
  "topics": [
    {"topic": "orders"},
    {"topic": "payments"}
  ],
  "version": 1
}
EOF

# Generate reassignment plan for specific brokers
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3" \
  --generate

# Output:
# Current partition assignment:
# {"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[0,1]},...]}
#
# Proposed partition reassignment:
# {"version":1,"partitions":[{"topic":"orders","partition":0,"replicas":[2,3]},...]}

Step 2: Execute Reassignment

# Save proposed reassignment to file
cat > reassignment.json <<EOF
{
  "version": 1,
  "partitions": [
    {"topic": "orders", "partition": 0, "replicas": [2, 3]},
    {"topic": "orders", "partition": 1, "replicas": [3, 0]},
    ...
  ]
}
EOF

# Execute reassignment
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# Monitor progress
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Using Kafka Admin API for Reassignment

import org.apache.kafka.clients.admin.*;
import org.apache.kafka.common.TopicPartitionReplica;

Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

try (AdminClient admin = AdminClient.create(props)) {
    // Define new replica assignment
    Map<TopicPartition, Optional<NewPartitionReassignment>> reassignments = Map.of(
        new TopicPartition("orders", 0),
        Optional.of(new NewPartitionReassignment(Arrays.asList(2, 3, 0))),
        
        new TopicPartition("orders", 1),
        Optional.of(new NewPartitionReassignment(Arrays.asList(3, 0, 1)))
    );

    // Execute reassignment
    AlterPartitionReassignmentsResult result = 
        admin.alterPartitionReassignments(reassignments);
    
    result.all().get(); // Wait for completion
    
    System.out.println("Partition reassignment initiated");
    
    // List ongoing reassignments
    ListPartitionReassignmentsResult listResult = 
        admin.listPartitionReassignments();
    
    Map<TopicPartition, PartitionReassignment> ongoing = 
        listResult.reassignments().get();
    
    ongoing.forEach((tp, reassignment) -> {
        System.out.printf("Partition %s: %s -> %s%n",
            tp,
            reassignment.replicas(),
            reassignment.addingReplicas());
    });
}

Throttling Reassignment Traffic

Problem: Partition reassignment can saturate network bandwidth, affecting production traffic.

Solution: Throttle replication bandwidth:

# Set replication throttle to 50 MB/s per broker
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute \
  --throttle 50000000

# Remove throttle after reassignment completes
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

# Manually remove throttle if needed
kafka-configs.sh \
  --bootstrap-server localhost:9092 \
  --entity-type brokers \
  --entity-name 0 \
  --alter \
  --delete-config leader.replication.throttled.rate,follower.replication.throttled.rate

Throttle Configuration:

4. Rack Awareness

What is Rack Awareness?

Rack Awareness ensures replicas are distributed across different physical racks or availability zones (AZs) for fault tolerance.

Without Rack Awareness:
Rack A: Broker 1 (Leader), Broker 2 (Follower), Broker 3 (Follower)
Rack B: Broker 4, Broker 5, Broker 6

If Rack A fails → All replicas lost → Data unavailable

With Rack Awareness:
Rack A: Broker 1 (Leader), Broker 2
Rack B: Broker 3 (Follower), Broker 4
Rack C: Broker 5 (Follower), Broker 6

If Rack A fails → Replicas in Rack B and C still available

Configuring Rack Awareness

Broker Configuration

# server.properties on each broker
broker.rack=rack-a

# For AWS availability zones
broker.rack=us-east-1a

# For Kubernetes clusters
broker.rack=zone-1

Topic Creation with Rack Awareness

# Create topic with rack-aware replica placement
kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2

# Kafka automatically spreads replicas across racks
# Partition 0: Replicas on [rack-a, rack-b, rack-c]
# Partition 1: Replicas on [rack-b, rack-c, rack-a]
# etc.

Rack Awareness Algorithm

Given:
- 3 racks: [A, B, C]
- 9 brokers: [0,1,2 in A], [3,4,5 in B], [6,7,8 in C]
- Topic with replication factor 3

Kafka's Rack-Aware Assignment:
1. Distribute leader replicas evenly across brokers
2. For each partition, place replicas in different racks
3. Balance replica count per broker

Example Result:
Partition 0: [0(A), 3(B), 6(C)]  ← 3 different racks
Partition 1: [1(A), 4(B), 7(C)]  ← 3 different racks
Partition 2: [2(A), 5(B), 8(C)]  ← 3 different racks

Benefits:

• ✅ Survives rack/AZ failure
• ✅ No single point of failure
• ✅ Higher availability

Trade-offs:

• ⚠️ Cross-rack replication may increase latency
• ⚠️ Higher network costs (cross-AZ traffic in cloud)

5. Monitoring Leader Distribution

JMX Metrics

// Key metrics to monitor

// Leaders per broker
kafka.server:type=ReplicaManager,name=LeaderCount

// Partitions per broker (leader + follower)
kafka.server:type=ReplicaManager,name=PartitionCount

// Offline partitions (no leader)
kafka.controller:type=KafkaController,name=OfflinePartitionsCount

// Preferred replica imbalance
kafka.controller:type=KafkaController,name=PreferredReplicaImbalanceCount

Using kafka-topics.sh to Inspect Leaders

# Describe topic to see leader distribution
kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

# Output:
# Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
# Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
# Topic: orders  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2

What to Look For:

• Balanced leaders: Leaders evenly distributed across brokers
• ISR = Replicas: All replicas in sync
• Preferred leaders: First replica in list is the current leader

Detecting Imbalance with Scripts

#!/bin/bash
# Check leader distribution across brokers

kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  | awk '/Leader:/ {print $6}' \
  | sort | uniq -c | sort -rn

# Output:
#  45 Leader:1  ← Broker 1 has 45 partitions as leader
#  38 Leader:2  ← Broker 2 has 38 partitions as leader
#  17 Leader:3  ← Broker 3 has 17 partitions as leader (IMBALANCED!)

Imbalance Indicators:

• Large variance in leader counts
• Preferred leader ≠ actual leader
• High PreferredReplicaImbalanceCount metric

Remediation:

1. Check auto.leader.rebalance.enable=true
2. Trigger manual preferred leader election:

   kafka-leader-election.sh \
     --bootstrap-server localhost:9092 \
     --election-type PREFERRED \
     --all-topic-partitions
   

6. Tuning Considerations for Large Clusters

Controller Overhead

The Kafka Controller (one broker elected as controller) manages:

• Partition leadership elections
• Broker liveness monitoring
• Metadata propagation

In large clusters:

• Frequent leader elections overwhelm controller
• Prefer stability over perfect balance

Tuning Recommendations:

# For clusters > 50 brokers
leader.imbalance.check.interval.ms=600000  # 10 minutes (default: 5 min)
leader.imbalance.per.broker.percentage=15  # More tolerant (default: 10%)
auto.leader.rebalance.enable=true          # Keep enabled but less aggressive

# For clusters > 100 brokers
leader.imbalance.check.interval.ms=900000  # 15 minutes
leader.imbalance.per.broker.percentage=20

Partition Count Limits

Per-Broker Partition Limits:

Why Limits Exist:

• Each partition has overhead (file handles, memory, metadata)
• Controller load increases with partition count
• Rebalancing takes longer with more partitions

Best Practices:

• Consolidate low-traffic topics
• Use fewer partitions for high-throughput topics (contrary to intuition)
• Monitor controller performance

7. Relationship to Consumer Coordination

Two Levels of Coordination

┌─────────────────────────────────────────────────────┐
│  BROKER-SIDE: Partition Leadership                  │
│  ├─ Which broker leads each partition?              │
│  ├─ Handled by Kafka Controller                     │
│  └─ Affects availability and performance            │
└─────────────────────────────────────────────────────┘
                      ↓
              Partitions exist
                      ↓
┌─────────────────────────────────────────────────────┐
│  CONSUMER-SIDE: Partition Assignment                │
│  ├─ Which consumer reads each partition?            │
│  ├─ Handled by Group Coordinator                    │
│  └─ Affects consumer group throughput                │
└─────────────────────────────────────────────────────┘

Key Differences:

Both Are Independent:

• Broker leadership can change without affecting consumer assignments
• Consumer rebalances don't affect broker leadership
• But both must be healthy for consumers to read data

8. Common Operational Scenarios

Scenario 1: Adding Brokers to Cluster

# 1. Start new broker with unique broker.id
# 2. New broker joins cluster but has no partitions

# 3. Manually reassign partitions to include new broker
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3,4"  ← Include new broker 4
  --generate

# 4. Execute reassignment with throttling
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute \
  --throttle 50000000

Scenario 2: Decommissioning a Broker

# 1. Generate reassignment excluding broker to remove
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2"  ← Exclude broker 3
  --generate

# 2. Execute reassignment
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# 3. Wait for reassignment to complete
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

# 4. Shut down broker safely
kafka-server-stop.sh

Scenario 3: Recovering from Leader Imbalance

# 1. Check current leader distribution
kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  | awk '/Leader:/ {print $6}' | sort | uniq -c

# 2. Trigger preferred leader election
kafka-leader-election.sh \
  --bootstrap-server localhost:9092 \
  --election-type PREFERRED \
  --all-topic-partitions

# 3. Verify rebalancing
kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  | awk '/Leader:/ {print $6}' | sort | uniq -c

Summary

You learned:

• ✅ Partition leadership is managed at the broker level by the Kafka Controller
• ✅ Preferred leader election automatically rebalances leader distribution
• ✅ Manual reassignment allows controlled partition movement for scaling and maintenance
• ✅ Rack awareness improves fault tolerance by spreading replicas across failure domains
• ✅ Broker-side and consumer-side coordination are independent but complementary
• ✅ Tuning large clusters requires balancing rebalancing frequency with controller overhead

Key Configurations:

Next up: Module 3.1 (Partition Assignment Strategies) — Learn how consumers assign partitions among themselves within a group.