4.1 Rebalancing & Group Stability

Module Overview

Goal: Provide the ONE authoritative explanation of rebalancing. This is the ONLY place where rebalancing is explained deeply.

What You'll Learn:

• Why rebalances happen
• Rebalance phases (JoinGroup, SyncGroup)
• Stop-the-world effect
• Eager vs Cooperative rebalancing
• Static membership
• Impact on latency and throughput
• Why rebalances are dangerous in production
• How to minimize rebalances

What's NOT Covered Here:

• Assignment strategies (covered in Module 3.1)
• Poll loop mechanics (covered in Module 6.1)
• Configuration tuning (covered in Module 7.1)

Key Insight: Rebalancing is necessary for elasticity but dangerous for stability. Understanding it is critical for production systems.

1. What is a Rebalance?

Definition

A rebalance is the process of:

• Redistributing partitions among consumers in a group
• Triggered by group membership or metadata changes
• Coordinated by the Group Coordinator

Think of it as reorganizing a team when:

• A new team member joins
• A team member leaves
• Tasks (partitions) are added or removed

Before Rebalance:
Consumer A → Partitions 0, 1
Consumer B → Partitions 2, 3

[Consumer C joins]

After Rebalance:
Consumer A → Partition 0
Consumer B → Partition 2
Consumer C → Partitions 1, 3

2. Why Rebalances Happen

Triggers for Rebalancing

1. Consumer Joins the Group

// New consumer starts
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("orders"));
consumer.poll(Duration.ofMillis(100)); // ← Triggers join → Rebalance

When: New consumer instance starts and calls poll().

2. Consumer Leaves the Group

Graceful shutdown:

consumer.close(); // Explicit leave → Rebalance

Ungraceful shutdown:

• Process crash
• Network partition
• Consumer stops sending heartbeats
• Consumer stops calling poll() within max.poll.interval.ms

Result: Coordinator detects dead consumer → Rebalance

3. Consumer Subscription Changes

// Consumer changes what it's subscribed to
consumer.subscribe(Collections.singletonList("orders"));
// Later...
consumer.subscribe(Arrays.asList("orders", "payments")); // → Rebalance

When: Consumer modifies its topic subscription.

4. Topic Partition Count Changes

# Admin adds partitions to topic
kafka-topics.sh --alter --topic orders --partitions 10

Result: New partitions available → Rebalance to assign them.

5. Heartbeat Timeout

Consumer                    Coordinator
   │                            │
   ├──── heartbeat ───────────→ │
   │                            │
   ├──── heartbeat ───────────→ │
   │                            │
   │  (heartbeat stops)          │
   │                            │
   │                            ├─ Timeout!
   │                            │  session.timeout.ms elapsed
   │                            └─ Trigger rebalance

When: Consumer misses heartbeats for session.timeout.ms.

6. Poll Timeout

Consumer                    Coordinator
   │                            │
   ├──── heartbeat ───────────→ │  (background thread)
   │                            │
   │  poll()                    │
   │  ... processing ...        │
   │  ... still processing ...  │
   │  (no poll for 5+ minutes)  │
   │                            │
   │                            ├─ Timeout!
   │                            │  max.poll.interval.ms elapsed
   │                            └─ Trigger rebalance

When: Consumer doesn't call poll() within max.poll.interval.ms.

Summary of Triggers

3. The Rebalance Protocol (Eager)

Eager Rebalancing Phases

Eager rebalancing (used by Range, RoundRobin, Sticky) has two phases:

Phase 1: JoinGroup
├─ All consumers stop consuming
├─ Consumers send JoinGroupRequest
├─ Coordinator waits for all consumers
├─ Coordinator selects group leader
└─ Leader receives member list

Phase 2: SyncGroup
├─ Leader computes partition assignment
├─ Leader sends assignment to coordinator
├─ Coordinator sends assignment to all consumers
└─ All consumers resume consuming

Key characteristic: All consumers stop consuming during the rebalance (stop-the-world).

Phase 1: JoinGroup

Time  Consumer A    Consumer B    Consumer C    Coordinator
────┼─────────────┼─────────────┼─────────────┼──────────────────
  1 │ Consuming   │ Consuming   │ Consuming   │
  2 │             │             │ (dies)      │
  3 │             │             │             │ Detects C is dead
  4 │             │             │             │ Trigger rebalance
  5 │ ← Rebalance │ ← Rebalance │             │ Send to A, B
  6 │ Stop ⏸️      │ Stop ⏸️      │             │
  7 │ Revoke all  │ Revoke all  │             │
  8 │ JoinGroup →│             │             │
  9 │             │ JoinGroup →│             │
 10 │             │             │             │ Wait...
 11 │             │             │             │ All joined
 12 │ ← Leader    │ ← Member    │             │ Select leader (A)

What happens:

1. Coordinator detects change
2. Sends rebalance signal to all consumers
3. Consumers stop consuming
4. Consumers revoke all partitions
5. Consumers send JoinGroupRequest
6. Coordinator waits for all consumers (or timeout)
7. Coordinator selects group leader

Phase 2: SyncGroup

Time  Leader (A)    Member (B)     Coordinator
────┼─────────────┼──────────────┼─────────────────────
 13 │ Receives    │              │
    │ member list │              │
 14 │ Computes    │              │
    │ assignment  │              │
 15 │ SyncGroup → │              │ (with assignment)
 16 │             │ SyncGroup → │ (empty)
 17 │             │              │ Distribute assignment
 18 │ ← Your      │ ← Your       │
    │   partitions│   partitions │
 19 │ Resume ▶️    │ Resume ▶️     │

What happens:

1. Leader computes partition assignment
2. Leader sends assignment to coordinator
3. Coordinator distributes to all consumers
4. Consumers receive their partitions
5. Consumers resume consuming

Total Rebalance Time

Total rebalance time =
    JoinGroup phase
  + SyncGroup phase
  + Time to fetch initial records

Typical: 5-30 seconds
Slow:    30 seconds - 2 minutes

During this time: Zero consumption from the group!

4. Stop-the-World Effect

The Problem

Consumer Group: 10 consumers
Topic: 100 partitions
Rebalance duration: 10 seconds

Impact:
- 10 consumers × 10 seconds = 100 consumer-seconds of downtime
- All 100 partitions paused
- Lag increases during rebalance
- Downstream systems starved

Key Insight: Rebalances are not free. They cause consumption pauses.

Latency Impact

Normal operation:
Consumer → poll() → [records] → process → commit
Latency: ~100ms

During rebalance:
Consumer → poll() → [rebalance!] → wait... → wait... → [records]
Latency: 10+ seconds

Result: p99 latency spikes!

Throughput Impact

Normal: 10,000 msgs/sec consumed
During rebalance: 0 msgs/sec
After rebalance: Catch up at full speed

Metrics:
┌────────────────────────────────┐
│ Throughput                     │
│                                │
│ 10k─┐        ┌────┐           │
│     │        │    │            │
│  5k─┤        │    │            │
│     │        │    │            │
│  0──┴────────┴────┴────────────│
│     Normal  Rebal Resume       │
└────────────────────────────────┘

5. Cooperative Rebalancing

The Evolution

Problem with eager rebalancing:

• All consumers stop, even if they're not affected
• Consumer keeping same partitions still stops

Solution: Cooperative rebalancing (Kafka 2.4+)

• Only consumers losing partitions stop
• Others keep consuming
• May require multiple rounds

Cooperative Protocol

Phase 1: Revoke
├─ Consumers losing partitions revoke them
├─ Other consumers keep consuming ▶️
└─ Partial assignment

Phase 2: Assign
├─ Revoked partitions assigned to new consumers
└─ All consumers now have correct partitions

Key difference: Incremental instead of stop-the-world.

Example: Cooperative vs Eager

Scenario

Initial:
Consumer A: Partitions 0, 1, 2
Consumer B: Partitions 3, 4, 5

Consumer C joins
Goal:
Consumer A: Partitions 0, 1
Consumer B: Partitions 3, 4
Consumer C: Partitions 2, 5

Eager Rebalancing

Time  Consumer A      Consumer B      Consumer C
────┼───────────────┼───────────────┼─────────────
  1 │ Consuming 0-2 │ Consuming 3-5 │
  2 │ STOP ⏸️        │ STOP ⏸️        │ Joins
  3 │ Revoke all    │ Revoke all    │
  4 │ (waiting)     │ (waiting)     │ (waiting)
  5 │ (waiting)     │ (waiting)     │ (waiting)
  6 │ Assigned 0-1  │ Assigned 3-4  │ Assigned 2, 5
  7 │ Resume ▶️      │ Resume ▶️      │ Resume ▶️

Downtime: A, B both stopped for full rebalance

Cooperative Rebalancing

Time  Consumer A      Consumer B      Consumer C
────┼───────────────┼───────────────┼─────────────
  1 │ Consuming 0-2 │ Consuming 3-5 │
  2 │ Revoke 2      │ Revoke 5      │ Joins
  3 │ Consuming 0-1▶│ Consuming 3-4▶│ (waiting)
  4 │ Consuming 0-1▶│ Consuming 3-4▶│ Assigned 2, 5
  5 │ Consuming 0-1▶│ Consuming 3-4▶│ Consuming 2, 5▶

Downtime: None! A and B never fully stopped

Result: Minimal disruption, most consumers keep running.

Cooperative Rebalancing Pros and Cons

Pros:

• ✅ Minimal disruption — most consumers keep consuming
• ✅ Better for high-availability systems
• ✅ Reduced lag spikes

Cons:

• ❌ May require multiple rounds
• ❌ Slightly more complex protocol
• ❌ All consumers must support it (can't mix with eager)

Recommendation: Use CooperativeStickyAssignor for Kafka 2.4+.

6. Static Membership

The Problem with Dynamic Membership

Consumer restarts (e.g., deployment):
1. Consumer A shuts down
2. Rebalance triggered (A removed)
3. Consumer A starts up
4. Rebalance triggered (A joins)

Result: 2 rebalances for 1 restart!

Impact:

• Rolling restarts cause many rebalances
• Each rebalance pauses consumption
• Stateful consumers lose local state

Static Membership Solution

props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "consumer-1"); // Static ID

Behavior:

• Consumer has persistent identity
• If consumer rejoins within session.timeout.ms, no rebalance
• Partitions retained across restarts

Static Membership Example

Consumer A (group.instance.id = "consumer-1"):
1. Consumes partitions 0, 1, 2
2. Shuts down (graceful)
3. Coordinator marks as offline (but doesn't remove)
4. Wait session.timeout.ms...
5. If A rejoins within timeout:
   → No rebalance, same partitions
6. If A doesn't rejoin:
   → Rebalance triggered

Key: Rebalance delayed by session.timeout.ms, allowing quick restarts.

Static vs Dynamic Membership

When to Use Static Membership

✅ Use when:

• Consumers maintain local state (caches, RocksDB)
• Rolling restarts are common
• Minimizing rebalances is critical
• You can tolerate delayed failover

❌ Don't use when:

• Consumers are stateless
• Fast failover is critical
• Ad-hoc consumer additions

7. Impact on Production Systems

Why Rebalances Are Dangerous

1. Latency Spikes

p99 latency:
Normal:   50ms
Rebalance: 15 seconds ← SLA violation!

2. Lag Accumulation

During 10-second rebalance:
New messages: 100,000 (at 10k msgs/sec)
Lag increases: +100,000 messages

Time to catch up: Depends on consumer capacity

3. Cascading Failures

Consumer slow → Rebalance triggered
→ Other consumers take over partitions
→ Overload other consumers
→ More slow consumers
→ More rebalances
→ Rebalance storm! 🌪️

4. Duplicate Processing

Consumer A processing message at offset 100
→ Rebalance triggered before commit
→ Consumer B takes over partition
→ Starts from last committed offset (95)
→ Messages 95-100 processed twice!

Rebalance Storms

Rebalance storm: Continuous rebalancing with no stable state.

Causes:

1. Consumer processing too slow (max.poll.interval.ms too short)
2. GC pauses (heartbeat timeouts)
3. Network issues
4. Cascading failures

Symptoms:

• Constant rebalancing in logs
• No stable consumption
• Ever-increasing lag
• Metrics show 0 throughput

Solution:

• Fix root cause (slow processing, GC, network)
• Increase timeouts
• Reduce max.poll.records
• Add more consumers (if underprovisioned)

8. Minimizing Rebalances

Strategy 1: Tune Timeouts

// Increase session timeout (slower failover, fewer false positives)
props.put("session.timeout.ms", 30000);        // 30 seconds (default: 10s)
props.put("heartbeat.interval.ms", 10000);     // 10 seconds (default: 3s)

// Increase poll interval (allow longer processing)
props.put("max.poll.interval.ms", 600000);     // 10 minutes (default: 5m)

Trade-off: Slower failure detection vs fewer false positives.

Strategy 2: Use Cooperative Rebalancing

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    CooperativeStickyAssignor.class.getName());

Benefit: Partial rebalances instead of stop-the-world.

Strategy 3: Use Static Membership

props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "consumer-" + instanceId);

Benefit: Rolling restarts don't trigger rebalances.

Strategy 4: Reduce Slow Processing

// Reduce batch size (faster poll loop)
props.put("max.poll.records", 100);  // Default: 500

Trade-off: Lower throughput vs faster polls.

Strategy 5: Optimize Consumer Code

// Bad: Slow processing
while (true) {
    records = consumer.poll(Duration.ofMillis(100));
    for (record : records) {
        slowDatabaseCall(record);  // 5 seconds per record!
    }
}

// Good: Fast processing
while (true) {
    records = consumer.poll(Duration.ofMillis(100));
    for (record : records) {
        queue.add(record);  // Fast, async
    }
}
// Separate thread processes queue

Strategy 6: Plan Deployments

For dynamic membership:

• Accept that rolling restarts cause rebalances
• Deploy during low-traffic windows
• Monitor lag during deployments

For static membership:

• Restart quickly (< session.timeout.ms)
• Use health checks to ensure fast startup
• Consider blue-green deployments

9. Detecting Rebalances

In Consumer Logs

[main] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=my-group] Discovered group coordinator broker1:9092
[main] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=my-group] (Re-)joining group
[main] INFO  o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=my-group] Successfully joined group with generation 5
[main] INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=my-group] Finished rebalance
[main] INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=my-group] Setting offset for partition orders-0 to position 12345

Look for:

• "(Re-)joining group"
• "Successfully joined group with generation X"
• "Finished rebalance"

Using ConsumerRebalanceListener

consumer.subscribe(
    Collections.singletonList("orders"),
    new ConsumerRebalanceListener() {
        @Override
        public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
            System.out.println("Partitions revoked: " + partitions);
            // Commit offsets before losing partitions
            consumer.commitSync();
        }

        @Override
        public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
            System.out.println("Partitions assigned: " + partitions);
            // Initialize state for new partitions
        }
    }
);

Use cases:

• Commit offsets before revocation
• Save local state
• Initialize state for new partitions
• Metrics/logging

Monitoring Metrics

JMX Metrics:
- kafka.consumer:type=consumer-coordinator-metrics,client-id=*,attribute=rebalance-total
- kafka.consumer:type=consumer-coordinator-metrics,client-id=*,attribute=rebalance-rate-per-hour
- kafka.consumer:type=consumer-coordinator-metrics,client-id=*,attribute=rebalance-latency-avg

Alert on:

• High rebalance rate
• Increasing rebalance frequency
• Long rebalance latency

10. Mental Model Summary

Key Concepts

REBALANCE = Redistribution of partitions

Triggers:
├─ Consumer joins/leaves
├─ Heartbeat timeout
├─ Poll timeout
└─ Partition count change

Types:
├─ Eager: Stop-the-world
└─ Cooperative: Incremental

Impact:
├─ Consumption pause
├─ Latency spike
├─ Lag accumulation
└─ Potential duplicates

The Rebalance Protocol

Eager:
1. All consumers stop
2. Revoke all partitions
3. Compute new assignment
4. All consumers resume

Cooperative:
1. Some consumers revoke partitions
2. Others keep consuming
3. Revoked partitions reassigned
4. Minimal disruption

Questions You Should Be Able to Answer

✅ What is a rebalance?
✅ What triggers a rebalance?
✅ What are the two rebalance phases?
✅ What is the stop-the-world effect?
✅ What's the difference between eager and cooperative rebalancing?
✅ What is static membership and when should you use it?
✅ Why are rebalances dangerous in production?
✅ How can you minimize rebalances?

Questions We Haven't Answered Yet

❌ How are offsets managed during rebalances? → Module 5
❌ What happens in the poll loop? → Module 6.1
❌ How do I tune consumer configs? → Module 7.1

11. What's Next?

Now that you understand rebalancing deeply, the next modules will cover:

• Module 5: Offset management and delivery semantics
• Module 6.1: Poll loop internals
• Module 7.1: Configuration and performance tuning
• Module 8.1: Error handling strategies

Summary

You learned:

• ✅ Rebalancing redistributes partitions among consumers
• ✅ Triggered by membership changes, timeouts, or metadata changes
• ✅ Eager rebalancing stops all consumers (stop-the-world)
• ✅ Cooperative rebalancing minimizes disruption (Kafka 2.4+)
• ✅ Static membership reduces rebalances during restarts
• ✅ Rebalances cause latency spikes, lag, and potential duplicates
• ✅ Minimize rebalances with tuning, cooperative protocols, and static membership

Next up: Offsets Explained (Module 5.1) — Learn what offsets are and how they track consumption progress.