KafkaIQ

10.4.2 – Partition Sizing and Rebalancing Strategy

Module: M10 – Advanced Topics & Architecture
Section: 10.4 – Cluster Governance & Architecture Decisions
Goal: Determine optimal partition count and manage partition rebalancing

Introduction

Partitions are Kafka's unit of parallelism. Too few = underutilized, too many = overhead. This module covers sizing and rebalancing strategies.

Partition Sizing Formula

Formula

Partitions = max(
  Throughput-based partitions,
  Consumer parallelism,
  Desired max partition size
)

Throughput-Based Calculation

Formula:

Partitions = Target Throughput / Single-Partition Throughput

Example:
  Target: 100 MB/sec
  Single partition limit: 10 MB/sec
  Partitions = 100 / 10 = 10

Single-Partition Limits:

  • Producers: ~10-20 MB/sec per partition
  • Consumers: ~10-20 MB/sec per partition
  • Replication: ~10 MB/sec per partition

Consumer Parallelism

Rule: Partitions ≥ Max consumer instances (for full parallelism)

Example:

Consumer group needs 20 instances for processing capacity
→ Topic needs ≥ 20 partitions

Why: Each consumer instance handles ≥1 partition. If partitions < instances, some instances idle.

Partition Size Limit

Best Practice: Keep partitions < 25 GB (uncompressed)

Why:

  • Faster recovery (leader election)
  • Faster compaction
  • Lower memory overhead

Formula:

Max Retention Size Per Partition = 25 GB
Partitions = Total Data Size / 25 GB

Example:
  Topic generates 500 GB/day, retention = 7 days
  Total = 3.5 TB
  Partitions = 3500 GB / 25 GB = 140

Partition Count Examples

Low-Throughput Topic

Use Case: Configuration updates (1 message/minute)

Calculation:

  • Throughput: < 1 KB/sec
  • Consumer parallelism: 1-2 instances

Result: 1-2 partitions

Medium-Throughput Topic

Use Case: Order events (1000 orders/sec, 10 KB each)

Calculation:

  • Throughput: 10 MB/sec
  • Single partition limit: 10 MB/sec
  • Consumer parallelism: 10 instances

Result: 10 partitions

High-Throughput Topic

Use Case: IoT telemetry (100k events/sec, 1 KB each)

Calculation:

  • Throughput: 100 MB/sec
  • Single partition limit: 10 MB/sec
  • Consumer parallelism: 50 instances

Result: 50 partitions

Partition Rebalancing

When Needed

Triggers:

  • New brokers added
  • Brokers removed/failed
  • Hot partitions (uneven load)
  • Leader imbalance (all leaders on few brokers)

Manual Reassignment

Tool: kafka-reassign-partitions.sh

Step 1: Generate Plan

# Move partitions for topic to specific brokers
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics-to-move.json \
  --broker-list "1,2,3" \
  --generate

topics-to-move.json:

{
  "topics": [{"topic": "orders"}],
  "version": 1
}

Output: Current assignment + Proposed assignment

Step 2: Execute Reassignment

kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

Step 3: Verify

kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Cruise Control (Automated)

What It Is: LinkedIn's automated partition rebalancer.

Features:

  • Auto-detects imbalances
  • Generates rebalance proposals
  • Executes with throttling
  • Monitors cluster health

Architecture:

Cruise Control
  ├── Monitors broker metrics
  ├── Detects imbalances (CPU, disk, network)
  ├── Generates rebalance plan
  └── Executes via Kafka Admin API

Use Cases:

  • Auto-scale clusters (add/remove brokers)
  • Balance partition leaders
  • Fix hot partitions

Limitations:

  • Requires separate deployment
  • Learning curve (complex config)

Partition Increase Considerations

Can Partitions Be Decreased?

Answer: ❌ No – Kafka does not support decreasing partition count.

Workaround: Create new topic with fewer partitions, migrate data.

Risks of Increasing Partitions

1. Key-to-Partition Mapping Changes

Before (3 partitions):
  key="order-123" → hash % 3 = partition 0

After (6 partitions):
  key="order-123" → hash % 6 = partition 3

Result: Same key now in different partition (order lost)

Mitigation: Don't increase partitions if ordering by key is critical.

2. Custom Offsets/State Stores

  • Consumers using external offsets (e.g., DB) must handle new partitions
  • Kafka Streams state stores must be rebuilt

Leader Rebalancing

Preferred Leader Election

Problem: Leaders concentrated on few brokers (after failures/restarts).

Solution: Trigger preferred leader election.

kafka-leader-election.sh \
  --bootstrap-server localhost:9092 \
  --election-type preferred \
  --all-topic-partitions

What It Does: Moves leadership back to "preferred replica" (first in ISR list).

When to Run: After broker restarts, failovers.

Monitoring Partition Distribution

Key Metrics

1. Partitions Per Broker

kafka-topics.sh --describe --bootstrap-server localhost:9092 \
  | grep "Leader:" | awk '{print $6}' | sort | uniq -c

Healthy: Even distribution (±10%)

2. Leader Distribution

kafka-topics.sh --describe --bootstrap-server localhost:9092 \
  | grep "Leader:" | awk '{print $6}' | sort | uniq -c

Healthy: Leaders evenly distributed

3. Under-Replicated Partitions

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

Healthy: 0

Key Takeaways

  1. Partition Formula: max(throughput/10 MB/sec, consumer parallelism, data size/25 GB).
  2. Throughput Limit: ~10-20 MB/sec per partition.
  3. Partition Size: Keep < 25 GB for faster recovery.
  4. Cannot Decrease: Kafka doesn't support reducing partition count.
  5. Increasing Risks: Key-to-partition mapping changes (order lost).
  6. Rebalancing: Manual (kafka-reassign-partitions.sh) or automated (Cruise Control).
  7. Leader Rebalancing: Use preferred leader election after failures.

What's Next?

Next Module: 10.4.3 – Quota and Tenant Governance

References:

  • Topic design → M2: Brokers & Topics
  • Consumer parallelism → M4: Consumers