4.1 Partitioning Strategy & Message Ordering

Module Overview

Goal: Make partitioning a design decision, not an accident. Understand ordering guarantees and their limits.

What You'll Learn:

• How default partitioning works
• Key-based partitioning and guarantees
• Ordering guarantees (and limitations)
• Custom partitioners and when to use them
• When NOT to use keys
• Consequences of repartitioning

What's NOT Covered Here:

• Performance tuning (covered in Module 8)
• Broker-side partition management (covered elsewhere)
• Consumer partition assignment (consumer module)

1. Partitioning Fundamentals

What is Partitioning?

Partitioning is the process of determining which partition a record should be sent to.

Topic: "orders" (3 partitions)

Producer sends record → Which partition?

Partition 0: [order-1][order-4][order-7]...
Partition 1: [order-2][order-5][order-8]...
Partition 2: [order-3][order-6][order-9]...

Why Partitioning Matters

1. Parallelism: Multiple consumers can read different partitions
2. Ordering: Records in same partition are ordered
3. Scalability: Distribute load across brokers
4. Fault Tolerance: Each partition can be replicated

2. The Partitioning Decision Tree

┌────────────────────────────────────────────┐
│     How is partition determined?          │
├────────────────────────────────────────────┤
│                                            │
│  1. Partition number explicitly set?      │
│     ├─ YES → Use that partition           │
│     └─ NO ↓                               │
│                                            │
│  2. Key provided?                         │
│     ├─ YES → hash(key) % numPartitions   │
│     └─ NO ↓                               │
│                                            │
│  3. Use default partitioner               │
│     └─ Sticky/Round-robin                │
│                                            │
└────────────────────────────────────────────┘

3. Explicit Partition Assignment

Manual Partition Selection

// Explicitly send to partition 2
ProducerRecord<String, String> record = 
    new ProducerRecord<>(
        "my-topic",    // topic
        2,             // partition (explicit)
        "key",         // key (ignored for partitioning)
        "value"        // value
    );

producer.send(record);

When to Use:

• Rare use case (usually let partitioner decide)
• Special routing requirements
• Testing specific scenarios

⚠️ Warning: You're responsible for:

• Knowing partition count
• Handling partition changes (if topic scales)
• Load balancing

4. Key-Based Partitioning

How It Works

When a key is provided, Kafka uses a hash function to determine the partition:

partition = hash(key) % numberOfPartitions

Key Property: Same key always goes to same partition (given same partition count).

Example: Key-Based Partitioning

// All records with key "user-123" go to same partition
ProducerRecord<String, String> record1 = 
    new ProducerRecord<>("user-events", "user-123", "login");

ProducerRecord<String, String> record2 = 
    new ProducerRecord<>("user-events", "user-123", "logout");

producer.send(record1); // → Partition X
producer.send(record2); // → Partition X (same as record1)

Default Hash Function

Kafka uses MurmurHash2 by default:

public class DefaultPartitioner {
    public int partition(String topic, Object key, byte[] keyBytes,
                        Object value, byte[] valueBytes, Cluster cluster) {
        
        if (keyBytes == null) {
            // No key: sticky or round-robin
            return stickyPartitionCache.partition(topic, cluster);
        }
        
        // Key provided: hash it
        int numPartitions = cluster.partitionCountForTopic(topic);
        return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    }
}

Key Points:

• Hash is computed on serialized key bytes
• Partition assignment is deterministic
• Uses modulo to map to partition number

Practical Example: User Events

public void sendUserEvent(long userId, String eventType) {
    String key = String.valueOf(userId); // Key: user ID
    
    ProducerRecord<String, String> record = 
        new ProducerRecord<>("user-events", key, eventType);
    
    producer.send(record);
}

// Usage
sendUserEvent(123L, "login");      // → Partition 2 (example)
sendUserEvent(123L, "view_page");  // → Partition 2 (same)
sendUserEvent(456L, "login");      // → Partition 0 (different key)
sendUserEvent(123L, "logout");     // → Partition 2 (same key)

Result: All events for user 123 are in partition 2, in order.

5. Null Key (No Key)

Round-Robin and Sticky Partitioning

When no key is provided, Kafka uses one of two strategies:

Kafka < 2.4: Round-Robin

ProducerRecord<String, String> record = 
    new ProducerRecord<>("topic", null, "value"); // null key

// Distribution pattern:
// Record 1 → Partition 0
// Record 2 → Partition 1
// Record 3 → Partition 2
// Record 4 → Partition 0
// ...

Behavior: Cycles through partitions.

Kafka ≥ 2.4: Sticky Partitioning (Default)

ProducerRecord<String, String> record = 
    new ProducerRecord<>("topic", null, "value"); // null key

// Distribution pattern:
// Records 1-100 → Partition 0 (until batch full)
// Records 101-200 → Partition 1 (until batch full)
// Records 201-300 → Partition 2 (until batch full)
// ...

Behavior: Sticks to one partition until batch is full, then switches.

Why Better?

• Fewer, larger batches
• Better throughput
• Reduced latency

Deep Dive: Sticky Partitioner

Introduced in Kafka 2.4 to improve performance for records with null keys.

Problem with Round-Robin (Kafka < 2.4)

Time  Record  Partition  Batch State
────┼───────┼──────────┼─────────────────
  1 │ msg1  │    0     │ Batch 0: [msg1]
  2 │ msg2  │    1     │ Batch 1: [msg2]
  3 │ msg3  │    2     │ Batch 2: [msg3]
  4 │ msg4  │    0     │ Batch 0: [msg1, msg4]
  5 │ msg5  │    1     │ Batch 1: [msg2, msg5]
  6 │ msg6  │    2     │ Batch 2: [msg3, msg6]

Result: 6 messages → 3 small batches (2 msgs each)

Problem: Every record goes to a different partition, creating many small batches.

Solution: Sticky Partitioner (Kafka ≥ 2.4)

Time  Record  Partition  Batch State
────┼───────┼──────────┼─────────────────────────────
  1 │ msg1  │    0     │ Batch 0: [msg1]
  2 │ msg2  │    0     │ Batch 0: [msg1, msg2]
  3 │ msg3  │    0     │ Batch 0: [msg1, msg2, msg3]
  4 │ msg4  │    0     │ Batch 0: [msg1, msg2, msg3, msg4]
  5 │ msg5  │    0     │ Batch 0: [msg1, msg2, msg3, msg4, msg5]
  6 │ msg6  │    0     │ Batch 0: [msg1, msg2, msg3, msg4, msg5, msg6]
  
  (Batch 0 full, send it, switch partition)
  
  7 │ msg7  │    1     │ Batch 1: [msg7]
  8 │ msg8  │    1     │ Batch 1: [msg7, msg8]
  ...

Result: 6 messages → 1 large batch (6 msgs)

Benefit: All records go to same partition until batch is full, creating fewer, larger batches.

Performance Impact

Benchmark: 1 million records, null keys

Sticky partitioner is 4x faster with 3x lower latency!

When Sticky Partitioner Switches

The sticky partitioner switches to a new partition when:

1. Current batch is full (batch.size reached)
2. Current batch is sent (due to linger.ms timeout)
3. Partition becomes unavailable (leader failure)

// Sticky partitioner in action
producer.send(new ProducerRecord<>("topic", null, "msg1")); // → Partition 0
producer.send(new ProducerRecord<>("topic", null, "msg2")); // → Partition 0
producer.send(new ProducerRecord<>("topic", null, "msg3")); // → Partition 0
// ... (batch fills up)
producer.send(new ProducerRecord<>("topic", null, "msg100")); // → Partition 0

// Batch full, sent, switch partition
producer.send(new ProducerRecord<>("topic", null, "msg101")); // → Partition 1
producer.send(new ProducerRecord<>("topic", null, "msg102")); // → Partition 1

Configuration

Sticky partitioner is the default in Kafka 2.4+. No configuration needed.

To explicitly use it:

props.put("partitioner.class", 
    "org.apache.kafka.clients.producer.internals.DefaultPartitioner");

To revert to round-robin (not recommended):

props.put("partitioner.class", 
    "org.apache.kafka.clients.producer.RoundRobinPartitioner");

Recommendation: Use the default (sticky) unless you have a specific reason not to.

Important: Sticky Only Applies to Null Keys

// ✅ Sticky partitioner applies (null key)
producer.send(new ProducerRecord<>("topic", null, "value"));

// ❌ Sticky partitioner does NOT apply (key provided)
producer.send(new ProducerRecord<>("topic", "key", "value"));
// → Uses key-based hashing instead

Key-based partitioning always uses hash(key) % numPartitions, regardless of sticky partitioner.

6. Ordering Guarantees

The Golden Rule

Kafka guarantees ordering within a partition only.

Partition 0: [A][B][C][D] ← Order guaranteed
Partition 1: [E][F][G][H] ← Order guaranteed
Partition 2: [I][J][K][L] ← Order guaranteed

Cross-partition: No ordering guarantee between A and E

Scenario 1: Same Key (Ordered)

producer.send(new ProducerRecord<>("orders", "user-123", "order-1"));
producer.send(new ProducerRecord<>("orders", "user-123", "order-2"));
producer.send(new ProducerRecord<>("orders", "user-123", "order-3"));

// Result: All go to same partition → Ordered
Partition X: [order-1][order-2][order-3]

✅ Ordering preserved

Scenario 2: Different Keys (Not Ordered)

producer.send(new ProducerRecord<>("orders", "user-123", "order-1"));
producer.send(new ProducerRecord<>("orders", "user-456", "order-2"));
producer.send(new ProducerRecord<>("orders", "user-789", "order-3"));

// Result: Different partitions → No ordering
Partition 0: [order-1]
Partition 1: [order-2]
Partition 2: [order-3]

❌ No cross-partition ordering

Scenario 3: Null Keys (Not Ordered)

producer.send(new ProducerRecord<>("logs", null, "log-1"));
producer.send(new ProducerRecord<>("logs", null, "log-2"));
producer.send(new ProducerRecord<>("logs", null, "log-3"));

// Result: Sticky or round-robin → May be ordered within batch
Partition 0: [log-1][log-2]  (if batched together)
Partition 1: [log-3]

⚠️ Ordering not guaranteed (depends on batching)

Ordering with Retries

The Problem

props.put("max.in.flight.requests.per.connection", 5); // Multiple in-flight

// Send sequence
send(record-1); // Sent
send(record-2); // Sent
// record-1 fails, retries
// record-2 succeeds

// Result in Kafka: [record-2][record-1] ❌ OUT OF ORDER

Solution 1: Enable Idempotence

props.put("enable.idempotence", "true");

Effect:

• Producer tracks sequence numbers
• Broker reorders on arrival
• Ordering guaranteed even with retries
• Can use max.in.flight.requests.per.connection=5

✅ Recommended for production

Solution 2: Limit In-Flight Requests (Old Way)

props.put("max.in.flight.requests.per.connection", 1);

Effect:

• Only one request in flight at a time
• Guarantees ordering
• Low throughput (serializes requests)

❌ Not recommended (use idempotence instead)

7. Custom Partitioners

When to Create Custom Partitioners

✅ Good Reasons:

• Custom business logic (e.g., VIP users to specific partitions)
• Load balancing based on key distribution
• Geographic routing
• Special key formats

❌ Bad Reasons:

• "Just to be different"
• Optimization without measurement
• Over-engineering

Creating a Custom Partitioner

Example: VIP Partitioner

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;

public class VipPartitioner implements Partitioner {
    
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                        Object value, byte[] valueBytes, Cluster cluster) {
        
        int numPartitions = cluster.partitionCountForTopic(topic);
        
        if (key == null) {
            // No key: random
            return ThreadLocalRandom.current().nextInt(numPartitions);
        }
        
        String keyStr = (String) key;
        
        // VIP users go to partition 0 (dedicated partition)
        if (keyStr.startsWith("VIP-")) {
            return 0;
        }
        
        // Regular users: hash to remaining partitions
        int partition = (Utils.toPositive(Utils.murmur2(keyBytes)) % (numPartitions - 1)) + 1;
        return partition;
    }
    
    @Override
    public void close() {
        // Cleanup
    }
    
    @Override
    public void configure(Map<String, ?> configs) {
        // Initialize with configs
    }
}

Using Custom Partitioner

props.put("partitioner.class", "com.myapp.VipPartitioner");

// VIP user
producer.send(new ProducerRecord<>("orders", "VIP-123", "order-data"));
// → Partition 0

// Regular user
producer.send(new ProducerRecord<>("orders", "user-456", "order-data"));
// → Partition 1-N (hashed)

Custom Partitioner Best Practices

✅ Be Deterministic

// ✅ GOOD: Same key always produces same partition
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

// ❌ BAD: Non-deterministic
return ThreadLocalRandom.current().nextInt(numPartitions);

✅ Handle Null Keys

if (keyBytes == null) {
    return 0; // or round-robin, or sticky
}

✅ Handle Partition Count Changes

int numPartitions = cluster.partitionCountForTopic(topic);
// Use numPartitions, don't hardcode

✅ Keep It Fast

// ❌ BAD: Slow logic
public int partition(...) {
    // Call external service
    int partition = httpClient.get("/api/partition?key=" + key);
    return partition;
}

// ✅ GOOD: Pure computation
public int partition(...) {
    return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}

Why? Partitioner is called for every record.

8. When NOT to Use Keys

Use Null Keys When:

1. Order doesn't matter

   // Log entries (no ordering needed)
   producer.send(new ProducerRecord<>("logs", null, logEntry));
   

2. Maximum throughput needed

   // Metrics (distribute evenly)
   producer.send(new ProducerRecord<>("metrics", null, metricData));
   

3. Load balancing is more important than ordering

   // Analytics events (don't care about order)
   producer.send(new ProducerRecord<>("analytics", null, event));
   

Key Selection Anti-Patterns

❌ Anti-Pattern 1: High-Cardinality Keys

// ❌ BAD: UUID as key (every record to different partition)
String key = UUID.randomUUID().toString();
producer.send(new ProducerRecord<>("orders", key, orderData));

// Effect: Defeats purpose of keys, no ordering benefit

Fix: Use entity ID (user ID, order ID) as key.

❌ Anti-Pattern 2: Low-Cardinality Keys

// ❌ BAD: Status as key (only 2 values)
String key = order.getStatus(); // "PENDING" or "COMPLETED"
producer.send(new ProducerRecord<>("orders", key, orderData));

// Effect: Only 2 partitions used (uneven distribution)

Fix: Use high-cardinality ID as key.

❌ Anti-Pattern 3: Timestamp as Key

// ❌ BAD: Timestamp as key
String key = String.valueOf(System.currentTimeMillis());
producer.send(new ProducerRecord<>("events", key, eventData));

// Effect: All recent events to same partition (hot partition)

Fix: Use entity ID or null key.

9. Partition Count and Repartitioning

Choosing Initial Partition Count

Factors to Consider:

• Throughput requirements
• Consumer parallelism needs
• Broker capacity
• Future growth

Rule of Thumb:

Target partitions = (Target throughput) / (Single consumer throughput)

Example:

• Target: 100 MB/s
• Consumer throughput: 10 MB/s
• Partitions needed: 10

Start conservatively, can add partitions later.

Adding Partitions (Repartitioning)

kafka-topics.sh --alter --topic orders --partitions 6

Effect on Existing Data:

• Old data stays in original partitions
• New data may hash to different partitions

The Repartitioning Problem

// Before: 3 partitions
hash("user-123") % 3 = Partition 1

// After: 6 partitions
hash("user-123") % 6 = Partition 5  // Different!

Impact:

• Keys rehash to different partitions
• Breaks ordering guarantee across partition change
• Historical data in old partition, new data in new partition

Handling Repartitioning

Strategy 1: Accept the Break

// New records go to new partition
// Old records stay in old partition
// Ordering broken at partition change point

Use When: Ordering only needed for recent data.

Strategy 2: Custom Partitioner with Stable Mapping

public class StablePartitioner implements Partitioner {
    private final int originalPartitionCount = 3;
    
    @Override
    public int partition(...) {
        // Always hash against original count
        int partition = Utils.toPositive(Utils.murmur2(keyBytes)) % originalPartitionCount;
        return partition;
    }
}

Effect: Only use subset of partitions (not ideal).

Strategy 3: Reprocess from Beginning

// 1. Stop producers
// 2. Add partitions
// 3. Reprocess all data from offset 0
// 4. Restart producers

Use When: Must maintain ordering, can reprocess.

10. Ordering Checklist

To Guarantee Ordering:

• Use keys for entities that need ordering
• Enable idempotence (enable.idempotence=true)
• Same key always for related records
• Don't change partition count for topics requiring ordering
• Handle producer retries (idempotence handles this)
• Single producer instance per key (or use transactions)

When Ordering Breaks:

❌ Different keys → different partitions
❌ Partition count changes → keys rehash
❌ Multiple producers without transactions → race conditions
❌ Retries without idempotence (old Kafka)
❌ Consumer rebalancing (not a producer issue, but affects end-to-end ordering)

11. Practical Examples

Example 1: E-Commerce Orders

public class OrderProducer {
    
    public void sendOrder(Order order) {
        // Key: Order ID (all events for same order in same partition)
        String key = order.getOrderId();
        String value = serializeOrder(order);
        
        ProducerRecord<String, String> record = 
            new ProducerRecord<>("orders", key, value);
        
        producer.send(record, (metadata, exception) -> {
            if (exception == null) {
                logger.info("Order {} sent to partition {}", 
                    order.getOrderId(), metadata.partition());
            } else {
                logger.error("Failed to send order {}", order.getOrderId(), exception);
            }
        });
    }
}

Guarantee: All events for order-123 are ordered.

Example 2: User Activity Tracking

public class ActivityTracker {
    
    public void trackActivity(long userId, String activity) {
        // Key: User ID (all activities for same user in same partition)
        String key = String.valueOf(userId);
        
        ProducerRecord<String, String> record = 
            new ProducerRecord<>("user-activities", key, activity);
        
        producer.send(record);
    }
}

// Usage
tracker.trackActivity(123L, "login");
tracker.trackActivity(123L, "view_product");
tracker.trackActivity(123L, "add_to_cart");
tracker.trackActivity(123L, "checkout");

// Result: All activities for user 123 in order in same partition

Example 3: Log Aggregation (No Ordering)

public class LogProducer {
    
    public void sendLog(String logLevel, String message) {
        // No key: distribute evenly for throughput
        ProducerRecord<String, String> record = 
            new ProducerRecord<>("logs", null, message);
        
        producer.send(record);
    }
}

// Result: Logs distributed across partitions for maximum throughput

Summary

Key Takeaways

✅ Partitioning determines which partition a record goes to
✅ Same key → same partition (ordering guaranteed)
✅ Null key → sticky or round-robin (no ordering)
✅ Ordering guaranteed within partition only
✅ Enable idempotence for ordering with retries
✅ Custom partitioners for special routing logic
✅ Repartitioning breaks ordering (keys rehash)
✅ Choose keys based on entity ID, not random values

Partitioning Decision Guide

Need ordering? 
  ├─ YES → Use entity ID as key
  └─ NO → Use null key for max throughput

Need custom routing?
  ├─ YES → Custom partitioner
  └─ NO → Default partitioner

Multiple partition count changes expected?
  ├─ YES → Avoid keys (or plan reprocessing)
  └─ NO → Use keys freely

Ordering Requirements Checklist

What's Next

• Module 5: Reliability, Delivery Guarantees & Semantics
  Deep dive into acks, idempotence, exactly-once

• Module 6.1: Error Handling & Failure Scenarios
  Handle partition failures and errors

Previous Module: ← 3.1 - Serialization & Data Modeling
Next Module: 5.1 - Delivery Semantics Explained →