5.1 – Designing Topics & Partition Strategy

Why Topic Design Matters

Poor topic design leads to:

• ❌ Throughput bottlenecks (too few partitions)
• ❌ Operational complexity (too many partitions)
• ❌ Ordering violations (wrong partition key strategy)
• ❌ Consumer imbalance (uneven partition sizes)
• ❌ Expensive repartitioning (changing partition count breaks key mapping)

Good topic design ensures:

• ✅ Scalable throughput (right partition count)
• ✅ Parallelism (consumers can scale horizontally)
• ✅ Ordering guarantees (per-key ordering where needed)
• ✅ Operational simplicity (stable partition count)

Topic Design Checklist

Before creating a topic, answer these questions:

1. Data Model: Events or state? (Determines cleanup policy)
2. Throughput: How much data per second?
3. Parallelism: How many consumers will read concurrently?
4. Ordering: Do you need global order, per-key order, or no order?
5. Retention: How long should data be retained?
6. Growth: Will throughput grow 2x, 10x, 100x in the next year?

Partition Count: How Many?

The Fundamental Trade-off

Key Point: Partition count is the upper bound on consumer parallelism. You can have at most N consumers in a consumer group for a topic with N partitions.

Formula 1: Throughput-Based Sizing

Goal: Ensure each partition can handle the write/read load.

Formula:

partitions = max(
  target_write_throughput / partition_write_throughput,
  target_read_throughput / partition_read_throughput
)

Example:

Target Write Throughput: 100 MB/sec
Per-Partition Write Throughput: 10 MB/sec (broker disk limit)

Partitions Needed (Write): 100 / 10 = 10 partitions

Target Read Throughput: 200 MB/sec (2 consumer groups reading simultaneously)
Per-Partition Read Throughput: 10 MB/sec

Partitions Needed (Read): 200 / 10 = 20 partitions

Result: max(10, 20) = 20 partitions

Typical Per-Partition Throughput (benchmarks):

• Write: 10-50 MB/sec (limited by broker disk)
• Read: 10-50 MB/sec (limited by network)

Formula 2: Consumer Parallelism-Based Sizing

Goal: Allow N consumers to read in parallel.

Formula:

partitions = max_concurrent_consumers

Example:

You need 20 consumers to process messages fast enough.

Result: 20 partitions (one consumer per partition)

Key Point: If you create 10 partitions and have 20 consumers, 10 consumers will be idle (no partition to assign).

Combined Formula

Formula:

partitions = max(
  target_write_throughput / partition_write_throughput,
  target_read_throughput / partition_read_throughput,
  max_concurrent_consumers
)

Example:

Target Write: 50 MB/sec → 50/10 = 5 partitions
Target Read: 100 MB/sec → 100/10 = 10 partitions
Max Consumers: 15 → 15 partitions

Result: max(5, 10, 15) = 15 partitions

Rule of Thumb: Start with 10-30 Partitions

For Most Use Cases:

• Start with 10-30 partitions
• This supports:
  • 10-30 consumers (good parallelism)
  • 100-500 MB/sec throughput (enough for most apps)
  • Room for growth (2-3x traffic increase)

When to Use More:

• High throughput (> 500 MB/sec) → 50-100 partitions
• Many consumer groups (> 10 groups) → 50-100 partitions

When to Use Fewer:

• Low throughput (< 10 MB/sec) → 3-5 partitions
• Need global ordering → 1 partition (last resort)

Partition Count: Upper Limits

Kafka Cluster Limits

Practical Limits:

• Per Broker: 2,000-4,000 partitions (including replicas)
• Per Cluster: 200,000 partitions (across all brokers)

Why Limits?

1. Metadata Size: Each partition has metadata (leader, ISR, configs). Too many partitions = large metadata.
2. File Handles: Each partition = 2-3 open files (log, index, timeindex). Too many partitions = too many file handles.
3. Leader Election Time: More partitions = longer failover (controller must elect leaders for all partitions).

Example:

Cluster: 10 brokers
Partitions per Broker: 2,000
Total Partitions: 10 brokers × 2,000 = 20,000 partitions

Replication Factor: 3
Total Partition Replicas: 20,000 × 3 = 60,000 replicas

Warning: Over-Partitioning

Symptoms:

• Slow leader elections (> 30 seconds)
• High metadata transfer time (clients take long to fetch metadata)
• Broker CPU usage high (managing many partitions)

Solution: Consolidate topics (fewer partitions per topic).

Partition Key Strategy

Partition key determines which partition a message goes to.

Three Strategies:

1. Keyed (message has a key) → hash(key) % num_partitions
2. Non-Keyed (key is null) → Round-robin or sticky partitioning
3. Custom (custom partitioner logic) → Your own function

Strategy 1: Keyed Partitioning (Per-Key Ordering)

Use Case: You need ordering per entity (per user, per device, per session).

Example:

ProducerRecord<String, String> record = 
  new ProducerRecord<>("user-events", "user-123", "login");

Result: All messages with key user-123 go to the same partition → ordering is preserved.

Partition Assignment:

hash("user-123") % 10 = 3
hash("user-456") % 10 = 7
hash("user-789") % 10 = 1

Partition 1: user-789 events
Partition 3: user-123 events
Partition 7: user-456 events

Key Point: Ordering is per-key, not global. Different keys go to different partitions (can be processed in parallel).

Strategy 2: Non-Keyed Partitioning (No Ordering)

Use Case: Order doesn't matter, maximize throughput.

Example:

ProducerRecord<String, String> record = 
  new ProducerRecord<>("logs", null, "log message");

Result: Messages are distributed round-robin (or sticky, Kafka 2.4+) across all partitions.

Partition Assignment:

Message 1 → Partition 0
Message 2 → Partition 1
Message 3 → Partition 2
Message 4 → Partition 0 (wraps around)

Benefit: Maximum throughput (all partitions are used evenly).

Strategy 3: Custom Partitioner

Use Case: You have special partitioning logic (e.g., geographic partitioning, priority partitioning).

Example: Route US users to partitions 0-4, EU users to partitions 5-9.

public class GeoPartitioner implements Partitioner {
  @Override
  public int partition(String topic, Object key, byte[] keyBytes, 
                       Object value, byte[] valueBytes, Cluster cluster) {
    String keyStr = (String) key;
    if (keyStr.startsWith("US-")) {
      return hash(keyStr) % 5;  // Partitions 0-4
    } else if (keyStr.startsWith("EU-")) {
      return 5 + (hash(keyStr) % 5);  // Partitions 5-9
    }
    return hash(keyStr) % 10;  // Default
  }
}

Use Case: Geographically isolated processing (US consumers only read partitions 0-4).

Choosing the Right Partition Key

Example 1: User Events

Requirement: Preserve order of events per user.

Key: user_id

Result: All events for user-123 go to the same partition → ordering preserved.

Example 2: IoT Device Telemetry

Requirement: Process data from each device independently.

Key: device_id

Result: All telemetry from device-456 goes to the same partition → can track device state.

Example 3: Financial Transactions

Requirement: Process transactions per account in order.

Key: account_id

Result: All transactions for account-789 go to the same partition → balance updates are ordered.

Example 4: Logs (No Ordering Needed)

Requirement: High throughput, order doesn't matter.

Key: null (no key)

Result: Messages distributed evenly across all partitions → maximum throughput.

Partition Count Changes: Repartitioning Risks

Problem: Changing Partition Count Breaks Key Mapping

Original (3 partitions):

hash("user-123") % 3 = 1  →  Partition 1

After adding partition (4 partitions):

hash("user-123") % 4 = 3  →  Partition 3

Impact:

• Old data for user-123 is in Partition 1
• New data for user-123 goes to Partition 3
• Per-key ordering is broken (old messages in p1, new messages in p3)

Repartitioning Options

Option 1: Accept Broken Ordering

Use Case: You only need ordering for recent data (not historical).

Example: User session tracking (sessions are short-lived, old sessions don't matter).

Action: Just add partitions, accept that old data is in different partitions.

Option 2: Dual-Write to New Topic

Use Case: You need to preserve ordering for all data.

Steps:

1. Create new topic with desired partition count
2. Dual-write to both old and new topics
3. Migrate consumers to new topic
4. Stop writing to old topic
5. Delete old topic (after retention expires)

Downside: Operational complexity (need to manage two topics temporarily).

Option 3: Use Kafka Streams Repartition

Use Case: You're using Kafka Streams.

Kafka Streams automatically creates repartition topics when you repartition data.

stream
  .selectKey((key, value) -> value.getUserId())  // Change key
  .groupByKey()  // Triggers repartitioning
  .count();

Kafka Streams creates an internal repartition topic with the correct partition count.

Best Practice: Plan Partition Count Upfront

Rule: Always over-provision partitions (it's easier to have extra partitions than to add more later).

Example:

• Current throughput: 50 MB/sec → need 5 partitions
• Expected growth: 10x in 1 year → need 50 partitions
• Recommendation: Create 50 partitions today (even if only 5 are actively used)

Trade-off: Extra partitions have small overhead (metadata, file handles), but repartitioning has high operational cost.

⚠️ Critical Warning: Increasing Partitions Breaks Key-to-Partition Mapping

Important: When you increase partition count, existing keys will map to different partitions.

Why This Happens:

Kafka uses this formula to assign keys to partitions:

partition = hash(key) % num_partitions

Example:

Before (10 partitions):

key="user-123" → hash("user-123") = 54321
partition = 54321 % 10 = 1

After (20 partitions):

key="user-123" → hash("user-123") = 54321
partition = 54321 % 20 = 1  ← Same partition (lucky!)

But:

key="user-456" → hash("user-456") = 98765
Before: partition = 98765 % 10 = 5
After:  partition = 98765 % 20 = 5  ← Same partition (lucky!)

However:

key="user-789" → hash("user-789") = 12345
Before: partition = 12345 % 10 = 5
After:  partition = 12345 % 20 = 5  ← Same partition (lucky!)

Wait, that's not right. Let me recalculate:

key="order-123" → hash("order-123") = 67890
Before: partition = 67890 % 10 = 0
After:  partition = 67890 % 20 = 10  ← Different partition!

Result: After increasing partitions, order-123 now goes to partition 10 (was partition 0).

Impact on Consumers

Problem: If consumers rely on partition-level ordering, increasing partitions breaks this guarantee.

Example:

Before (10 partitions):

Partition 0:
  offset 0: {"orderId": "123", "status": "created"}
  offset 1: {"orderId": "123", "status": "paid"}
  offset 2: {"orderId": "123", "status": "shipped"}

After (20 partitions):

Partition 0:
  offset 0: {"orderId": "123", "status": "created"}
  offset 1: {"orderId": "123", "status": "paid"}
  offset 2: {"orderId": "123", "status": "shipped"}
  (no new messages for orderId=123)

Partition 10:
  offset 0: {"orderId": "123", "status": "delivered"}  ← New messages go here!

Result: Consumer reading partition 0 sees "shipped" as the final status, but "delivered" is in partition 10.

This breaks ordering guarantees for orderId=123.

Impact on Stateful Consumers (Kafka Streams)

Problem: Kafka Streams maintains local state stores keyed by partition.

Example:

Before (10 partitions):

Consumer instance 1 (partition 0):
  Local state store:
    orderId=123 → status="shipped"

After (20 partitions):

Consumer instance 1 (partition 0):
  Local state store:
    orderId=123 → status="shipped"  ← Stale!

Consumer instance 2 (partition 10):
  Local state store:
    orderId=123 → status="delivered"  ← New state!

Result: Two consumer instances have conflicting state for orderId=123.

Kafka Streams will NOT automatically migrate state when you increase partitions.

Impact on Compacted Topics

Problem: Log compaction relies on key-to-partition mapping to keep the latest value for each key.

Example:

Before (10 partitions):

Partition 0:
  offset 0: {"userId": "123", "name": "Alice"}
  offset 1: {"userId": "123", "name": "Alice Smith"}  ← Latest value

After (20 partitions):

Partition 0:
  offset 0: {"userId": "123", "name": "Alice"}
  offset 1: {"userId": "123", "name": "Alice Smith"}  ← Old latest value

Partition 10:
  offset 0: {"userId": "123", "name": "Alice Johnson"}  ← New latest value

Result: Compaction keeps both values (one in partition 0, one in partition 10).

Consumers reading the compacted topic will see duplicate keys (one from partition 0, one from partition 10).

When Increasing Partitions is Safe

Safe Scenarios:

1. No keys (all messages use key=null, random partition assignment)
2. No ordering requirements (consumers don't care about message order)
3. No stateful processing (no Kafka Streams state stores, no local caches)
4. Not a compacted topic (cleanup.policy=delete, not compact)

Example Safe Use Case:

Topic: application-logs
Key: null (random partitioning)
Use Case: Logs are independent (no ordering needed)

Increasing partitions: Safe (no impact on consumers)

When Increasing Partitions is Dangerous

Dangerous Scenarios:

1. Keyed messages (messages have keys, order matters)
2. Stateful consumers (Kafka Streams, local state stores)
3. Compacted topics (cleanup.policy=compact)
4. Idempotency/deduplication (consumers track processed keys)

Example Dangerous Use Case:

Topic: user-events
Key: userId (partition by user)
Use Case: Maintain user state (Kafka Streams)

Increasing partitions: Dangerous (breaks state stores, ordering)

Mitigation Strategies

Option 1: Accept Temporary Inconsistency

Approach: Increase partitions, accept that old keys may be split across partitions.

Steps:

1. Increase partition count
2. Wait for retention period to expire (old messages deleted)
3. After retention, all messages for a key are in the same partition again

Use Case: Short retention (e.g., 1 day). After 1 day, old messages are gone.

Option 2: Create a New Topic

Approach: Create a new topic with the desired partition count, migrate consumers.

Steps:

1. Create orders-v2 with 20 partitions
2. Dual-write to both orders and orders-v2
3. Migrate consumers to orders-v2
4. Stop writing to orders
5. Delete orders after retention period

Use Case: Critical topics where ordering/state must be preserved.

Option 3: Use a Custom Partitioner

Approach: Use a custom partitioner that remains consistent across partition count changes.

Example (consistent hashing):

public class ConsistentHashPartitioner implements Partitioner {
  @Override
  public int partition(String topic, Object key, byte[] keyBytes, 
                       Object value, byte[] valueBytes, Cluster cluster) {
    // Use consistent hashing (e.g., Murmur3 hash + consistent hash ring)
    // This minimizes key remapping when partition count changes
  }
}

Limitation: Requires custom code, not supported by all clients.

Recommendation: Plan partition count upfront to avoid increasing partitions later.

Hot Partitions (Partition Skew)

Problem: Uneven Key Distribution

Scenario: Some keys are "hot" (many messages), some keys are "cold" (few messages).

Example:

key="celebrity-123" → 1 million messages/day
key="user-456" → 10 messages/day

Both keys hash to different partitions, but:
Partition 3 (celebrity-123): 1 million messages/day (hot)
Partition 7 (user-456): 10 messages/day (cold)

Result: Partition 3 is overloaded (high disk usage, high CPU), Partition 7 is underutilized.

Solution 1: Add Random Suffix to Hot Keys

Strategy: Split hot keys across multiple partitions.

Example:

String key = "celebrity-123";
if (isHotKey(key)) {
  key = key + "-" + random.nextInt(10);  // Append random suffix (0-9)
}
producer.send(new ProducerRecord<>("topic", key, value));

Result: celebrity-123-0, celebrity-123-1, ... celebrity-123-9 go to different partitions.

Trade-off: Ordering is lost for hot keys (split across partitions).

Solution 2: Dedicated Topic for Hot Keys

Strategy: Route hot keys to a separate topic with more partitions.

Example:

Normal users → "user-events" (10 partitions)
Celebrities → "celebrity-events" (50 partitions)

Result: Hot keys have more partitions (better parallelism).

Solution 3: Custom Partitioner

Strategy: Manually assign hot keys to specific partitions.

Example:

if (key.equals("celebrity-123")) {
  return 0;  // Always partition 0
} else {
  return hash(key) % numPartitions;
}

Use Case: You know hot keys in advance and want to control their placement.

Topic Naming Conventions

Best Practices

1. Use lowercase (avoid uppercase: Orders vs orders)
2. Use hyphens for word separation: user-events, payment-transactions
3. Use namespaces for multi-tenancy: team-a.orders, team-b.orders
4. Avoid dots (if using Kafka Streams, which uses dots for internal topics)
5. Descriptive names: user-login-events, not topic1

Naming Patterns

Topic Design Anti-Patterns

Anti-Pattern 1: One Topic Per User

Example: user-123-events, user-456-events (thousands of topics)

Problem:

• Metadata explosion (one topic = one partition minimum)
• Consumer complexity (need to subscribe to thousands of topics)
• Operational nightmare (can't manage thousands of topics)

Solution: Use one topic with user_id as key → user-events (partitioned by user_id)

Anti-Pattern 2: Single Partition for Global Ordering

Example: Topic with 1 partition (to get global order)

Problem:

• Throughput bottleneck (single partition = single writer/reader)
• No parallelism (only 1 consumer can read)

Solution: Use per-key ordering (partition by entity ID) instead of global ordering.

Anti-Pattern 3: Too Many Partitions

Example: 1000 partitions for a 1 MB/sec topic

Problem:

• Metadata overhead (slow leader elections, slow metadata fetch)
• Wasted resources (most partitions are idle)

Solution: Start with 10-30 partitions, scale up only if needed.

Key Takeaways

1. Partition Count = Parallelism: You can have at most N consumers for a topic with N partitions.

2. Throughput Formula: partitions = target_throughput / partition_throughput (typically 10-50 MB/sec per partition).

3. Start with 10-30 Partitions: Good balance for most use cases. Easier to over-provision than to add partitions later.

4. Partition Key Strategy: Keyed (per-key ordering), non-keyed (maximum throughput), or custom (special logic).

5. Repartitioning is Hard: Changing partition count breaks key-to-partition mapping. Plan partition count upfront.

6. Hot Partitions: Uneven key distribution causes partition skew. Use random suffixes or custom partitioner to balance load.

7. Cluster Limits: ~2,000-4,000 partitions per broker, ~200,000 per cluster. Don't over-partition.

8. Topic Naming: Use lowercase, hyphens, namespaces. Avoid one-topic-per-entity anti-pattern.

What's Next?

Next Module: 6.1 – Broker-Level Configurations

What We Covered:

• ✅ Partition count sizing (throughput, parallelism)
• ✅ Partition key strategy (keyed, non-keyed, custom)
• ✅ Repartitioning risks and solutions
• ✅ Hot partitions and mitigation
• ✅ Topic naming conventions
• ✅ Anti-patterns (one topic per user, single partition, too many partitions)

What We Didn't Cover (Explicitly Deferred):

• ❌ Broker and topic configuration parameters → Module 6.x
• ❌ Performance tuning → M6: Operations

Designing Topics & Partition Strategy: Complete ✅