2.1 – Topics & Partitions Explained

What is a Kafka Topic?

A topic is a logical category or feed name to which messages are published.

Analogy:

• A topic is like a table in a database
• A topic is like a channel in a messaging system
• A topic is like a directory in a file system

Key Characteristics:

• Topics are named (e.g., orders, user-clicks, payments)
• Topics are append-only (messages are added, never updated in-place)
• Topics are durable (messages are stored on disk for a configured retention period)

Topics as Logical Containers

Important: A topic is NOT a single file or queue. A topic is a logical grouping of one or more partitions.

Topic: "orders"
   ├── Partition 0
   ├── Partition 1
   ├── Partition 2
   └── Partition 3

Key Point: The topic name is what producers and consumers use, but the actual data storage happens at the partition level.

What is a Partition?

A partition is a single, ordered, immutable sequence of messages (an append-only log).

Key Characteristics:

1. Ordered: Messages within a partition have a total order (defined by offset)
2. Immutable: Once written, messages cannot be changed
3. Append-Only: New messages are always added to the end
4. Parallelism Unit: Partitions enable parallel processing (one consumer per partition)

Partitions as Parallel Logs

Each partition is an independent log:

Topic: "orders" (3 partitions)

Partition 0:
┌───────┬───────┬───────┬───────┬───────┐
│ msg 0 │ msg 1 │ msg 2 │ msg 3 │ msg 4 │
└───────┴───────┴───────┴───────┴───────┘
Offsets:  0       1       2       3       4

Partition 1:
┌───────┬───────┬───────┬───────┐
│ msg 0 │ msg 1 │ msg 2 │ msg 3 │
└───────┴───────┴───────┴───────┘
Offsets:  0       1       2       3

Partition 2:
┌───────┬───────┬───────┬───────┬───────┬───────┐
│ msg 0 │ msg 1 │ msg 2 │ msg 3 │ msg 4 │ msg 5 │
└───────┴───────┴───────┴───────┴───────┴───────┘
Offsets:  0       1       2       3       4       5

Key Points:

• Each partition has its own offset sequence (starting from 0)
• Partitions can have different message counts (no requirement to be balanced)
• Messages in different partitions are independent (no ordering guarantee across partitions)

Ordering Guarantees (Critical Understanding)

1. Within a Partition: Total Order

Messages sent to the same partition are guaranteed to be ordered.

Example:

producer.send(new ProducerRecord<>("orders", "key-A", "msg-1"));
producer.send(new ProducerRecord<>("orders", "key-A", "msg-2"));
producer.send(new ProducerRecord<>("orders", "key-A", "msg-3"));

If all three messages go to partition 0:

• They will be stored in the order: msg-1, msg-2, msg-3
• A consumer reading partition 0 will see them in that exact order

2. Across Partitions: No Order Guarantee

Messages in different partitions have no ordering relationship.

Example:

Partition 0: msg-A (timestamp: 10:00:01)
Partition 1: msg-B (timestamp: 10:00:00)
Partition 2: msg-C (timestamp: 10:00:02)

If a consumer reads all three partitions in parallel:

• It might see: msg-B, msg-A, msg-C (not time-ordered)
• It might see: msg-A, msg-C, msg-B (not guaranteed)

Key Takeaway: If you need global ordering, use a single partition. If you need per-key ordering, use a keyed message (covered next).

Message Keys and Partition Assignment

Concept: Key → Partition Mapping

Kafka uses message keys to determine which partition a message goes to.

Two Scenarios:

1. Keyed Message (message has a key):

   • Kafka computes hash(key) % num_partitions to determine partition
   • All messages with the same key go to the same partition
   • Ordering is preserved per key

2. Non-Keyed Message (message key is null):

   • Kafka uses round-robin or sticky partitioning
   • No ordering guarantee

Example: Keyed Messages

Scenario: Track user actions per user.

Messages:

producer.send(new ProducerRecord<>("user-actions", "user-123", "login"));
producer.send(new ProducerRecord<>("user-actions", "user-456", "click"));
producer.send(new ProducerRecord<>("user-actions", "user-123", "purchase"));

Result (assuming 3 partitions):

hash("user-123") % 3 = 1  →  Partition 1: [login, purchase]
hash("user-456") % 3 = 0  →  Partition 0: [click]

Key Point: All actions for user-123 are in partition 1, so they are ordered. Actions for different users are in different partitions, so they can be processed in parallel.

Example: Non-Keyed Messages

Messages:

producer.send(new ProducerRecord<>("logs", null, "log-entry-1"));
producer.send(new ProducerRecord<>("logs", null, "log-entry-2"));
producer.send(new ProducerRecord<>("logs", null, "log-entry-3"));

Result (round-robin across 3 partitions):

Partition 0: log-entry-1
Partition 1: log-entry-2
Partition 2: log-entry-3

Key Point: No key = no ordering guarantee. Use this for high throughput when order doesn't matter.

Important: Partitioner is a Producer-Side Concept

Note: The logic that determines which partition a message goes to is called the partitioner. This is a producer-side mechanism.

Producer partitioner configs and custom partitioners are covered in M3: Kafka Producers.

What You Need to Know Here:

• Messages with the same key go to the same partition (by default)
• This enables per-key ordering

Partition Count Implications

The number of partitions in a topic has major implications for parallelism, throughput, and ordering.

1. Parallelism = Partition Count

Rule: You can have at most N consumers in a consumer group reading a topic with N partitions.

Example:

• Topic orders has 4 partitions
• Consumer group order-processors has 4 consumers
• Result: Each consumer reads 1 partition (maximum parallelism)

If you add a 5th consumer:

• It will be idle (no partition to assign)

Key Point: Partition count is the upper bound on consumer parallelism.

2. Throughput = Sum of Partition Throughput

Rule: Each partition can handle ~N MB/sec (limited by disk I/O, network, and CPU).

Example:

• One partition can handle ~10 MB/sec
• Topic with 10 partitions can handle ~100 MB/sec

Key Point: More partitions = higher throughput (up to broker capacity limits).

3. Ordering vs Parallelism Trade-off

Key Takeaway: You cannot have both global ordering and high parallelism. You must choose.

4. Repartitioning is Expensive

Problem: Changing the partition count breaks key-to-partition assignment.

Example:

Original: 3 partitions
hash("user-123") % 3 = 1  →  Partition 1

After adding partition (4 partitions):
hash("user-123") % 4 = 3  →  Partition 3

Impact:

• Existing data stays in old partition (partition 1)
• New data goes to new partition (partition 3)
• Per-key ordering is broken (old messages in p1, new messages in p3)

Key Point: Always plan partition count upfront. Changing it later requires full topic rebalancing or data migration.

Partition Count Design Guidelines

How Many Partitions Should You Create?

General Formula:

Partitions = max(target_throughput / partition_throughput, max_consumers)

Example 1: Throughput-Driven:

• Target throughput: 100 MB/sec
• Per-partition throughput: 10 MB/sec
• Partitions needed: 100 / 10 = 10 partitions

Example 2: Consumer-Driven:

• You need 20 parallel consumers for fast processing
• Partitions needed: 20 partitions

Over-Partitioning vs Under-Partitioning

Best Practice: Start with moderate partition count (e.g., 10-30 partitions) and scale up if needed.

Detailed partition sizing strategy is covered in Module 5.1: Designing Topics & Partition Strategy.

Topic and Partition Naming

Topic Naming Conventions

Best Practices:

• Use lowercase letters, numbers, hyphens, underscores (avoid special characters)
• Use descriptive names: user-events, order-updates, payment-notifications
• Use namespaces for multi-tenancy: team-a.orders, team-b.orders

Avoid:

• Dots (.) in topic names if using Kafka Streams (Streams uses dots for internal topics)
• Very long names (> 249 characters, Kafka's limit)

Partition Naming (Internal)

On disk, partitions are stored as directories:

/var/lib/kafka-logs/
  orders-0/
  orders-1/
  orders-2/

Format: <topic-name>-<partition-number>

Key Point: You don't manually name partitions. Kafka assigns partition numbers automatically (0, 1, 2, ...).

Replication and Partitions

Each partition can have multiple replicas (copies of the data).

Example:

Topic: "orders"
Partitions: 3
Replication Factor: 2

Partition 0:
  - Leader: Broker 1
  - Follower: Broker 2

Partition 1:
  - Leader: Broker 2
  - Follower: Broker 3

Partition 2:
  - Leader: Broker 3
  - Follower: Broker 1

Key Points:

• Replication factor determines how many copies of each partition exist
• Leader replica handles all reads and writes
• Follower replicas replicate data from the leader (for fault tolerance)

Deep replication mechanics are covered in Module 3.1: Replication & Fault Tolerance.

Topic Metadata

When you create a topic, Kafka stores metadata about it:

Clients fetch metadata to know:

• Which broker hosts the leader for partition X?
• How many partitions does the topic have?

Metadata Flow:

Producer                    Any Broker
   │                           │
   ├──Metadata Request─────────▶│
   │  (topic: orders)           │
   │                            │
   │◀──Metadata Response─────────┤
   │  {                          │
   │    partitions: 3,           │
   │    p0: leader=broker-1,     │
   │    p1: leader=broker-2,     │
   │    p2: leader=broker-3      │
   │  }                          │

Key Point: Clients cache metadata locally and refresh it periodically (or when they get errors indicating metadata is stale).

Key Takeaways

1. Topic = Logical Category: A topic is a named feed (e.g., orders). It's made up of one or more partitions.

2. Partition = Ordered Log: Each partition is an independent, append-only, ordered log of messages.

3. Ordering Guarantee: Messages within a partition are totally ordered. Messages across partitions are not ordered.

4. Message Keys: Messages with the same key go to the same partition, enabling per-key ordering.

5. Partition Count = Parallelism Limit: You can have at most N consumers reading a topic with N partitions.

6. Repartitioning is Hard: Changing partition count breaks key-to-partition mapping. Plan partition count upfront.

7. Replication Factor: Each partition can have multiple replicas for fault tolerance (one leader, N followers).

What's Next?

Next Module: 2.2 – Log Structure & Message Storage

What We Covered:

• ✅ Topics as logical containers
• ✅ Partitions as parallel, ordered logs
• ✅ Ordering guarantees (within partition vs across partitions)
• ✅ Message keys and partition assignment
• ✅ Partition count implications (parallelism, throughput, ordering)
• ✅ Topic metadata

What We Didn't Cover (Explicitly Deferred):

• ❌ Producer partitioner configs → M3: Kafka Producers
• ❌ Consumer group partition assignment → M4: Kafka Consumers
• ❌ Deep replication protocol → Module 3.1 & M7: Internals
• ❌ Partition sizing strategy → Module 5.1

Topics & Partitions: Complete ✅