2.1 Topic & Partition Internals

Module Overview

Goal: Explain how partitions exist internally — the fundamental storage unit in Kafka.

What You'll Learn:

Topic to partition directory mapping
Partition as append-only log structure
Internal offset assignment mechanics
Ordering guarantees from log design
Partition state machine

What's NOT Covered Here:

Replication protocol (covered in Module 3.x)
ISR mechanics (covered in Module 3.2)
Log segments and indexes (covered in Module 2.2)
Compaction details (covered in Module 2.3)

Key Insight: A partition is not just a concept — it's a concrete append-only log on disk with specific internal mechanics.

1. Topics as Partition Collections

Logical vs Physical View

LOGICAL VIEW (Application):

Topic: "orders"
└─ Logical channel for order events

PHYSICAL VIEW (Broker):

Topic: "orders"
├─ Partition 0 (directory on disk)
├─ Partition 1 (directory on disk)
└─ Partition 2 (directory on disk)

Each partition is:
├─ Independent append-only log
├─ Separate directory on broker filesystem
└─ Unit of parallelism and ordering

Topic Metadata

TOPIC METADATA (stored in cluster state):

Topic: orders
├─ Partition count: 3
├─ Replication factor: 3
├─ Configuration:
│   ├─ retention.ms: 604800000 (7 days)
│   ├─ cleanup.policy: delete
│   └─ segment.bytes: 1073741824 (1 GB)
└─ Partition assignments:
    ├─ Partition 0: Replicas [1, 2, 3], Leader: 1
    ├─ Partition 1: Replicas [2, 3, 1], Leader: 2
    └─ Partition 2: Replicas [3, 1, 2], Leader: 3

2. Partition Directory Structure

Filesystem Layout

BROKER FILESYSTEM:

/var/kafka/data/
├─ orders-0/          # Partition 0 of topic "orders"
│   ├─ 00000000000000000000.log
│   ├─ 00000000000000000000.index
│   ├─ 00000000000000000000.timeindex
│   ├─ 00000000000012345678.log
│   ├─ 00000000000012345678.index
│   ├─ 00000000000012345678.timeindex
│   └─ leader-epoch-checkpoint
│
├─ orders-1/          # Partition 1 of topic "orders"
│   ├─ 00000000000000000000.log
│   ├─ 00000000000000000000.index
│   └─ ...
│
└─ orders-2/          # Partition 2 of topic "orders"
    ├─ 00000000000000000000.log
    ├─ 00000000000000000000.index
    └─ ...

Directory naming:
└─ {topic}-{partition}

File Types in Partition Directory

.log files:
└─ Actual message data (records)
└─ Segment of the append-only log

.index files:
└─ Offset index (offset → file position)
└─ Enables fast offset lookup

.timeindex files:
└─ Timestamp index (timestamp → offset)
└─ Enables time-based lookup

.txnindex files (if transactions enabled):
└─ Transaction index

leader-epoch-checkpoint:
└─ Tracks leader epochs
└─ Used during recovery and truncation

partition.metadata:
└─ Partition-level metadata

3. Partition as Append-Only Log

The Log Abstraction

PARTITION LOG STRUCTURE:

Offset: 0    1    2    3    4    5    6    7    8    9   ...
        ├────┼────┼────┼────┼────┼────┼────┼────┼────┼────┤
Data:   │ M0 │ M1 │ M2 │ M3 │ M4 │ M5 │ M6 │ M7 │ M8 │ M9 │ ...
        └────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
        │                                                   │
    Base Offset                                        Log End Offset (LEO)

Properties:
├─ Append-only: New records added at end
├─ Immutable: Existing records never modified
├─ Sequential offsets: 0, 1, 2, 3, ...
└─ Ordered: Read order = write order

Why Append-Only?

BENEFITS:

1. FAST WRITES:
   └─ Sequential disk I/O (fastest disk access pattern)
   └─ No seeking, no fragmentation

2. SIMPLE REPLICATION:
   └─ Just copy the log from leader to follower
   └─ No complex conflict resolution

3. IMMUTABILITY:
   └─ Once written, data never changes
   └─ Simplifies consistency model
   └─ Supports time-travel (replay from any offset)

4. CACHE-FRIENDLY:
   └─ OS page cache holds recent writes
   └─ Reads often served from memory

4. Offset Assignment Internals

Log End Offset (LEO)

DEFINITION:

Log End Offset (LEO):
└─ Offset of the next record to be written
└─ Points to end of log

Example:
├─ Last record written: offset 99
└─ LEO: 100 (next available offset)

LEO advances:
└─ LEO += number of records appended

Offset Assignment Algorithm

WHEN RECORDS ARRIVE:

Current state:
├─ LEO: 1000
└─ Incoming batch: 5 records

Offset assignment:
1. Assign offsets: 1000, 1001, 1002, 1003, 1004
2. Append records to log
3. Update LEO: 1005

CRITICAL PROPERTIES:
├─ Offsets are dense (no gaps)
├─ Offsets are sequential (monotonically increasing)
├─ Offsets are unique (within partition)
└─ Offsets are immutable (never change)

Offset Space

OFFSET RANGE:

Type: Long (64-bit signed integer)
Range: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807

Practical limits:
└─ At 1 million messages/second:
    └─ Would take 292,471 years to exhaust offset space
    └─ Never a concern in practice

Special offsets:
├─ -1: Latest (end of log)
├─ -2: Earliest (start of log, after retention cleanup)
└─ >= 0: Specific offset

5. Ordering Guarantees

Partition-Level Ordering

KAFKA'S ORDERING GUARANTEE:

"Messages sent by a producer to a particular partition 
 are appended in the order they are sent."

Within a partition:
├─ Record A written before Record B
└─ Record A has lower offset than Record B
└─ Consumers read A before B (guaranteed)

Example:
Producer sends: M1, M2, M3 (to partition 0)
Log contains: [M1(offset 100), M2(offset 101), M3(offset 102)]
Consumer reads: M1, then M2, then M3 (always)

Cross-Partition Ordering

NO GUARANTEE ACROSS PARTITIONS:

Topic: orders (3 partitions)

Producer sends:
├─ Order1 → Partition 0 (offset 50)
├─ Order2 → Partition 1 (offset 30)
└─ Order3 → Partition 2 (offset 70)

Consumer may read in ANY order:
├─ Could be: Order2, Order1, Order3
├─ Could be: Order3, Order2, Order1
└─ No guarantee on cross-partition order

If ordering across entire topic is needed:
└─ Use single partition (sacrifices parallelism)

Keyed Messages and Ordering

WITH MESSAGE KEYS:

Records with same key → same partition
└─ Guarantees ordering per key

Example:
Key: "user123"
Messages: Login, Click, Purchase

All "user123" messages:
├─ Go to same partition (based on key hash)
└─ Maintain order: Login → Click → Purchase

This enables:
└─ Partition tolerance (multiple partitions)
└─ Ordering guarantee (per key)
└─ Parallelism (different keys, different partitions)

6. Partition States

Leader vs Follower State

PARTITION REPLICA STATES:

LEADER:
├─ Accepts writes from producers
├─ Assigns offsets
├─ Serves reads from consumers
├─ Coordinates replication
└─ Tracks ISR membership

FOLLOWER:
├─ Replicates from leader (pull-based)
├─ Does NOT accept client writes
├─ Does NOT serve client reads (normally)
└─ Can become leader if current leader fails

Partition State Machine

STATES:

NewPartition:
└─ Partition created, no leader elected yet

OnlinePartition:
└─ Leader elected, accepting writes

OfflinePartition:
└─ No leader available (all ISR members down)

NonExistentPartition:
└─ Partition deleted

TRANSITIONS:

NewPartition → OnlinePartition
└─ Leader election completes

OnlinePartition → OfflinePartition
└─ Leader failure, no eligible followers

OnlinePartition → NonExistentPartition
└─ Topic/partition deletion

7. Partition-Level Metrics

Internal Tracking

EACH PARTITION MAINTAINS:

Log End Offset (LEO):
└─ Next offset to be written (leader)
└─ Latest offset replicated (follower)

High Watermark (HW):
└─ Last offset fully replicated to all ISR
└─ Visible to consumers

Last Stable Offset (LSO):
└─ For transactional messages
└─ Last offset of committed transactions

Start Offset:
└─ Earliest available offset
└─ Advances as old segments are deleted

Offset Metrics Example

PARTITION STATE SNAPSHOT:

Partition: orders-0
├─ Leader: Broker 1
├─ ISR: [1, 2, 3]
│
├─ Start Offset: 0 (earliest available)
├─ High Watermark: 1008 (last committed)
├─ Log End Offset: 1010 (leader)
│
└─ Follower LEOs:
    ├─ Broker 2: 1009
    └─ Broker 3: 1008

Consumer visibility:
└─ Can read offsets 0 through 1007
└─ Cannot read 1008, 1009 (not yet committed)

8. Partition Loading and Recovery

Broker Startup: Partition Loading

WHEN BROKER STARTS:

1. Scan log.dirs for partition directories
   └─ Example: orders-0, orders-1, users-2

2. For each partition:
   a. Read partition metadata
   b. Load log segments
   c. Rebuild indexes (if needed)
   d. Validate integrity

3. Determine role (leader or follower)
   └─ Based on metadata from controller

4. Start serving partition
   └─ Accept writes (if leader)
   └─ Start replication (if follower)

Unclean Shutdown Recovery

IF BROKER CRASHED (unclean shutdown):

1. Detect incomplete writes
   └─ Last segment may be corrupted

2. Truncate incomplete records
   └─ Remove partial writes

3. Rebuild indexes
   └─ Scan log files
   └─ Regenerate offset and time indexes

4. Validate log integrity
   └─ Check CRCs
   └─ Verify offset sequence

5. Mark partition as recovered
   └─ Ready to serve

9. Partition Assignment

How Partitions are Distributed

ASSIGNMENT ALGORITHM (simplified):

Goal: Balance partitions across brokers

For topic with N partitions, RF replication factor:
1. Select N brokers as leaders (round-robin)
2. For each partition:
   ├─ Assign leader broker
   └─ Assign RF-1 follower brokers
   └─ Prefer different racks (if rack awareness)

Example:
Topic: orders (6 partitions, RF=3)
Brokers: 1, 2, 3

Assignments:
├─ Partition 0: Leader=1, Followers=[2,3]
├─ Partition 1: Leader=2, Followers=[3,1]
├─ Partition 2: Leader=3, Followers=[1,2]
├─ Partition 3: Leader=1, Followers=[2,3]
├─ Partition 4: Leader=2, Followers=[3,1]
└─ Partition 5: Leader=3, Followers=[1,2]

Result:
└─ Even leadership distribution
└─ Even replica distribution

Rack-Aware Replica Placement

RACK AWARENESS:

Purpose:
└─ Distribute replicas across failure domains (racks, availability zones)
└─ Survive rack-level failures (power, network, cooling)

Configuration:
└─ broker.rack property (per broker)

Example:
broker.rack=us-east-1a  # Broker 1
broker.rack=us-east-1b  # Broker 2
broker.rack=us-east-1c  # Broker 3

Controller uses rack info during replica assignment

Rack-Aware Assignment Algorithm

WHEN RACK AWARENESS ENABLED:

1. Distribute partition leaders evenly across brokers (standard)

2. For each partition's replicas:
   a. Place leader on selected broker
   b. For each follower replica:
      ├─ Prefer brokers in different racks
      ├─ Maximize rack diversity
      └─ Fall back to same-rack if needed

3. Result: Replicas spread across racks

Example (3 brokers, 3 racks, RF=3):
├─ Partition 0: [Broker1(rack-a), Broker2(rack-b), Broker3(rack-c)]
├─ Partition 1: [Broker2(rack-b), Broker3(rack-c), Broker1(rack-a)]
└─ Partition 2: [Broker3(rack-c), Broker1(rack-a), Broker2(rack-b)]

Each partition has replicas in all 3 racks
└─ Survives entire rack failure

Benefits of Rack Awareness

ADVANTAGES:

1. Fault Tolerance:
   └─ Entire rack can fail (power, network, cooling)
   └─ Partition remains available (replicas in other racks)

2. Network Locality (with KIP-392):
   └─ Consumers can read from nearest replica (same rack/AZ)
   └─ Reduced cross-AZ data transfer costs
   └─ Lower latency

3. Failure Domain Isolation:
   └─ Correlated failures contained to single rack
   └─ Independent failure modes across racks

4. Cloud Benefits:
   └─ AWS: Availability zones = racks
   └─ Survive AZ outage
   └─ Comply with multi-AZ requirements

Trade-offs

CONSIDERATIONS:

Performance Impact:
├─ Cross-rack writes may have higher latency
├─ Network bandwidth between racks may be limited
└─ Typically acceptable for durability gain

Rack Imbalance:
├─ If racks have different broker counts
├─ May lead to uneven partition distribution
└─ Plan rack sizing carefully

Resource Usage:
└─ More network traffic across racks
└─ Consider in capacity planning

Configuration Example

# Broker config (server.properties)
broker.rack=us-east-1a  # Set per broker

# Verify rack awareness
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

# Output shows replicas across racks:
# Partition 0: Leader: 1, Replicas: 1,2,3
# (where brokers 1,2,3 are in different racks)

When to Use Rack Awareness

RECOMMENDED FOR:

✅ Production clusters (high availability)
✅ Cloud deployments (multi-AZ)
✅ Environments with rack-level failures
✅ Compliance requirements (data locality)

OPTIONAL FOR:

⚠️ Development clusters (single rack OK)
⚠️ Low-criticality data
⚠️ Latency-sensitive workloads (if cross-rack latency high)

10. Partition Reads and Writes

Write Path (Internal)

PRODUCER WRITE TO PARTITION:

1. Producer determines target partition
   └─ Based on key hash or round-robin

2. Producer sends ProduceRequest to leader
   └─ Includes records for partition

3. Leader validates request
   ├─ Authorization (if enabled)
   └─ Message size limits

4. Leader appends to log
   ├─ Assigns sequential offsets
   ├─ Writes to active log segment
   └─ Updates LEO

5. If acks=all, wait for ISR replication
   └─ Followers fetch and append

6. Leader responds to producer
   └─ Success: base offset of batch
   └─ Error: error code

Read Path (Internal)

CONSUMER READ FROM PARTITION:

1. Consumer sends FetchRequest to leader
   └─ Includes: partition, offset, max bytes

2. Leader validates request
   ├─ Is this broker the leader?
   └─ Authorization (if enabled)

3. Leader checks offset vs High Watermark
   └─ Only committed data (offset < HW) is readable

4. Leader reads from log
   ├─ Uses offset index to find segment
   ├─ Reads records from segment
   └─ Applies max.bytes limit

5. Leader responds to consumer
   └─ Records + metadata (HW, next offset)

6. Consumer processes records
   └─ Deserializes and hands to application

11. Partition Compaction (Preview)

Compaction-Eligible Partitions

IF cleanup.policy=compact:

Partition becomes:
├─ Log-compacted (key-based retention)
├─ Retains latest value per key
└─ Deletes old values for same key

Structure changes:
├─ Still append-only for writes
├─ Background cleaner removes old records
└─ Offsets remain (gaps may appear)

Example:
Before: [K1:V1(offset 0), K2:V2(1), K1:V3(2), K2:V4(3)]
After:  [_deleted______, _deleted__, K1:V3(2), K2:V4(3)]

Details covered in Module 2.3

12. Key Takeaways

You've learned:

✅ Partitions are physical directories on broker filesystems
✅ Each partition is an append-only log with sequential offsets
✅ Log End Offset (LEO) tracks the next offset to be written
✅ Offsets are assigned sequentially by the partition leader
✅ Ordering is guaranteed within a partition but not across partitions
✅ Leader accepts writes, followers replicate
✅ High Watermark determines consumer visibility
✅ Partition state machine transitions: New → Online → Offline
✅ Recovery process handles unclean shutdowns and rebuilds indexes
✅ Partition assignment balances load across brokers

13. Partition Internals Checklist

PARTITION STRUCTURE:

☑ Directory per partition: {topic}-{partition}
☑ Log files: *.log (actual data)
☑ Index files: *.index, *.timeindex
☑ Metadata files: leader-epoch-checkpoint

OFFSET MECHANICS:

☑ Sequential assignment (0, 1, 2, ...)
☑ Dense (no gaps in normal operation)
☑ Immutable (never change)
☑ LEO tracks end of log

ORDERING GUARANTEES:

☑ Within partition: Total order guaranteed
☑ Across partitions: No ordering guarantee
☑ Per-key: Ordering guaranteed (same key → same partition)

PARTITION ROLES:

☑ Leader: Accepts writes, assigns offsets
☑ Follower: Replicates from leader
☑ State machine: New → Online → Offline

Summary

Partitions are Kafka's fundamental storage unit:

Partition Structure =
├─ Append-only log on disk
├─ Sequential offset assignment
├─ Directory with .log and .index files
├─ Leader/follower replication
└─ Unit of ordering and parallelism

Offset Mechanics =
├─ LEO (Log End Offset): Next write position
├─ HW (High Watermark): Last committed
├─ Offsets are dense, sequential, immutable
└─ Ordering guaranteed within partition

Partition Lifecycle =
├─ Created: NewPartition state
├─ Leader elected: OnlinePartition state
├─ Leader fails: May go OfflinePartition
└─ Deleted: NonExistentPartition

Remember: Understanding partition internals is key to grasping Kafka's performance, ordering guarantees, and replication model. Every Kafka concept builds on the partition abstraction.

Next up: Module 2.2 — Log Segments, Indexes & Storage Layout — Deep dive into Kafka's storage engine.