2.3 – Segments & Physical Storage Layout

Why Segments?

Problem: A partition log can grow to terabytes in size. Storing this as a single file would be impractical:

• Can't delete old data (need to scan entire file)
• Can't efficiently search by offset or timestamp
• File recovery after crash would be slow

Solution: Kafka splits each partition into segments (smaller log files).

What is a Segment?

A segment is a fixed-size chunk of a partition's log.

Key Characteristics:

• Each segment is a separate file on disk
• Segments are sequential (segment 0, segment 1, segment 2, ...)
• Only the active segment is writable (new messages go here)
• Closed segments are immutable (no more writes)

Analogy:

• A partition is like a book (the full log)
• A segment is like a chapter (a portion of the log)
• You write to the last chapter (active segment), previous chapters are read-only (closed segments)

Segment Rolling (Creating New Segments)

Segment rolling is the process of closing the current active segment and starting a new one.

Triggers for Rolling:

Example Flow:

Time 0:
  orders-0/
    00000000000000000000.log (active, 500 MB)
    00000000000000000000.index

Time 1 (after 1 GB of writes):
  orders-0/
    00000000000000000000.log (closed, 1 GB)
    00000000000000000000.index
    00000000000000123456.log (active, 200 MB)  ← New segment
    00000000000000123456.index

Time 2 (after another 1 GB of writes):
  orders-0/
    00000000000000000000.log (closed, 1 GB)
    00000000000000000000.index
    00000000000000123456.log (closed, 1 GB)
    00000000000000123456.index
    00000000000000456789.log (active, 100 MB)  ← New segment
    00000000000000456789.index

Key Point: Segment filenames are the base offset (first offset in that segment). This makes it easy to find which segment contains a given offset.

Physical Storage Layout

Directory Structure

Each partition is stored as a directory on the broker's disk.

Example (3 partitions, replication factor 1):

/var/lib/kafka-logs/
  ├── orders-0/
  │   ├── 00000000000000000000.log
  │   ├── 00000000000000000000.index
  │   ├── 00000000000000000000.timeindex
  │   ├── 00000000000000123456.log
  │   ├── 00000000000000123456.index
  │   ├── 00000000000000123456.timeindex
  │   ├── 00000000000000456789.log (active segment)
  │   ├── 00000000000000456789.index
  │   └── 00000000000000456789.timeindex
  │
  ├── orders-1/
  │   ├── 00000000000000000000.log
  │   ├── 00000000000000000000.index
  │   └── ...
  │
  └── orders-2/
      ├── 00000000000000000000.log
      ├── 00000000000000000000.index
      └── ...

Naming Convention:

• <topic-name>-<partition-number>/ (directory)
• <base-offset>.log (message data)
• <base-offset>.index (offset index)
• <base-offset>.timeindex (time index)

Segment Files

Each segment consists of three files:

The .log File (Message Data)

Contents: The .log file contains the actual messages (record batches).

Structure:

00000000000000000000.log
┌─────────────────────────────────────────────┐
│ Record Batch (offsets 0-9)                  │  500 bytes
├─────────────────────────────────────────────┤
│ Record Batch (offsets 10-19)                │  800 bytes
├─────────────────────────────────────────────┤
│ Record Batch (offsets 20-29)                │  600 bytes
├─────────────────────────────────────────────┤
│ ...                                          │
└─────────────────────────────────────────────┘

Key Point: Messages are not stored individually. They're stored in record batches (for compression and efficiency).

The .index File (Offset Index)

Purpose: Maps logical offsets to physical file positions (byte offsets).

Why Needed?

Without Index:

• To find message at offset 123456, you'd have to scan the entire log from offset 0.
• For a 1 GB log file, this could take seconds.

With Index:

• Binary search the index to find the closest offset ≤ 123456.
• Jump directly to that position in the .log file.
• Read forward to find exact message.

Index File Structure

Format: Pairs of (offset, file_position)

Example (00000000000000000000.index):

┌─────────────────────────────────────┐
│ Offset  │ File Position (bytes)    │
├─────────────────────────────────────┤
│    0    │         0                │
│   10    │       500                │
│   20    │      1300                │
│   30    │      1900                │
│   40    │      2500                │
│   50    │      3200                │
│  ...    │       ...                │
└─────────────────────────────────────┘

Interpretation:

• Message at offset 0 is at byte position 0 in the .log file
• Message at offset 10 is at byte position 500
• Message at offset 20 is at byte position 1300

Key Point: The index is sparse (not every message is indexed, only sample offsets). This keeps index size small.

Finding a Message by Offset

Query: Find message at offset 25.

Steps:

1. Binary search .index file:

   • Closest offset ≤ 25 is offset 20 (at file position 1300)

2. Jump to position 1300 in .log file

3. Read forward from position 1300 until offset 25:

   • Read record batch at position 1300 (offsets 20-29)
   • Extract message at offset 25

Performance:

• Without index: O(n) (scan entire log)
• With index: O(log n) (binary search) + O(k) (read a few batches)

Result: Sub-millisecond lookups even for multi-GB log files.

The .timeindex File (Time Index)

Purpose: Maps timestamps to offsets.

Why Needed?

Use Case: Consumer wants to read "all messages since 2 hours ago."

Without Time Index:

• Scan entire log, check each message's timestamp (slow)

With Time Index:

• Binary search time index to find offset corresponding to "2 hours ago"
• Jump to that offset, start reading

Time Index File Structure

Format: Pairs of (timestamp, offset)

Example (00000000000000000000.timeindex):

┌─────────────────────────────────────┐
│ Timestamp       │ Offset           │
├─────────────────────────────────────┤
│ 1620000000000   │     0            │
│ 1620003600000   │   100            │
│ 1620007200000   │   200            │
│ 1620010800000   │   300            │
│  ...            │   ...            │
└─────────────────────────────────────┘

Interpretation:

• Messages at timestamp 1620000000000 (epoch ms) start at offset 0
• Messages at timestamp 1620003600000 (1 hour later) start at offset 100

Finding Messages by Timestamp

Query: Find messages with timestamp ≥ 1620005000000.

Steps:

1. Binary search .timeindex file:

   • Closest timestamp ≤ 1620005000000 is 1620003600000 (offset 100)

2. Jump to offset 100 (using .index to find file position)

3. Read forward from offset 100, skipping messages with timestamp < 1620005000000

Use Case: Time-based log deletion (covered in Module 4.1) relies on time index.

Segment Lookup Example (End-to-End)

Scenario: Consumer wants to read message at offset 456789.

Partition Directory:

orders-0/
  00000000000000000000.log (offsets 0-123455)
  00000000000000123456.log (offsets 123456-456788)
  00000000000000456789.log (offsets 456789+)  ← Active segment

Steps:

1. Find Segment:

   • Offset 456789 → Look for segment with base offset ≤ 456789
   • Result: 00000000000000456789.log

2. Find Position in Segment:

   • Binary search 00000000000000456789.index
   • Find closest offset ≤ 456789 (e.g., offset 456789 → position 0)

3. Read from .log File:

   • Seek to position 0 in 00000000000000456789.log
   • Read record batch containing offset 456789

Performance: O(log n) segment lookup + O(log m) index search + O(1) read = sub-millisecond latency

Segment Deletion (Retention)

When segments are deleted (covered in detail in Module 4.1):

1. Retention policy expires (e.g., data older than 7 days)
2. Kafka deletes entire segments (not individual messages)
3. Segment deletion is atomic (either delete all 3 files or none)

Example:

Before retention cleanup:
  orders-0/
    00000000000000000000.log (8 days old)  ← Eligible for deletion
    00000000000000123456.log (5 days old)
    00000000000000456789.log (1 day old, active)

After retention cleanup:
  orders-0/
    00000000000000123456.log (5 days old)
    00000000000000456789.log (1 day old, active)

Key Point: Kafka deletes entire segments, not individual messages. This is why segment size matters (smaller segments = finer-grained retention).

Segment Compaction (Deduplications)

For compacted topics (covered in detail in Module 4.2):

Kafka runs a log cleaner that:

1. Scans segments
2. Keeps only the latest value for each key
3. Deletes old values for the same key

Before Compaction:

Segment 0:
  key=A, value=1, offset=0
  key=B, value=2, offset=1
  key=A, value=3, offset=2  ← Newer value for key=A

Segment 1:
  key=A, value=4, offset=3  ← Newest value for key=A
  key=C, value=5, offset=4

After Compaction:

Segment 0:
  key=B, value=2, offset=1  ← Only message without newer value

Segment 1:
  key=A, value=4, offset=3  ← Latest value for key=A
  key=C, value=5, offset=4  ← Latest value for key=C

Key Point: Compaction rewrites segments to remove old values, freeing disk space.

Active vs Closed Segments

Active Segment

Characteristics:

• Writable (new messages are appended here)
• Not eligible for deletion (even if retention policy would delete it)
• Not compacted (compaction only runs on closed segments)
• Index is updated as messages arrive

Closed Segments

Characteristics:

• Immutable (no more writes)
• Eligible for deletion (if retention policy expires)
• Eligible for compaction (if topic is compacted)
• Index is final (no more updates)

Segment Configuration

Key Parameters

Choosing Segment Size

Small Segments (e.g., 100 MB):

• ✅ Pros: Finer-grained retention (delete data more precisely), faster compaction
• ❌ Cons: More segment files (higher metadata overhead), more file handles

Large Segments (e.g., 10 GB):

• ✅ Pros: Fewer segment files, less metadata overhead
• ❌ Cons: Coarse-grained retention (can't delete old data until entire segment expires)

Recommendation: 1 GB (default) is a good balance for most use cases.

Use smaller segments (100-500 MB) if:

• You have aggressive retention policies (e.g., delete data older than 1 hour)
• You use compaction heavily

Use larger segments (2-5 GB) if:

• You have very long retention (e.g., 90 days)
• You have high write throughput and want to minimize metadata overhead

Disk Layout Best Practices

1. Use Separate Disks for Each Broker

Why: Kafka is disk I/O intensive. Don't share disks with other applications.

Configuration:

log.dirs=/mnt/kafka-disk1/logs,/mnt/kafka-disk2/logs,/mnt/kafka-disk3/logs

Result: Kafka distributes partitions across disks (for parallelism).

2. Use RAID 10 (or JBOD)

RAID 10 (striping + mirroring):

• ✅ Pros: Good balance of performance and redundancy
• ❌ Cons: 50% disk utilization (mirroring overhead)

JBOD (Just a Bunch of Disks):

• ✅ Pros: 100% disk utilization, Kafka handles replication
• ❌ Cons: If one disk fails, all partitions on that disk are lost (rely on Kafka replication)

Recommendation: JBOD if you have replication factor ≥ 3. RAID 10 if replication factor = 2 or you want extra safety.

3. SSD vs HDD

Key Point: Kafka's sequential I/O pattern works well even on HDDs. SSDs provide better performance but aren't strictly required.

4. Monitor Disk Usage

Key Metrics:

• Disk utilization (df -h /var/lib/kafka-logs)
• Disk I/O (iostat -x 1)
• Inode usage (lots of segment files = lots of inodes)

Alert Thresholds:

• Disk > 80% full → Add more brokers or reduce retention
• Disk I/O > 80% → Too many partitions on one broker, rebalance

Segment File Inspection (CLI Tools)

1. Dump Segment Contents

kafka-dump-log --files /var/lib/kafka-logs/orders-0/00000000000000000000.log --print-data-log

Output:

baseOffset: 0 lastOffset: 9 count: 10 baseSequence: 0 lastSequence: 9 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 0 CreateTime: 1620000000000 size: 500 magic: 2 compresscodec: NONE crc: 123456789
  offset: 0 CreateTime: 1620000000000 keysize: 8 valuesize: 10 sequence: 0 headerKeys: [] key: user-123 payload: order-456
  offset: 1 CreateTime: 1620000001000 keysize: 8 valuesize: 10 sequence: 1 headerKeys: [] key: user-456 payload: order-789
  ...

2. Dump Index Contents

kafka-dump-log --files /var/lib/kafka-logs/orders-0/00000000000000000000.index

Output:

offset: 0 position: 0
offset: 10 position: 500
offset: 20 position: 1300
offset: 30 position: 1900
...

3. Dump Time Index Contents

kafka-dump-log --files /var/lib/kafka-logs/orders-0/00000000000000000000.timeindex

Output:

timestamp: 1620000000000 offset: 0
timestamp: 1620003600000 offset: 100
timestamp: 1620007200000 offset: 200
...

Key Takeaways

1. Segments: Partitions are split into segments (smaller log files) for efficient deletion, compaction, and indexing.

2. Segment Rolling: Kafka closes the active segment and starts a new one when it reaches log.segment.bytes or log.segment.ms.

3. Three Files per Segment: .log (message data), .index (offset → position), .timeindex (timestamp → offset).

4. Sparse Index: The .index file is sparse (not every message is indexed) to keep index size small.

5. Segment-Based Deletion: Kafka deletes entire segments (not individual messages) when retention policy expires.

6. Active vs Closed: Only the active segment is writable. Closed segments are immutable and eligible for deletion/compaction.

7. Disk Layout: Use multiple disks (log.dirs), prefer JBOD with replication factor ≥ 3, SSDs improve performance but aren't required.

8. Segment Size Trade-off: Smaller segments = finer-grained retention, larger segments = less metadata overhead. Default 1 GB is a good balance.

What's Next?

Next Module: 3.1 – Replication & Fault Tolerance (Conceptual Model)

What We Covered:

• ✅ Segment files and rolling
• ✅ Physical storage layout (directories, .log, .index, .timeindex)
• ✅ Offset and time index lookups
• ✅ Segment deletion and compaction (overview)
• ✅ Active vs closed segments
• ✅ Disk layout best practices

What We Didn't Cover (Explicitly Deferred):

• ❌ Retention policies (time, size) → Module 4.1
• ❌ Log compaction details → Module 4.2
• ❌ Replication protocol → Module 3.1 & M7: Internals
• ❌ Performance tuning → M6: Operations

Segments & Physical Storage Layout: Complete ✅