KafkaIQ

2.2 Log Segments, Indexes & Storage Layout

Module Overview

Goal: Deep dive into Kafka's storage engine — how data is actually stored and retrieved.

What You'll Learn:

  • Segment file structure and rolling
  • Offset index internals
  • Time index internals
  • File layout on disk
  • Fast lookup mechanics
  • Segment lifecycle

What's NOT Covered Here:

  • Compaction algorithm (covered in Module 2.3)
  • Replication mechanics (covered in Module 3.x)
  • Topic configuration strategy (covered in Operations)

Key Insight: Kafka's storage engine is highly optimized for sequential I/O and fast offset lookups. Understanding segments and indexes explains why Kafka performs so well.

1. Why Segments?

The Problem with a Single File

IF PARTITION USED ONE HUGE FILE:

Problems:
├─ File grows unbounded
├─ Deletion of old data is expensive (rewrite entire file)
├─ Index becomes huge (slow lookups)
├─ Recovery takes forever (must scan entire file)
└─ OS limits on file size

SOLUTION: Split log into segments

Benefits:
├─ Delete old segments cheaply (just rm file)
├─ Smaller indexes (per segment)
├─ Faster recovery (only active segment needs validation)
└─ Easier to manage (rotate segments)

2. Segment File Structure

Segment Components

EACH SEGMENT CONSISTS OF 3 FILES:

00000000000012345678.log
└─ Actual message data (records)

00000000000012345678.index
└─ Offset index (offset → file position)

00000000000012345678.timeindex
└─ Timestamp index (timestamp → offset)

Filename = base offset (20 digits, zero-padded)

Log Segment File Layout

DIRECTORY VIEW:

/var/kafka/data/orders-0/
├─ 00000000000000000000.log      # Base offset: 0
├─ 00000000000000000000.index
├─ 00000000000000000000.timeindex
│
├─ 00000000000001000000.log      # Base offset: 1000000
├─ 00000000000001000000.index
├─ 00000000000001000000.timeindex
│
├─ 00000000000002000000.log      # Base offset: 2000000 (active segment)
├─ 00000000000002000000.index
└─ 00000000000002000000.timeindex

Active segment:
└─ Currently being written to
└─ All new records appended here

Older segments:
└─ Immutable (no more writes)
└─ Candidates for deletion or compaction

Log File Format

LOG FILE CONTENTS (.log):

┌─────────────────────────────────────────────────────┐
│  Record Batch 1                                     │
│  ├─ baseOffset: 12345678                            │
│  ├─ length: 1024 bytes                              │
│  ├─ crc: checksum                                   │
│  ├─ attributes: compression, timestamp type         │
│  ├─ lastOffsetDelta: 9 (10 records in batch)       │
│  ├─ baseTimestamp: 1234567890000                    │
│  └─ records:                                        │
│      ├─ Record 0 (offsetDelta=0, key, value)       │
│      ├─ Record 1 (offsetDelta=1, key, value)       │
│      └─ ...                                         │
├─────────────────────────────────────────────────────┤
│  Record Batch 2                                     │
│  ├─ baseOffset: 12345688                            │
│  └─ ...                                             │
└─────────────────────────────────────────────────────┘

Record Batch = multiple records compressed together
└─ Produced by producer batching
└─ Kafka stores batches as-is (no re-batching)

3. Segment Rolling

When to Create New Segment

SEGMENT ROLLING TRIGGERS:

1. Size-based (segment.bytes):
   └─ Current segment >= 1 GB (default)

2. Time-based (segment.ms):
   └─ Current segment open for >= 7 days (default)

3. Index full (segment.index.bytes):
   └─ Index file reaches max size (default: 10 MB)

When triggered:
├─ Close current active segment
├─ Create new segment with next offset as base
└─ New segment becomes active

Rolling Process

STEP-BY-STEP:

Before rolling:
├─ Active segment: 00000000000001000000.log (LEO: 1500000)
└─ Writing new records

Rolling triggered (size limit reached):

1. Close active segment
   ├─ Flush to disk
   └─ Mark as immutable

2. Create new segment files:
   ├─ 00000000000001500000.log (base offset = LEO)
   ├─ 00000000000001500000.index
   └─ 00000000000001500000.timeindex

3. New segment becomes active
   └─ All new writes go to this segment

After rolling:
├─ Old segment: 00000000000001000000.* (immutable)
└─ Active segment: 00000000000001500000.* (accepting writes)

4. Offset Index Internals

Purpose

PROBLEM:

To find record at offset 1234567:
└─ Without index: Must scan log from beginning (slow!)

SOLUTION:

Offset index: Maps offset → file position
└─ Quickly jump to approximate location in log file

Index File Structure

OFFSET INDEX FORMAT (.index):

Binary file with fixed-size entries (8 bytes each):

┌──────────────┬──────────────┐
│ Relative     │ File         │
│ Offset (4B)  │ Position (4B)│
├──────────────┼──────────────┤
│ 0            │ 0            │
│ 100          │ 12480        │
│ 200          │ 24960        │
│ 300          │ 37440        │
│ ...          │ ...          │
└──────────────┴──────────────┘

Relative offset:
└─ Offset relative to base offset of segment
└─ Saves space (4 bytes instead of 8)

Example:
├─ Segment base offset: 1000000
├─ Index entry: relative offset 100
└─ Actual offset: 1000100

Index Building

INDEX ENTRY CREATION:

Not every record has index entry!
└─ Index is sparse (sampled)

Index entry created every log.index.interval.bytes (default: 4096 bytes):
├─ After writing 4 KB of data
├─ Add entry: (current offset, current file position)
└─ Continue writing

Example:
Write records:
├─ Offset 1000: Position 0 → Index entry [0, 0]
├─ Offsets 1001-1100: No index entries (< 4 KB)
├─ Offset 1101: Position 4200 → Index entry [101, 4200]
├─ Offsets 1102-1200: No index entries
└─ ... continue

Result:
└─ Sparse index (not every offset indexed)
└─ Small index size

Lookup Algorithm

TO FIND OFFSET 1234567:

1. Determine segment
   └─ Find segment with base offset <= 1234567
   └─ Result: 00000000000001000000.log (base: 1000000)

2. Calculate relative offset
   └─ 1234567 - 1000000 = 234567

3. Binary search in index file
   └─ Find largest relative offset <= 234567
   └─ Result: Entry [234500, 105820400]

4. Seek to file position 105820400

5. Scan forward from position
   └─ Read records sequentially
   └─ Until offset 1234567 found

6. Return record

Time complexity:
└─ O(log N) for binary search in index
└─ O(1) on average for sequential scan (small gap)

5. Time Index Internals

Purpose

PROBLEM:

Find records from timestamp 1609459200000 (Jan 1, 2021):
└─ Without time index: Must scan all records (very slow!)

SOLUTION:

Time index: Maps timestamp → offset
└─ Then use offset index to find record

Time Index File Structure

TIME INDEX FORMAT (.timeindex):

Binary file with fixed-size entries (12 bytes each):

┌──────────────┬──────────────┐
│ Timestamp    │ Relative     │
│ (8 bytes)    │ Offset (4B)  │
├──────────────┼──────────────┤
│ 1609459200000│ 0            │
│ 1609459260000│ 100          │
│ 1609459320000│ 200          │
│ ...          │ ...          │
└──────────────┴──────────────┘

Also sparse:
└─ Sampled at same frequency as offset index

Timestamp Lookup

TO FIND RECORDS FROM TIMESTAMP T:

1. Search all segments' time indexes
   └─ Find segment where max timestamp >= T

2. Binary search in time index
   └─ Find largest timestamp <= T
   └─ Get corresponding offset

3. Use offset index
   └─ Seek to approximate position

4. Scan forward
   └─ Read records until timestamp >= T

Use case:
└─ Consumer reset to timestamp
└─ Time-based retention (find records older than N days)

6. Segment Lifecycle

State Transitions

SEGMENT STATES:

1. ACTIVE:
   ├─ Currently being written to
   ├─ Records appended
   └─ Index updated

2. CLOSED:
   ├─ No longer active (segment rolled)
   ├─ Immutable
   └─ Eligible for cleanup

3. DELETED:
   └─ Removed from disk (retention or compaction)

TRANSITION:

Active → Closed:
└─ When rolling occurs

Closed → Deleted:
└─ When retention policy triggers deletion

Segment Deletion

RETENTION-BASED DELETION:

If segment's max timestamp < (now - retention.ms):
└─ Delete segment files:
    ├─ rm *.log
    ├─ rm *.index
    └─ rm *.timeindex

Size-based deletion:
If total partition size > retention.bytes:
└─ Delete oldest segments until under limit

Deletion is atomic:
└─ All segment files deleted together
└─ Start offset of partition advances

7. Index Recovery and Rebuilding

When Indexes are Lost or Corrupted

REBUILD PROCESS:

1. Scan .log file from beginning
2. For each record batch:
   ├─ Read offset and timestamp
   ├─ Determine if index entry should be created
   │   └─ Based on log.index.interval.bytes
   └─ Write index entry

3. Fsync index files
4. Mark as recovered

When recovery is needed:
├─ Broker crash (indexes may be incomplete)
├─ Index corruption detected (CRC failure)
└─ Index files deleted accidentally

Broker Startup Recovery

ON BROKER START:

For each partition:

1. Load all segment files
2. For active segment only:
   ├─ Validate records
   ├─ Truncate incomplete records (if crash)
   └─ Rebuild index

3. For closed segments:
   └─ Indexes assumed valid (already immutable)

4. Mark partition as recovered
   └─ Ready to serve

Fast recovery:
└─ Only active segment needs validation
└─ Closed segments are immutable

8. Storage Layout on Disk

Partition Directory Structure

COMPLETE VIEW:

/var/kafka/data/orders-0/
│
├─ 00000000000000000000.log          # Segment 1 (log)
├─ 00000000000000000000.index        # Segment 1 (offset index)
├─ 00000000000000000000.timeindex    # Segment 1 (time index)
│
├─ 00000000000001000000.log          # Segment 2
├─ 00000000000001000000.index
├─ 00000000000001000000.timeindex
│
├─ 00000000000002000000.log          # Segment 3 (active)
├─ 00000000000002000000.index
├─ 00000000000002000000.timeindex
│
├─ leader-epoch-checkpoint           # Leader epoch tracking
├─ partition.metadata                # Partition metadata
│
└─ (optional files):
    ├─ 00000000000000000000.txnindex # Transaction index
    ├─ 00000000000000000000.snapshot # Snapshot (if compacted)
    └─ cleaner-offset-checkpoint     # Compaction progress

File Sizes

TYPICAL SIZES:

.log files:
└─ Up to segment.bytes (default: 1 GB)
└─ Can be larger if single record batch exceeds limit

.index files:
└─ Up to segment.index.bytes (default: 10 MB)
└─ Smaller if segment has fewer records

.timeindex files:
└─ Similar size to .index files

Leader-epoch-checkpoint:
└─ Small (few KB)
└─ One entry per leader epoch

Total disk usage:
└─ Primarily .log files (data)
└─ Indexes are ~1% of data size

9. Read Performance Optimizations

OS Page Cache

KAFKA RELIES ON OS PAGE CACHE:

Write path:
├─ Producer writes → Kafka → OS page cache
└─ OS flushes to disk asynchronously

Read path:
├─ Recent writes still in page cache
├─ Consumer reads → Kafka → OS page cache → return
└─ No disk I/O (served from RAM)

Benefits:
├─ No large JVM heap needed
├─ OS manages memory efficiently
├─ Zero-copy possible (sendfile)
└─ Fast access to recent data

Read-Ahead

OS READ-AHEAD:

Sequential access pattern detected:
└─ OS prefetches next pages into cache

Kafka benefits:
├─ Consumers typically read sequentially
├─ OS loads data ahead of requests
└─ Further reduces disk I/O

Tuning:
└─ Linux: /sys/block/sda/queue/read_ahead_kb
└─ Increase for consumer-heavy workloads

10. Write Performance Optimizations

Batching

KAFKA WRITE PATH:

Producer:
└─ Batches multiple records
└─ Compresses batch
└─ Sends single batch to broker

Broker:
└─ Writes entire batch to log (one operation)
└─ No re-batching (stores as-is)

Benefits:
├─ Fewer disk writes
├─ Better compression ratio
├─ Fewer index entries
└─ Higher throughput

Flush Policy

KAFKA DOES NOT FLUSH IMMEDIATELY:

Default behavior:
├─ Writes to OS page cache
├─ OS flushes to disk asynchronously (lazy)
└─ Relies on replication for durability

Configuration (not recommended):
├─ log.flush.interval.messages
├─ log.flush.interval.ms
└─ Explicit flush (reduces throughput)

Why lazy flush is OK:
└─ Replication factor = 3 means data on 3 brokers
└─ Even if 1 broker crashes, data is safe
└─ No need to wait for disk sync

11. Index Memory Mapping

mmap for Fast Access

INDEXES ARE MEMORY-MAPPED:

mmap():
└─ Maps file directly into process address space
└─ OS handles paging (load pages as needed)

Benefits:
├─ No explicit read() syscalls
├─ Indexes feel like in-memory arrays
├─ OS manages caching automatically
└─ Very fast lookups (binary search in memory)

Kafka implementation:
└─ All index files mmap'ed on broker start
└─ Lookups are simple array accesses

12. Segment Compaction (Preview)

Compacted Segment Structure

IF cleanup.policy=compact:

Segments still follow same structure:
├─ .log, .index, .timeindex files
└─ But semantics change

Compaction process:
├─ Creates new segment with only latest keys
├─ Old segment deleted
└─ Offsets preserved (gaps may appear)

Details in Module 2.3

13. Key Takeaways

You've learned:

✅ Segments split the log into manageable chunks (default: 1 GB)
✅ Each segment has 3 files: .log (data), .index (offset), .timeindex (timestamp)
✅ Offset index is sparse (sampled every 4 KB)
✅ Lookup is O(log N) binary search in index + sequential scan
✅ Segments roll based on size, time, or index size
✅ Active segment is mutable, closed segments are immutable
✅ Deletion is simple: just remove old segment files
✅ Indexes are memory-mapped for fast access
✅ OS page cache is Kafka's primary cache (not JVM heap)
✅ Recovery only validates active segment (fast startup)

14. Storage Layout Checklist

SEGMENT FILES:

☑ *.log: Actual record data
☑ *.index: Offset → file position mapping
☑ *.timeindex: Timestamp → offset mapping
☑ Filename = base offset (20 digits)

SEGMENT PROPERTIES:

☑ Active segment: Currently being written
☑ Closed segments: Immutable
☑ Segments roll based on size or time
☑ Deletion: Remove entire segment (atomic)

INDEX PROPERTIES:

☑ Sparse (not every record indexed)
☑ Fixed-size entries
☑ Memory-mapped for fast access
☑ Binary search for lookups

PERFORMANCE OPTIMIZATIONS:

☑ OS page cache (primary cache)
☑ Zero-copy (sendfile for reads)
☑ Batching (producer and broker)
☑ Sequential I/O (fast on disk)

Summary

Kafka's storage engine is optimized for throughput:

Segment Design =
├─ Split log into fixed-size segments
├─ Simple deletion (remove old segments)
├─ Fast recovery (only active segment)
└─ Manageable index sizes

Index Design =
├─ Sparse (sampled)
├─ Binary searchable
├─ Memory-mapped
└─ Small size (~1% of data)

Performance =
├─ Sequential I/O (disk-friendly)
├─ OS page cache (RAM utilization)
├─ Batching (fewer I/O operations)
└─ Zero-copy (kernel optimizations)

Remember: Kafka's storage engine trades off some complexity (multiple files per partition) for massive performance gains (sequential I/O, simple deletion, fast lookups). Understanding segments and indexes explains why Kafka can sustain millions of messages per second.

Next up: Module 2.3 — Compaction & Retention Mechanics — How Kafka manages log cleanup at the engine level.