2.2 – Log Structure & Message Storage

The Append-Only Log Model

Kafka's core storage model is an append-only log.

Definition: An append-only log is a data structure where:

• New records are always added to the end
• Existing records are never modified
• Records are read sequentially from beginning to end

Analogy:

• Like a write-ahead log (WAL) in databases
• Like a journal in accounting (you never erase entries, you add corrections)
• Like a ledger in blockchain (immutable history)

Why Append-Only?

Performance Benefits:

1. Sequential Disk Writes: Much faster than random writes (500 MB/sec vs 10 MB/sec)
2. OS Page Cache: Sequential reads/writes leverage OS caching effectively
3. Zero-Copy: Data can be transferred directly from disk to network without CPU copying

Operational Benefits:

1. Simplicity: No update-in-place logic (no locking, no race conditions)
2. Replayability: Consumers can re-read old data (time travel)
3. Durability: Once written, data is immutable (no accidental corruption)

Message Storage: From Producer to Disk

High-Level Flow

Producer                    Broker
   │                           │
   ├──Produce Request──────────▶│
   │  {                         │
   │    topic: orders,          │
   │    partition: 0,           │
   │    key: "user-123",        │
   │    value: "order-456",     │
   │    timestamp: 1620000000   │
   │  }                         │
   │                            │
   │                   ┌────────▼─────────┐
   │                   │ Validate request │
   │                   │ (auth, ACLs)     │
   │                   └────────┬─────────┘
   │                            │
   │                   ┌────────▼─────────┐
   │                   │ Append to log    │
   │                   │ (orders-0/log)   │
   │                   └────────┬─────────┘
   │                            │
   │                   ┌────────▼─────────┐
   │                   │ Replicate to     │
   │                   │ followers        │
   │                   └────────┬─────────┘
   │                            │
   │◀───Produce Response─────────┤
   │  {                          │
   │    offset: 12345,           │
   │    timestamp: 1620000001    │
   │  }                          │

Offsets: The Address in the Log

What is an Offset?

An offset is a unique, sequential ID assigned to each message within a partition.

Key Characteristics:

• Offsets start at 0 for each partition
• Offsets are monotonically increasing (always go up, never skip)
• Offsets are partition-local (each partition has its own offset sequence)

Example:

Topic: orders, Partition 0
┌────────┬────────┬────────┬────────┬────────┐
│ offset │ offset │ offset │ offset │ offset │
│   0    │   1    │   2    │   3    │   4    │
└────────┴────────┴────────┴────────┴────────┘

Offset as a Position

Think of offset as a "byte position" in a file.

Consumer Reading Flow:

1. Consumer starts at offset 0
2. Consumer reads message at offset 0
3. Consumer advances to offset 1
4. Consumer reads message at offset 1
5. ...

Key Point: Consumers track their own offset position. Kafka doesn't automatically move consumers forward.

Offset Durability

Where Offsets Are Stored:

Key Point: The broker assigns offsets when writing messages. The consumer stores its own "read position" separately.

Message Structure (On Disk)

Kafka Message Format (Record Batch Format)

Kafka stores messages in batches (not individual messages). This improves compression and throughput.

Single Message Structure (v2, Kafka 0.11+):

┌─────────────────────────────────────────────────┐
│               Record Batch                      │
├─────────────────────────────────────────────────┤
│ baseOffset: 12345                               │  # First offset in batch
│ batchLength: 512 bytes                          │  # Total batch size
│ partitionLeaderEpoch: 5                         │  # Leader epoch (replication)
│ magic: 2                                        │  # Message format version
│ crc: 0x1a2b3c4d                                 │  # Checksum for integrity
│ attributes: 0x00                                │  # Compression, transaction info
│ lastOffsetDelta: 10                             │  # Offset of last record in batch
│ firstTimestamp: 1620000000                      │  # Timestamp of first message
│ maxTimestamp: 1620000010                        │  # Timestamp of last message
│ producerId: 98765                               │  # For idempotence/transactions
│ producerEpoch: 3                                │  # For idempotence/transactions
│ baseSequence: 0                                 │  # For idempotence/transactions
│ records: [                                      │  # Array of messages
│   {                                             │
│     length: 50,                                 │
│     attributes: 0x00,                           │
│     timestampDelta: 0,                          │
│     offsetDelta: 0,                             │
│     keyLength: 8,                               │
│     key: "user-123",                            │
│     valueLength: 32,                            │
│     value: "order-456",                         │
│     headers: []                                 │
│   },                                            │
│   { ... },                                      │
│   { ... }                                       │
│ ]                                               │
└─────────────────────────────────────────────────┘

Key Fields:

Message Batching

Why Batching?

Result: Batching improves throughput by 10-100x.

Producer Batching (covered in M3: Producers):

• Producers batch messages before sending to broker
• Config: batch.size, linger.ms

Broker Batching:

• Broker writes entire batch to disk at once
• Broker replicates entire batch to followers

Key Point: Batching is transparent to consumers. Consumers see individual messages, not batches.

Immutability: Messages Never Change

Once a message is written to the log, it cannot be changed.

Consequences:

1. No Update Operations: You can't modify an existing message

   • If you need to "update" a record, you write a new message with the new value
   • For key-value updates, use log compaction (covered in Module 4.2)

2. No Delete Operations: You can't delete a specific message

   • You can only delete all messages older than a retention period
   • For key-based deletes, use tombstones (covered in Module 4.2)

3. No Overwrite: Disk space is only freed when retention policy expires or compaction runs

Why Immutability Matters

Benefits:

• No Locks: No need for complex locking (no concurrent updates)
• Replayability: Consumers can re-read old data (for debugging, reprocessing)
• Audit Trail: Full history of all events (compliance, forensics)

Trade-offs:

• Storage Cost: Every event is stored (can't update in-place to save space)
• Delete Semantics: Can't selectively delete one message (only bulk retention or compaction)

Sequential I/O: Kafka's Performance Secret

Why Sequential I/O is Fast

Random I/O (typical database):

• Disk seek time: ~10ms per seek
• 100 seeks/sec = max 100 operations/sec

Sequential I/O (Kafka):

• No seeks (just write to end of file)
• Modern SSD: 500 MB/sec sequential write
• Modern HDD: 100 MB/sec sequential write

Performance Comparison:

Key Insight: Sequential writes are 100-1000x faster than random writes, even on modern SSDs.

How Kafka Leverages Sequential I/O

1. Append-Only Writes: All writes go to the end of the log (no random seeks)
2. Batch Writes: Multiple messages written in one disk operation
3. OS Page Cache: Linux page cache keeps hot data in memory (no disk reads for recent data)
4. Zero-Copy: sendfile() syscall transfers data directly from disk to network (no CPU copying)

Result: Kafka can sustain millions of messages per second on commodity hardware.

OS Page Cache (The Hidden Layer)

How Page Cache Works

Without Page Cache (naive approach):

1. Broker writes message to application buffer
2. Application calls write() syscall
3. OS writes to disk
4. Consumer calls read() syscall
5. OS reads from disk
6. Data copied to application buffer

With Page Cache (Kafka's approach):

1. Broker writes message to application buffer
2. Application calls write() syscall
3. OS caches write in page cache (not written to disk yet)
4. Consumer calls read() syscall (milliseconds later)
5. OS serves read from page cache (no disk read!)
6. Data copied to application buffer

Key Benefit: Recent messages are served entirely from RAM, even though Kafka "writes to disk."

Page Cache vs JVM Heap

Why Kafka Doesn't Cache in JVM Heap:

Kafka's Design: Keep JVM heap small (4-8 GB), let OS page cache use the rest of RAM (e.g., 56 GB on a 64 GB machine).

Result: Kafka can cache 100 GB of hot data in page cache without GC issues.

Zero-Copy Optimization

Traditional Data Transfer

Copying Data from Disk to Network (without zero-copy):

1. Disk → OS kernel buffer (DMA transfer)
2. OS kernel buffer → JVM heap (CPU copy)
3. JVM heap → OS socket buffer (CPU copy)
4. OS socket buffer → Network (DMA transfer)

Total Copies: 2 CPU copies + 2 DMA transfers

Zero-Copy with sendfile()

Kafka uses sendfile() syscall (Linux/Unix):

1. Disk → OS kernel buffer (DMA transfer)
2. OS kernel buffer → Network (DMA transfer, no CPU copy!)

Total Copies: 0 CPU copies + 2 DMA transfers

Performance Impact:

• Without zero-copy: ~500 MB/sec (CPU-bound)
• With zero-copy: ~2 GB/sec (network-bound)

Key Point: Kafka can serve consumers at wire speed (limited only by network bandwidth, not CPU).

Message Timestamps

Kafka supports two types of timestamps:

1. LogAppendTime (Broker-Set)

Definition: Broker sets timestamp when message is written to log.

Config:

message.timestamp.type=LogAppendTime

Use Case:

• Clients are untrusted (producer could send fake timestamps)
• Need consistent, server-side timestamps

Trade-off: All messages in a batch get the same timestamp (batch write time).

2. CreateTime (Producer-Set)

Definition: Producer sets timestamp when message is created.

Config:

message.timestamp.type=CreateTime

Use Case:

• Need event-time semantics (e.g., "when did the user click?")
• Producers are trusted

Trade-off: Clock skew between producers can cause out-of-order timestamps.

Timestamp Index

Kafka maintains a time index to enable efficient timestamp-based lookups.

Query: "Give me all messages after timestamp 1620000000"

Without Time Index: Scan entire log (slow)

With Time Index: Binary search time index → jump to offset → read from there (fast)

Time index is covered in Module 2.3: Segments & Physical Storage Layout.

Message Headers (Kafka 0.11+)

Headers are optional key-value metadata attached to messages.

Example:

ProducerRecord<String, String> record = new ProducerRecord<>("orders", "key", "value");
record.headers().add("trace-id", "abc-123".getBytes());
record.headers().add("content-type", "application/json".getBytes());
producer.send(record);

Use Cases:

• Distributed Tracing: Propagate trace IDs across services
• Content Type: Indicate message format (JSON, Avro, Protobuf)
• Routing Metadata: SMTs (Single Message Transforms) in Kafka Connect

Key Point: Headers do not affect partitioning (only the key does). Headers are just metadata.

Key Takeaways

1. Append-Only Log: Kafka stores messages in an append-only log (never updates in-place).

2. Offsets: Each message has a unique, sequential offset within its partition. Offsets are the "position" in the log.

3. Immutability: Once written, messages cannot be changed. This enables replayability and simplifies concurrency.

4. Sequential I/O: Kafka writes messages sequentially to disk, achieving 100-1000x better performance than random I/O.

5. OS Page Cache: Kafka relies on the OS page cache to serve recent messages from RAM (no JVM heap caching).

6. Zero-Copy: Kafka uses sendfile() to transfer data from disk to network without CPU copying, achieving wire-speed throughput.

7. Batching: Messages are stored in batches (not individually) to improve compression and reduce disk writes.

8. Timestamps: Kafka supports both producer-set timestamps (CreateTime) and broker-set timestamps (LogAppendTime).

What's Next?

Next Module: 2.3 – Segments & Physical Storage Layout

What We Covered:

• ✅ Append-only log model
• ✅ Offsets as positions in the log
• ✅ Message structure (record batches)
• ✅ Immutability and its consequences
• ✅ Sequential I/O benefits
• ✅ OS page cache and zero-copy
• ✅ Message timestamps and headers

What We Didn't Cover (Explicitly Deferred):

• ❌ Segment files and rolling → Module 2.3
• ❌ Retention and compaction → Module 4.x
• ❌ Replication protocol → Module 3.1 & M7: Internals
• ❌ Producer batching configs → M3: Producers

Log Structure & Message Storage: Complete ✅