KafkaIQ1.1 End-to-End Message Flow & Core Architecture
Module Overview
Goal: Show the complete internal path of a record through Kafka's machinery.
What You'll Learn:
- Producer → Broker → Partition → Log → Replica → Consumer flow
- Request handling pipeline inside brokers
- Network layer and I/O thread architecture
- Data plane vs control plane separation
- High-level internal architecture
What's NOT Covered Here:
- Producer configuration details (covered in Producer modules)
- Consumer configuration details (covered in Consumer modules)
- Performance tuning (covered in Operations modules)
- Specific topic design (covered in Operations modules)
Key Insight: Understanding the full internal message journey reveals why Kafka behaves the way it does. This is "inside the machine" knowledge.
1. The Complete Message Journey
High-Level Flow
PRODUCER → BROKER → PARTITION → LOG → REPLICAS → CONSUMER
Step-by-step:
1. Producer constructs ProducerRecord
└─ Serializes key and value to bytes
2. Producer sends batch to broker (leader for target partition)
└─ Network request with ProduceRequest
3. Broker receives request on network thread
└─ Validates and queues for processing
4. I/O thread writes to partition log
└─ Appends to active log segment on disk
5. Followers fetch data from leader
└─ Replication to maintain durability
6. High watermark advances
└─ Data becomes visible to consumers
7. Consumer fetches data
└─ FetchRequest to broker
8. Consumer receives records
└─ Deserializes and processes
Visual Architecture
┌─────────────────────────────────────────────────────────────┐
│ PRODUCER │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Serializer │→ │ Partitioner│→ │RecordBatch │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└─────────────────────────┬───────────────────────────────────┘
│ ProduceRequest
↓
┌─────────────────────────────────────────────────────────────┐
│ KAFKA BROKER │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ NETWORK LAYER │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Acceptor │→ │ Processor │→ Request Queue │ │
│ │ │ (port 9092) │ │ (N threads) │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ REQUEST HANDLER POOL │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ I/O Thread 1 │ │ I/O Thread N │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ LOG MANAGER │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Partition 0 │ │ Partition 1 │ ... │ │
│ │ │ (Log) │ │ (Log) │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ REPLICATION │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Leader │→ │ Follower │ │ │
│ │ │ (this broker)│ │ (other broker│ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────┬───────────────────────────────────┘
│ FetchRequest
↓
┌─────────────────────────────────────────────────────────────┐
│ CONSUMER │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Fetcher │→ │Deserializer│→ │ Application│ │
│ └────────────┘ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────┘
2. Broker Internal Architecture
Three Core Subsystems
1. NETWORK LAYER
└─ Handles TCP connections and protocol parsing
2. REQUEST PROCESSING LAYER
└─ Executes business logic (produce, fetch, metadata)
3. STORAGE LAYER
└─ Manages partition logs on disk
Network Layer Details
COMPONENTS:
Acceptor Thread:
├─ Binds to configured port (default: 9092)
├─ Accepts new client connections
└─ Assigns connections to processor threads
Processor Threads (num.network.threads):
├─ Read requests from sockets
├─ Parse request protocol
├─ Queue requests for processing
└─ Send responses back to clients
Request Queue:
├─ Bounded queue between network and I/O threads
└─ Backpressure mechanism if handlers are slow
Flow:
Client Connection
↓
Acceptor Thread (1 thread)
↓
Processor Thread (3-8 threads)
↓ (enqueue)
Request Queue (in-memory)
↓ (dequeue)
I/O Handler Thread (8-16 threads)
Request Processing Layer
HANDLER THREADS (num.io.threads):
Responsibilities:
├─ Dequeue requests from request queue
├─ Execute request logic:
│ ├─ ProduceRequest → write to log
│ ├─ FetchRequest → read from log
│ ├─ MetadataRequest → return cluster metadata
│ └─ OffsetCommitRequest → update consumer offsets
├─ Interact with log manager
└─ Enqueue response for network layer
Thread Pool Size:
└─ Typically 8-16 threads
└─ More threads = higher throughput (up to a point)
Storage Layer (Log Manager)
LOG MANAGER:
Manages all partition logs for broker:
├─ One log per partition replica hosted on this broker
├─ Each log = sequence of segment files
└─ Coordinates:
├─ Writes (appends)
├─ Reads (fetch)
├─ Log rolling (new segments)
├─ Log cleanup (retention/compaction)
└─ Index management
Partition Log:
└─ Append-only structure
└─ Segments rotated based on size or time
└─ Indexes for fast offset lookup
3. Producer Request Handling
ProduceRequest Internal Flow
STEP-BY-STEP:
1. ProduceRequest arrives at broker
└─ Contains: topic, partition, records, acks level
2. Network thread parses request
└─ Validates protocol
└─ Enqueues for I/O thread
3. I/O thread dequeues request
└─ Authorization check (if security enabled)
└─ Partition validation
4. Request routed to Log Manager
└─ Log.append(records)
5. Log Manager writes to active segment
├─ Assigns offsets sequentially
├─ Writes to OS page cache
└─ fsync (if required by config)
6. If acks=all, wait for ISR replicas
└─ Followers fetch data
└─ Leader tracks acknowledgments
7. Response sent back to producer
└─ Success: offset of first record
└─ Error: error code
Offset Assignment
INTERNAL MECHANISM:
Each partition maintains:
└─ Log End Offset (LEO): Next offset to be written
When new records arrive:
1. Current LEO: 1000
2. Batch has 10 records
3. Assign offsets: 1000, 1001, ..., 1009
4. Append to log
5. New LEO: 1010
Offsets are:
├─ Sequential (per partition)
├─ Immutable (never change)
├─ Unique (within partition)
└─ Dense (no gaps in normal operation)
Acknowledgment Levels (Internal Behavior)
acks=0 (No acknowledgment):
└─ Broker accepts request
└─ Immediately returns success
└─ No guarantee data is written
acks=1 (Leader acknowledgment):
└─ Broker writes to leader's log
└─ Returns success after leader write
└─ No guarantee of replication
acks=all (ISR acknowledgment):
└─ Broker writes to leader's log
└─ Waits for all ISR replicas to fetch
└─ Returns success after ISR acknowledgment
└─ Highest durability guarantee
4. Consumer Fetch Request Handling
FetchRequest Internal Flow
STEP-BY-STEP:
1. FetchRequest arrives at broker
└─ Contains: topic, partition, offset, max bytes
2. Network thread parses request
└─ Enqueues for I/O thread
3. I/O thread dequeues request
└─ Validates: Is this broker the leader for partition?
4. Check offset vs High Watermark
├─ If offset > HW: Wait or return empty
└─ Only committed data is visible
5. Log Manager reads from log
├─ Uses offset index to find segment
├─ Reads from segment file
└─ Applies max.bytes limit
6. Zero-copy optimization (if possible)
└─ sendfile() system call
└─ Data goes directly from disk to network
└─ No intermediate buffer copies
7. Response sent back to consumer
└─ Records + metadata (HW, next offset)
High Watermark Visibility
KEY CONCEPT:
High Watermark (HW):
└─ Last offset fully replicated to all ISR replicas
└─ Only records up to HW are visible to consumers
Example:
├─ Leader LEO: 1010
├─ Follower 1 LEO: 1008
├─ Follower 2 LEO: 1009
└─ High Watermark: 1008 (minimum of ISR LEOs)
Consumer fetch at offset 1005:
└─ Can read up to offset 1007
└─ Records 1008, 1009, 1010 are NOT visible yet
└─ (Not yet replicated to all ISR members)
Zero-Copy Transfer
TRADITIONAL COPY:
Disk → OS Buffer → Kafka Buffer → Socket Buffer → Network
└─ 4 copies, 2 context switches
ZERO-COPY (sendfile):
Disk → OS Buffer → Network
└─ 1 copy, 1 context switch
└─ 10x faster for large sequential reads
Enabled when:
├─ SSL/TLS is NOT enabled (encrypts in user space)
└─ Data is not compressed differently than stored
5. Data Plane vs Control Plane
Separation of Concerns
DATA PLANE:
└─ Handles client requests (produce, fetch)
└─ High throughput, low latency critical
└─ Bulk of broker traffic
CONTROL PLANE:
└─ Handles cluster coordination
└─ Leader elections, metadata updates
└─ Lower throughput, consistency critical
Data Plane Operations
CLIENT-FACING REQUESTS:
ProduceRequest:
└─ Write records to partition logs
FetchRequest:
└─ Read records from partition logs (consumers & followers)
OffsetFetch/OffsetCommit:
└─ Consumer offset management
MetadataRequest:
└─ Topic/partition/broker discovery
ListOffsets:
└─ Find offsets by timestamp
Control Plane Operations
CLUSTER COORDINATION:
Controller → Broker:
├─ LeaderAndIsrRequest (partition leadership changes)
├─ StopReplicaRequest (remove replica)
└─ UpdateMetadataRequest (cluster state sync)
Broker → Controller:
├─ BrokerHeartbeat (liveness)
└─ AlterPartitionRequest (ISR changes)
Broker → ZooKeeper/KRaft:
└─ Metadata reads/writes
Why Separation Matters
BENEFITS:
1. Resource Isolation:
└─ Control plane issues don't block data plane
2. Different Performance Profiles:
├─ Data plane: Throughput optimized
└─ Control plane: Consistency optimized
3. Different Failure Modes:
├─ Data plane: Can continue if metadata is stale
└─ Control plane: Needs strong consistency
6. Request Pipeline Architecture
Threading Model
KAFKA BROKER THREADS:
1. Acceptor (1 thread):
└─ Accept new connections
2. Processor (num.network.threads):
└─ Read/write from sockets
└─ Default: 3-8 threads
3. Request Handler (num.io.threads):
└─ Process requests (produce, fetch, etc.)
└─ Default: 8-16 threads
4. Log Cleaner (log.cleaner.threads):
└─ Compact logs
└─ Default: 1 thread
5. Replication Fetcher (num.replica.fetchers):
└─ Fetch from leaders (follower side)
└─ Default: 1 thread
6. Background (background.threads):
└─ Miscellaneous tasks
└─ Default: 10 threads
Request Queue Mechanics
BOUNDED QUEUE:
RequestQueue (size: queued.max.requests):
├─ Producers: Processor threads enqueue
├─ Consumers: I/O threads dequeue
└─ Backpressure: If full, network threads block
Flow control:
├─ Full queue → producers experience increased latency
├─ Empty queue → I/O threads idle
└─ Balanced: Queue depth 10-50% of max
Request Priority (Optional)
SOME KAFKA VERSIONS SUPPORT PRIORITY:
High Priority:
└─ Controller requests (LeaderAndIsr, etc.)
└─ Ensures control plane works even under load
Normal Priority:
└─ Client requests (produce, fetch)
Implementation:
└─ Separate queues or priority queue
7. Partition Routing
How Producers Find the Right Broker
METADATA DISCOVERY:
1. Producer connects to any broker (bootstrap server)
└─ MetadataRequest
2. Broker returns cluster metadata:
├─ All brokers (id, host, port)
├─ All topics and partitions
└─ Leader for each partition
3. Producer caches metadata
└─ Knows which broker hosts leader for partition X
4. Producer sends ProduceRequest to correct broker
└─ Directly to leader (no proxy)
5. Metadata refresh on errors
└─ NOT_LEADER_FOR_PARTITION error triggers refresh
Leader-Only Writes
CRITICAL RULE:
All writes go to the partition leader:
├─ Leader assigns offsets
├─ Leader appends to log
└─ Leader coordinates replication
Followers are read-only (internally):
└─ Followers only fetch from leader
└─ Followers do NOT accept client writes
Why?
└─ Ensures single source of truth
└─ Simplifies consistency model
8. Replication Flow (High-Level)
Follower Fetch Loop
FOLLOWER BEHAVIOR:
While (true) {
1. Send FetchRequest to leader
└─ Includes current offset
2. Leader responds with new records
└─ If any available
3. Follower appends to local log
└─ Same offsets as leader
4. Follower sends next FetchRequest
└─ With updated offset
}
Key Points:
├─ Follower pulls (not pushed)
├─ Fetch includes current offset (for leader tracking)
└─ Leader tracks follower progress
ISR Membership
IN-SYNC REPLICA SET (ISR):
Leader maintains ISR set:
├─ All replicas that are "caught up"
├─ "Caught up" = within replica.lag.time.max.ms
If follower falls behind:
├─ Leader removes from ISR
└─ ISR shrinks
When follower catches up:
├─ Leader adds back to ISR
└─ ISR expands
ISR impacts:
└─ acks=all waits for all ISR members
└─ Smaller ISR = faster acks (but less durability)
9. Broker Startup Sequence
Internal Initialization
BROKER STARTUP PHASES:
1. Load Configuration
└─ server.properties
└─ Validate settings
2. Initialize Log Manager
├─ Scan log.dirs
├─ Load partition logs
├─ Recover unclean shutdown
└─ Build offset indexes
3. Register with ZooKeeper/KRaft
├─ Broker ID registration
├─ Create ephemeral node (ZK)
└─ Join controller quorum (KRaft)
4. Fetch Metadata from Controller
└─ Partition assignments
└─ Leader/follower roles
5. Start Network Layer
└─ Bind to listeners
└─ Start acceptor and processor threads
6. Start Replication Threads
└─ For partitions where broker is follower
7. Mark Broker as Alive
└─ Ready to serve traffic
10. Key Internal Invariants
Core Guarantees
KAFKA INTERNAL RULES:
1. Offsets are sequential per partition
└─ No gaps (in normal operation)
2. Leader assigns offsets
└─ Followers replicate exact same offsets
3. High watermark never decreases
└─ (except on unclean leader election)
4. Data up to HW is immutable
└─ Safe to read by consumers
5. Leader election preserves committed data
└─ New leader must have all HW data
6. Replicas diverge → truncation happens
└─ Follower truncates to leader's HW
11. Why This Architecture?
Design Decisions
APPEND-ONLY LOG:
└─ Sequential writes are fast (even on HDD)
└─ Simplifies replication (just copy log)
└─ Immutability simplifies consistency
PULL-BASED CONSUMPTION:
└─ Consumer controls pace (backpressure)
└─ Broker doesn't track consumer state (stateless)
└─ Multiple consumers at different speeds
LEADER-BASED REPLICATION:
└─ Single writer (leader) per partition
└─ No distributed consensus per write
└─ Fast writes, eventually consistent reads
ZERO-COPY:
└─ Leverage OS page cache
└─ Avoid user-space copies
└─ Kernel handles disk → network
12. Performance Characteristics
Why Kafka is Fast
1. SEQUENTIAL DISK I/O:
└─ Append-only log = sequential writes
└─ Sequential reads for consumers
└─ Modern disks excel at sequential I/O
2. BATCH PROCESSING:
├─ Producers batch records
├─ Brokers write batches
└─ Consumers fetch batches
└─ Amortizes overhead
3. ZERO-COPY:
└─ sendfile() system call
└─ No user-space buffer copies
4. PAGE CACHE:
└─ OS caches recently written data
└─ Reads often served from memory
└─ No need for large JVM heap
5. COMPRESSION:
└─ Stored compressed on disk
└─ Transferred compressed over network
└─ Reduces I/O and bandwidth
13. Key Takeaways
You've learned:
✅ Complete message journey from producer to consumer through broker internals
✅ Network layer handles connections and protocol parsing
✅ I/O threads execute request logic (produce, fetch)
✅ Log Manager appends to partition logs on disk
✅ Offsets are assigned sequentially by the partition leader
✅ High Watermark determines consumer visibility
✅ Replication is pull-based (followers fetch from leader)
✅ Data plane vs control plane separate client traffic from coordination
✅ Zero-copy optimization for fast data transfer
✅ Sequential I/O is the foundation of Kafka's performance
14. Architecture Checklist
BROKER INTERNAL COMPONENTS:
☑ Network Layer (Acceptor, Processors)
☑ Request Queue (bounded, backpressure)
☑ I/O Threads (request handlers)
☑ Log Manager (partition logs)
☑ Replication Manager (ISR tracking)
☑ Controller Client (leadership updates)
REQUEST FLOW:
☑ Producer → Network → Queue → I/O → Log → Disk
☑ Consumer ← Network ← I/O ← Log ← Disk (or cache)
☑ Follower → Fetch → Leader → Replicate → Local Log
PERFORMANCE OPTIMIZATIONS:
☑ Sequential disk I/O
☑ Batching (producer, broker, consumer)
☑ Zero-copy (sendfile)
☑ OS page cache utilization
☑ Compression
Summary
Kafka's internal architecture prioritizes:
Core Design =
├─ Append-only log (simple, fast, durable)
├─ Leader-based replication (no consensus per write)
├─ Pull-based consumption (consumer controls pace)
├─ Zero-copy transfers (leverage kernel)
└─ Sequential I/O (disk-friendly)
Request Pipeline =
├─ Network threads (parse protocol)
├─ Request queue (buffer, backpressure)
├─ I/O threads (execute logic)
└─ Log Manager (manage storage)
Performance =
├─ Batching everywhere
├─ OS page cache as primary cache
├─ Compression in flight and at rest
└─ Minimal data copies
Remember: Kafka's architecture makes trade-offs favoring throughput, simplicity, and durability. Understanding these internals explains why Kafka behaves the way it does under load, failure, and scale.
Next up: Module 2.1 — Topic & Partition Internals — Deep dive into partition structure and offset mechanics.