1.2 – Event Streaming and Kafka Architecture

Module: M1 – Kafka Fundamentals
Goal: Understand event-driven architecture and Kafka's high-level system design

Event-Driven Architecture

What Is Event-Driven Architecture?

Event-Driven Architecture (EDA) = Systems communicate by producing and consuming events, rather than making direct API calls.

Traditional Request-Response:

Service A → (HTTP POST) → Service B → Response → Service A

Event-Driven:

Service A → Event (Order Created) → Event Bus
                                       ↓
                           Service B reads event (async)

Key Difference:

• Request-Response: Synchronous, tightly coupled, blocking
• Event-Driven: Asynchronous, decoupled, non-blocking

Benefits of Event-Driven Architecture

Example: E-commerce Order Flow

Traditional (Synchronous):

User clicks "Buy"
  ↓
Order Service calls Payment Service (waits)
  ↓
Payment Service calls Inventory Service (waits)
  ↓
Inventory Service calls Shipping Service (waits)
  ↓
Response to user (5 seconds later)

Problem: If Shipping Service is slow, entire flow is slow.

Event-Driven (Asynchronous):

User clicks "Buy"
  ↓
Order Service → Kafka Topic (order-created event)
  ↓
Response to user (200ms, "Order received!")

Meanwhile:
  Payment Service reads event → processes payment → publishes (payment-completed)
  Inventory Service reads event → reserves stock → publishes (stock-reserved)
  Shipping Service reads event → creates shipment → publishes (shipment-created)

Benefit: User gets instant response. Services process events independently.

Pub/Sub vs Log Model

Traditional Pub/Sub (e.g., Google Pub/Sub, AWS SNS)

Model: Fire-and-forget

Publisher → Topic → Subscriber 1 (gets message)
                  → Subscriber 2 (gets message)

Behavior:

• Messages are delivered once to each subscriber
• Messages are deleted after delivery
• No replay: Can't re-read old messages

Use Case: Notifications, alerts (one-time delivery)

Kafka's Log Model

Model: Persistent, ordered log

Producer → Topic (log: [msg1, msg2, msg3, ...]) → Consumer 1 (reads from offset 0)
                                                 → Consumer 2 (reads from offset 2)

Behavior:

• Events are stored durably (not deleted immediately)
• Events are ordered (append-only log)
• Replay: Consumers can re-read from any offset
• Multiple consumers: Independent reads (each tracks own offset)

Use Case: Event streaming, data pipelines, state reconstruction

Pub/Sub vs Log: Comparison

Key Insight: Kafka is a distributed commit log, not a traditional pub/sub system.

Kafka Cluster Components (High-Level)

Kafka is a distributed system made up of multiple components.

The Four Main Components

┌─────────────────────────────────────────────────┐
│              Kafka Cluster                      │
│                                                 │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐        │
│  │ Broker 1│  │ Broker 2│  │ Broker 3│        │
│  │         │  │         │  │         │        │
│  │ Topics  │  │ Topics  │  │ Topics  │        │
│  │ Data    │  │ Data    │  │ Data    │        │
│  └─────────┘  └─────────┘  └─────────┘        │
│                                                 │
└─────────────────────────────────────────────────┘
         ↑                           ↓
    ┌────────┐                  ┌──────────┐
    │Producer│                  │ Consumer │
    └────────┘                  └──────────┘

1. Producers

What They Are: Applications that write events to Kafka.

Examples:

• Web server logging user clicks
• Microservice publishing order events
• Sensor sending temperature readings

Key Point: Producers append events to topics (write-only).

2. Brokers

What They Are: Servers that store events and serve them to consumers.

Key Responsibilities:

• Store events on disk
• Replicate events for fault tolerance
• Serve read requests from consumers

Key Point: A Kafka cluster is made up of multiple brokers (e.g., 3-10 brokers).

Example:

Broker 1: Stores partitions 0, 3, 6
Broker 2: Stores partitions 1, 4, 7
Broker 3: Stores partitions 2, 5, 8

3. Topics

What They Are: Logical categories or channels for events.

Analogy: Topics are like folders in a file system.

Examples:

• orders (all order events)
• payments (all payment events)
• user-activity (all user clicks, views)

Key Point: Topics are divided into partitions for scalability.

4. Consumers

What They Are: Applications that read events from Kafka.

Examples:

• Analytics service reading user clicks
• Fraud detection reading payment events
• Notification service reading order events

Key Point: Consumers poll for events (pull-based, not push-based).

Data Flow Overview

High-Level Flow

1. Producer writes event to topic "orders"
   ↓
2. Kafka stores event in partition 0 (on Broker 1)
   ↓
3. Event is replicated to Broker 2 (for fault tolerance)
   ↓
4. Consumer reads event from partition 0
   ↓
5. Consumer commits offset (marks event as read)

Key Concepts:

• Partition: Topic is divided into partitions (e.g., orders has 10 partitions)
• Offset: Position of an event in a partition (e.g., event at offset 42)
• Replication: Events are copied to multiple brokers (for durability)

Detailed Flow: Producer to Consumer

Step 1: Producer Sends Event

producer.send(new ProducerRecord<>("orders", "key123", "order data"));

What Happens:

1. Producer serializes event (converts to bytes)
2. Producer determines which partition to write to (based on key)
3. Producer sends event to the leader broker for that partition

Step 2: Broker Stores Event

Broker 1 (Leader for Partition 0):
  - Appends event to log file (on disk)
  - Assigns offset (e.g., offset 42)
  - Replicates event to follower brokers (Broker 2, Broker 3)

What Happens:

1. Event is written to disk (durable)
2. Event is replicated to followers (fault tolerance)
3. Acknowledgment is sent to producer ("event saved")

Step 3: Consumer Reads Event

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

What Happens:

1. Consumer sends fetch request to leader broker
2. Broker returns events starting from consumer's last offset (e.g., offset 40)
3. Consumer processes events (e.g., save to database, send notification)
4. Consumer commits offset (e.g., "I've read up to offset 45")

End-to-End Message Journey (Conceptual)

Let's trace a single event through Kafka.

Scenario: User places an order for $99.

Step 1: Producer Creates Event

// Order Service (Producer)
OrderEvent event = new OrderEvent(
  orderId: "12345",
  userId: "user-789",
  amount: 99.00,
  timestamp: "2023-01-15T10:30:00Z"
);

producer.send(new ProducerRecord<>("orders", event.userId, event));

Key Point: Event has a key (userId) and a value (order data).

Step 2: Kafka Determines Partition

Kafka uses the key to determine which partition:

hash(key) % num_partitions = partition

hash("user-789") % 10 = 3

Result: Event goes to partition 3 of topic orders.

Step 3: Leader Broker Receives Event

Broker 2 (Leader for Partition 3):
  - Receives event from producer
  - Appends event to log:
    Offset 0: {orderId: "12340", ...}
    Offset 1: {orderId: "12341", ...}
    Offset 2: {orderId: "12344", ...}
    Offset 3: {orderId: "12345", amount: 99.00}  ← New event

Key Point: Event is assigned offset 3 (next available offset).

Step 4: Event Is Replicated

Broker 2 (Leader) → Replicates to Broker 1 (Follower)
                  → Replicates to Broker 3 (Follower)

Result: Event is stored on 3 brokers (leader + 2 followers).

Why?: If Broker 2 crashes, Broker 1 or Broker 3 can take over.

Step 5: Producer Receives Acknowledgment

Producer: "Event saved! Offset: 3"

What This Means: Event is safely stored (replicated to 3 brokers).

Step 6: Consumer Reads Event

// Payment Service (Consumer)
ConsumerRecords<String, OrderEvent> records = consumer.poll(Duration.ofMillis(100));

for (ConsumerRecord<String, OrderEvent> record : records) {
  OrderEvent event = record.value();
  processPayment(event);  // Charge $99.00
}

What Happens:

1. Consumer polls Broker 2 (leader for partition 3)
2. Broker returns event at offset 3
3. Consumer processes event (charges payment)
4. Consumer commits offset 4 ("I've read up to offset 3")

Step 7: Another Consumer Reads Same Event

// Notification Service (Consumer)
ConsumerRecords<String, OrderEvent> records = consumer.poll(Duration.ofMillis(100));

for (ConsumerRecord<String, OrderEvent> record : records) {
  OrderEvent event = record.value();
  sendNotification(event);  // Send "Order confirmed" email
}

Key Point: Both Payment Service and Notification Service read the same event.

Why This Works: Each consumer tracks its own offset independently.

Partition 3:
  Offset 0: {orderId: "12340", ...}
  Offset 1: {orderId: "12341", ...}
  Offset 2: {orderId: "12344", ...}
  Offset 3: {orderId: "12345", amount: 99.00}
           ↑                    ↑
    Payment Service         Notification Service
    (offset 3)              (offset 3)

Kafka Architecture Principles

1. Distributed System

Kafka runs on multiple servers (brokers) for:

• Scalability: More brokers = more storage and throughput
• Fault tolerance: If one broker fails, others take over

Example Cluster:

Broker 1 (10.0.0.1)
Broker 2 (10.0.0.2)
Broker 3 (10.0.0.3)

2. Append-Only Log

Events are never updated or deleted (immediately).

Log Structure:

Offset 0: Event A
Offset 1: Event B
Offset 2: Event C
Offset 3: Event D  ← New events appended here

Benefits:

• Fast writes: Sequential disk writes (very fast)
• Immutability: Events can't be changed (audit trail)
• Replay: Re-read old events

3. Partitioning for Scalability

Topics are divided into partitions.

Example: Topic orders with 10 partitions.

orders-0 (Broker 1)
orders-1 (Broker 2)
orders-2 (Broker 3)
orders-3 (Broker 1)
...
orders-9 (Broker 3)

Benefits:

• Parallel writes: Producers write to multiple partitions simultaneously
• Parallel reads: Consumers read from multiple partitions simultaneously
• Scalability: Add more partitions to handle more traffic

4. Replication for Fault Tolerance

Each partition is replicated to multiple brokers.

Example: Partition orders-0 with replication factor 3.

orders-0 (Leader: Broker 1)
orders-0 (Follower: Broker 2)
orders-0 (Follower: Broker 3)

Benefits:

• Durability: No data loss if one broker fails
• High availability: If leader fails, follower becomes leader

5. Consumer Groups for Scalability

Multiple consumers can work together to read a topic.

Example: Consumer group payment-processors with 3 consumers.

Consumer 1 → Reads partitions 0, 1, 2
Consumer 2 → Reads partitions 3, 4, 5
Consumer 3 → Reads partitions 6, 7, 8

Benefit: Parallel processing (3x faster than single consumer).

Kafka vs Traditional Architecture

Traditional Architecture (Without Kafka)

Service A → (HTTP POST) → Service B
Service A → (HTTP POST) → Service C
Service A → (HTTP POST) → Service D

Problems:
- Tight coupling (A must know B, C, D)
- If B is down, A fails
- No replay (if D crashes, data is lost)

Event-Driven Architecture (With Kafka)

Service A → Kafka Topic → Service B (reads independently)
                        → Service C (reads independently)
                        → Service D (reads independently)

Benefits:
- Loose coupling (A only knows Kafka)
- If B is down, events wait in Kafka
- Replay (D can re-read after crash)

Why Kafka is Fast and Durable

Kafka achieves exceptional performance (millions of messages/sec) and durability (no data loss) through several key architectural optimizations:

Performance Optimizations

1. Sequential Disk Writes

How It Works: Kafka appends messages to the end of partition logs, writing sequentially to disk rather than seeking random locations.

Why It's Fast: Modern disk drives achieve 100+ MB/sec for sequential writes, nearly as fast as RAM writes. Random writes, by contrast, are 100-1000x slower due to disk seek time.

Sequential Write:    [msg1][msg2][msg3][msg4]  →  Fast (no disk seeking)
Random Write:        [msg1]....[msg3]....[msg2]  →  Slow (constant seeking)

Impact: Kafka can write millions of messages per second on commodity hardware.

2. Zero-Copy Transfer

How It Works: Kafka uses the operating system's sendfile() system call to transfer data directly from disk to network socket without copying through application memory.

Traditional Approach (4 copies):

Disk → Kernel Buffer → Application Buffer → Socket Buffer → Network Card

Kafka's Zero-Copy (2 copies):

Disk → Kernel Buffer → Network Card (DMA)

Impact: Reduces CPU usage by 50-70% and doubles throughput for consumers reading data.

3. Batching and Compression

Batching: Producers batch multiple messages before sending to Kafka, amortizing network overhead.

Configuration:

batch.size=16384        # 16 KB batch size
linger.ms=10            # Wait 10ms to fill batch

Benefit: Sending 100 messages in one batch is 10-100x faster than sending 100 individual messages.

Compression: Kafka compresses batches using Snappy, LZ4, Zstd, or Gzip, reducing:

• Network bandwidth (50-80% reduction typical)
• Disk storage (same compression ratio)

Example:

Uncompressed batch: 100 KB
Compressed (Snappy): 30 KB  → 70% reduction

Impact: 3-5x increase in effective throughput for text/JSON data.

4. Efficient Network Utilization

Pipelining: Kafka allows multiple in-flight requests (producer and follower replicas), maximizing network utilization.

No Acknowledgment Per Message: Producers wait for batch acknowledgment, not per-message acks, reducing protocol overhead.

5. Asynchronous Replication

How It Works: Follower replicas replicate from the leader asynchronously (pull-based), without blocking the leader's write path.

Producer → Leader (writes to log, returns ack)
           ↓ (asynchronous)
         Follower 1 (fetches and replicates)
         Follower 2 (fetches and replicates)

Impact: Leader can accept writes at full speed while replication happens in parallel.

Durability Features

1. Write-Ahead Log

How It Works: Kafka writes messages to disk before acknowledging success to the producer.

Configuration:

acks=all  # Wait for leader + all ISR replicas to acknowledge

Guarantee: Once acknowledged, messages are durable (even if broker crashes immediately after).

2. Replication and In-Sync Replicas (ISR)

How It Works: Each partition is replicated to multiple brokers (typically RF=3). Only replicas that are fully caught up (ISR) can become leaders.

Partition 0:
  - Leader: Broker 1 (latest offset: 1000)
  - Follower: Broker 2 (offset: 1000, in ISR)
  - Follower: Broker 3 (offset: 1000, in ISR)

Guarantee: If the leader fails, one of the ISR replicas (with all data) becomes the new leader. No data loss.

3. Configurable Durability vs Latency Tradeoff

Kafka lets you choose your durability guarantee:

Production Recommendation: Always use acks=all for critical data.

4. Retention Policies

Time-Based Retention: Keep events for N days/hours (default: 7 days).

retention.ms=604800000  # 7 days

Size-Based Retention: Keep events until partition size exceeds N GB.

retention.bytes=1073741824  # 1 GB

Log Compaction: Keep only the latest value per key (for state/changelog topics).

cleanup.policy=compact

Guarantee: Events are never deleted before retention period, allowing replay and recovery.

Why These Features Matter

For Producers:

• Sequential writes + batching + compression = millions of messages/sec
• Configurable durability (choose latency vs guarantee tradeoff)

For Consumers:

• Zero-copy + sequential reads = millions of messages/sec
• Independent consumer groups (no contention)

For Operations:

• Replication + ISR = fault tolerance (no data loss on broker failure)
• Retention = replay capability (debug bugs, recover from errors)

Performance Benchmark Example:

Hardware: 3 brokers, commodity servers (12 cores, 64 GB RAM, SSD)

Throughput:

• Writes: 1 million messages/sec (10 KB messages)
• Reads: 2 million messages/sec (zero-copy advantage)
• Latency: 5ms (p99) for acks=all

Key Insight: Kafka's performance comes from minimizing disk seeks, reducing copies, and maximizing batch processing — not from expensive hardware.

Key Takeaways

1. Event-Driven Architecture: Systems communicate via events, not direct API calls.

2. Kafka Is a Log: Kafka stores events in an ordered, append-only log (not a queue).

3. Four Main Components: Producers (write), Brokers (store), Topics (organize), Consumers (read).

4. Data Flow: Producer → Partition → Replication → Consumer → Offset commit.

5. Distributed System: Multiple brokers for scalability and fault tolerance.

6. Partitioning: Topics are divided into partitions for parallel processing.

7. Replication: Partitions are replicated for durability.

8. Decoupling: Producers and consumers don't know about each other (only Kafka).

What's Next?

Next Module: 1.3 – Core Concepts and Building Blocks

What We Covered:

• ✅ Event-driven architecture
• ✅ Pub/sub vs log model
• ✅ Kafka cluster components (high-level)
• ✅ Producers, brokers, topics, consumers
• ✅ Data flow overview
• ✅ End-to-end message journey

What We Didn't Cover (Coming in Later Modules):

• ❌ Topics and partitions deep dive → M2: Brokers & Topics
• ❌ Producer internals → M3: Producers
• ❌ Consumer internals → M4: Consumers
• ❌ Replication mechanics → M7: Kafka Internals

Event Streaming and Kafka Architecture: Complete ✅