1.1 Introduction to Kafka Streams & Processing Model

Module Overview

Goal: Build the mental model of stream processing before APIs.

What You'll Learn:

What Kafka Streams is (library, not a cluster)
Stream processing vs batch processing
Record-at-a-time processing paradigm
Relation to Kafka topics and partitions
High-level lifecycle of a Streams application
When to use Kafka Streams

What's NOT Covered Here:

DSL operations (covered in Modules 4 and 5)
State stores (covered in Module 6)
Exactly-once semantics (covered in Module 9)
Specific APIs and code (covered in Module 3)

Key Insight: After this module, you'll understand what Kafka Streams does — not how to code it. That comes later.

1. What is Kafka Streams?

Definition

Kafka Streams is a client library for building real-time stream processing applications on top of Apache Kafka.

Kafka Streams is:
├─ A Java/Scala library
├─ Embedded in your application
├─ Reads from Kafka topics
├─ Processes records in real-time
└─ Writes back to Kafka topics

What Kafka Streams is NOT

❌ NOT a cluster (like Kafka itself)
❌ NOT a separate service (like Spark)
❌ NOT a broker component
❌ NOT a heavyweight framework

Key Point: Kafka Streams runs inside your application as a library dependency. You deploy it like any other JVM application.

Kafka Streams vs Kafka Connect vs Kafka Consumer API

Before diving deeper, let's clarify how Kafka Streams differs from other Kafka tools:

┌─────────────────┬───────────────────────────────┬─────────────────────────────┬──────────────────────────────┐
│ Feature         │ Kafka Streams                 │ Kafka Connect               │ Kafka Consumer API           │
├─────────────────┼───────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ Purpose         │ Real-time process/transform   │ Moving data in/out of Kafka │ Consuming messages manually  │
│                 │ data                          │                             │ from topics                  │
├─────────────────┼───────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ Infrastructure  │ None (library inside app)     │ Needs Kafka Connect cluster │ None (library inside app)    │
│ Needed          │                               │                             │                              │
├─────────────────┼───────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ Use Case        │ Aggregate sales by region     │ Pull DB data into Kafka     │ Build a custom consumer      │
│ Example         │ live                          │                             │                              │
├─────────────────┼───────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ Complexity      │ Medium (DSL + state mgmt)     │ Low (configuration-based)   │ Low to High (you write all)  │
├─────────────────┼───────────────────────────────┼─────────────────────────────┼──────────────────────────────┤
│ State           │ Built-in state management     │ Stateless                   │ You manage state manually    │
│ Management      │                               │                             │                              │
└─────────────────┴───────────────────────────────┴─────────────────────────────┴──────────────────────────────┘

When to use each:

Kafka Streams: Real-time computation on streaming data (filtering, aggregations, joins, windowing)
Kafka Connect: Moving data between Kafka and external systems (databases, S3, Elasticsearch)
Kafka Consumer API: Building custom consumers with specific logic not covered by Streams or Connect

Comparison: Library vs Cluster

Traditional Stream Processing (Flink, Spark Streaming):
┌──────────────────────────────────────────┐
│     Separate Cluster                     │
│  ┌─────┐  ┌─────┐  ┌─────┐              │
│  │Node1│  │Node2│  │Node3│              │
│  └─────┘  └─────┘  └─────┘              │
│     ↑        ↑        ↑                  │
│     └────────┴────────┘                  │
│    Your app submits jobs here            │
└──────────────────────────────────────────┘

Kafka Streams (Library):
┌──────────────────────────────────────────┐
│  Your Application (JAR)                  │
│  ┌────────────────────────────────────┐  │
│  │  Business Logic                    │  │
│  │  + Kafka Streams Library           │  │
│  └────────────────────────────────────┘  │
│     ↓ reads from / writes to ↓          │
│        Kafka Cluster                     │
└──────────────────────────────────────────┘

Implications:

✅ No separate cluster to manage
✅ Deploy as microservice (Docker, K8s, etc.)
✅ Simpler operations
⚠️ Scaling = deploying more app instances

2. Stream Processing vs Batch Processing

The Paradigm Difference

BATCH PROCESSING:
├─ Collect data for time period
├─ Process entire dataset at once
├─ Wait for results
└─ Repeat

STREAM PROCESSING:
├─ Process data as it arrives
├─ Continuous, never-ending
├─ Results available immediately
└─ Always running

Visual Comparison

Batch Processing:
┌─────────────────────────────────────────────┐
│ Day 1 Data                                  │
│ [Record 1] [Record 2] ... [Record 1000]    │
└─────────────────────────────────────────────┘
            ↓ (process at end of day)
      [Batch Job Runs]
            ↓
      Results at 11:59 PM


Stream Processing:
[Record 1] → [Process] → [Result 1]
[Record 2] → [Process] → [Result 2]
[Record 3] → [Process] → [Result 3]
    ...         ...         ...
(continuous, real-time, never stops)

Latency Comparison

Batch Processing:
Event happens → Wait hours/days → See result
Example: Daily sales report

Stream Processing:
Event happens → Process immediately → See result
Example: Fraud detection (milliseconds)

Aspect	Batch	Stream
Latency	Hours to days	Milliseconds to seconds
Data scope	Bounded (finite)	Unbounded (infinite)
Processing model	Process once	Process continuously
Use case	Historical analysis	Real-time reactions

3. Record-at-a-Time Processing

The Core Mental Model

Kafka Streams processes one record at a time as they flow through the pipeline.

Topic: "orders"
[Order 1] → [Process] → [Transformed Order 1] → Output
[Order 2] → [Process] → [Transformed Order 2] → Output
[Order 3] → [Process] → [Transformed Order 3] → Output

Not like this (batch):

[Order 1, Order 2, Order 3] → [Process All] → [Results]

The Infinite Stream

Time →
┌────────────────────────────────────────────────→
│
│ [Event 1] [Event 2] [Event 3] ... [Event N] ... ∞
│
│ Your Kafka Streams app processes this continuous flow
│
└────────────────────────────────────────────────→

Key Concepts:

Stream Never Ends
- Kafka topics are unbounded
- Your app runs continuously
- No concept of "done"
Process as Data Arrives
- Don't wait for "all data"
- React to each record
- Low latency by design
State Accumulates Over Time
- Can maintain aggregations
- Running totals, counts, etc.
- State persists between records

4. Relation to Kafka Topics & Partitions

Input and Output

Kafka Cluster:
┌──────────────────────────────────────────────┐
│  Topic: "input-orders"                       │
│  ├─ Partition 0: [Record][Record][Record]   │
│  ├─ Partition 1: [Record][Record][Record]   │
│  └─ Partition 2: [Record][Record][Record]   │
└──────────────────────────────────────────────┘
            ↓ reads from
┌──────────────────────────────────────────────┐
│  Your Kafka Streams Application             │
│  (processes records)                         │
└──────────────────────────────────────────────┘
            ↓ writes to
┌──────────────────────────────────────────────┐
│  Topic: "output-processed-orders"            │
│  ├─ Partition 0: [Result][Result][Result]   │
│  ├─ Partition 1: [Result][Result][Result]   │
│  └─ Partition 2: [Result][Result][Result]   │
└──────────────────────────────────────────────┘

Key Points:

Kafka Streams reads from topics (just like a consumer)
Kafka Streams writes to topics (just like a producer)
It's a producer + consumer + processing logic combined

Partition-Level Parallelism

Topic with 3 partitions:

Partition 0 → [Streams Task 0] → Process independently
Partition 1 → [Streams Task 1] → Process independently
Partition 2 → [Streams Task 2] → Process independently

Each task is single-threaded
Parallelism = partition count

Critical Understanding:

Important: If you process data from a topic with 10 partitions, the number of tasks will be equal to the number of partitions in the source topic, with each task processing messages from exactly one partition.

Tasks = Partitions (1:1 relationship)

Example:
├─ Topic with 10 partitions → 10 tasks created
├─ Topic with 3 partitions  → 3 tasks created
└─ Each task handles exactly 1 partition

This means:
├─ Max parallelism = number of partitions
├─ Can't have more tasks than partitions
└─ Adding more app instances beyond partition count = idle instances

Critical Understanding:

More partitions = More parallelism
└─ Each partition processed by one task
└─ Tasks can run across multiple instances
└─ Ordering preserved within partition

Scaling Model

Scenario: 6 partitions, 2 app instances

Instance 1:
├─ Task 0 → Partition 0
├─ Task 1 → Partition 1
└─ Task 2 → Partition 2

Instance 2:
├─ Task 3 → Partition 3
├─ Task 4 → Partition 4
└─ Task 5 → Partition 5

Each instance processes 3 partitions

Scaling Rules:

Max parallelism = number of input topic partitions
Adding more instances than partitions = idle instances
Kafka Streams handles task distribution automatically

5. High-Level Lifecycle of a Streams Application

The Five Phases

1. DEFINE TOPOLOGY
   ├─ Describe processing logic
   ├─ Input topics
   ├─ Transformations
   └─ Output topics

2. BUILD APPLICATION
   ├─ Create KafkaStreams instance
   ├─ Configure properties
   └─ Wire everything together

3. START PROCESSING
   ├─ Call streams.start()
   ├─ Join consumer group
   └─ Begin consuming

4. PROCESS RECORDS
   ├─ Read → Transform → Write
   ├─ Continuous loop
   └─ Runs until stopped

5. SHUT DOWN
   ├─ Graceful cleanup
   ├─ Commit state
   └─ Close resources

Conceptual Code Flow

// 1. Define Topology (what to do)
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> input = builder.stream("input-topic");

input
  .filter((key, value) -> value.contains("important"))
  .mapValues(value -> value.toUpperCase())
  .to("output-topic");

// 2. Build Application
Topology topology = builder.build();
KafkaStreams streams = new KafkaStreams(topology, props);

// 3. Start Processing
streams.start();

// 4. Processing happens continuously...
// (Kafka Streams runs in background)

// 5. Shut Down (when needed)
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Don't worry about the code yet! This is just to show the flow. We'll dive into APIs in Module 3.

State Transitions

┌─────────────────────────────────────────────┐
│  State: CREATED                             │
│  (Topology defined, not started yet)        │
└─────────────────────────────────────────────┘
            ↓ streams.start()
┌─────────────────────────────────────────────┐
│  State: REBALANCING                         │
│  (Joining consumer group, assigning tasks)  │
└─────────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────────┐
│  State: RUNNING                             │
│  (Processing records continuously)          │
└─────────────────────────────────────────────┘
            ↓ streams.close() or failure
┌─────────────────────────────────────────────┐
│  State: PENDING_SHUTDOWN                    │
│  (Cleaning up, committing state)            │
└─────────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────────┐
│  State: NOT_RUNNING                         │
│  (Application stopped)                      │
└─────────────────────────────────────────────┘

6. Data Flow Architecture

From Kafka to Kafka

┌────────────────────────────────────────────────────┐
│              KAFKA CLUSTER                         │
│                                                    │
│  Input Topics        Processing        Output     │
│  ┌──────────┐       ┌─────────┐      ┌────────┐  │
│  │ orders   │───→   │ Streams │ ───→ │results │  │
│  │ users    │───→   │   App   │ ───→ │metrics │  │
│  │ products │───→   └─────────┘ ───→ └────────┘  │
│  └──────────┘                                      │
│      ↑                                      │      │
│      │                                      ↓      │
│   Producers                            Consumers   │
│                                                    │
└────────────────────────────────────────────────────┘

Kafka Streams sits in the middle:

Consumes from input topics
Processes in real-time
Produces to output topics
Other apps can consume the results

Example: Order Processing Pipeline

Step 1: Raw Orders Arrive
Topic: "raw-orders"
[{"id": 1, "amount": 100}]
[{"id": 2, "amount": 50}]

        ↓ (Streams App 1: Validation)

Step 2: Validated Orders
Topic: "valid-orders"
[{"id": 1, "amount": 100, "valid": true}]
[{"id": 2, "amount": 50, "valid": true}]

        ↓ (Streams App 2: Enrichment)

Step 3: Enriched Orders
Topic: "enriched-orders"
[{"id": 1, "amount": 100, "customer": "Alice"}]
[{"id": 2, "amount": 50, "customer": "Bob"}]

        ↓ (Streams App 3: Aggregation)

Step 4: Order Totals
Topic: "order-totals"
[{"customer": "Alice", "total": 100}]
[{"customer": "Bob", "total": 50}]

Key Pattern: Chain of Streams applications, each reading and writing topics.

7. Processing Guarantees (High-Level)

Three Levels

Kafka Streams supports three processing guarantees:

1. At-Most-Once
   └─ Messages may be lost, never duplicated

2. At-Least-Once (default)
   └─ Messages may be duplicated, never lost

3. Exactly-Once (configurable)
   └─ Each message processed exactly one time

Note: Exactly-once semantics will be covered in detail in Module 9. For now, just know it's possible.

Fault Tolerance

Scenario: Streams app crashes

Without Kafka Streams:
├─ Lose progress
├─ Lose state
└─ Manual recovery

With Kafka Streams:
├─ State backed up to Kafka
├─ Automatic recovery
├─ Resume from last checkpoint
└─ No data loss

Key Point: Kafka Streams provides built-in fault tolerance through Kafka itself.

8. When to Use Kafka Streams

Ideal Use Cases

✅ Real-time transformations
   Example: Convert JSON to Avro

✅ Filtering and routing
   Example: Route high-value orders to special topic

✅ Aggregations and analytics
   Example: Count events by user per minute

✅ Enrichment
   Example: Join orders with user data

✅ Stateful processing
   Example: Maintain running totals

✅ Event-driven microservices
   Example: React to events, emit new events

When NOT to Use Kafka Streams

❌ Batch processing (use Spark, Hadoop)
   └─ If you need to process historical data once

❌ Complex ML pipelines (use Flink, Spark)
   └─ If you need advanced ML libraries

❌ Low-latency microsecond processing (use custom code)
   └─ Kafka Streams is milliseconds, not microseconds

❌ Non-Kafka data sources (use Flink, Spark)
   └─ Kafka Streams only reads from Kafka topics

❌ Extremely high throughput (millions/sec per partition)
   └─ Consider custom consumers

Kafka Streams vs Alternatives

Feature	Kafka Streams	Kafka Consumers	Flink	Spark Streaming
Deployment	Library (embedded)	Library (embedded)	Cluster	Cluster
State management	Built-in	Manual	Built-in	Built-in
Exactly-once	Yes	Manual	Yes	Yes
Operations	Simple	Simple	Complex	Complex
Learning curve	Moderate	Low	Steep	Steep
Best for	Kafka-centric apps	Simple consumption	Complex CEP	Batch + Stream

9. The Kafka Streams Philosophy

Core Principles

1. Kafka-Native
   └─ Built for Kafka, leverages Kafka's strengths

2. Library, Not Framework
   └─ You control the main() method

3. Elastic Scalability
   └─ Add instances = more throughput

4. Stateful by Design
   └─ State is first-class citizen

5. Fault-Tolerant
   └─ Uses Kafka for durability

6. No External Dependencies
   └─ Only needs Kafka cluster

Design Philosophy

┌──────────────────────────────────────────────┐
│  "If you have Kafka, you have all you need" │
│                                              │
│  ├─ Message transport: Kafka                │
│  ├─ State storage: Kafka                    │
│  ├─ Fault tolerance: Kafka                  │
│  └─ Coordination: Kafka                     │
│                                              │
│  No external databases, caches, or systems  │
└──────────────────────────────────────────────┘

This is powerful: Your entire stateful stream processing stack is just Kafka + your app.

10. Mental Model Summary

Think of Kafka Streams as...

A library that:
├─ Reads from Kafka topics (like a consumer)
├─ Processes records one-at-a-time
├─ Can maintain state (aggregations, joins)
├─ Writes back to Kafka topics (like a producer)
├─ Scales by adding more instances
└─ Handles failures automatically

The Processing Model

CONTINUOUS LOOP:

1. Poll record from input topic
2. Process it (transform, aggregate, etc.)
3. Update state (if needed)
4. Produce result to output topic
5. Commit progress
6. Go to step 1

This happens for every record, forever.

Key Characteristics

REAL-TIME:
└─ Process as data arrives, not in batches

STATEFUL:
└─ Can remember things across records

FAULT-TOLERANT:
└─ Crashes don't lose data or state

SCALABLE:
└─ Add instances to process more partitions

KAFKA-CENTRIC:
└─ Input and output are Kafka topics

11. Questions You Should Be Able to Answer

After this module, you should be able to answer:

✅ What is Kafka Streams? (library, not cluster)
✅ How does stream processing differ from batch processing?
✅ What does "record-at-a-time" processing mean?
✅ How does Kafka Streams relate to Kafka topics and partitions?
✅ What is the high-level lifecycle of a Streams application?
✅ When should I use Kafka Streams vs alternatives?
✅ How does Kafka Streams scale?
✅ What guarantees does Kafka Streams provide?

12. What's Next?

Now that you understand the mental model, the next modules will teach you how to build Streams applications:

Module 2.1: Core abstractions (KStream, KTable, GlobalKTable) — the building blocks
Module 3.1: Building a Streams application — actual code structure
Module 4.1: Stateless operations — your first transformations
Module 5: Stateful processing — aggregations and windows

Summary

You learned:

✅ Kafka Streams is a library that runs embedded in your application
✅ Stream processing is continuous and real-time, not batch
✅ Kafka Streams processes records one-at-a-time from Kafka topics
✅ Parallelism comes from partitions (more partitions = more parallelism)
✅ Applications have a lifecycle: define → build → start → process → shutdown
✅ Kafka Streams is fault-tolerant and stateful by design
✅ Use it for real-time Kafka-centric stream processing

Next up: Module 2.1 — Core Abstractions (KStream, KTable & GlobalKTable) — Learn the fundamental building blocks for stream processing.