KafkaIQ1.1 Introduction & Consumer Mental Model
Module Overview
Goal: Build a correct mental model of Kafka consumption without coordination, offsets, or tuning.
What You'll Learn:
- What a Kafka Consumer is and its role in the ecosystem
- Pull-based consumption model (vs push)
- Topic → Partition → Record flow
- ConsumerRecord structure and components
- The high-level consume loop: poll → process → (commit)
- Single consumer, single partition intuition
What's NOT Covered Here:
- Consumer groups and coordination (covered in Module 2.1)
- Offset commit mechanics (covered in Module 5.2)
- Rebalancing (covered in Module 4.1)
- Performance tuning (covered in Module 7.1)
Key Insight: This module focuses on the fundamental mechanics of reading data from Kafka, not how it scales.
Prerequisites & Setup
Before you start, you'll need the Kafka client library in your project.
Maven
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.7.0</version>
</dependency>
Gradle
implementation 'org.apache.kafka:kafka-clients:3.7.0'
Note: Use the latest stable version compatible with your Kafka cluster version.
1. What is a Kafka Consumer?
Definition
A Kafka Consumer is a client application that:
- Reads records (messages) from Kafka topics
- Pulls data from specific partitions
- Tracks its position in each partition (via offsets)
- Processes records according to application logic
Think of a consumer as a reader at a library that:
- Picks books (partitions) to read
- Remembers which page (offset) they're on
- Reads at their own pace (pull-based)
- Can resume where they left off
Consumer's Core Responsibilities
┌─────────────────────────────────────────────────┐
│ CONSUMER RESPONSIBILITIES │
├─────────────────────────────────────────────────┤
│ 1. Request data from brokers (pull) │
│ 2. Deserialize bytes into objects │
│ 3. Process records │
│ 4. Track consumption position (offsets) │
│ 5. Commit offsets (checkpoint progress) │
│ 6. Handle errors and failures │
└─────────────────────────────────────────────────┘
2. Pull-Based Consumption Model
Pull vs Push
Kafka uses a pull-based model, unlike many messaging systems.
Pull-Based (Kafka)
┌──────────────┐ ┌──────────────┐
│ CONSUMER │ ──── poll() ────→ │ BROKER │
│ │ ←─── records ──── │ │
└──────────────┘ └──────────────┘
Consumer controls:
- When to fetch
- How much to fetch
- Processing speed
Advantages:
- ✅ Consumer controls the pace (backpressure)
- ✅ Batch processing efficient
- ✅ Consumer can catch up at full speed
- ✅ No risk of overwhelming consumers
Disadvantages:
- ❌ Introduces latency (polling interval)
- ❌ Consumer must actively poll
- ❌ Empty polls waste resources
Push-Based (Traditional MQ)
┌──────────────┐ ┌──────────────┐
│ CONSUMER │ ←─── push() ───── │ BROKER │
│ │ │ │
└──────────────┘ └──────────────┘
Broker controls:
- When to send
- How much to send
Advantages:
- ✅ Low latency (immediate delivery)
- ✅ No polling overhead
Disadvantages:
- ❌ Can overwhelm consumers
- ❌ Complex backpressure mechanisms
- ❌ Broker must track consumer capacity
Why Kafka Chose Pull
Quote from Kafka documentation:
"A push-based system must choose to either send a request immediately or accumulate messages and send them later... If tuned for low latency, this leads to sending a single message at a time only for the transfer to end up being buffered anyway, which is wasteful."
Key Insight: Pull-based allows consumers to:
- Batch aggressively when catching up
- Process one-at-a-time when real-time
- Control their own throughput
3. The Consumer → Broker Flow (Mental Model)
High-Level Architecture
┌──────────────────┐
│ APPLICATION │
│ │
│ consumer.poll() │
└────────┬─────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ KAFKA CONSUMER CLIENT │
│ ┌────────────┐ ┌────────────┐ ┌───────────┐ │
│ │ Fetch │→│Deserializer│→│ Process │ │
│ │ Request │ │ │ │ Records │ │
│ └────────────┘ └────────────┘ └───────────┘ │
└────────┬─────────────────────────────────────────┘
│ Network
▼
┌─────────────────────────────────────────────────┐
│ KAFKA CLUSTER │
│ │
│ Topic: "orders" │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Partition0│ │Partition1│ │Partition2│ │
│ │ Broker 1 │ │ Broker 2 │ │ Broker 3 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────┘
The Journey of Reading a Record
- Application calls
consumer.poll() - Consumer sends fetch request to broker
- Broker reads records from partition log
- Broker returns batch of records
- Consumer deserializes keys and values
- Application processes records
- Consumer tracks new position (offset)
- Consumer commits offset (eventually)
4. Understanding Topics, Partitions, and Records
Topic Consumption View
Topic: "orders" (3 partitions)
Partition 0: [R0] [R1] [R2] [R3] [R4] [R5] ...
↑
Consumer position (offset 0)
Partition 1: [R0] [R1] [R2] [R3] [R4] [R5] ...
↑
Consumer position (offset 0)
Partition 2: [R0] [R1] [R2] [R3] [R4] [R5] ...
↑
Consumer position (offset 0)
Key Points:
- Topics are divided into partitions
- Each partition is an ordered, immutable log
- Consumers read from specific partitions
- Each partition has independent offsets
Single Consumer, Single Partition (Simplest Case)
┌──────────────┐ Partition 0
│ CONSUMER │ ┌────────────────────────┐
│ │ ←── reads ──── │ [0][1][2][3][4][5][6] │
└──────────────┘ └────────────────────────┘
↑
offset 3 (next to read)
Mental Model:
- Consumer reads records sequentially
- Offset tracks position in the log
- Consumer moves forward (usually never backward)
5. ConsumerRecord Structure
The ConsumerRecord
public class ConsumerRecord<K, V> {
String topic();
int partition();
long offset();
long timestamp();
K key();
V value();
Headers headers();
}
Components Explained
Topic (String)
- Which topic this record came from
- Important when subscribing to multiple topics
String topic = record.topic(); // "orders"
Partition (int)
- Which partition this record came from
- Important for understanding ordering
int partition = record.partition(); // 2
Offset (long)
- Position of this record in its partition
- Unique within partition (not globally)
- Monotonically increasing
long offset = record.offset(); // 12345
Key Insight: Offset is the position in the log, not a timestamp or sequence number across the topic.
Timestamp (long)
- When the record was created
- Can be producer-set or broker-set
- Milliseconds since epoch
long timestamp = record.timestamp(); // 1674567890123
Key (K)
- Optional identifier for the record
- Used for partitioning (by producer)
- Used for grouping/aggregation (by consumer)
String key = record.key(); // "user-123"
Can be null — not all records have keys.
Value (V)
- The actual data payload
- What your application cares about
String value = record.value(); // "{\"orderId\": 123, ...}"
Headers (Headers)
- Optional metadata attached to record
- Key-value pairs (both are byte arrays)
- Not used for partitioning
Headers headers = record.headers();
Header traceId = headers.lastHeader("trace-id");
if (traceId != null) {
String trace = new String(traceId.value());
}
6. The Basic Consume Loop
Minimal Consumer Example
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class SimpleConsumer {
public static void main(String[] args) {
// 1. Configuration
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
// 2. Create consumer
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// 3. Subscribe to topic(s)
consumer.subscribe(Collections.singletonList("orders"));
try {
// 4. Poll loop
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("key=%s, value=%s, partition=%d, offset=%d%n",
record.key(),
record.value(),
record.partition(),
record.offset()
);
// Process record here
}
}
} finally {
// 5. Close consumer
consumer.close();
}
}
}
The Poll Loop Explained
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
// Process each record
}
}
What happens in poll():
- Fetch data from brokers
- Deserialize keys and values
- Return batch of records
- Update internal state (position tracking)
Important: poll() is blocking up to the timeout.
consumer.poll(Duration.ofMillis(100));
// Waits up to 100ms for records
// Returns immediately if records available
// Returns empty if no records after 100ms
Poll Timeout Considerations
Short Timeout (e.g., 100ms)
consumer.poll(Duration.ofMillis(100));
Pros:
- ✅ Fast reaction to shutdown signals
- ✅ Tight control loop
Cons:
- ❌ More frequent empty polls
- ❌ Slightly more overhead
Long Timeout (e.g., 1000ms)
consumer.poll(Duration.ofMillis(1000));
Pros:
- ✅ Fewer empty polls
- ✅ More efficient
Cons:
- ❌ Slower shutdown response
- ❌ Blocks longer when no data
Recommended: Start with 100-500ms and adjust based on your needs.
7. Subscription vs Assignment
Kafka consumers can consume partitions in two ways:
Subscribe (Dynamic, Recommended)
consumer.subscribe(Collections.singletonList("orders"));
Behavior:
- Consumer joins a consumer group
- Partitions assigned automatically
- Rebalancing handles changes
- Typical for scalable applications
Use when: You want automatic partition management.
Assign (Manual, Advanced)
TopicPartition partition0 = new TopicPartition("orders", 0);
TopicPartition partition1 = new TopicPartition("orders", 1);
consumer.assign(Arrays.asList(partition0, partition1));
Behavior:
- Consumer reads specific partitions
- No consumer group coordination
- No automatic rebalancing
- Full control over partitions
Use when:
- Custom partition assignment logic
- Single-consumer scenarios
- Testing or debugging
- Reading from specific partitions
Key Difference:
subscribe() → "I want to read from this topic, assign me partitions"
assign() → "I will read from these specific partitions"
8. Consumer Lifecycle
Consumer State Machine
CREATE → CONFIGURE → SUBSCRIBE → POLL (loop) → CLOSE
│ │ │ │ │
│ │ │ │ └─ Release resources
│ │ │ └─ Process records
│ │ └─ Join group (if subscribing)
│ └─ Set properties
└─ new KafkaConsumer<>(props)
Basic Lifecycle Example
// 1. CREATE & CONFIGURE
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-group");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
// 2. INITIALIZE
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// 3. SUBSCRIBE
consumer.subscribe(Collections.singletonList("orders"));
// 4. POLL LOOP
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
// Process
}
}
// 5. CLOSE
consumer.close();
9. Mental Model: Key Takeaways
The Consumer's Job
REQUEST DATA
↓
[Poll from broker]
↓
[Deserialize bytes]
↓
[Process records]
↓
[Track position]
↓
[Commit offset (eventually)]
Three Levels of Abstraction
- User Level: Call
consumer.poll(), process records - Consumer Level: Fetch data, deserialize, track offsets
- Broker Level: Serve records from partition logs
You control level 1.
The consumer handles level 2.
The broker handles level 3.
Questions You Should Be Able to Answer
✅ What is a Kafka Consumer?
✅ What is the pull-based model?
✅ What are the components of a ConsumerRecord?
✅ What is the basic poll loop?
✅ What's the difference between subscribe() and assign()?
✅ What is an offset?
Questions We Haven't Answered Yet
❌ How do multiple consumers coordinate? → Module 2.1
❌ How do I commit offsets safely? → Module 5.2
❌ What happens during a rebalance? → Module 4.1
❌ How do I tune consumer performance? → Module 7.1
10. Common Beginner Misconceptions
❌ Misconception 1: "poll() sends records to my consumer"
Reality: Consumer pulls records by calling poll(). Broker never pushes.
❌ Misconception 2: "Offsets are global across the topic"
Reality: Offsets are per-partition. Partition 0 offset 100 ≠ Partition 1 offset 100.
❌ Misconception 3: "Kafka guarantees processing exactly once"
Reality: Kafka guarantees at-least-once by default. Exactly-once requires careful offset management (covered in Module 5.3).
❌ Misconception 4: "Records are processed in the order they were produced"
Reality: Order is guaranteed within a partition, not across partitions.
❌ Misconception 5: "poll() returns one record"
Reality: poll() returns a batch (0 to many records), not a single record.
11. Simple Working Example
Reading Orders from Kafka
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class OrderConsumer {
public static void main(String[] args) {
// Configuration
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processors");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
// Create consumer
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
try {
// Subscribe to topic
consumer.subscribe(Collections.singletonList("orders"));
// Poll loop
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.printf(
"Processing order: key=%s, value=%s, partition=%d, offset=%d%n",
record.key(),
record.value(),
record.partition(),
record.offset()
);
// Your business logic here
processOrder(record.key(), record.value());
}
}
} finally {
consumer.close();
}
}
private static void processOrder(String orderId, String orderData) {
// Process the order
System.out.println("Order " + orderId + " processed successfully");
}
}
Output:
Processing order: key=order-123, value={"userId":42,...}, partition=2, offset=1543
Order order-123 processed successfully
Processing order: key=order-124, value={"userId":43,...}, partition=0, offset=890
Order order-124 processed successfully
12. What's Next?
Now that you understand the fundamentals of consuming, the next modules will cover:
- Module 2.1: How multiple consumers coordinate (consumer groups)
- Module 3.1: How partitions are assigned to consumers
- Module 4.1: What happens when consumers join/leave (rebalancing)
- Module 5: How to manage offsets and delivery semantics
- Module 6.1: The internal mechanics of the poll loop
- Module 7.1: How to tune consumer performance
- Module 8.1: How to handle errors and failures
- Module 9.1: How to monitor consumers in production
- Module 10.1: Real-world patterns and best practices
Summary
You learned:
- ✅ Kafka uses a pull-based consumption model
- ✅ Consumers read from partitions (ordered logs)
- ✅ The poll loop is the heart of consumption
- ✅ Offsets track position in each partition
- ✅ ConsumerRecord contains key, value, and metadata
- ✅ subscribe() for automatic assignment, assign() for manual control
Next up: Consumer Groups & Coordination (Module 2.1) — Learn how multiple consumers work together to scale consumption.