KafkaIQ

1.3 – Core Concepts and Building Blocks

Module: M1 – Kafka Fundamentals
Goal: Define Kafka vocabulary (intro-level definitions, not deep mechanics)

Introduction

This module introduces Kafka's core building blocks. We'll define each concept at a high level only.

Important: This is vocabulary building, not deep-dive mechanics. Deep dives come in later modules.

Topics

What Is a Topic?

Topic = A logical category or channel for events.

Analogy: Like a folder in a file system or a table in a database.

Examples:

  • orders (all order events)
  • user-activity (all user clicks, page views)
  • payments (all payment transactions)
  • sensor-readings (all IoT sensor data)

Topic Characteristics

Characteristic Description
Name Unique identifier (e.g., orders)
Events Stream of events (e.g., order created, payment processed)
Multi-producer Many producers can write to the same topic
Multi-consumer Many consumers can read from the same topic
Durable Events are stored on disk (not deleted immediately)

Example:

Topic: orders
  - Producer 1: Web API writes order events
  - Producer 2: Mobile API writes order events
  - Consumer 1: Payment Service reads order events
  - Consumer 2: Analytics Service reads order events

Topic Naming Conventions

Best Practice: Use descriptive, hierarchical names.

Good Names:

  • orders.created
  • user.activity.clicks
  • payment.processed
  • sensor.temperature.celsius

Avoid:

  • data (too generic)
  • topic1 (meaningless)
  • my-topic (what data is in it?)

When to Create a New Topic?

Rule of Thumb: One topic per event type.

Example:

  • orders.created (when order is created)
  • orders.confirmed (when order is confirmed)
  • orders.shipped (when order is shipped)

Alternative: Use a single topic orders with different event types (via message schema).

Deep Dive Deferred: Topic design strategy is covered in M2: Brokers & Topics.

Partitions

What Is a Partition?

Partition = A division of a topic into smaller, parallel logs.

Analogy: Like splitting a large file into smaller chunks for parallel processing.

Key Point: Topics are divided into partitions for scalability.

Partition Characteristics

Characteristic Description
Ordered Events within a partition are ordered (by offset)
Append-only New events are always added to the end
Distributed Partitions are spread across multiple brokers
Parallel Multiple partitions allow parallel reads/writes

Example:

Topic: orders (10 partitions)
  Partition 0: [event1, event2, event3, ...]
  Partition 1: [event4, event5, event6, ...]
  Partition 2: [event7, event8, event9, ...]
  ...
  Partition 9: [event28, event29, event30, ...]

Partition Physical Storage Layout

Partitions are append-only log files stored on disk. Understanding the physical structure helps you troubleshoot storage issues and configure retention policies.

Disk Directory Structure:

/var/kafka/data/
├── orders-0/                           ← Topic "orders", Partition 0
│   ├── 00000000000000000000.log        ← Segment file (messages)
│   ├── 00000000000000000000.index      ← Offset index (offset → byte position)
│   ├── 00000000000000000000.timeindex  ← Time index (timestamp → offset)
│   ├── 00000000000000010000.log        ← Next segment (starts at offset 10000)
│   ├── 00000000000000010000.index
│   ├── 00000000000000010000.timeindex
│   └── leader-epoch-checkpoint         ← Leader epoch metadata
│
├── orders-1/                           ← Topic "orders", Partition 1
│   ├── 00000000000000000000.log
│   ├── 00000000000000000000.index
│   └── ...
│
└── payments-0/                         ← Topic "payments", Partition 0
    ├── 00000000000000000000.log
    └── ...

Key Files:

1. .log Files (Message Data)

Contains the actual message data (keys, values, headers, timestamps).

Example:

Offset 0: key="user-123", value="order-1", timestamp=1673784123456
Offset 1: key="user-456", value="order-2", timestamp=1673784123789
Offset 2: key="user-123", value="order-3", timestamp=1673784124012
...

File Size: Controlled by log.segment.bytes (default: 1 GB). When a segment reaches this size, Kafka creates a new segment.

2. .index Files (Offset Index)

Maps offsets to byte positions in the .log file for fast random access.

Example:

Offset 0 → Byte position 0
Offset 100 → Byte position 12,500
Offset 200 → Byte position 25,000

Why It Matters: Consumers can jump to any offset without reading the entire log file.

3. .timeindex Files (Time Index)

Maps timestamps to offsets for time-based searches.

Example:

Timestamp 1673784123000 (2023-01-15 10:00:00) → Offset 0
Timestamp 1673784183000 (2023-01-15 10:01:00) → Offset 1000

Use Case: Find messages from a specific time (e.g., "replay from 10 AM yesterday").

4. leader-epoch-checkpoint

Tracks leader epochs (how many times leadership changed for this partition).

Use Case: Helps detect data inconsistencies after leader failover.

Segment Rolling:

Kafka creates a new segment when:

  • Current segment reaches log.segment.bytes (default: 1 GB)
  • Current segment is older than log.segment.ms (default: 7 days)

Example Timeline:

Day 0: Create segment 00000000000000000000.log (messages 0-9,999)
Day 1: Segment reaches 1 GB → roll to new segment 00000000000000010000.log
Day 2: Segment reaches 1 GB → roll to new segment 00000000000000020000.log
...
Day 8: Delete segment 00000000000000000000.log (retention expired)

Why This Matters:

1. Retention: Kafka deletes entire segments, not individual messages. If retention is 7 days, segments older than 7 days are deleted.

2. Performance: Smaller segments (e.g., 512 MB) allow finer-grained retention but increase metadata overhead. Larger segments (e.g., 2 GB) reduce overhead but delay retention cleanup.

3. Disk Usage: You can calculate disk usage:

Disk Usage = (message rate) × (avg message size) × (retention time) × (replication factor)

Example:
  1000 msg/sec × 1 KB × 7 days × 3 (RF=3) = 1.8 TB

Common Misconceptions:

  • ❌ "Kafka stores messages in a database" → No, it's just append-only log files on disk.
  • ❌ "Kafka deletes messages immediately after consumption" → No, retention is time/size-based, not consumption-based.
  • ❌ "All partitions share the same log file" → No, each partition has its own directory and log files.

Why Partitions?

Scalability: More partitions = more parallelism.

Example:

  • 1 partition: Only 1 consumer can read (bottleneck)
  • 10 partitions: 10 consumers can read in parallel (10x faster)

Ordering Guarantee: Events within a partition are ordered, but not across partitions.

Example:

Partition 0: [event1, event2, event3] ← Ordered
Partition 1: [event4, event5, event6] ← Ordered

But: No guarantee that event3 happens before event4 (different partitions).

Deep Dive Deferred: Partition strategy is covered in M2: Brokers & Topics.

Offsets

What Is an Offset?

Offset = A position or index of an event within a partition.

Analogy: Like a page number in a book or a row number in a spreadsheet.

Key Point: Offsets start at 0 and increment by 1 for each new event.

Offset Example

Partition 0:
  Offset 0: {orderId: "12345", amount: 99.00}
  Offset 1: {orderId: "12346", amount: 150.00}
  Offset 2: {orderId: "12347", amount: 75.00}
  Offset 3: {orderId: "12348", amount: 200.00}  ← Next event will be offset 4

How Consumers Use Offsets

Consumers track their position using offsets.

Example:

Consumer 1: Last read offset 2 (next will read offset 3)
Consumer 2: Last read offset 0 (next will read offset 1)

Key Point: Each consumer group tracks its own offset independently.

Benefit: Consumers can replay events by resetting their offset.

Example:

Consumer was at offset 100
Consumer crashes
Consumer restarts at offset 95 (replay last 5 events)

Deep Dive Deferred: Offset management is covered in M4: Consumers.

Brokers

What Is a Broker?

Broker = A Kafka server that stores events and serves them to clients.

Analogy: Like a database server or a file server.

Key Point: A Kafka cluster is made up of multiple brokers (typically 3-10+).

Broker Responsibilities

Responsibility Description
Store data Write events to disk
Serve reads Send events to consumers
Replicate data Copy events to other brokers (for fault tolerance)
Leader election Elect new leader if current leader fails

Example Cluster:

Broker 1 (10.0.0.1)
Broker 2 (10.0.0.2)
Broker 3 (10.0.0.3)

Broker IDs

Each broker has a unique ID.

Example:

Broker 1 (ID: 1)
Broker 2 (ID: 2)
Broker 3 (ID: 3)

Key Point: Broker ID is set in configuration (broker.id=1).

Partition Distribution Across Brokers

Partitions are distributed across brokers for load balancing.

Example: Topic orders with 9 partitions, 3 brokers.

Broker 1: Partitions 0, 3, 6
Broker 2: Partitions 1, 4, 7
Broker 3: Partitions 2, 5, 8

Benefit: Load is evenly distributed (each broker handles 3 partitions).

Deep Dive Deferred: Broker internals are covered in M2: Brokers & Topics and M7: Kafka Internals.

Consumer Groups

What Is a Consumer Group?

Consumer Group = A set of consumers that work together to read a topic.

Analogy: Like a team of workers sharing tasks.

Key Point: Consumers in the same group divide partitions among themselves.

Consumer Group Behavior

Example: Topic orders with 10 partitions, consumer group payment-processors with 3 consumers.

Consumer 1 → Reads partitions 0, 1, 2, 3
Consumer 2 → Reads partitions 4, 5, 6
Consumer 3 → Reads partitions 7, 8, 9

Key Guarantee: Each partition is read by only one consumer in the group.

Why?: Ensures events are processed exactly once per consumer group.

Multiple Consumer Groups

Different consumer groups read the same events independently.

Example:

Topic: orders

Consumer Group 1 (payment-processors):
  - Consumer A reads partitions 0-4
  - Consumer B reads partitions 5-9

Consumer Group 2 (analytics):
  - Consumer C reads partitions 0-4
  - Consumer D reads partitions 5-9

Key Point: Both groups read the same events, but track separate offsets.

When Consumer Count Changes (Rebalancing)

Rebalancing = Partitions are reassigned when consumers join or leave the group.

Example: Consumer 2 crashes.

Before:

Consumer 1 → Partitions 0, 1, 2, 3
Consumer 2 → Partitions 4, 5, 6        ← Crashes
Consumer 3 → Partitions 7, 8, 9

After Rebalancing:

Consumer 1 → Partitions 0, 1, 2, 3, 4, 5
Consumer 3 → Partitions 6, 7, 8, 9

Result: Remaining consumers pick up the work.

Deep Dive Deferred: Consumer groups and rebalancing are covered in M4: Consumers.

Delivery Semantics (Consumer Guarantees)

Kafka supports three delivery semantics, allowing you to choose the right tradeoff between performance and reliability.

Semantic Description Behavior Use Case Example
At-most-once Messages may be lost, but never duplicated Consumer commits offset before processing Logging where data loss is tolerable (metrics, analytics)
At-least-once Messages may be duplicated, but never lost Consumer commits offset after processing Payment processing (handle duplicates with idempotency)
Exactly-once No duplicates, no loss (guaranteed) Uses Kafka transactions + idempotent producer Financial transactions, mission-critical data

How to Achieve Each Semantic:

At-Most-Once (Commit First, Process Later):

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
consumer.commitSync();  // Commit BEFORE processing
for (ConsumerRecord<String, String> record : records) {
    process(record);  // If this crashes, message is lost
}

Risk: If process() crashes, offset is already committed → message lost.

At-Least-Once (Process First, Commit Later):

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
    process(record);  // Process first
}
consumer.commitSync();  // Commit AFTER processing

Risk: If commit fails, consumer will reprocess messages → duplicates.

Best Practice: Make process() idempotent (processing same message twice has no side effects).

Exactly-Once (Kafka Transactions):

producer.initTransactions();
producer.beginTransaction();
producer.send(record);
producer.sendOffsetsToTransaction(offsets, consumerGroupId);
producer.commitTransaction();

Guarantee: Messages are written exactly once, even if producer retries or consumer rebalances.

Use Case: Financial transactions, event sourcing, critical data pipelines.

Common Misconceptions:

  • ❌ "At-least-once is bad because of duplicates" → It's the most common pattern! Just make your processing idempotent.
  • ❌ "Exactly-once is free" → No, it has overhead (transactions, extra protocol round-trips). Use only when needed.

Offset Storage: __consumer_offsets Internal Topic

Where Are Offsets Stored?

Consumer groups track their offsets in a special Kafka topic called __consumer_offsets.

Key Points:

  • Internal Topic: Created automatically by Kafka, you don't manage it directly
  • Compacted: Uses log compaction to keep only the latest offset for each consumer group + partition
  • Replicated: Typically RF=3 for durability
  • 50 Partitions (default): Offsets are distributed across partitions for scalability

How It Works:

Consumer commits offset 100 for partition 0
   ↓
Kafka writes to __consumer_offsets:
  key: (group=payment-processors, topic=orders, partition=0)
  value: offset=100
   ↓
Log compaction keeps only latest offset per key

Example Offset Entry:

Key: (consumer-group-1, orders, partition-0)
Value: offset=1500, timestamp=1673784123456

Why This Matters:

1. Fault Tolerance: If consumer crashes, it can resume from the last committed offset (stored in __consumer_offsets).

2. Consumer Lag: Monitoring tools compare the last committed offset (from __consumer_offsets) with the log-end-offset to calculate lag.

3. Offset Reset: You can reset offsets by writing new values to __consumer_offsets (using kafka-consumer-groups.sh --reset-offsets).

Viewing Offsets:

# List all consumer groups
kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

# Describe consumer group offsets
kafka-consumer-groups.sh --describe \
  --group payment-processors \
  --bootstrap-server localhost:9092

Output:

GROUP               TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
payment-processors  orders    0          1500            1550            50
payment-processors  orders    1          1200            1200            0

Explanation:

  • CURRENT-OFFSET: Last committed offset (from __consumer_offsets)
  • LOG-END-OFFSET: Latest offset in partition
  • LAG: How far behind the consumer is (50 messages for partition 0)

Common Misconceptions:

  • ❌ "Offsets are stored in ZooKeeper" → No (this was true in old Kafka versions, but since 0.9, offsets are in Kafka itself).
  • ❌ "I need to manually manage __consumer_offsets" → No, Kafka handles it automatically.
  • ❌ "Offsets are per consumer, not per group" → No, offsets are per consumer group + topic + partition.

Replication Factor

What Is Replication Factor?

Replication Factor (RF) = The number of copies of each partition.

Example: Replication factor 3 means each partition is stored on 3 brokers.

Key Point: Replication provides fault tolerance (no data loss if a broker fails).

Replication Example

Topic: orders with 3 partitions, RF=3.

Partition 0:
  - Broker 1 (Leader)
  - Broker 2 (Follower)
  - Broker 3 (Follower)

Partition 1:
  - Broker 2 (Leader)
  - Broker 3 (Follower)
  - Broker 1 (Follower)

Partition 2:
  - Broker 3 (Leader)
  - Broker 1 (Follower)
  - Broker 2 (Follower)

Key Point: If Broker 1 fails, Broker 2 or Broker 3 can serve Partition 0 (they have a copy).

Replication Factor Best Practices

Replication Factor Fault Tolerance Use Case
1 None (no copies) Dev/test only
2 Tolerates 1 failure Not recommended (split-brain risk)
3 Tolerates 1 failure Production standard
4+ Tolerates 2+ failures Critical data (rare)

Recommendation: Always use RF=3 in production.

Deep Dive Deferred: Replication mechanics are covered in M2: Brokers & Topics and M7: Kafka Internals.

Leaders and Followers

What Are Leaders and Followers?

Leader = The primary replica of a partition that handles all reads and writes.

Follower = A backup replica that replicates data from the leader.

Key Point: Only the leader serves client requests. Followers are for fault tolerance only.

Leader and Follower Example

Partition 0:
  - Broker 1 (Leader) ← Producers write here, consumers read here
  - Broker 2 (Follower) ← Replicates from Broker 1
  - Broker 3 (Follower) ← Replicates from Broker 1

What Happens If Leader Fails?

Broker 1 crashes
   ↓
Controller elects new leader (Broker 2)
   ↓
Partition 0:
  - Broker 2 (NEW Leader) ← Now handles reads/writes
  - Broker 3 (Follower) ← Replicates from Broker 2

Failover Time: Typically 5-10 seconds (ZooKeeper mode), < 1 second (KRaft mode).

Deep Dive Deferred: Leader election is covered in M7: Kafka Internals.

In-Sync Replicas (ISR)

What Is ISR?

ISR (In-Sync Replicas) = The set of replicas that are fully caught up with the leader.

Key Point: Only ISR replicas can become the new leader if the current leader fails.

ISR Example

Partition 0 (Leader: Broker 1, RF=3):
  - Broker 1 (Leader, offset 1000) ← In ISR
  - Broker 2 (Follower, offset 1000) ← In ISR (caught up)
  - Broker 3 (Follower, offset 950) ← Out of ISR (lagging)

What Happens If Broker 3 Catches Up?

Broker 3 fetches events 950-1000
   ↓
Broker 3 is added back to ISR
   ↓
ISR: [Broker 1, Broker 2, Broker 3]

Why ISR Matters:

  • Producers with acks=all wait for all ISR replicas to acknowledge
  • Only ISR replicas can become leader (prevents data loss)

Deep Dive Deferred: ISR mechanics are covered in M2: Brokers & Topics and M7: Kafka Internals.

Retention

What Is Retention?

Retention = How long Kafka keeps events before deleting them.

Key Point: Kafka is not a traditional queue (events are not deleted immediately after consumption).

Retention Policies

Kafka supports two retention policies:

Policy Description Use Case
Time-based Delete events older than N days Event logs (e.g., 7 days)
Size-based Delete events when partition exceeds N GB Limit disk usage

Example:

Retention: 7 days

Day 0: Event A written
Day 1: Event B written
Day 7: Event C written
Day 8: Event A deleted (older than 7 days)

Retention vs Consumption

Important: Retention is independent of consumption.

Example:

Retention: 7 days
Consumer reads event on Day 1
Event is still retained until Day 7 (even though it was read)

Why?: Other consumers may need to read the same event, or the consumer may need to replay.

Deep Dive Deferred: Retention policies are covered in M2: Brokers & Topics.

Compaction (Optional Retention Policy)

What Is Compaction?

Compaction = Kafka keeps only the latest value for each key, discarding old values.

Use Case: When you care about current state, not history.

Compaction Example

Before Compaction:

Partition 0:
  Offset 0: {userId: "123", name: "Alice"}
  Offset 1: {userId: "456", name: "Bob"}
  Offset 2: {userId: "123", name: "Alice Smith"}  ← Updated name
  Offset 3: {userId: "789", name: "Charlie"}

After Compaction:

Partition 0:
  Offset 1: {userId: "456", name: "Bob"}
  Offset 2: {userId: "123", name: "Alice Smith"}  ← Latest value for user 123
  Offset 3: {userId: "789", name: "Charlie"}

Result: Offset 0 is deleted (old value for user 123).

Use Cases:

  • Database change data capture (CDC)
  • Kafka Streams state stores
  • Configuration management

Deep Dive Deferred: Log compaction is covered in M2: Brokers & Topics.

Putting It All Together

Kafka Component Relationships

┌───────────────────────────────────────────────────────┐
│                  Kafka Cluster                        │
│                                                       │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐    │
│  │  Broker 1  │  │  Broker 2  │  │  Broker 3  │    │
│  │            │  │            │  │            │    │
│  │ Partition  │  │ Partition  │  │ Partition  │    │
│  │ 0 (Leader) │  │ 1 (Leader) │  │ 2 (Leader) │    │
│  │ 1 (Follower│  │ 2 (Follower│  │ 0 (Follower│    │
│  │ 2 (Follower│  │ 0 (Follower│  │ 1 (Follower│    │
│  └────────────┘  └────────────┘  └────────────┘    │
│                                                       │
│  Topic: orders (3 partitions, RF=3)                 │
└───────────────────────────────────────────────────────┘
           ↑                              ↓
    ┌──────────┐                   ┌──────────────┐
    │ Producer │                   │ Consumer     │
    │          │                   │ Group        │
    │ (writes  │                   │ (reads with  │
    │  events) │                   │  offsets)    │
    └──────────┘                   └──────────────┘

Example Scenario: Order Processing

Setup:

  • Topic: orders (10 partitions, RF=3)
  • Producers: Web API, Mobile API
  • Consumers: Payment Service, Analytics Service

Flow:

  1. User places order via Web API
  2. Web API writes event to topic orders (partition 3)
  3. Event is replicated to 3 brokers (leader + 2 followers)
  4. Payment Service reads event from partition 3 (offset 42)
  5. Payment Service commits offset 43 ("I've read up to offset 42")
  6. Analytics Service reads the same event (independent offset tracking)

Key Points:

  • Multiple consumers read the same event
  • Partitions enable parallel processing
  • Replication ensures no data loss
  • Offsets allow consumers to replay events

Configuration Reference (Quick Start)

This section provides a quick reference for key Kafka configurations. These configs control performance, reliability, and behavior.

Note: Deep configuration tuning is covered in M2: Brokers & Topics and M3/M4: Producers/Consumers. This is an intro-level reference.

Topic and Partition Configuration

Configuration Controls When You Care Why It Matters
num.partitions Number of partitions per topic When creating topics Balances load and determines consumer parallelism
replication.factor How many brokers hold copies of each partition For fault tolerance in production Ensures durability and availability (RF=3 recommended)

Example:

kafka-topics.sh --create \
  --topic orders \
  --partitions 10 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

Impact: 10 partitions = up to 10 consumers can read in parallel. RF=3 = can tolerate 2 broker failures.

Producer Configuration

Configuration Controls When You Care Why It Matters
acks Level of acknowledgment required For message durability guarantees Tradeoff between latency and data safety
batch.size Max batch size in bytes for messages When tuning throughput Larger batches reduce network overhead
linger.ms Time to wait before sending batched messages Controls latency vs throughput Balances message batching benefits against delay
compression.type Compression codec (snappy, lz4, gzip, zstd) To optimize bandwidth Reduces network and storage costs

acks Values:

  • acks=0: Fire-and-forget (no acknowledgment, fastest but least safe)
  • acks=1: Leader acknowledges (moderate safety, low latency)
  • acks=all: Leader + all ISR replicas acknowledge (strongest guarantee, higher latency)

Example:

Properties props = new Properties();
props.put("acks", "all");               // Strong durability
props.put("batch.size", 16384);         // 16 KB batch size
props.put("linger.ms", 10);             // Wait 10ms to fill batch
props.put("compression.type", "snappy"); // Compress with Snappy

Impact: acks=all ensures no data loss, batching improves throughput by 5-10x, compression reduces bandwidth by 50-70%.

Consumer Configuration

Configuration Controls When You Care Why It Matters
enable.auto.commit Enables automatic offset committing Simplifies offset management Important for correct fault recovery
auto.commit.interval.ms Frequency of automatic commits Controls commit overhead Tradeoff between commit frequency and risk of reprocessing
group.id Consumer group name For parallel processing Determines partition assignment
auto.offset.reset Where to start if no offset exists (earliest, latest, none) When consumer first starts Controls initial read position

Auto-Commit vs Manual Commit:

Auto-Commit (enable.auto.commit=true):

// Kafka automatically commits offsets every 5 seconds (default)
consumer.poll(Duration.ofMillis(100));  // Process messages
// Offset committed automatically in background

Benefit: Simple, no code needed.
Risk: If consumer crashes after processing but before auto-commit, messages are reprocessed (at-least-once).

Manual Commit (enable.auto.commit=false):

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
    process(record);
}
consumer.commitSync();  // Explicitly commit after processing

Benefit: Full control (commit exactly when you want).
Use Case: At-least-once with idempotent processing, exactly-once with transactions.

Retention and Cleanup Configuration

Configuration Controls When You Care Why It Matters
retention.ms How long to keep events (time-based) For storage management Balance replay window vs disk usage
retention.bytes Max size per partition (size-based) For disk space limits Hard cap on partition size
cleanup.policy delete (time/size-based) or compact (keep latest per key) For state vs event topics Determines retention strategy

Example:

# Time-based retention (7 days)
retention.ms=604800000

# Size-based retention (100 GB per partition)
retention.bytes=107374182400

# Delete policy (default)
cleanup.policy=delete

Use Cases:

  • Event logs: cleanup.policy=delete, retention.ms=7 days (replay for 7 days, then delete)
  • State/changelog: cleanup.policy=compact (keep latest value per key forever)
  • Hybrid: cleanup.policy=compact,delete, retention.ms=30 days (keep latest, but delete after 30 days)

Configuration Best Practices

1. Production Topics:

replication.factor=3           # Fault tolerance
min.insync.replicas=2          # At least 2 replicas must acknowledge
acks=all                       # Producer waits for all ISR

2. High-Throughput Topics:

batch.size=65536               # 64 KB batches
linger.ms=10                   # Wait 10ms to fill batch
compression.type=snappy        # Fast compression

3. Low-Latency Topics:

batch.size=1024                # Small batches (1 KB)
linger.ms=0                    # Send immediately
acks=1                         # Leader-only acknowledgment

4. Storage-Constrained Topics:

retention.ms=86400000          # 1 day retention
compression.type=zstd          # Best compression ratio

Common Configuration Mistakes:

  • ❌ replication.factor=1 in production → Single point of failure
  • ❌ acks=0 for critical data → Data loss on broker failure
  • ❌ enable.auto.commit=true without idempotent processing → Duplicate processing on crashes
  • ❌ Too many partitions (> 100 per topic) → High metadata overhead, slow rebalancing
  • ❌ Infinite retention without compaction → Disk fills up

Core Concepts Cheat Sheet

Concept Definition Example
Topic Logical category for events orders, payments
Partition Division of a topic for parallelism orders-0, orders-1, ...
Offset Position of an event in a partition Event at offset 42
Broker Kafka server that stores events Broker 1, Broker 2, Broker 3
Consumer Group Set of consumers sharing partition reads payment-processors
Replication Factor Number of copies of each partition RF=3 (3 copies)
Leader Primary replica handling reads/writes Broker 1 (leader for partition 0)
Follower Backup replica replicating from leader Broker 2 (follower for partition 0)
ISR Replicas fully caught up with leader [Broker 1, Broker 2]
Retention How long events are kept 7 days, 100 GB

What's Next?

Next Module: 1.4 – Kafka APIs and Client Landscape

What We Covered:

  • ✅ Topics (logical categories)
  • ✅ Partitions (divisions for scalability)
  • ✅ Offsets (positions in partitions)
  • ✅ Brokers (Kafka servers)
  • ✅ Consumer groups (parallel consumers)
  • ✅ Replication factor (fault tolerance)
  • ✅ Leaders and followers (primary vs backup)
  • ✅ ISR (in-sync replicas)
  • ✅ Retention (how long to keep events)

What We Didn't Cover (Coming in Later Modules):

  • ❌ Topic design strategy → M2: Brokers & Topics
  • ❌ Partition assignment logic → M2: Brokers & Topics
  • ❌ Consumer rebalancing mechanics → M4: Consumers
  • ❌ Replication protocol → M7: Kafka Internals
  • ❌ Retention and compaction tuning → M2: Brokers & Topics

Core Concepts and Building Blocks: Complete ✅