KafkaIQ5.2 Idempotent Producer
Module Overview
Goal: Understand how idempotent producers prevent duplicates and when to use them.
What You'll Learn:
- What idempotence means in Kafka
- How idempotent producers work internally
- Producer ID (PID) and sequence numbers
- Configuration and enabling idempotence
- Limitations and edge cases
- When idempotence is not enough
Prerequisites:
- Module 5.1 (Delivery Semantics Explained) — understand at-least-once and duplicates
1. The Duplicate Problem
Scenario: Network Failure After Write
Time Producer Leader Broker
─────┼────────────────────────────┼────────────────
1 │ send(record-100) │
2 │ ───── record-100 ─────→ │ Writes to log (offset 50)
3 │ │
4 │ X←─── ack ──── │ Ack lost in network!
5 │ TIMEOUT! Retry... │
6 │ ───── record-100 ─────→ │ Duplicate write (offset 51)
7 │ ←───── ack ──── │
8 │ ✅ Success │ ⚠️ Duplicate!
Result: record-100 written twice (offsets 50 and 51).
Problem: At-least-once delivery causes duplicates on retries.
2. What is Idempotence?
Definition
Idempotence: Same operation can be applied multiple times without changing the result beyond the initial application.
In Kafka: Sending the same message multiple times results in exactly one write.
send(message) → Write once
send(message) → No-op (already written)
send(message) → No-op (already written)
Result: Message appears exactly once in the log
Mathematical Analogy
Non-idempotent: x = x + 1
Call 1: x = 5 + 1 = 6
Call 2: x = 6 + 1 = 7 ← Different result!
Idempotent: x = 5
Call 1: x = 5
Call 2: x = 5 ← Same result
Call 3: x = 5 ← Same result
3. How Idempotent Producers Work
Core Mechanism
Kafka uses two identifiers to detect duplicates:
- Producer ID (PID): Unique ID assigned to producer
- Sequence Number: Monotonically increasing per partition
┌──────────────────────────────────────────────┐
│ Producer (PID = 12345) │
├──────────────────────────────────────────────┤
│ │
│ Partition 0: seq=0, 1, 2, 3, ... │
│ Partition 1: seq=0, 1, 2, 3, ... │
│ Partition 2: seq=0, 1, 2, 3, ... │
│ │
└──────────────────────────────────────────────┘
Each record sent includes:
- Producer ID (PID)
- Sequence number for that partition
Broker-Side Duplicate Detection
┌──────────────────────────────────────────────┐
│ Broker (Leader) │
├──────────────────────────────────────────────┤
│ Last known sequence for PID 12345: │
│ Partition 0: seq=99 │
│ Partition 1: seq=42 │
│ Partition 2: seq=15 │
└──────────────────────────────────────────────┘
Incoming record: PID=12345, Partition=0, Seq=100
├─ Expected: 99 + 1 = 100 ✅
└─ Action: Write to log, update last known seq to 100
Incoming record: PID=12345, Partition=0, Seq=100 (retry)
├─ Expected: 100 + 1 = 101, got 100 ⚠️
└─ Action: Duplicate detected → Ack without writing
Result: Broker acknowledges but doesn't write duplicate.
Detailed Example
Producer (PID=12345) sends to Partition 0:
Time Producer Broker
─────┼────────────────────────────┼────────────────────────────
1 │ send(msg-1, seq=0) │
2 │ ────────────────→ │ seq=0: write (offset 10)
3 │ ←──── ack ────── │ last_seq=0
4 │ │
5 │ send(msg-2, seq=1) │
6 │ ────────────────→ │ seq=1 (expected 0+1): write (offset 11)
7 │ X←─── ack ───── │ ack lost, last_seq=1
8 │ TIMEOUT! Retry msg-2 │
9 │ send(msg-2, seq=1) [retry] │
10 │ ────────────────→ │ seq=1 (duplicate! expected 1+1=2)
11 │ ←──── ack ────── │ Ack without writing
12 │ ✅ Success │ ✅ No duplicate in log
Broker log:
offset 10: msg-1 (seq=0)
offset 11: msg-2 (seq=1)
← No duplicate of msg-2
4. Enabling Idempotent Producer
Configuration
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Enable idempotence
props.put("enable.idempotence", "true");
That's it! One configuration property.
Automatic Configuration Changes
When enable.idempotence=true, Kafka automatically sets:
// These are set automatically (you don't need to set them):
props.put("acks", "all"); // Was: can be 0, 1, all
props.put("retries", Integer.MAX_VALUE); // Was: default 0
props.put("max.in.flight.requests.per.connection", 5); // Was: default 5
Why?
acks=all: Ensures durability (required for idempotence)retries=MAX_VALUE: Allows retries (needed to recover from failures)max.in.flight.requests=5: Allows pipelining while maintaining order
Default in Modern Kafka
// Kafka 3.0+
// Idempotence is ENABLED BY DEFAULT
props.put("enable.idempotence", "true"); // Default!
You can explicitly disable it (not recommended):
props.put("enable.idempotence", "false"); // Opt out
5. Producer ID (PID) Assignment
How PID is Assigned
1. Producer starts
2. Producer requests PID from broker
3. Broker assigns unique PID (e.g., 12345)
4. Producer uses this PID for all records
PID is session-scoped (lost on restart)
PID Lifecycle
// Create producer
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// → Broker assigns PID=12345
producer.send(record1); // PID=12345, seq=0
producer.send(record2); // PID=12345, seq=1
producer.close();
// Create new producer
KafkaProducer<String, String> producer2 = new KafkaProducer<>(props);
// → Broker assigns NEW PID=67890 (different from before)
producer2.send(record3); // PID=67890, seq=0 (resets)
Key Point: PID is NOT persistent. New producer → New PID.
6. Sequence Numbers
Per-Partition Sequence
Each producer maintains a separate sequence number per partition:
Producer (PID=12345):
Partition 0: seq=0 → seq=1 → seq=2 → seq=3 ...
Partition 1: seq=0 → seq=1 → seq=2 → seq=3 ...
Partition 2: seq=0 → seq=1 → seq=2 → seq=3 ...
Why per-partition?
- Records to different partitions are independent
- Allows parallel sending to multiple partitions
Sequence Number Wrap-Around
Sequence numbers are 32-bit integers (0 to 2^31 - 1).
seq=0, 1, 2, ..., 2147483647 (max int)
seq=0, 1, 2, ... (wraps around)
Broker handles wrap-around correctly (no duplicates).
7. Ordering with Idempotence
Ordering Guarantee
Idempotent producer guarantees:
- Records sent to same partition are delivered in order
- Even with retries and multiple in-flight requests
producer.send(new ProducerRecord<>("topic", 0, "key", "msg-1"));
producer.send(new ProducerRecord<>("topic", 0, "key", "msg-2"));
producer.send(new ProducerRecord<>("topic", 0, "key", "msg-3"));
// Partition 0 will have: [msg-1][msg-2][msg-3] (in order)
Even if:
- msg-1 is retried
- msg-2 and msg-3 are sent before msg-1 completes
- Network reorders packets
How Ordering is Preserved
Broker uses sequence numbers to reorder:
Time Producer sends Broker receives Broker writes
─────┼───────────────────────┼───────────────────────┼──────────────
1 │ msg-1 (seq=0) │ │
2 │ msg-2 (seq=1) │ │
3 │ msg-3 (seq=2) │ │
4 │ │ msg-2 (seq=1) arrives │ Wait for seq=0
5 │ │ msg-3 (seq=2) arrives │ Wait for seq=0
6 │ │ msg-1 (seq=0) arrives │ Write seq=0,1,2
7 │ ←── all acks ──────── │ │
Result: Broker writes in sequence order, not arrival order.
8. Limitations of Idempotent Producer
Limitation 1: Session-Scoped Only
Problem: PID is lost on producer restart.
// Producer 1
KafkaProducer producer1 = new KafkaProducer<>(props); // PID=12345
producer1.send(record1); // seq=0
// Producer 1 crashes and restarts
KafkaProducer producer2 = new KafkaProducer<>(props); // PID=67890 (new!)
producer2.send(record1); // seq=0 (new sequence)
// Result: If record1 was not acked, it may be duplicated
Limitation: Duplicates can occur across producer restarts.
Solution: Use transactional producer (Module 5.3).
Limitation 2: Single Producer Instance
Problem: Multiple producers can create duplicates.
// Producer A
KafkaProducer producerA = new KafkaProducer<>(props); // PID=12345
producerA.send(record1);
// Producer B (different instance, same application)
KafkaProducer producerB = new KafkaProducer<>(props); // PID=67890
producerB.send(record1);
// Result: record1 written twice (different PIDs)
Limitation: Idempotence is per-producer instance, not application-wide.
Solution: Use transactional producer with same transactional ID.
Limitation 3: Single Partition Only (for duplicate prevention)
Problem: Idempotence prevents duplicates within a partition, not across partitions.
producer.send(new ProducerRecord<>("topic", 0, "key", "data")); // Partition 0
producer.send(new ProducerRecord<>("topic", 1, "key", "data")); // Partition 1
// If both fail and retry, both may duplicate
// No cross-partition duplicate detection
Limitation: Each partition tracked independently.
Solution: Use transactions for atomic multi-partition writes.
9. Error Scenarios with Idempotent Producer
OutOfOrderSequenceException
When it happens:
// Expected sequence: 0, 1, 2, 3, ...
// Broker receives: 0, 1, 3 (missing 2!)
// Broker throws: OutOfOrderSequenceException
Causes:
- Bug in producer logic
- Network issues causing lost messages
- Broker/producer version mismatch
Producer action:
- Fatal error
- Producer enters error state
- Must be closed and recreated
UnknownProducerIdException
When it happens:
// Producer sends with PID=12345, seq=10
// But broker has no state for PID=12345 (expired or never seen)
// Broker throws: UnknownProducerIdException
Causes:
- Producer inactive too long (broker expired state)
- Broker restarted
Producer action:
- Fatal error
- Producer must be recreated
10. Performance Impact
Overhead
Additional work:
- Producer tracks sequence numbers (in-memory)
- Broker tracks last sequence per (PID, partition)
- Broker performs duplicate check on every write
Performance impact:
- Minimal (<5% throughput reduction in most cases)
- No disk I/O overhead (metadata in memory)
- Slightly higher latency (duplicate check)
Benchmark Example
Configuration: acks=all, 100 byte messages
Without idempotence:
Throughput: 105,000 msg/sec
Latency (p99): 10ms
With idempotence:
Throughput: 102,000 msg/sec (-3%)
Latency (p99): 11ms (+1ms)
Conclusion: Overhead is negligible for the guarantee provided.
11. Practical Usage
Recommended Configuration
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Idempotence (default in Kafka 3.0+)
props.put("enable.idempotence", "true");
// These are set automatically, but shown for clarity:
// props.put("acks", "all");
// props.put("retries", Integer.MAX_VALUE);
// props.put("max.in.flight.requests.per.connection", 5);
// Other recommended settings
props.put("delivery.timeout.ms", 120000);
props.put("compression.type", "lz4");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
When to Use Idempotent Producer
✅ Use Idempotent Producer When:
- You need at-least-once delivery without duplicates
- Single producer instance per topic/partition
- No cross-partition atomicity needed
- No need for exactly-once across restarts
Use Cases:
- Standard production applications
- Event logging
- Metrics ingestion
- User activity tracking
When Idempotence is NOT Enough
❌ Need Transactional Producer When:
- Writing to multiple topics/partitions atomically
- Exactly-once across producer restarts
- Read-process-write patterns (e.g., Kafka Streams)
Next Module: 5.3 — Exactly-Once Semantics (Transactions)
12. Comparing Idempotence Modes
Non-Idempotent vs Idempotent
┌─────────────────────────────────────────────────────────┐
│ NON-IDEMPOTENT │
├─────────────────────────────────────────────────────────┤
│ acks=1 │
│ retries=0 │
│ │
│ send(msg) → ack lost → LOST │
│ Delivery: At-most-once │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ NON-IDEMPOTENT WITH RETRIES │
├─────────────────────────────────────────────────────────┤
│ acks=all │
│ retries=MAX_VALUE │
│ enable.idempotence=false │
│ │
│ send(msg) → ack lost → retry → DUPLICATE │
│ Delivery: At-least-once (may duplicate) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ IDEMPOTENT │
├─────────────────────────────────────────────────────────┤
│ acks=all │
│ retries=MAX_VALUE │
│ enable.idempotence=true │
│ │
│ send(msg) → ack lost → retry → DETECTED, NOT WRITTEN │
│ Delivery: Exactly-once (per producer session) │
└─────────────────────────────────────────────────────────┘
Summary
Key Takeaways
✅ Idempotent producer prevents duplicates from retries
✅ Enabled via enable.idempotence=true (default in Kafka 3.0+)
✅ Uses Producer ID (PID) and sequence numbers
✅ Broker detects duplicates and acks without writing
✅ Preserves ordering even with retries
✅ Session-scoped: Lost on producer restart
✅ Per-partition: Duplicates prevented within partition
✅ Minimal performance overhead (<5%)
✅ Automatically sets: acks=all, retries=MAX_VALUE
When to Use
Standard production application? → ✅ Use idempotence
Need exactly-once across restarts? → ❌ Use transactions (5.3)
Need atomic multi-partition writes? → ❌ Use transactions (5.3)
OK with duplicates on producer restart? → ✅ Use idempotence
Single producer instance? → ✅ Use idempotence
Configuration Checklist
-
enable.idempotence=true(default) -
acks=all(automatic) -
retries=Integer.MAX_VALUE(automatic) -
delivery.timeout.msset appropriately - Error handling for
OutOfOrderSequenceException - Error handling for
UnknownProducerIdException
What's Next
Module 5.3: Exactly-Once Semantics (Producer Side — Transactions)
Learn transactional producers for cross-partition exactly-onceModule 6.1: Error Handling & Failure Scenarios
Handle producer errors and failures
Previous Module: ← 5.1 - Delivery Semantics Explained
Next Module: 5.3 - Exactly-Once Semantics (Producer Side) →