5.1 Delivery Semantics Explained

Module Overview

Goal: Understand the three delivery semantics and how Kafka achieves them.

What You'll Learn:

At-most-once delivery
At-least-once delivery
Exactly-once delivery
The role of acks configuration
Producer retries and their impact
Message duplication vs loss scenarios
Timeouts and delivery guarantees

This is the PRIMARY MODULE where:

acks is explained in depth
Delivery semantics are defined
Trade-offs are discussed

Other modules reference this, not re-explain.

1. The Three Delivery Semantics

Overview

┌──────────────────────────────────────────────────┐
│         DELIVERY SEMANTICS SPECTRUM              │
├──────────────────────────────────────────────────┤
│                                                  │
│  At-Most-Once    At-Least-Once    Exactly-Once  │
│       ↓               ↓                 ↓        │
│   May lose       May duplicate     No loss,     │
│   messages         messages      no duplicates  │
│                                                  │
│   Fastest ←──────────────────────────→ Slowest  │
│   Least reliable ←────────────→ Most reliable   │
│                                                  │
└──────────────────────────────────────────────────┘

2. At-Most-Once Delivery

Definition

"Messages may be lost, but never duplicated."

Also called: "Fire-and-forget"

Characteristics

Producer → Broker
    │
    └─ Don't wait for acknowledgment
    └─ Don't retry on failure
    
Result: Some messages may be lost

How to Achieve At-Most-Once

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// At-most-once configuration
props.put("acks", "0");           // Don't wait for ack
props.put("retries", 0);          // Don't retry
props.put("enable.idempotence", "false"); // No idempotence needed

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// Send without waiting
producer.send(new ProducerRecord<>("topic", "key", "value"));

When Messages Are Lost

Scenario: Network failure after send

Time  Producer                    Broker
─────┼────────────────────────────┼────────────────
  1  │ send(record)                │
  2  │ ──── record ──X             │ Network fails
  3  │ Returns immediately         │ Never received
  4  │ ✅ Producer thinks success  │ ❌ Message lost

Result: Message lost, no retry.

Use Cases for At-Most-Once

✅ Appropriate When:

Message loss is acceptable
Ultra-low latency required
High throughput is critical
Messages are not critical

Examples:

Real-time metrics (missing one data point OK)
Sensor readings (next reading coming soon)
Log aggregation (some logs can be lost)
Monitoring dashboards

Pros and Cons

✅ Pros:

Fastest (no waiting)
Highest throughput
Simplest logic
No duplicates

❌ Cons:

Data loss on failures
No delivery confirmation
Not suitable for critical data

3. At-Least-Once Delivery

Definition

"Messages are never lost, but may be duplicated."

Also called: "Guaranteed delivery"

Characteristics

Producer → Broker → Acknowledgment
    │          ↓
    └─ Wait for ack
    └─ Retry on failure

Result: No loss, possible duplicates

How to Achieve At-Least-Once

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// At-least-once configuration
props.put("acks", "all");                      // Wait for all replicas
props.put("retries", Integer.MAX_VALUE);       // Retry indefinitely
props.put("delivery.timeout.ms", 120000);      // 2 minutes total
props.put("enable.idempotence", "false");      // Idempotence disabled (allows duplicates)

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// Send with callback
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        // Handle error (producer auto-retries first)
    }
});

When Messages Are Duplicated

Scenario: Acknowledgment lost after broker writes

Time  Producer                    Broker
─────┼────────────────────────────┼────────────────
  1  │ send(record-1)              │
  2  │ ──── record-1 ─────────→   │ Writes successfully
  3  │ ←─── ack ──────X            │ Ack lost in network
  4  │ Timeout! Retry              │
  5  │ ──── record-1 ─────────→   │ Duplicate write
  6  │ ←─────── ack ──────────     │ Ack received
  7  │ ✅ Success                  │ ⚠️ Duplicate written

Result: Message written twice.

Use Cases for At-Least-Once

✅ Appropriate When:

Data loss is unacceptable
Duplicates can be handled downstream
Idempotent processing on consumer side

Examples:

Financial transactions (with deduplication)
Order events (idempotent processing)
Inventory updates (handle duplicates)
Critical business events

Handling Duplicates

Consumer-Side Deduplication

Set<String> processedIds = new HashSet<>();

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    
    for (ConsumerRecord<String, String> record : records) {
        String messageId = extractMessageId(record);
        
        if (!processedIds.contains(messageId)) {
            process(record); // Process only once
            processedIds.add(messageId);
        } else {
            // Skip duplicate
            logger.info("Duplicate detected: {}", messageId);
        }
    }
}

Requires: Message contains unique ID.

Pros and Cons

✅ Pros:

No data loss
Delivery guaranteed
Good throughput
Standard for most applications

❌ Cons:

Possible duplicates
Consumer must handle duplicates
Slightly higher latency than at-most-once

4. Exactly-Once Delivery

Definition

"Messages are never lost and never duplicated."

Also called: "Exactly-once semantics (EOS)"

Characteristics

Producer → Broker → Acknowledgment
    │          ↓
    └─ Wait for ack
    └─ Retry on failure
    └─ Idempotence prevents duplicates

Result: No loss, no duplicates

Two Levels of Exactly-Once

1. Idempotent Producer (Single partition, single session)

Prevents duplicates within producer session.

props.put("enable.idempotence", "true");

Guarantees:

No duplicates from retries
Ordering preserved
Single producer session

2. Transactional Producer (Multiple partitions, exactly-once end-to-end)

Prevents duplicates across partitions and sessions.

props.put("enable.idempotence", "true");
props.put("transactional.id", "my-transactional-id");

Guarantees:

Atomic writes across partitions
Exactly-once across application restarts
Read-committed isolation

Quick Comparison

Feature	Idempotent	Transactional
Duplicates prevented?	✅ Yes	✅ Yes
Across partitions?	❌ No	✅ Yes
Across sessions?	❌ No	✅ Yes
Atomic multi-partition writes?	❌ No	✅ Yes
Overhead	Low	Medium

When to Use Each

Idempotent Producer:

Standard production use
Single topic/partition writes
No cross-partition atomicity needed

Transactional Producer:

Writing to multiple topics/partitions atomically
Exactly-once stream processing
Critical financial systems

5. Understanding `acks` (Acknowledgments)

The `acks` Configuration

Purpose: Control how many replicas must acknowledge before producer considers send successful.

props.put("acks", "all"); // or "0", "1", "all"

`acks=0` (No Acknowledgment)

Producer                    Leader Broker
   │                            │
   │──── send(record) ────→     │
   │                            │
   │← Returns immediately       │ (writes to log)
   │                            │

Behavior:

Producer doesn't wait for broker response
Broker may not have received the message
Fastest, least durable

Delivery Semantic: At-most-once

`acks=1` (Leader Acknowledgment)

Producer              Leader Broker        Follower Brokers
   │                      │                      │
   │─── send(record) ──→  │                      │
   │                      │ (writes to log)      │
   │                      │                      │
   │                      │─ replicate ──→       │ (async)
   │                      │                      │
   │← ack ─────────────   │                      │
   │                      │                      │

Behavior:

Leader writes to local log
Leader acknowledges immediately
Replication to followers happens asynchronously

Durability: Medium

Risk: If leader fails before replication, message lost.

`acks=all` or `acks=-1` (All In-Sync Replicas)

Producer         Leader Broker          Follower Brokers (ISR)
   │                 │                          │
   │─ send(record) → │                          │
   │                 │ (writes to log)          │
   │                 │                          │
   │                 │──── replicate ────→      │
   │                 │                          │ (write to log)
   │                 │←──── ack ─────────       │
   │                 │                          │
   │← ack ────────   │                          │
   │                 │                          │

Behavior:

Leader waits for all ISR (In-Sync Replicas) to acknowledge
Highest durability
Slowest

Durability: Highest

Note: "All" means all replicas in the ISR, not necessarily all replicas.

`acks` Comparison

`acks`	Leader Writes	ISR Writes	Durability	Latency	Use Case
`0`	Maybe	No	Lowest	Lowest	Logs, metrics
`1`	Yes	No	Medium	Medium	General use
`all`	Yes	Yes	Highest	Highest	Critical data

ISR (In-Sync Replicas)

Definition: Replicas that are fully caught up with the leader.

Topic: orders
Partitions: 3
Replication factor: 3

Partition 0:
  Leader: Broker 1
  ISR: [Broker 1, Broker 2, Broker 3]  ← All in sync
  
Partition 1:
  Leader: Broker 2
  ISR: [Broker 2, Broker 3]  ← Broker 1 lagging (not in ISR)

acks=all waits for: Leader + all replicas in ISR

Minimum ISR (`min.insync.replicas`)

Broker/Topic Configuration:

min.insync.replicas=2

Effect:

With acks=all, at least 2 replicas must acknowledge
If ISR drops below 2, writes fail (better than data loss)

Example:

Replication factor: 3
min.insync.replicas: 2

ISR: [Broker 1, Broker 2, Broker 3] → Writes succeed
ISR: [Broker 1, Broker 2]           → Writes succeed
ISR: [Broker 1]                     → Writes FAIL (not enough ISR)

Recommendation: Use acks=all + min.insync.replicas=2

6. Producer Retries

Why Retries Matter

Transient failures are common:

Network glitches
Broker temporarily unavailable
Leader election in progress

Retries ensure delivery despite temporary failures.

Configuring Retries

props.put("retries", Integer.MAX_VALUE);      // Retry indefinitely
props.put("retry.backoff.ms", 100);           // Wait 100ms between retries
props.put("delivery.timeout.ms", 120000);     // Total timeout: 2 minutes

Modern Best Practice: Let producer retry indefinitely within delivery timeout.

Retry Behavior

Time  Producer                              Broker
─────┼──────────────────────────────────────┼────────────────
  1  │ send(record)                         │
  2  │ ─────→ X                              │ Network error
  3  │ Retry attempt 1 (after 100ms)        │
  4  │ ─────→                                │ Success
  5  │ ←──── ack                             │
  6  │ ✅ Delivered                          │

Without retries: Message lost
With retries: Message delivered

Retriable vs Non-Retriable Errors

Retriable Errors (Producer auto-retries)

NOT_LEADER_FOR_PARTITION — Leader election in progress
NETWORK_EXCEPTION — Temporary network issue
REQUEST_TIMED_OUT — Broker slow to respond
NOT_ENOUGH_REPLICAS — ISR temporarily low

Action: Producer automatically retries.

Non-Retriable Errors (Fail immediately)

INVALID_TOPIC_EXCEPTION — Topic doesn't exist
RECORD_TOO_LARGE — Message exceeds max.message.bytes
AUTHORIZATION_FAILED — No permission
UNSUPPORTED_VERSION — Incompatible client/broker versions

Action: Producer fails immediately, callback receives exception.

Error handling details: Module 6.1

7. Timeouts and Delivery Guarantees

Key Timeout Configurations

props.put("delivery.timeout.ms", 120000);    // 2 minutes total
props.put("request.timeout.ms", 30000);      // 30 seconds per request
props.put("linger.ms", 10);                  // 10ms batch wait

`delivery.timeout.ms`

Purpose: Maximum time from send() to success or failure.

Includes:

Time in buffer
Retry attempts
Network time

Relationship:

delivery.timeout.ms ≥ linger.ms + request.timeout.ms

Example:

props.put("delivery.timeout.ms", 120000);  // 2 minutes

// Kafka will retry for up to 2 minutes
// After 2 minutes, if not delivered → TIMEOUT exception

`request.timeout.ms`

Purpose: How long to wait for a single request to complete.

Example:

props.put("request.timeout.ms", 30000);  // 30 seconds

// If broker doesn't respond within 30s → Retry

Timeout Scenarios

Scenario 1: Success Within Timeout

Time  Action
────  ──────────────────────────────
0ms   send(record)
10ms  Request sent to broker
50ms  Ack received
✅    Success (50ms total)

Scenario 2: Retry Within Timeout

Time     Action
───────  ──────────────────────────────
0ms      send(record)
10ms     Request sent to broker
30010ms  request.timeout.ms expires → Retry
30110ms  Retry sent
30200ms  Ack received
✅       Success (30200ms total)

Scenario 3: Delivery Timeout

Time      Action
────────  ──────────────────────────────
0ms       send(record)
10ms      Request sent → Timeout
30110ms   Retry 1 → Timeout
60210ms   Retry 2 → Timeout
90310ms   Retry 3 → Timeout
120000ms  delivery.timeout.ms expires
❌        TIMEOUT exception

Result: Producer gives up, callback receives TimeoutException.

8. Delivery Guarantees Summary Table

Semantic	Config	Duplicates?	Loss?	Use Case
At-most-once	`acks=0`, `retries=0`	No	Yes	Logs, metrics
At-least-once	`acks=all`, `retries>0`, no idempotence	Yes	No	Most applications
Exactly-once	`acks=all`, `retries>0`, `idempotence=true`	No	No	Critical systems

9. Configuration Templates

Template 1: At-Most-Once (Fast, Lossy)

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// At-most-once
props.put("acks", "0");
props.put("retries", 0);

Template 2: At-Least-Once (Reliable, May Duplicate)

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// At-least-once
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("delivery.timeout.ms", 120000);
props.put("enable.idempotence", "false"); // Allows duplicates

Template 3: Exactly-Once (No Loss, No Duplicates)

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Exactly-once (idempotent)
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("delivery.timeout.ms", 120000);
props.put("enable.idempotence", "true");  // Prevents duplicates

Recommended for production.

10. Trade-offs and Recommendations

Latency vs Durability

Latency:   Low ←────────────────────→ High
           acks=0    acks=1      acks=all

Durability: Low ←────────────────────→ High
            acks=0    acks=1      acks=all

Throughput vs Reliability

Throughput: High ←──────────────────→ Low
            acks=0    acks=1      acks=all
            retries=0             retries=∞

Reliability: Low ←──────────────────→ High
             acks=0    acks=1      acks=all
             retries=0             retries=∞

Recommended Configuration (Production)

// ✅ Production-ready configuration
props.put("acks", "all");                     // Max durability
props.put("retries", Integer.MAX_VALUE);      // Retry indefinitely
props.put("enable.idempotence", "true");      // No duplicates
props.put("delivery.timeout.ms", 120000);     // 2 minutes
props.put("max.in.flight.requests.per.connection", 5); // Good throughput

Why?

No data loss (acks=all)
No duplicates (idempotence=true)
Automatic retries
Good performance

Summary

Key Takeaways

✅ At-most-once: Fast, may lose messages
✅ At-least-once: Reliable, may duplicate messages
✅ Exactly-once: No loss, no duplicates (use idempotence)
✅ acks=0: No waiting (at-most-once)
✅ acks=1: Leader ack (balanced)
✅ acks=all: All ISR ack (most durable)
✅ Retries: Enable with Integer.MAX_VALUE
✅ Idempotence: Prevents duplicates from retries
✅ delivery.timeout.ms: Total time before giving up

Decision Matrix

Need speed, OK with loss?          → acks=0
Need reliability, can handle dups? → acks=all + retries (at-least-once)
Need no loss, no dups?             → acks=all + retries + idempotence

What's Next

Module 5.2: Idempotent Producer
Deep dive into how idempotence prevents duplicates
Module 5.3: Exactly-Once Semantics (Transactions)
Multi-partition atomic writes

Previous Module: ← 4.1 - Partitioning Strategy & Message Ordering
Next Module: 5.2 - Idempotent Producer →

5.1 Delivery Semantics Explained

Module Overview

1. The Three Delivery Semantics

Overview

2. At-Most-Once Delivery

Definition

Characteristics

How to Achieve At-Most-Once

When Messages Are Lost

Use Cases for At-Most-Once

Pros and Cons

3. At-Least-Once Delivery

Definition

Characteristics

How to Achieve At-Least-Once

When Messages Are Duplicated

Use Cases for At-Least-Once

Handling Duplicates

Consumer-Side Deduplication

Pros and Cons

4. Exactly-Once Delivery

Definition

Characteristics

Two Levels of Exactly-Once

1. Idempotent Producer (Single partition, single session)

2. Transactional Producer (Multiple partitions, exactly-once end-to-end)

Quick Comparison

When to Use Each

5. Understanding acks (Acknowledgments)

The acks Configuration

acks=0 (No Acknowledgment)

acks=1 (Leader Acknowledgment)

acks=all or acks=-1 (All In-Sync Replicas)

acks Comparison

ISR (In-Sync Replicas)

Minimum ISR (min.insync.replicas)

6. Producer Retries

Why Retries Matter

Configuring Retries

Retry Behavior

Retriable vs Non-Retriable Errors

Retriable Errors (Producer auto-retries)

Non-Retriable Errors (Fail immediately)

7. Timeouts and Delivery Guarantees

Key Timeout Configurations

delivery.timeout.ms

request.timeout.ms

Timeout Scenarios

Scenario 1: Success Within Timeout

Scenario 2: Retry Within Timeout

Scenario 3: Delivery Timeout

8. Delivery Guarantees Summary Table

9. Configuration Templates

Template 1: At-Most-Once (Fast, Lossy)

Template 2: At-Least-Once (Reliable, May Duplicate)

Template 3: Exactly-Once (No Loss, No Duplicates)

10. Trade-offs and Recommendations

Latency vs Durability

Throughput vs Reliability

Recommended Configuration (Production)

Summary

Key Takeaways

Decision Matrix

What's Next

5. Understanding `acks` (Acknowledgments)

The `acks` Configuration

`acks=0` (No Acknowledgment)

`acks=1` (Leader Acknowledgment)

`acks=all` or `acks=-1` (All In-Sync Replicas)

`acks` Comparison

Minimum ISR (`min.insync.replicas`)

`delivery.timeout.ms`

`request.timeout.ms`