10.1.6 – EOS Performance Tradeoffs

Module: M10 – Advanced Topics & Architecture
Section: 10.1 – Transactions & Exactly-Once Semantics
Goal: Understand the performance cost of transactions and when to use them

Introduction

Exactly-once semantics provide strong guarantees but come with performance costs. This module analyzes the throughput, latency, and resource tradeoffs, helping you decide when transactions are worth the overhead.

Performance Impact Overview

Throughput Comparison

Key Insight: Batching messages per transaction dramatically improves throughput.

Latency Comparison

Impact: Transactions add ~4-5ms median latency and 10-40ms tail latency.

Cost Breakdown

1. Transaction Coordinator Overhead

What Happens:

• Producer sends InitProducerIdRequest (once per restart)
• Producer sends AddPartitionsToTxnRequest (first write per partition)
• Producer sends EndTxnRequest (every commit/abort)
• Coordinator writes to __transaction_state topic

Cost: ~2-3ms per transaction (coordinator communication + log write).

2. Control Marker Writes

What Happens:

• Coordinator sends COMMIT/ABORT markers to all partitions in transaction
• Each partition leader writes marker to log
• Followers replicate markers

Cost: ~1-2ms per partition involved (can be significant for fan-out writes).

Example:

Transaction writes to 10 partitions
→ 10 COMMIT markers must be written
→ ~10-20ms overhead (1-2ms per partition)

3. Consumer-Side Overhead (read_committed)

What Happens:

• Consumer waits for COMMIT/ABORT markers before reading uncommitted messages
• Broker filters aborted messages server-side
• Consumer tracks LSO (Last Stable Offset)

Cost: ~5-10ms additional latency (waiting for markers).

4. Replication Overhead

What Happens:

• Transaction metadata (PID, epoch, sequence numbers) added to every message
• Increases message size by ~20-30 bytes per message
• Replication must replicate metadata

Cost: ~2-5% network/disk overhead.

Batching: The Performance Multiplier

Impact of Batch Size on Throughput

Batch Size (messages/transaction) → Throughput (messages/sec)

1      →  50,000   (5% of baseline)
10     → 300,000  (30%)
100    → 800,000  (80%)
1,000  → 900,000  (90%)
10,000 → 950,000  (95%)

Key Insight: Batching 100-1000 messages per transaction recovers most of the performance.

Optimal Batch Size Formula

Optimal Batch Size = (Throughput Target × Transaction Commit Time) / Message Processing Time

Example:
  Target: 100,000 msg/sec
  Commit Time: 10ms
  Processing Time: 0.1ms/msg

  Optimal Batch = (100,000 × 0.01) / 0.0001 = 10,000 messages/transaction

Real-World Benchmark Data

Test Setup

Hardware:

• 3 brokers (AWS r5.xlarge instances)
• CPU: 4 vCPUs per broker
• Memory: 32 GB per broker
• Storage: NVMe SSD (GP3)
• Network: 10 Gbps

Cluster Configuration:

• Replication factor: 3
• Min ISR: 2
• acks=all

Workload:

• Producer: 1M messages (1KB each)
• Topic: 3 partitions
• Consumer: Single consumer with read_committed

Benchmark Results

Key Insights:

1. Throughput: ~39% reduction due to transaction coordination overhead
2. Latency: P99 latency more than doubles (12ms → 28ms)
3. CPU: Significant increase due to transaction state management
4. Network: Slightly lower due to better batching
5. Disk: Slightly higher due to transaction markers and state log writes

Impact of Batch Size on EOS Performance

Conclusion: Batching 100-1000 messages per transaction recovers most throughput (80-90% of baseline).

Tuning for Performance

Complete Tuning Configuration Table

Producer Configuration

Maximize Throughput:

# Increase batch size
batch.size=65536          # 64 KB (default: 16 KB)
linger.ms=10              # Wait 10ms to fill batch

# Compression reduces network/disk I/O
compression.type=snappy   # or lz4, zstd

# Increase transaction timeout for large batches
transaction.timeout.ms=300000  # 5 minutes

# Max parallelism for EOS
max.in.flight.requests.per.connection=5

Maximize Latency:

# Smaller batches (faster commits)
batch.size=16384          # 16 KB
linger.ms=0               # Don't wait

# Shorter transaction timeout
transaction.timeout.ms=30000  # 30 seconds

# Reduce parallelism for lower latency
max.in.flight.requests.per.connection=1

Partition Scaling for EOS

Problem: EOS serializes transactions per partition.

Solution: Scale partitions to parallelize transactions.

# Double partition count to improve parallelism
kafka-topics --alter --topic orders --partitions 6 --bootstrap-server localhost:9092

Impact:

• 3 partitions: 52,000 msg/s
• 6 partitions: 95,000 msg/s (~83% improvement)
• 12 partitions: 120,000 msg/s (diminishing returns)

Tradeoff: More partitions → higher ZooKeeper/controller overhead.

Rule of Thumb: 100-300 partitions per broker for balanced performance.

Consumer Configuration

Reduce Latency:

# Use read_uncommitted if EOS not required
isolation.level=read_uncommitted

# Increase fetch size to reduce round-trips
fetch.min.bytes=1048576   # 1 MB

Maximize Consistency:

# Use read_committed for EOS
isolation.level=read_committed

# Increase timeout to tolerate slow transactions
max.poll.interval.ms=600000  # 10 minutes

When to Use Transactions

Use Transactions If:

✅ Financial transactions (payments, orders)
✅ Critical data pipelines (no duplicates tolerated)
✅ Multi-partition atomicity (writes to multiple topics must succeed or fail together)
✅ Read-process-write pipelines (consume, transform, produce atomically)
✅ State management (state updates + Kafka writes must be consistent)

Don't Use Transactions If:

❌ Logging and metrics (duplicates acceptable)
❌ High-throughput, low-latency requirements (< 1ms latency target)
❌ Single partition writes (idempotent producer sufficient)
❌ Fire-and-forget use cases (no need for acknowledgment)
❌ Cost-sensitive deployments (transaction overhead not justified)

Alternative Approaches

1. Idempotent Producer (Without Transactions)

Use Case: Single partition writes, no multi-topic atomicity needed.

Config:

enable.idempotence=true  # Default in Kafka 3.0+
# No transactional.id

Benefit: ~95% throughput (vs 80-90% with transactions), ~1ms latency (vs 5ms).

Limitation: No atomicity across partitions.

2. Application-Level Deduplication

Use Case: Consumer can detect and skip duplicates.

Approach:

Set<String> processedIds = new HashSet<>();

for (ConsumerRecord<String, String> record : records) {
    if (processedIds.contains(record.key())) {
        continue;  // Skip duplicate
    }
    
    process(record);
    processedIds.add(record.key());
}

Benefit: No transaction overhead.

Limitation: Requires storing processed IDs (memory/disk cost).

3. Idempotent Processing

Use Case: Processing same message multiple times has no side effects.

Example: Idempotent database UPDATE

-- Idempotent (running twice = same result)
UPDATE accounts SET balance = 1000 WHERE id = 123;

-- Not idempotent (running twice = different result)
UPDATE accounts SET balance = balance - 100 WHERE id = 123;

Benefit: At-least-once delivery sufficient (no transactions needed).

Cost-Benefit Analysis

Scenario 1: Payment Processing

Requirements:

• No duplicates (double charges unacceptable)
• Multi-topic writes (orders + payments + notifications)
• Read-process-write pipeline

Cost of Transactions:

• 20% throughput reduction
• 5ms added latency

Cost of NOT Using Transactions:

• Potential duplicates → customer complaints, refunds
• Inconsistent state → manual reconciliation
• Regulatory risk (PCI-DSS compliance)

Decision: Use transactions (cost justified by business requirements).

Scenario 2: Application Logging

Requirements:

• High throughput (1M logs/sec)
• Low latency (< 1ms)
• Duplicates acceptable

Cost of Transactions:

• 80% throughput reduction (200k logs/sec)
• 5ms added latency

Cost of NOT Using Transactions:

• Occasional duplicate logs (minimal impact)

Decision: Don't use transactions (overhead not justified).

Monitoring Transaction Performance

Key Metrics

1. Transaction Commit Time

kafka.producer:type=producer-metrics,client-id=*,attribute=txn-commit-time-avg

Healthy: < 10ms
Warning: > 50ms
Critical: > 100ms

2. Transaction Throughput

Committed Transactions / Second = 
  kafka.producer:attribute=txn-commit-total (delta) / time

Target: Match application throughput requirements.

3. Abort Rate

Abort Rate = txn-abort-total / (txn-commit-total + txn-abort-total)

Healthy: < 1%
Warning: > 5%
Critical: > 10%

4. Consumer Lag (with LSO)

Blocked Lag = LEO - LSO

Healthy: 0
Warning: > 1000 messages
Critical: > 10000 messages

Performance Benchmarking

Benchmark Setup

Hardware: 3 brokers, 16 cores, 64 GB RAM, NVMe SSD

Test 1: Throughput (No Transactions)

kafka-producer-perf-test.sh \
  --topic perf-test \
  --num-records 10000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 acks=all

Result: ~1,000,000 messages/sec

Test 2: Throughput (With Transactions, Batch=1)

# Custom test with transactions (1 msg/txn)
# Result: ~50,000 messages/sec (-95%)

Test 3: Throughput (With Transactions, Batch=100)

# Custom test with transactions (100 msg/txn)
# Result: ~800,000 messages/sec (-20%)

Real-World Example: E-Commerce Platform

Before Transactions:

• Throughput: 500,000 orders/sec
• Latency: p99 = 10ms
• Duplicate rate: 0.1% (500 duplicates/sec)

After Transactions:

• Throughput: 400,000 orders/sec (-20%)
• Latency: p99 = 50ms (+40ms)
• Duplicate rate: 0% (zero duplicates)

ROI:

• Cost: 20% throughput loss, 40ms latency increase
• Benefit: Zero duplicates, eliminated manual reconciliation (saved 10 eng hours/week)

Decision: Worth the cost.

Key Takeaways

1. Transaction Overhead: ~10-20% throughput reduction, ~5ms median latency increase.

2. Batching Critical: Batch 100-1000 messages per transaction to recover 80-90% throughput.

3. Consumer Impact: read_committed adds ~5-10ms latency (waiting for markers).

4. Use Cases: Financial transactions, critical pipelines, multi-partition atomicity.

5. Alternatives: Idempotent producer (single partition), application-level dedup, idempotent processing.

6. Cost-Benefit: Justify transaction cost based on business requirements (duplicate cost vs performance cost).

7. Monitoring: Track commit time, abort rate, and blocked lag.

8. Tuning: Increase batch.size, linger.ms, and transaction.timeout.ms for throughput.

What's Next?

Next Section: 10.2 – Multi-Datacenter Replication & Geo Architecture

What We Covered in Section 10.1:

• ✅ Transaction protocol and coordinator (10.1.1)
• ✅ Producer transactional API (10.1.2)
• ✅ Consumer isolation levels (10.1.3)
• ✅ End-to-end EOS pipelines (10.1.4)
• ✅ EOS anti-patterns and debugging (10.1.5)
• ✅ EOS performance tradeoffs (10.1.6)

What's Coming:

• ❌ Multi-cluster design models
• ❌ MirrorMaker and MM2 architecture
• ❌ Active-passive vs active-active replication
• ❌ Geo-DR patterns

References:

• Transaction protocol → 10.1.1
• Producer configs → M3: Producers
• Consumer configs → M4: Consumers
• Performance tuning basics → M6: Operations