10.1.5 – EOS Anti-Patterns and Debugging

Module: M10 – Advanced Topics & Architecture
Section: 10.1 – Transactions & Exactly-Once Semantics
Goal: Identify common EOS anti-patterns and master debugging transactional issues

Introduction

Implementing exactly-once semantics is complex. This module covers common mistakes, anti-patterns, and debugging techniques to help you avoid and fix transactional issues.

Common Anti-Patterns

Anti-Pattern 1: Non-Unique transactional.id

Problem: Multiple producers using the same transactional.id.

Example:

// ❌ All instances use same transactional.id
Properties props = new Properties();
props.put("transactional.id", "my-producer");  // Same for all instances

// Instance 1 starts
// Instance 2 starts with same ID
// → Instance 1 fenced, fails

Result: Producers constantly fence each other, causing transaction failures.

Solution: Unique transactional.id per logical producer instance.

// ✅ Unique ID per instance
String instanceId = System.getenv("INSTANCE_ID");  // e.g., from Kubernetes pod name
props.put("transactional.id", "my-producer-" + instanceId);

Best Practice:

• Single-threaded app: One transactional.id
• Multi-threaded app: One transactional.id per thread
• Distributed app: One transactional.id per instance/partition

Anti-Pattern 2: Randomized transactional.id

Problem: transactional.id changes on every restart.

Example:

// ❌ New ID on every restart
props.put("transactional.id", UUID.randomUUID().toString());

Result: Each restart creates a new producer ID, defeating zombie fencing and state recovery.

Solution: Stable transactional.id across restarts.

// ✅ Stable ID (based on consumer partition)
String transactionalId = "pipeline-1-partition-" + partition.partition();
props.put("transactional.id", transactionalId);

Anti-Pattern 3: Transaction Per Message

Problem: Committing a transaction for every single message.

Example:

// ❌ One transaction per message (expensive)
for (Message msg : messages) {
    producer.beginTransaction();
    producer.send(msg);
    producer.commitTransaction();  // 5-10ms overhead per message
}

Result: 10-100x throughput reduction due to transaction overhead.

Solution: Batch messages per transaction.

// ✅ Batch 100-1000 messages per transaction
producer.beginTransaction();
for (Message msg : batch) {
    producer.send(msg);
}
producer.commitTransaction();  // Overhead amortized

Rule of Thumb: 100-1000 messages per transaction for optimal throughput.

Anti-Pattern 4: Mixing Transactional and Non-Transactional Writes

Problem: Writing to same topic with both transactional and non-transactional producers.

Example:

Producer A (transactional) → Topic
Producer B (non-transactional) → Same Topic

Result: read_committed consumers see inconsistent data (non-transactional messages visible immediately, transactional messages delayed).

Solution: Use same transaction semantics for all producers writing to a topic.

All producers → transactional (if EOS required)
Or all producers → non-transactional (if EOS not needed)

Anti-Pattern 5: Not Handling ProducerFencedException

Problem: Retrying after being fenced.

Example:

try {
    producer.commitTransaction();
} catch (ProducerFencedException e) {
    // ❌ Retrying won't help (fenced = terminated)
    retry();
}

Result: Producer continues writing with outdated epoch, all writes rejected.

Solution: Close producer immediately on fencing.

catch (ProducerFencedException e) {
    log.error("Producer fenced, shutting down", e);
    producer.close();
    System.exit(1);  // Let orchestration restart
}

Anti-Pattern 6: Ignoring Transaction Timeout

Problem: Long-running transactions exceeding transaction.timeout.ms.

Example:

producer.beginTransaction();

// Slow processing (e.g., external API call takes 5 minutes)
for (Message msg : largeNetwork) {
    String result = slowApiCall(msg);  // 5 min
    producer.send(result);
}

producer.commitTransaction();  // ❌ Already timed out

Result: Transaction auto-aborted by coordinator, commit fails.

Solution: Tune transaction.timeout.ms or break into smaller transactions.

// ✅ Option 1: Increase timeout
props.put("transaction.timeout.ms", "600000");  // 10 minutes

// ✅ Option 2: Smaller transactions
for (Batch batch : batches) {
    producer.beginTransaction();
    processBatch(batch);  // < 1 minute
    producer.commitTransaction();
}

Anti-Pattern 7: Not Using read_committed with Transactional Producer

Problem: Producer uses transactions, consumer uses read_uncommitted.

Example:

// Producer
props.put("transactional.id", "my-producer");

// Consumer
props.put("isolation.level", "read_uncommitted");  // ❌ Defeats EOS

Result: Consumer sees aborted messages, defeating exactly-once guarantees.

Solution: Match isolation level to producer.

// ✅ Consumer reads committed messages only
props.put("isolation.level", "read_committed");

Anti-Pattern 8: Committing Offsets Outside Transaction

Problem: Committing consumer offsets separately from producing messages.

Example:

producer.beginTransaction();
producer.send(...);
producer.commitTransaction();

consumer.commitSync();  // ❌ Separate commit (not atomic)

Result: Producer commit succeeds but consumer commit fails → messages reprocessed (duplicates).

Solution: Use sendOffsetsToTransaction().

// ✅ Atomic offset commit
producer.beginTransaction();
producer.send(...);
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();  // Offsets committed atomically

Debugging Transactional Issues

Issue 1: Transactions Constantly Aborting

Symptoms:

• High txn-abort-total metric
• Consumer lag increasing
• Logs showing KafkaException repeatedly

Possible Causes:

1. Network instability (coordinator unreachable)
2. Broker failures during transaction
3. Application errors (uncaught exceptions)
4. Transaction timeout too short

Debugging Steps:

1. Check broker logs:

grep "transaction" /var/log/kafka/server.log

Look for:

• TransactionCoordinator: Aborting transaction
• Transaction timed out

2. Check producer metrics:

// Transaction abort rate
kafka.producer:type=producer-metrics,client-id=*,attribute=txn-abort-total

// Transaction commit time
kafka.producer:type=producer-metrics,client-id=*,attribute=txn-commit-time-avg

Alert If: Abort rate > 5% of total transactions.

3. Increase transaction timeout:

transaction.timeout.ms=300000  # 5 minutes (from 60 seconds)

4. Add detailed logging:

try {
    producer.commitTransaction();
    log.info("Transaction committed successfully");
} catch (KafkaException e) {
    log.error("Transaction failed: {}", e.getMessage(), e);
    producer.abortTransaction();
}

Issue 2: Consumer Stuck (Not Making Progress)

Symptoms:

• Consumer lag not decreasing
• read_committed consumer appears frozen
• No errors in logs

Possible Causes:

1. Stuck transaction (producer didn't commit/abort)
2. LSO (Last Stable Offset) not advancing
3. Transaction timeout not expired yet

Debugging Steps:

1. Check LSO vs LEO:

# Get log-end-offset (LEO)
kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 \
  --topic my-topic \
  --time -1  # Latest offset

# Compare with consumer position (LSO)
kafka-consumer-groups.sh --describe \
  --group my-group \
  --bootstrap-server localhost:9092

If LEO - Consumer Position > 1000, LSO is stuck.

2. Find stuck transaction:

# Check __transaction_state topic
kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic __transaction_state \
  --from-beginning \
  --formatter "kafka.coordinator.transaction.TransactionLog\$TransactionLogMessageFormatter"

Look for: Transactions in "Ongoing" state for > transaction.timeout.ms.

3. Wait for transaction timeout:

• Transaction coordinator will auto-abort after transaction.timeout.ms
• Default: 60 seconds
• Check broker config: transaction.max.timeout.ms

4. Restart problematic producer:

• If producer crashed without aborting, restart will trigger abort

Issue 3: Producer Fenced Unexpectedly

Symptoms:

• ProducerFencedException in logs
• Producer stops writing
• No obvious duplicate instances

Possible Causes:

1. Another producer with same transactional.id started
2. Coordinator failover incremented epoch
3. Producer took too long to send InitProducerIdRequest

Debugging Steps:

1. Check for duplicate instances:

# Search logs for multiple initializations
grep "Invoking InitProducerId" producer.log | wc -l

If > 1, multiple instances are starting with same ID.

2. Check coordinator logs:

grep "Fenced producer" /var/log/kafka/server.log

Look for: Fenced producer pid=12345 with epoch 1 because new producer with epoch 2 started

3. Verify transactional.id uniqueness:

# In Kubernetes
kubectl get pods -o json | jq '.items[].spec.containers[].env[] | select(.name=="TRANSACTIONAL_ID") | .value'

4. Add fencing detection:

catch (ProducerFencedException e) {
    log.error("Producer fenced! Instance ID: {}, Transactional ID: {}",
              instanceId, transactionalId, e);
    // Investigate duplicate instances
    producer.close();
    System.exit(1);
}

Issue 4: High Latency with read_committed

Symptoms:

• p99 latency increased by 50-100ms
• Consumer processing slow
• No errors

Cause: Waiting for transaction markers (LSO blocking).

Debugging Steps:

1. Measure transaction commit time:

kafka.producer:type=producer-metrics,client-id=*,attribute=txn-commit-time-avg

If > 100ms, transactions are slow.

2. Reduce transaction size:

// Smaller batches = faster commits
int batchSize = 100;  // Reduce from 1000

3. Increase consumer timeout:

max.poll.interval.ms=600000  # 10 minutes (tolerate longer waits)

4. Monitor LSO lag:

LEO - LSO = Blocked lag

Alert If: Blocked lag > 1000 messages.

Debugging Tools

1. Transaction CLI Commands

List Active Transactions:

kafka-transactions.sh --bootstrap-server localhost:9092 --list

Output:

transactionalId | state    | producerId | producerEpoch | timeout
eos-producer-1  | Ongoing  | 1001       | 3             | 60000ms
payment-proc-2  | Complete | 1002       | 1             | 60000ms

Describe Specific Transaction:

kafka-transactions.sh --bootstrap-server localhost:9092 \
  --describe --transactional-id eos-producer-1

Output:

TransactionalId: eos-producer-1
ProducerId: 1001
ProducerEpoch: 3
State: Ongoing
TopicPartitions: [orders-0, payments-2, notifications-1]
TransactionStartTime: 2026-02-01T10:15:30Z
TransactionTimeout: 60000ms

Force-Abort a Stuck Transaction:

kafka-transactions.sh --bootstrap-server localhost:9092 \
  --abort --transactional-id eos-producer-1

Use Case: Transaction stuck in "Ongoing" state for longer than timeout (zombie producer).

Warning: Only use this if you're certain the producer is dead. Aborting an active producer's transaction will cause it to be fenced.

Check for Zombie Transactions in Logs:

grep "ProducerFencedException" /var/log/kafka/server.log

# Or for transaction timeouts
grep "Transaction timed out" /var/log/kafka/server.log

2. JMX Metrics for EOS Monitoring

Enable JMX on Producer:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false"

Critical EOS Metrics Table:

Access Metrics via JConsole:

jconsole localhost:9999
# Navigate to: MBeans → kafka.producer → producer-metrics

Access Metrics via JMX Exporter (Prometheus):

# Download JMX exporter
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.17.0/jmx_prometheus_javaagent-0.17.0.jar

# Start producer with JMX exporter
java -javaagent:jmx_prometheus_javaagent-0.17.0.jar=8080:kafka-producer.yml \
  -jar my-producer-app.jar

Sample kafka-producer.yml:

rules:
- pattern: kafka.producer<type=producer-metrics, client-id=(.+)><>txn-commit-time-avg
  name: kafka_producer_txn_commit_time_avg
  type: GAUGE

- pattern: kafka.producer<type=producer-metrics, client-id=(.+)><>txn-abort-rate
  name: kafka_producer_txn_abort_rate
  type: GAUGE

Grafana Dashboard Query (PromQL):

# Alert on high transaction abort rate
sum(rate(kafka_producer_txn_abort_rate[1m])) by (transactional_id) > 0

# Alert on slow transaction commits
kafka_producer_txn_commit_time_avg > 1000

# Alert on high transaction duration
kafka_producer_txn_duration_avg > 10000

JMX Query via Command Line:

# Using jmxterm
echo "get -b kafka.producer:type=producer-metrics,client-id=my-producer txn-commit-time-avg" | \
  java -jar jmxterm.jar -l localhost:9999

3. Transaction State Inspection

View transaction log:

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic __transaction_state \
  --from-beginning \
  --formatter "kafka.coordinator.transaction.TransactionLog\$TransactionLogMessageFormatter"

Output:

TransactionalId: my-producer-1
ProducerId: 12345
ProducerEpoch: 1
State: Ongoing
Partitions: [orders-0, payments-2]
Timeout: 60000ms
LastUpdateTimestamp: 1643723400000

4. Consumer Offset Inspection

View committed offsets:

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic __consumer_offsets \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

5. Broker Logs

Enable debug logging:

# log4j.properties
log4j.logger.kafka.coordinator.transaction=DEBUG
log4j.logger.org.apache.kafka.clients.producer=DEBUG

Useful Log Searches:

# Find transaction coordinator activity
grep "TransactionCoordinator" /var/log/kafka/server.log

# Find producer fencing events
grep "Fenced producer" /var/log/kafka/server.log

# Find transaction timeouts
grep "Transaction.*timed out" /var/log/kafka/server.log

6. Monitoring Transaction Health - Complete Checklist

Daily Monitoring:

• Check txn-abort-rate (should be near 0)
• Check txn-duration-avg (should be < 10s)
• Check consumer lag for read_committed consumers
• Verify no stuck transactions (check __transaction_state)

Alert Configuration:

# Prometheus AlertManager rules
groups:
- name: kafka_transactions
  rules:
  - alert: HighTransactionAbortRate
    expr: rate(kafka_producer_txn_abort_rate[5m]) > 0.05
    for: 5m
    annotations:
      summary: "Transaction abort rate > 5% for {{ $labels.transactional_id }}"
      
  - alert: SlowTransactionCommits
    expr: kafka_producer_txn_commit_time_avg > 5000
    for: 5m
    annotations:
      summary: "Transaction commits taking > 5s for {{ $labels.client_id }}"
      
  - alert: TransactionTimeouts
    expr: rate(kafka_producer_txn_timeout_rate[5m]) > 0
    for: 1m
    annotations:
      summary: "Transaction timeouts detected for {{ $labels.transactional_id }}"

Best Practices

1. Monitor Transaction Metrics

// Alert on high abort rate
if (abortRate > 0.05) {  // > 5%
    alert("High transaction abort rate");
}

// Alert on slow commits
if (commitTimeAvg > 10000) {  // > 10 seconds
    alert("Slow transaction commits");
}

2. Implement Circuit Breaker

int consecutiveFailures = 0;
int maxFailures = 3;

while (true) {
    try {
        producer.beginTransaction();
        // ... process ...
        producer.commitTransaction();
        consecutiveFailures = 0;  // Reset on success
    } catch (KafkaException e) {
        consecutiveFailures++;
        if (consecutiveFailures >= maxFailures) {
            log.error("Circuit breaker tripped after {} failures", maxFailures);
            break;  // Stop retrying
        }
        producer.abortTransaction();
    }
}

3. Graceful Shutdown

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    try {
        producer.abortTransaction();  // Abort in-flight transaction
    } catch (Exception e) {
        log.warn("Error aborting transaction on shutdown", e);
    } finally {
        producer.close();
        consumer.close();
    }
}));

Key Takeaways

1. Unique transactional.id: Per logical producer instance, stable across restarts.

2. Batch Messages: 100-1000 per transaction to amortize overhead.

3. Match Isolation Level: Use read_committed with transactional producers.

4. Handle Fencing: Close producer immediately on ProducerFencedException.

5. Transaction Timeout: Tune transaction.timeout.ms for workload.

6. Atomic Offsets: Use sendOffsetsToTransaction() for read-process-write.

7. Monitor Metrics: Track abort rate, commit time, and LSO lag.

8. Debug Tools: Inspect __transaction_state and broker logs for issues.

What's Next?

Next Module: 10.1.6 – EOS Performance Tradeoffs

What We Covered:

• ✅ Common EOS anti-patterns
• ✅ Debugging transactional issues
• ✅ Debugging tools and techniques
• ✅ Best practices

What's Coming:

• ❌ Performance impact of transactions
• ❌ Latency vs throughput tradeoffs
• ❌ When to use (and not use) transactions

References:

• Transaction protocol → 10.1.1
• Producer transactions → 10.1.2
• Consumer isolation → 10.1.3
• End-to-end pipelines → 10.1.4