10.6 – Graceful Lifecycle and Testing Strategies

Module: M10 – Advanced Topics & Architecture
Section: 10.6 – Graceful Lifecycle & Testing Strategies
Goal: Implement graceful shutdown and comprehensive testing strategies

Introduction

Production Kafka systems require graceful lifecycle management (startup, shutdown, restarts) and rigorous testing (functional, performance, chaos) to ensure reliability.

Graceful Broker Shutdown

Why Graceful Shutdown Matters

Ungraceful Shutdown (e.g., kill -9):

Partitions marked offline immediately
Consumer rebalancing triggered
Under-replicated partitions until followers catch up
Potential data loss (uncommitted messages)

Graceful Shutdown:

Broker transfers leadership to followers
ISR replicas fully synced before shutdown
No under-replicated partitions
Zero data loss

Performing Graceful Shutdown

Enable Controlled Shutdown (server.properties):

controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000

Shutdown Command:

/opt/kafka/bin/kafka-server-stop.sh

What Happens:

Broker stops accepting new connections
Transfers partition leadership to ISR followers
Waits for followers to sync (up to replica.lag.time.max.ms)
Cleanly shuts down

Time: 30-60 seconds (vs 1-2 seconds for ungraceful).

Monitoring Shutdown

Logs (/var/log/kafka/server.log):

[Kafka Server], Shutting down broker
[Kafka Server], Transferred leadership for partitions: [orders-0, payments-2]
[Kafka Server], Shut down completed

Metrics:

kafka.server:type=KafkaServer,name=BrokerState (should go to 0 = NOT_RUNNING)

Graceful Producer Shutdown

Producer Lifecycle

Shutdown Steps:

Stop sending new messages
Flush in-flight batches
Commit transactions (if transactional)
Close producer

Code Example:

// Signal shutdown
shutdownFlag.set(true);

// Flush pending messages
producer.flush();

// If transactional
if (transactional) {
    try {
        producer.commitTransaction();
    } catch (Exception e) {
        producer.abortTransaction();
    }
}

// Close producer (waits for in-flight requests)
producer.close(Duration.ofSeconds(30));

Timeout: 30 seconds (allows time to complete in-flight requests).

Graceful Consumer Shutdown

Consumer Lifecycle

Shutdown Steps:

Stop polling
Commit offsets
Leave consumer group
Close consumer

Code Example:

// Signal shutdown
shutdownFlag.set(true);

// Stop polling (break out of poll loop)

// Commit offsets
consumer.commitSync();

// Close consumer (triggers rebalance)
consumer.close(Duration.ofSeconds(30));

What Happens:

Consumer sends LeaveGroup request
Rebalancing triggered (partitions reassigned to remaining consumers)
Clean shutdown (no partition "theft")

Rolling Restarts

Zero-Downtime Rolling Restart

Use Cases:

Config changes (that require restart)
Binary upgrades
Maintenance (OS patches)

Procedure:

Select one broker (e.g., Broker 1)
Gracefully stop broker: kafka-server-stop.sh
Wait for ISR sync (check no under-replicated partitions)
Start broker: kafka-server-start.sh
Wait for broker to rejoin (5-10 minutes)
Repeat for next broker

Key: Only restart one broker at a time.

Monitoring Rolling Restart

Metric: Under-replicated partitions

kafka-topics.sh --describe --under-replicated-partitions \
  --bootstrap-server localhost:9092

Healthy: 0 under-replicated partitions
Alert: > 0 (wait before restarting next broker)

Pre-Production Testing

Testing Layers

1. Unit Tests (application code)

Mock Kafka (e.g., MockProducer, MockConsumer)
Test producer/consumer logic

2. Integration Tests (embedded Kafka)

Use kafka-junit or Testcontainers
Test end-to-end producer → Kafka → consumer

3. Staging Tests (full Kafka cluster)

Mirror production environment
Full regression tests

4. Production Validation (canary deployments)

Deploy to 1% of traffic
Monitor for errors before full rollout

Performance Testing

Load Testing

Tool: kafka-producer-perf-test.sh

Example:

kafka-producer-perf-test.sh \
  --topic test-topic \
  --num-records 10000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 acks=all

Metrics:

Throughput (records/sec, MB/sec)
Latency (avg, p50, p99, p999)

Goal: Establish baseline performance (e.g., 100k msg/sec, p99 < 50ms).

Consumer Performance Test

Tool: kafka-consumer-perf-test.sh

Example:

kafka-consumer-perf-test.sh \
  --broker-list localhost:9092 \
  --topic test-topic \
  --messages 10000000 \
  --threads 10

Metrics:

Throughput (MB/sec)
Consumer lag

Goal: Verify consumers can keep up with producers.

Chaos Testing

What Is Chaos Testing?

Purpose: Simulate failures to validate resilience.

Scenarios:

Broker crash
Network partition
Disk failure
High CPU load
Memory exhaustion

Chaos Monkey for Kafka

Tool: Chaos Mesh, LitmusChaos, or custom scripts.

Example 1: Random broker kill

# Kill random broker
kill -9 $(pgrep -f kafka.Kafka | shuf -n 1)

# Monitor recovery
kafka-topics.sh --describe --under-replicated-partitions

Expected: Partition failover in 5-30 seconds, no data loss.

Example 2: Network partition

# Block traffic between Broker 1 and 2
iptables -A INPUT -s <broker2-ip> -j DROP
iptables -A OUTPUT -d <broker2-ip> -j DROP

# Wait 5 minutes

# Restore network
iptables -D INPUT -s <broker2-ip> -j DROP
iptables -D OUTPUT -d <broker2-ip> -j DROP

Expected: ISR shrinks, then recovers when network heals.

Failure Injection

Example 3: Disk full

# Fill disk to 95%
dd if=/dev/zero of=/var/kafka/data/filler bs=1G count=100

# Monitor broker (should stop accepting writes gracefully)

# Clean up
rm /var/kafka/data/filler

Expected: Broker stops accepting writes, alerts fired.

DR Drills

Disaster Recovery Testing

Goal: Validate DR plan (failover from primary to DR cluster).

Procedure:

Schedule DR drill (announce to teams)
Simulate primary cluster failure (shut down all brokers)
Execute failover (follow DR runbook, see 10.3.5)
Verify DR cluster (produce/consume test messages)
Measure RTO/RPO (actual vs target)
Fail back to primary (restore normal operations)
Document issues (update runbook)

Frequency: Quarterly (minimum).

Key Takeaways

Graceful Broker Shutdown: Enable controlled.shutdown.enable=true, use kafka-server-stop.sh.
Graceful Producer Shutdown: Flush, commit transactions, close with timeout.
Graceful Consumer Shutdown: Commit offsets, close consumer (triggers rebalance).
Rolling Restarts: One broker at a time, wait for ISR sync between restarts.
Testing Layers: Unit (mock), integration (embedded Kafka), staging (full cluster), production (canary).
Performance Testing: Use kafka-producer-perf-test.sh, kafka-consumer-perf-test.sh to establish baselines.
Chaos Testing: Simulate failures (broker crash, network partition, disk full), validate recovery.
DR Drills: Quarterly drills to validate failover procedures, measure RTO/RPO.

Course Complete!

Congratulations! You've completed M10 – Advanced Topics & Architecture.

What We Covered:

✅ Transactions & Exactly-Once Semantics (10.1)
✅ Multi-Datacenter Replication (10.2)
✅ High Availability & Disaster Recovery (10.3)
✅ Cluster Governance & Architecture Decisions (10.4)
✅ Advanced Performance Tuning (10.5)
✅ Graceful Lifecycle & Testing Strategies (10.6)

Total: 25 modules covering senior-level Kafka architecture, operations, and production best practices.

What's Next?

Apply Your Knowledge:

Design multi-region Kafka architecture
Implement EOS pipelines
Optimize JVM/OS/network for your workloads
Run DR drills and chaos tests
Establish governance policies for your organization

Keep Learning:

Kafka KIPs (Kafka Improvement Proposals)
Confluent blog and tech talks
Kafka Summit presentations
Community Slack/Discord

References:

All previous modules (M1-M9)
Apache Kafka documentation
Production deployment guides