2.1 Broker Lifecycle Management

Module Overview

Goal: Teach safe broker operations.

What You'll Learn:

• Broker startup and shutdown procedures
• Rolling restarts (zero-downtime maintenance)
• Adding brokers to a cluster
• Removing and decommissioning brokers
• Failure scenarios and recovery strategies
• Safe operational practices

What's NOT Covered Here:

• Deep dive into partition reassignment mechanics (covered in Module 2.3)
• Performance tuning (covered in Module 5.1)
• Monitoring broker health (covered in Module 4.x)

Key Insight: Broker lifecycle management is about controlled change — every operation should be deliberate, monitored, and reversible.

1. The Broker Lifecycle

State Transitions

BROKER LIFECYCLE STATES:

┌──────────┐
│  STOPPED │
└──────────┘
     ↓ start
┌──────────┐
│ STARTING │ ← Loading data, registering with cluster
└──────────┘
     ↓ ready
┌──────────┐
│ RUNNING  │ ← Serving requests, replicating data
└──────────┘
     ↓ shutdown
┌──────────┐
│ STOPPING │ ← Graceful shutdown, flush data
└──────────┘
     ↓
┌──────────┐
│ STOPPED  │
└──────────┘

Normal vs Abnormal Transitions

NORMAL (Controlled):
├─ Startup: Planned, graceful registration
├─ Shutdown: Controlled, data flushed
└─ Restart: Rolling, zero-downtime

ABNORMAL (Uncontrolled):
├─ Crash: Process killed, no cleanup
├─ Network partition: Broker isolated
└─ Disk failure: Data loss risk

2. Broker Startup

Pre-Startup Checklist

BEFORE STARTING A BROKER:

✅ Verify configuration file
   └─ Check broker.id, listeners, log.dirs

✅ Ensure disk space available
   └─ At least 10% free in log.dirs

✅ Check network connectivity
   └─ Can reach other brokers
   └─ Can reach coordination system (ZK/KRaft)

✅ Verify JVM settings
   └─ Heap size appropriate (4-16 GB typical)
   └─ GC settings tuned

✅ Check ports available
   └─ 9092 (default), no conflicts

✅ Review logs directory
   └─ Remove old logs if needed

Startup Sequence

KAFKA BROKER STARTUP PROCESS:

1. Load Configuration
   ├─ Read server.properties
   ├─ Validate required configs
   └─ Apply default values

2. Initialize Log Manager
   ├─ Scan log.dirs
   ├─ Load existing partitions
   └─ Validate log segments

3. Register with Coordination System
   ├─ ZooKeeper: Create /brokers/ids/{id}
   ├─ KRaft: Send registration request
   └─ Receive cluster metadata

4. Start Network Threads
   ├─ Bind to listeners
   ├─ Start acceptor threads
   └─ Ready to accept connections

5. Catch Up Replication
   ├─ Fetch missing data for follower partitions
   ├─ Join ISR for partitions
   └─ Report ready to controller

6. Enter RUNNING State
   └─ Broker now serves client requests

Startup Command

# Standard startup
kafka-server-start.sh /path/to/server.properties

# With JVM options
KAFKA_HEAP_OPTS="-Xms8g -Xmx8g" kafka-server-start.sh server.properties

# As systemd service
systemctl start kafka

# Background startup with logging
nohup kafka-server-start.sh server.properties > /var/log/kafka/server.log 2>&1 &

First-Time Startup

FIRST BROKER IN NEW CLUSTER:

1. Start coordination system first
   └─ ZooKeeper ensemble or KRaft quorum

2. Start first broker
   └─ Will become initial controller

3. Verify startup
   └─ Check logs for "started (kafka.server.KafkaServer)"

4. Validate cluster
   └─ kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Startup Validation

# Check broker is registered
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | head -5

# Check log for successful startup
tail -f /var/log/kafka/server.log | grep "started"

# Verify broker in cluster metadata (ZK mode)
zkCli.sh -server localhost:2181
ls /brokers/ids
# Should see broker ID

# Check metrics endpoint
curl localhost:9999/metrics | grep kafka_server

3. Broker Shutdown

Graceful Shutdown (RECOMMENDED)

PROPER SHUTDOWN PROCESS:

1. Initiate shutdown signal
   └─ SIGTERM (kill -15) or kafka-server-stop.sh

2. Broker enters STOPPING state
   ├─ Stop accepting new client connections
   ├─ Complete in-flight requests
   └─ Flush data to disk

3. Controlled leader migration
   ├─ Step down as leader for partitions
   ├─ Allow controller to elect new leaders
   └─ Minimize unavailability

4. Deregister from cluster
   ├─ Remove from coordination system
   └─ Update metadata

5. Shutdown complete
   └─ Process exits cleanly

Shutdown Commands

# Graceful shutdown (RECOMMENDED)
kafka-server-stop.sh

# Or send SIGTERM
kill -15 <kafka-pid>

# Systemd
systemctl stop kafka

# Wait for shutdown with timeout
timeout 60 kafka-server-stop.sh || kill -9 <kafka-pid>

Controlled Shutdown Configuration

# Enable controlled shutdown (recommended for production)
controlled.shutdown.enable=true

# Max time to wait for controlled shutdown (ms)
controlled.shutdown.max.retries=3
controlled.shutdown.retry.backoff.ms=5000

How it works:

WITH controlled.shutdown.enable=true:
├─ Broker tells controller it's shutting down
├─ Controller migrates leaders to other brokers
├─ Clients automatically reconnect to new leaders
└─ Zero data unavailability

WITHOUT controlled.shutdown.enable=true:
├─ Broker just exits
├─ Controller detects failure (takes seconds)
├─ Emergency leader election
└─ Brief unavailability for affected partitions

Ungraceful Shutdown (AVOID)

FORCED SHUTDOWN (ONLY AS LAST RESORT):

kill -9 <kafka-pid>

Consequences:
├─ Data may not be flushed to disk
├─ Unclean shutdown marker set
├─ Longer recovery time on restart
├─ Possible data loss for un-replicated data
└─ Controller must detect failure (timeout-based)

4. Rolling Restarts

What is a Rolling Restart?

Rolling Restart = Restarting brokers one at a time

Goal:
└─ Zero downtime for the cluster

Requirements:
├─ Replication factor ≥ 2 (better: 3)
├─ min.insync.replicas ≥ 1
└─ Enough capacity to handle load with N-1 brokers

Rolling Restart Procedure

STEP-BY-STEP ROLLING RESTART:

For each broker in cluster:

1. Verify Cluster Health
   ├─ Check under-replicated partitions = 0
   ├─ Check ISR healthy
   └─ Ensure other brokers running

2. Stop Broker (Gracefully)
   └─ kafka-server-stop.sh

3. Wait for Shutdown
   └─ Verify process stopped

4. Perform Maintenance
   ├─ Update config
   ├─ Upgrade binary
   └─ Patch OS

5. Start Broker
   └─ kafka-server-start.sh

6. Validate Startup
   ├─ Check broker registered
   ├─ Check logs for errors
   └─ Verify ISR rejoined

7. Wait for Replication Catch-Up
   └─ Monitor under-replicated partitions → 0

8. Repeat for Next Broker
   └─ Do NOT restart multiple brokers simultaneously

Rolling Restart Best Practices

✅ DO:
├─ Restart one broker at a time
├─ Wait for full catch-up before next broker
├─ Monitor cluster health between restarts
├─ Perform during low-traffic window
├─ Have rollback plan
└─ Document changes

❌ DON'T:
├─ Restart multiple brokers simultaneously
├─ Skip health checks between restarts
├─ Restart controller first (let it be last)
├─ Restart during peak traffic
└─ Rush the process

Automation Script Example

#!/bin/bash
# rolling-restart.sh

BROKERS=(1 2 3)
BOOTSTRAP_SERVER="localhost:9092"

for BROKER_ID in "${BROKERS[@]}"; do
  echo "=== Restarting Broker $BROKER_ID ==="
  
  # Stop broker
  ssh broker-$BROKER_ID "sudo systemctl stop kafka"
  
  # Wait for shutdown
  sleep 10
  
  # Start broker
  ssh broker-$BROKER_ID "sudo systemctl start kafka"
  
  # Wait for startup
  sleep 20
  
  # Check under-replicated partitions
  echo "Waiting for replication catch-up..."
  while true; do
    URP=$(kafka-topics.sh --bootstrap-server $BOOTSTRAP_SERVER \
      --describe --under-replicated-partitions | wc -l)
    
    if [ "$URP" -eq 0 ]; then
      echo "✅ Broker $BROKER_ID fully caught up"
      break
    fi
    
    echo "Under-replicated partitions: $URP (waiting...)"
    sleep 10
  done
  
  echo "Waiting 30s before next broker..."
  sleep 30
done

echo "✅ Rolling restart complete!"

5. Adding Brokers to a Cluster

When to Add Brokers

ADD BROKERS WHEN:
├─ Disk usage > 70%
├─ Network saturation
├─ CPU bottleneck
├─ Need more throughput
└─ Planning for growth

Adding a Broker: Step-by-Step

STEP 1: Prepare New Broker

1. Install Kafka on new server
2. Configure server.properties:
   ├─ broker.id=4 (unique!)
   ├─ log.dirs=/var/kafka
   ├─ listeners=PLAINTEXT://:9092
   └─ zookeeper.connect=... (or controller.quorum.voters for KRaft)

3. Ensure network connectivity
4. Verify disk space available

STEP 2: Start New Broker

kafka-server-start.sh server.properties

Broker will:
├─ Register with coordination system
├─ Receive cluster metadata
└─ Enter RUNNING state

BUT: No partitions assigned yet!

STEP 3: Verify Broker Joined

# Check broker list
kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Should see new broker ID

# Check logs
tail -f /var/log/kafka/server.log | grep "started"

STEP 4: Reassign Partitions (Optional but Recommended)

New broker has NO partitions initially.
Must manually reassign to balance load.

# Generate reassignment plan
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3,4" \
  --generate

# Execute reassignment
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# Verify completion
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

Adding Multiple Brokers

ADDING MULTIPLE BROKERS:

Option 1: Add one at a time
├─ Add broker 4
├─ Reassign partitions
├─ Wait for completion
├─ Add broker 5
└─ Repeat

Option 2: Add all, then reassign once
├─ Add brokers 4, 5, 6
├─ All join cluster (but no partitions)
├─ Single reassignment to all new brokers
└─ Faster, but higher risk

Recommendation: Add one at a time for critical production clusters.

6. Removing Brokers from Cluster

When to Remove Brokers

REMOVE BROKERS WHEN:
├─ Over-provisioned
├─ Decommissioning hardware
├─ Consolidating clusters
├─ Reducing costs
└─ Replacing failed broker permanently

Removing a Broker: Step-by-Step

STEP 1: Reassign Partitions OFF the Broker

CRITICAL: Must move partitions BEFORE stopping broker!

# Identify partitions on broker to remove
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "Leader: 4"

# Generate reassignment plan (excluding broker 4)
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "0,1,2,3" \  # Exclude broker 4
  --generate

# Execute reassignment
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --execute

# Wait for completion (can take hours for large data)
kafka-reassign-partitions.sh \
  --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json \
  --verify

STEP 2: Verify No Partitions on Broker

# Ensure broker has no leader partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "Leader: 4" | wc -l
# Should be 0

# Ensure broker has no replica partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "Replicas:.*4" | wc -l
# Should be 0

STEP 3: Stop Broker

# Graceful shutdown
kafka-server-stop.sh

# Verify stopped
ps aux | grep kafka

STEP 4: Remove from Cluster Metadata (KRaft only)

# ZooKeeper mode: Automatically removed after shutdown

# KRaft mode: May need explicit removal
# (Check KRaft documentation for your version)

Decommissioning Best Practices

✅ DO:
├─ Move ALL partitions before stopping
├─ Verify partition reassignment completed
├─ Graceful shutdown
├─ Backup data before decommissioning
└─ Update documentation

❌ DON'T:
├─ Stop broker before reassigning partitions (DATA LOSS!)
├─ Remove multiple brokers simultaneously
├─ Rush the reassignment (can take hours)
└─ Forget to update firewall rules, DNS, etc.

7. Failure Scenarios & Recovery

Scenario 1: Broker Crash

WHAT HAPPENS:

1. Broker process dies (OOM, crash, kill -9)
2. Coordination system detects failure
   └─ ZooKeeper: Ephemeral node deleted
   └─ KRaft: Missed heartbeats
3. Controller triggers leader election
   └─ New leaders elected for affected partitions
4. Clients reconnect to new leaders
5. Cluster continues operating (with N-1 brokers)

RECOVERY:

1. Investigate crash cause
   ├─ Check logs: /var/log/kafka/server.log
   ├─ Check system logs: dmesg, journalctl
   └─ Check heap dumps if OOM

2. Fix underlying issue
   ├─ Increase heap if OOM
   ├─ Fix disk issues
   └─ Patch bugs

3. Restart broker
   └─ kafka-server-start.sh

4. Monitor catch-up
   └─ Check under-replicated partitions

Scenario 2: Disk Failure

WHAT HAPPENS:

1. Disk becomes unavailable
2. Broker cannot read/write data
3. Broker may crash or hang
4. Partitions on failed disk become unavailable

RECOVERY:

Option 1: Repair Disk
├─ Stop broker
├─ Fix/replace disk
├─ Restore data from replicas
└─ Restart broker

Option 2: Move to New Disk
├─ Stop broker
├─ Update log.dirs config
├─ Start broker (will fetch data from replicas)
└─ Validate

Option 3: Remove Broker
└─ Follow decommissioning process

Scenario 3: Network Partition

WHAT HAPPENS:

1. Broker loses network connectivity
2. Cannot communicate with coordination system
3. Marked as failed by controller
4. Leaders migrated to other brokers

RECOVERY:

1. Fix network issue
2. Broker automatically rejoins cluster
3. Becomes follower for partitions
4. Catches up via replication
5. Rejoins ISR when caught up

NOTE: If network partition lasts long:
└─ May need to truncate and re-replicate data

Scenario 4: Controller Failure

WHAT HAPPENS:

1. Controller broker crashes
2. Other brokers detect failure
3. New controller election triggered
4. New controller takes over
5. Cluster operations resume

IMPACT:
├─ Data plane: UNAFFECTED (clients keep working)
├─ Control plane: PAUSED briefly (no admin ops)
└─ Duration: Typically < 10 seconds

RECOVERY:
└─ Automatic! No manual intervention needed.
   Just restart failed broker as normal.

8. Health Checks & Validation

Pre-Operation Health Check

# 1. Check under-replicated partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

# Should be empty

# 2. Check offline partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --unavailable-partitions

# Should be empty

# 3. Check broker status
kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# All brokers should respond

# 4. Check controller
# ZooKeeper mode:
zkCli.sh -server localhost:2181 get /controller

# 5. Check disk space on all brokers
df -h | grep kafka

# At least 20% free recommended

Post-Operation Validation

# 1. Verify broker started successfully
systemctl status kafka
# Should be "active (running)"

# 2. Check logs for errors
tail -100 /var/log/kafka/server.log | grep ERROR
# Should be empty

# 3. Verify broker registered
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 | grep "id: X"

# 4. Check ISR health
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-replicated-partitions

# Should return to 0 after catch-up

# 5. Monitor client metrics
# Check producer/consumer error rates

9. Common Operational Scenarios

Scenario: Upgrade Kafka Version

PROCEDURE:

1. Read release notes
2. Test in non-production first
3. Backup configuration files
4. Perform rolling restart:
   ├─ Stop broker
   ├─ Replace Kafka binaries
   ├─ Start broker
   ├─ Validate
   └─ Repeat for next broker
5. Update inter.broker.protocol.version (if needed)
6. Monitor for issues

Scenario: Change Broker Configuration

PROCEDURE:

For dynamic configs:
└─ kafka-configs.sh (no restart needed)

For static configs:
1. Update server.properties
2. Rolling restart required
3. Validate after each broker

Scenario: Replace Failed Hardware

PROCEDURE:

1. Add new broker with different broker.id
2. Reassign partitions from old to new
3. Wait for reassignment complete
4. Decommission old broker
5. Update inventory/documentation

10. Key Takeaways

You've learned:

✅ Broker lifecycle has startup, running, shutdown states
✅ Graceful shutdown prevents data loss and minimizes unavailability
✅ Rolling restarts enable zero-downtime maintenance
✅ Adding brokers requires partition reassignment to balance load
✅ Removing brokers requires moving partitions FIRST, then stopping
✅ Failure recovery depends on replication and proper configuration
✅ Health checks before/after operations prevent surprises
✅ Controlled operations minimize risk and downtime

11. Operational Checklist

BEFORE ANY BROKER OPERATION:
☑ Check cluster health
☑ Verify replication factor ≥ 2
☑ Check under-replicated partitions = 0
☑ Have rollback plan
☑ Schedule maintenance window
☑ Notify stakeholders

DURING OPERATION:
☑ One broker at a time
☑ Monitor logs in real-time
☑ Check metrics dashboards
☑ Wait for replication catch-up

AFTER OPERATION:
☑ Verify broker health
☑ Check under-replicated partitions
☑ Monitor client error rates
☑ Document changes
☑ Update runbooks

Summary

Broker lifecycle management is about control:

Safe Operations =
├─ Deliberate changes
├─ Health checks before/after
├─ One broker at a time
├─ Monitor continuously
└─ Always have rollback plan

Remember: Every broker operation should be planned, monitored, and validated. When in doubt, go slower — rushing causes outages.

Next up: Module 2.2 — Topic Design & Configuration — Learn how to design and configure topics for your use cases.