KafkaIQ

10.4.5 – Upgrade and Maintenance Strategies

Module: M10 – Advanced Topics & Architecture
Section: 10.4 – Cluster Governance & Architecture Decisions
Goal: Plan and execute zero-downtime upgrades and maintenance

Introduction

Kafka clusters require periodic upgrades (security patches, new features, bug fixes). This module covers strategies for zero-downtime upgrades.

Upgrade Types

1. Patch Upgrade (3.5.0 → 3.5.1)

Risk: Low (bug fixes only)
Downtime: None (rolling restart)
Testing: Minimal (sanity check)

2. Minor Upgrade (3.4.x → 3.5.x)

Risk: Medium (new features, deprecations)
Downtime: None (rolling restart with compatibility mode)
Testing: Moderate (test new features in dev/staging)

3. Major Upgrade (2.8.x → 3.0.x)

Risk: High (breaking changes possible)
Downtime: None (careful rolling upgrade with fallback plan)
Testing: Extensive (full regression in staging)

Rolling Upgrade Procedure

Pre-Upgrade Checklist

1. Review Release Notes

  • Breaking changes?
  • Deprecated features used?
  • New configs required?

2. Backup Metadata

# ZooKeeper
tar -czf zk-backup.tar.gz /var/zookeeper/data

# KRaft
tar -czf controller-backup.tar.gz /var/kafka/controller-logs

3. Test in Staging

  • Deploy to staging cluster
  • Run producer/consumer tests
  • Monitor for errors

4. Schedule Maintenance Window

  • Low-traffic period (e.g., 2-6 AM)
  • Notify teams

Rolling Upgrade Steps

Step 1: Upgrade One Broker

# Stop broker gracefully
systemctl stop kafka

# Upgrade binaries
tar -xzf kafka_3.5.1.tgz -C /opt/

# Update symlink
ln -sfn /opt/kafka_3.5.1 /opt/kafka

# Start broker
systemctl start kafka

Step 2: Verify Broker Health

# Check logs
tail -f /var/log/kafka/server.log

# Check under-replicated partitions
kafka-topics.sh --describe --under-replicated-partitions \
  --bootstrap-server localhost:9092

Step 3: Repeat for All Brokers

  • Upgrade one broker at a time
  • Wait for ISR sync before next broker (5-15 minutes)

Step 4: Enable New Features

# After all brokers upgraded, set inter.broker.protocol.version
inter.broker.protocol.version=3.5
log.message.format.version=3.5

Step 5: Rolling Restart (to apply new configs)

  • Restart brokers one by one

Compatibility Mode

What Is It?

Purpose: Run new broker version with old protocol (for gradual upgrade).

Config:

inter.broker.protocol.version=3.4  # Old version
log.message.format.version=3.4     # Old format

Why: Allows rollback if issues found (brokers can still communicate).

When to Remove Compatibility Mode?

After: All brokers upgraded, tested, and stable (typically 1-2 weeks).

How: Update configs, rolling restart.

inter.broker.protocol.version=3.5  # New version
log.message.format.version=3.5     # New format

Rollback Plan

When to Rollback?

Triggers:

  • Critical bugs discovered
  • Performance degradation
  • Incompatibility with clients

Rollback Procedure

Step 1: Verify Compatibility Mode Active

inter.broker.protocol.version=3.4  # Old version (must be set)

If NOT set: Rollback impossible (new message format not backward compatible).

Step 2: Downgrade Brokers

# Stop broker
systemctl stop kafka

# Revert to old binaries
ln -sfn /opt/kafka_3.4.0 /opt/kafka

# Start broker
systemctl start kafka

Step 3: Repeat for All Brokers

Maintenance Tasks

Task 1: Log Compaction

Frequency: Weekly (for compacted topics)

Command:

kafka-log-dirs.sh --trigger-log-compaction \
  --topic user-profiles \
  --bootstrap-server localhost:9092

Task 2: Leader Rebalancing

Frequency: After broker restarts/failures

Command:

kafka-leader-election.sh \
  --bootstrap-server localhost:9092 \
  --election-type preferred \
  --all-topic-partitions

Task 3: Partition Reassignment

Frequency: After adding/removing brokers

Tool: kafka-reassign-partitions.sh (see 10.4.2)

Task 4: Metadata Cleanup

ZooKeeper:

# Remove old controller epochs, etc.
zkCli.sh
deleteall /controller_epoch

KRaft: Automatic (no manual cleanup needed).

Zero-Downtime Upgrade Best Practices

1. Use Replication Factor ≥ 3

Why: Can lose 1-2 brokers during upgrade without downtime.

2. Rolling Upgrades Only

Why: Never upgrade all brokers simultaneously.

3. Monitor During Upgrade

Metrics:

  • Under-replicated partitions (should be 0)
  • ISR shrink/expand rate
  • Consumer lag

4. Gradual Rollout

Strategy: Upgrade 1 broker → wait 30 min → next broker.

5. Keep Compatibility Mode

Duration: 1-2 weeks (allows rollback).

6. Test in Staging First

Coverage: Full producer/consumer tests, schema evolution, EOS pipelines.

Key Takeaways

  1. Upgrade Types: Patch (low risk), minor (medium), major (high).
  2. Rolling Upgrade: One broker at a time, wait for ISR sync.
  3. Compatibility Mode: Run new brokers with old protocol (allows rollback).
  4. Rollback: Only possible if compatibility mode active.
  5. Maintenance: Leader rebalancing (after restarts), log compaction (weekly).
  6. Zero-Downtime: RF≥3, rolling upgrades, gradual rollout, monitor metrics.

What's Next?

Next Section: 10.5 – Advanced Performance Tuning Layer

What We Covered in Section 10.4:

  • ✅ Topic governance and standards (10.4.1)
  • ✅ Partition sizing and rebalancing (10.4.2)
  • ✅ Quota and tenant governance (10.4.3)
  • ✅ Broker placement and topology (10.4.4)
  • ✅ Upgrade and maintenance strategies (10.4.5)

What's Coming:

  • ❌ JVM tuning for Kafka
  • ❌ OS and disk tuning
  • ❌ Network stack optimization

References:

  • Broker configs → M2: Brokers & Topics
  • Operations → M6: Operations