KafkaIQ

10.3.4 – Backup and Restore Strategies

Module: M10 – Advanced Topics & Architecture
Section: 10.3 – High Availability & Disaster Recovery
Goal: Strategies for backing up and restoring Kafka data

What to Backup

1. Message Data (Topic Partitions)

Location: /var/kafka/data/<topic>-<partition>/
Files: .log, .index, .timeindex
Size: TB-scale (largest component)

2. Cluster Metadata

ZK Mode: /var/zookeeper/data
KRaft Mode: /var/kafka/controller-logs
Size: MB-scale

3. Configuration Files

Files: server.properties, log4j.properties, custom configs
Size: KB-scale

4. Consumer Offsets

Topic: __consumer_offsets
Purpose: Resume consumers from last position

Backup Strategies

Method: Use Kafka's built-in replication (no external backups).

Config:

replication.factor = 3
min.insync.replicas = 2

Pros: ✅ Real-time replication
✅ No backup overhead
✅ Fast recovery (replicas already available)

Cons: ❌ Doesn't protect against logical errors (accidental deletion, corruption)
❌ All replicas in same cluster

Use Case: Primary HA strategy

Strategy 2: Cross-Cluster Replication

Method: MirrorMaker 2 to DR cluster.

Config:

clusters = primary, dr
primary->dr.enabled = true
topics = critical-topic-.*

Pros: ✅ Geographic redundancy
✅ Fast failover (DR cluster ready)

Cons: ❌ 2x infrastructure cost
❌ Replication lag (RPO > 0)

Use Case: Disaster recovery

Strategy 3: Snapshot to Object Storage

Method: Kafka Connect S3 Sink Connector.

Config:

{
  "name": "s3-sink",
  "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "topics": "orders,payments",
  "s3.bucket.name": "kafka-backups",
  "flush.size": "1000",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage"
}

Pros: ✅ Long-term retention
✅ Low cost (~$0.023/GB/month)
✅ Compliance-friendly

Cons: ❌ Slow restore (hours)
❌ Not for real-time failover

Use Case: Compliance, audit logs, archival

Strategy 4: Disk-Level Snapshots

Method: EBS snapshots (AWS), disk snapshots (GCP, Azure).

Steps:

  1. Stop broker (or use consistent snapshot if available)
  2. Snapshot disk: aws ec2 create-snapshot --volume-id vol-xxx
  3. Resume broker

Pros: ✅ Full system backup (data + config)
✅ Fast restore (attach snapshot)

Cons: ❌ Requires broker downtime for consistent snapshot
❌ Cloud-specific

Use Case: Infrequent full backups (weekly)

Restore Strategies

Restore from Replication

Scenario: Single broker disk failure.

Steps:

  1. Replace disk
  2. Start broker
  3. Broker auto-syncs from partition leaders

RTO: 2-8 hours (data sync)
RPO: 0

Restore from DR Cluster

Scenario: Primary cluster lost.

Steps:

  1. Promote DR cluster to primary
  2. Update DNS/load balancer to point to DR
  3. Redirect producers/consumers
  4. Resume operations

RTO: 10-30 minutes
RPO: Replication lag (seconds to minutes)

Restore from S3

Scenario: Need historical data (months/years old).

Steps:

  1. Create new topic
  2. Use Kafka Connect S3 Source to replay from S3
  3. Consumers read from restored topic

RTO: Hours to days (depending on data volume)
RPO: Flush interval (minutes)

Restore from Disk Snapshot

Scenario: Full cluster loss, need fast recovery.

Steps:

  1. Create new volumes from snapshots
  2. Attach to new broker instances
  3. Update broker IDs/configs if needed
  4. Start brokers
  5. Verify cluster health

RTO: 1-2 hours
RPO: Snapshot age (hours to days)

Backup Automation

Backup Schedule

Daily:

  • Metadata snapshot (ZK/KRaft)
  • Consumer offsets backup
  • Config files backup

Weekly:

  • Full disk snapshots (EBS)

Continuous:

  • Cross-cluster replication (MirrorMaker)
  • S3 archival (Connect S3 Sink)

Backup Script Example

#!/bin/bash
# Daily metadata backup

DATE=$(date +%Y%m%d)
BACKUP_DIR=/backups/kafka/$DATE

# Backup ZooKeeper data
tar -czf $BACKUP_DIR/zk-data.tar.gz /var/zookeeper/data

# Backup Kafka configs
tar -czf $BACKUP_DIR/kafka-config.tar.gz /etc/kafka/

# Backup consumer offsets (export topic)
kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic __consumer_offsets \
  --from-beginning \
  --formatter kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter \
  > $BACKUP_DIR/consumer-offsets.txt

# Upload to S3
aws s3 sync $BACKUP_DIR s3://kafka-backups/$DATE/

# Cleanup old backups (>30 days)
find /backups/kafka/ -mtime +30 -delete

Restore Testing

DR Drill Procedure

Goal: Validate backups work.

Steps:

  1. Schedule drill (quarterly minimum)
  2. Select restore scenario (e.g., broker failure, cluster loss)
  3. Execute restore from backup (follow runbook)
  4. Verify data integrity (produce/consume test messages)
  5. Measure RTO/RPO (actual vs target)
  6. Document issues (update runbook)
  7. Cleanup test environment

Success Criteria: Restore completes within RTO target, data loss within RPO target.

Key Takeaways

  1. Replication: Primary HA strategy (RF=3, min.insync.replicas=2).
  2. Cross-Cluster: DR strategy (MirrorMaker to backup region).
  3. S3 Archival: Long-term retention, compliance (slow restore).
  4. Disk Snapshots: Full system backup (infrequent, cloud-dependent).
  5. Backup Schedule: Daily (metadata), weekly (disk), continuous (replication).
  6. Test Backups: Quarterly DR drills to validate restore procedures.
  7. Automation: Script backups, upload to S3, cleanup old backups.

What's Next?

Next Module: 10.3.5 – RTO-RPO Modeling

References:

  • MirrorMaker → 10.2.2
  • Kafka Connect → M8: Kafka Connect