10.2.5 – Geo-DR Patterns

Module: M10 – Advanced Topics & Architecture
Section: 10.2 – Multi-Datacenter Replication & Geo Architecture
Goal: Design disaster recovery strategies for geographically distributed Kafka clusters

Introduction

Disaster Recovery (DR) ensures business continuity when a datacenter or region fails. This module covers DR patterns, RTO/RPO targets, and failover procedures.

DR Pattern 1: Single Region Backup

Architecture:

Primary Region (US-East) → One-way replication → DR Region (US-West)

Characteristics:

• Active-passive
• DR region idle (standby only)
• Manual failover

RTO: 10-30 minutes (manual failover)
RPO: Replication lag (seconds to minutes)

Use Case: Simple DR for small deployments

Cost: 2x infrastructure (primary + DR)

DR Pattern 2: Multi-Region Active-Active

Architecture:

Region A ←→ Bidirectional ←→ Region B

Characteristics:

• Both regions active
• Automatic failover (users already in both regions)
• No idle resources

RTO: Seconds (redirect traffic)
RPO: 0 (eventual consistency)

Use Case: Global applications, high availability

Cost: 2x infrastructure, but fully utilized

DR Pattern 3: Backup to Cloud Storage

Architecture:

Primary Cluster → Kafka Connect S3 Sink → S3 Bucket

Characteristics:

• Archive Kafka data to S3/GCS/Azure Blob
• Not for real-time failover (restore required)
• Long-term retention

RTO: Hours (restore from S3)
RPO: Minutes (flush interval)

Use Case: Compliance, audit logs, long-term archive

Cost: Low (storage cost only, ~$0.023/GB/month for S3)

DR Pattern 4: Stretch Cluster (Cross-AZ)

Architecture:

Single Kafka Cluster
  ├── AZ-A (brokers 1-3)
  ├── AZ-B (brokers 4-6)
  └── AZ-C (brokers 7-9)

Characteristics:

• Single cluster across availability zones
• Synchronous replication (RF=3, min.insync.replicas=2)
• Rack awareness

RTO: 0 (automatic partition leader election)
RPO: 0 (synchronous replication)

Use Case: High availability within same region

Cost: ~10% latency overhead (cross-AZ writes)

DR Pattern 5: Hub-and-Spoke with DR

Architecture:

Edge Clusters (A, B, C) → Central Hub → DR Hub (backup)

Characteristics:

• Edge clusters replicate to central hub
• Hub replicated to DR hub (passive)
• Manual failover to DR hub

Use Case: Log aggregation, central analytics

RTO: 15-30 minutes
RPO: Hub replication lag

Failover Procedures

Active-Passive Failover

Pre-Failover Checklist:

1. ✅ DR cluster up-to-date (replication lag < 1 min)
2. ✅ Consumer groups synced (checkpoints current)
3. ✅ DNS/load balancer ready to switch
4. ✅ Runbook documented and tested

Failover Steps:

1. Detect primary failure (monitoring alert)
2. Stop MM2 replication (primary → DR)
3. Promote DR to primary:
   • Update DNS (point to DR cluster)
   • Update application configs (bootstrap.servers)
4. Redirect producers/consumers
5. Verify traffic on DR cluster
6. Start reverse replication (DR → old primary, when recovered)

Rollback (if primary recovers quickly):

1. Stop traffic to DR
2. Resume primary cluster
3. Sync data from DR to primary (if needed)
4. Redirect traffic back to primary

Active-Active Failover

Failover Steps:

1. Detect Region A failure
2. Update DNS/load balancer (redirect Region A traffic to Region B)
3. Region B handles all traffic (already active)
4. Monitor Region B load (scale if needed)
5. When Region A recovers, resume bidirectional replication

No Data Loss: Both clusters have data (eventual consistency)

RTO and RPO Targets

RTO (Recovery Time Objective)

Definition: Maximum acceptable downtime.

RPO (Recovery Point Objective)

Definition: Maximum acceptable data loss.

Example Requirements:

Financial trading system:
  RTO: < 1 minute (high availability required)
  RPO: 0 (no data loss tolerated)

Solution: Active-active or stretch cluster

Log aggregation:
  RTO: 30 minutes (acceptable downtime)
  RPO: 5 minutes (some log loss tolerated)

Solution: Active-passive

DR Testing

DR Drill Procedure

Goal: Validate DR plan works.

Steps:

1. Schedule drill (announce to team, non-prod hours)
2. Simulate primary failure (shut down primary cluster)
3. Execute failover (follow runbook)
4. Verify DR cluster (produce/consume test messages)
5. Measure RTO/RPO (actual vs target)
6. Document issues (update runbook)
7. Fail back to primary (restore normal operations)

Frequency: Quarterly (minimum)

Chaos Engineering

Goal: Test resilience under failure.

Tools:

• Chaos Monkey: Random broker shutdown
• Network partitioning: Simulate cross-region network failure
• Latency injection: Slow down replication

Example:

# Chaos test: Kill random broker
iptables -A INPUT -p tcp --dport 9092 -j DROP  # Block broker

Monitor: Replication lag, consumer lag, error rates

Cost Optimization

Active-Passive Cost Reduction

Problem: DR cluster idle 99% of time.

Options:

1. Smaller DR cluster (fewer brokers, smaller instances)
   • Trade-off: Lower throughput during failover
2. Spot instances for DR (AWS spot, preemptible VMs)
   • Trade-off: May lose DR during spot reclamation
3. Cross-region replication only for critical topics
   • Trade-off: Non-critical data not backed up

Active-Active Cost Optimization

Problem: 2x infrastructure cost.

Benefit: Resources fully utilized (no idle clusters).

ROI: Justify cost with latency improvement, availability SLA.

Key Takeaways

1. DR Patterns: Active-passive (simple), active-active (HA), stretch cluster (cross-AZ), backup to S3 (archive).

2. RTO: Active-active = seconds, active-passive = minutes, S3 restore = hours.

3. RPO: Stretch cluster = 0, active-active = 0 (eventual), active-passive = replication lag.

4. Failover: Active-passive requires manual steps, active-active is automatic (redirect traffic).

5. DR Testing: Run quarterly drills, measure actual RTO/RPO, update runbooks.

6. Cost: Active-passive wastes 50%, active-active utilizes 100%.

7. Chaos Engineering: Test resilience with failure injection.

What's Next?

Next Section: 10.3 – High Availability and Disaster Recovery

What We Covered in Section 10.2:

• ✅ Multi-cluster design models (10.2.1)
• ✅ MirrorMaker and MM2 architecture (10.2.2)
• ✅ Active-passive vs active-active (10.2.3)
• ✅ Replication consistency challenges (10.2.4)
• ✅ Geo-DR patterns (10.2.5)

What's Coming:

• ❌ Fault tolerance mechanisms
• ❌ Broker failure recovery playbooks
• ❌ KRaft vs ZooKeeper resilience

References:

• Multi-cluster patterns → 10.2.1
• MirrorMaker 2 → 10.2.2