10.2.1 – Multi-Cluster Design Models

Module: M10 – Advanced Topics & Architecture
Section: 10.2 – Multi-Datacenter Replication & Geo Architecture
Goal: Understand multi-cluster Kafka architectures and when to use them

Introduction

Most production Kafka deployments span multiple datacenters or regions. This module covers the architectural patterns, design decisions, and tradeoffs for multi-cluster Kafka deployments.

Note: This focuses on cross-cluster architecture, not within-cluster replication (covered in M7: Internals).

Why Multi-Cluster?

Single-Cluster Limitations

Single-cluster issues:

• Geographic latency: Cross-region writes have 50-200ms latency
• Blast radius: Single cluster failure affects all regions
• Compliance: Data residency laws require regional isolation
• Scale limits: Single cluster caps at ~100-200 brokers

Network Latency Comparison

Understanding network latency is critical for multi-cluster design decisions:

Key Insight: Cross-AZ replication offers lower latency than cross-region while still providing fault tolerance.

Example:

Single-cluster cross-region write:
  Producer (US-West) → Kafka (US-East) → Consumer (US-East)
  Latency: ~80ms (network) + 5ms (Kafka) = 85ms

Multi-cluster with local write:
  Producer (US-West) → Kafka (US-West) → Replication to US-East
  Producer latency: ~5ms (local Kafka)
  Consumer (US-East) reads from local cluster: ~5ms
  Total: 10ms (vs 85ms)

Design Implications:

• High-throughput, low-latency: Use cross-AZ replication within region, multi-cluster across regions
• Disaster recovery: Accept 60-80ms replication lag for cross-region DR
• Global apps: Deploy regional clusters, replicate selectively

Multi-Cluster Benefits

✅ Low latency: Producers/consumers connect to local cluster
✅ Fault isolation: Regional failures don't affect other regions
✅ Data sovereignty: Keep data within regulatory boundaries
✅ Scale: Independent scaling per region
✅ Disaster recovery: Cross-region replication for DR

Multi-Cluster Architecture Patterns

Pattern 1: Hub-and-Spoke

Architecture:

Region A (Producer) ─┐
Region B (Producer) ─┼─→ Central Hub Cluster → Analytics
Region C (Producer) ─┘

Characteristics:

• Central aggregation cluster (hub)
• Regional clusters (spokes) send data to hub
• One-way replication (spoke → hub)

Use Cases:

• Log aggregation from multiple regions
• Central analytics/data warehouse
• Compliance reporting

Pros:

• Simple topology
• Clear data flow

Cons:

• Hub is single point of failure
• Hub can become bottleneck
• Regional consumers can't read from hub (high latency)

Pattern 2: Active-Passive (Primary-Backup)

Architecture:

Primary Cluster (US-East)
  ↓ (one-way replication)
Passive Cluster (US-West) ← Standby, no writes

Characteristics:

• Primary cluster: All writes go here
• Passive cluster: Replica for disaster recovery
• One-way replication (primary → passive)
• Failover: Switch to passive if primary fails

Use Cases:

• Disaster recovery
• Business continuity
• Regulatory compliance (backup requirement)

Pros:

• Simple (only one active cluster)
• Clear failover plan

Cons:

• Passive cluster idle (wasted resources)
• Failover requires manual intervention
• RPO/RTO depends on replication lag

Pattern 3: Active-Active (Multi-Master)

Architecture:

Cluster A (US-East) ←──→ Bidirectional Replication ←──→ Cluster B (US-West)
     ↓                                                         ↓
Producers/Consumers                                  Producers/Consumers

Characteristics:

• Both clusters active: Writes and reads in both regions
• Bidirectional replication (A ↔ B)
• Loop prevention: Avoid replicating replicated data

Use Cases:

• Global applications (users in multiple regions)
• High availability (no single point of failure)
• Load distribution

Pros:

• Low latency (local reads/writes)
• High availability (both clusters active)
• Resource utilization (no idle cluster)

Cons:

• Complex (conflict resolution, loop prevention)
• Data consistency challenges
• Requires careful design

Pattern 4: Stretch Cluster (Synchronous Multi-Region)

Architecture:

Single Kafka Cluster
  ├── Brokers in US-East
  ├── Brokers in US-West
  └── Brokers in EU

Characteristics:

• Single logical cluster spanning multiple regions
• Synchronous replication across regions
• Rack awareness to distribute replicas geographically

Use Cases:

• Low RPO requirements (zero data loss)
• Compliance (data in multiple regions)
• Simplified operations (one cluster)

Pros:

• Zero data loss (synchronous replication)
• Single cluster to manage

Cons:

• High latency: Writes wait for cross-region acknowledgment (50-200ms)
• Network dependency: Cross-region network outage → cluster unavailable
• Complexity: Requires low-latency, high-bandwidth network

Recommendation: Rarely used due to latency impact. Consider active-passive or active-active instead.

Pattern 5: Aggregate-then-Stream

Architecture:

Region A Cluster → Aggregate → Global Cluster → Stream → Region B Cluster

Characteristics:

• Regional clusters produce local data
• Central aggregation cluster collects and processes
• Stream to regional clusters for local consumption

Use Cases:

• Real-time analytics
• Global dashboards
• Cross-region recommendations

Choosing a Pattern

Decision Matrix

Multi-Cluster Replication Mechanisms

Option 1: MirrorMaker 2 (MM2)

What It Is: Kafka Connect-based tool for cross-cluster replication.

Features:

• Topic replication
• Offset translation
• Automatic topic creation
• Group offset sync
• Bidirectional support

When to Use: Default choice for multi-cluster replication.

Reference: Detailed in 10.2.2 – MirrorMaker and MM2 Architecture.

Option 2: Confluent Replicator

What It Is: Commercial tool by Confluent for enterprise replication.

Features:

• Schema Registry integration
• Exactly-once replication
• Timestamp preservation
• ACL sync

When to Use: Enterprise deployments requiring Confluent support.

Option 3: Cluster Linking (Confluent Platform)

What It Is: Direct broker-to-broker replication (no Connect framework).

Features:

• Zero-copy replication
• Low latency (< 10ms)
• Byte-for-byte replica (identical offsets)

When to Use: Confluent Platform deployments needing ultra-low latency.

Option 4: Custom Replication (Application-Level)

What It Is: Custom consumers/producers replicating data.

When to Use: Special transformation logic required during replication.

Drawback: High operational burden.

Network and Latency Considerations

Cross-Region Latency

Impact: Synchronous replication adds full RTT to every write operation.

Network Bandwidth

Replication bandwidth formula:

Bandwidth = Throughput × Message Size × Replication Factor

Example:
  100 MB/sec throughput × 2 (replicating to 2 clusters) = 200 MB/sec bandwidth

Recommendation: Provision 2-3x expected bandwidth for spikes.

Data Sovereignty and Compliance

GDPR Example

Requirement: EU customer data must stay in EU.

Architecture:

EU Cluster (EU customers) ← No replication to US
US Cluster (US customers) ← No replication to EU

Implementation:

• Separate clusters per region
• Topic naming: eu-users, us-users
• MirrorMaker filters: Replicate only allowed topics

China Example

Requirement: Data generated in China must stay in China.

Architecture:

China Cluster (isolated)
  ↓ (one-way: aggregate metadata only, not raw data)
Global Cluster (analytics)

Cost Considerations

Network Costs

Cloud providers charge for cross-region transfer:

• AWS: $0.02/GB (cross-region)
• Azure: $0.02/GB (cross-region)
• GCP: $0.01/GB (cross-region)

Example:

Replicating 1 TB/day:
  Monthly cost = 1000 GB/day × 30 days × $0.02/GB = $600/month
  Annual cost = $7,200/year

Cluster Costs

Active-Passive:

• Primary: 10 brokers ($5k/month)
• Passive: 10 brokers ($5k/month)
• Total: $10k/month

Active-Active:

• Cluster A: 10 brokers ($5k/month)
• Cluster B: 10 brokers ($5k/month)
• Total: $10k/month (but both utilized)

Cost Efficiency: Active-Active maximizes utilization.

Key Takeaways

1. Multi-Cluster Patterns: Hub-and-spoke, active-passive, active-active, stretch cluster.

2. Active-Passive: Simple DR, one active cluster, manual failover.

3. Active-Active: Both clusters active, low latency, complex (conflict resolution).

4. Stretch Cluster: Synchronous cross-region replication, high latency (rarely used).

5. Replication Tools: MirrorMaker 2 (default), Confluent Replicator, Cluster Linking.

6. Latency: Cross-region RTT = 60-250ms (impacts synchronous replication).

7. Costs: Network transfer costs (~$0.02/GB cross-region), idle resources in active-passive.

8. Compliance: Data sovereignty requires regional isolation.

What's Next?

Next Module: 10.2.2 – MirrorMaker and MM2 Architecture

What We Covered:

• ✅ Multi-cluster patterns
• ✅ Active-passive vs active-active
• ✅ Network and latency considerations
• ✅ Compliance and cost

What's Coming:

• ❌ MirrorMaker 2 architecture
• ❌ Active-passive vs active-active details
• ❌ Replication consistency challenges

References:

• Replication mechanics → M7: Kafka Internals
• MirrorMaker config basics → M8: Kafka Connect