10.3.5 – RTO-RPO Modeling

Module: M10 – Advanced Topics & Architecture
Section: 10.3 – High Availability & Disaster Recovery
Goal: Model and calculate RTO/RPO for Kafka deployments

Definitions

RTO (Recovery Time Objective)

Definition: Maximum acceptable downtime after a disaster.

Example: "System must be back online within 30 minutes"

Measured In: Time (seconds, minutes, hours)

RPO (Recovery Point Objective)

Definition: Maximum acceptable data loss after a disaster.

Example: "Can lose at most 5 minutes of data"

Measured In: Time or messages

RTO Calculation

Formula

RTO = Detection Time + Decision Time + Recovery Time + Verification Time

Example: Broker Failure (Active-Passive DR)

Scenario: Primary broker fails, need to fail over to DR cluster.

Detection Time: 2 minutes (monitoring alert)
Decision Time: 3 minutes (verify failure, approve failover)
Recovery Time: 10 minutes (DNS update, redirect traffic)
Verification Time: 5 minutes (test produce/consume)

Total RTO: 2 + 3 + 10 + 5 = 20 minutes

RTO by Architecture

Architecture	RTO Target	Explanation
Single cluster, RF=3	30 seconds	Automatic leader election
Active-Passive (manual failover)	10-30 minutes	Manual DNS update, traffic redirect
Active-Passive (auto failover)	2-5 minutes	Automated DNS/load balancer update
Active-Active	Seconds	Traffic already in both clusters
Stretch Cluster	0 (no downtime)	Automatic partition failover

RPO Calculation

Formula

RPO = Replication Lag + Checkpoint Interval

Example: Asynchronous Replication

Replication Lag: 30 seconds (primary to DR cluster)
Checkpoint Interval: 60 seconds (consumer offset sync)

Total RPO: 30 + 60 = 90 seconds (up to 90 seconds of data loss)

RPO by Architecture

Architecture	RPO Target	Explanation
Single cluster, RF=3, acks=all	0	Synchronous replication
Active-Passive (async replication)	Replication lag (seconds-minutes)	MM2 lag
Active-Active (async)	0 (eventual consistency)	Both clusters have data
Stretch Cluster (sync)	0	Synchronous cross-region replication
S3 Backup	Flush interval (5-60 minutes)	Batch flush to S3

Modeling Scenarios

Scenario 1: E-Commerce Platform

Business Requirements:

RTO: < 5 minutes (revenue loss = $10k/minute downtime)
RPO: 0 (order data cannot be lost)

Recommended Architecture:

Primary: 3-broker cluster, RF=3, min.insync.replicas=2
DR: Active-active in two regions (US-East, US-West)
Failover: Automated DNS/load balancer

Actual RTO: 2 minutes (automated failover)
Actual RPO: 0 (both clusters active, eventual consistency)

Cost: 2x infrastructure, but justified by revenue protection

Scenario 2: Log Aggregation

Business Requirements:

RTO: 30 minutes (acceptable downtime for logs)
RPO: 10 minutes (some log loss acceptable)

Recommended Architecture:

Primary: 3-broker cluster
DR: Active-passive with MM2 (lower cost)
Backup: S3 archival for long-term retention

Actual RTO: 20 minutes (manual failover)
Actual RPO: 5 minutes (MM2 replication lag)

Cost: 1.5x infrastructure (smaller DR cluster)

Scenario 3: Financial Transactions

Business Requirements:

RTO: < 1 minute (regulatory requirement)
RPO: 0 (zero data loss tolerated)

Recommended Architecture:

Primary: Stretch cluster across 3 AZs
Replication: Synchronous (RF=3, min.insync.replicas=2)
Failover: Automatic partition leader election

Actual RTO: 30 seconds (automatic)
Actual RPO: 0 (synchronous replication)

Cost: 1x infrastructure, but higher latency (cross-AZ writes add 5-10ms)

Cost-Benefit Analysis

RTO/RPO vs Cost

RTO Target	RPO Target	Architecture	Cost Multiplier
30 min	10 min	Active-Passive	1.5x
5 min	0	Active-Active	2x
1 min	0	Stretch Cluster	1x (but higher latency)
Hours	Minutes	S3 Backup	1.1x (storage cost only)

Downtime Cost Calculation

Formula:

Downtime Cost = Revenue Loss per Minute × RTO (minutes)

Example:
  Revenue = $1M/day = $694/minute
  RTO = 30 minutes
  Downtime Cost = $694 × 30 = $20,820

ROI Decision: If active-active costs $50k/year but prevents $20k downtime incidents (2-3 incidents/year expected), it's cost-justified.

Improving RTO

Techniques

1. Automate Failover

Manual → Automated: Reduce RTO by 10-20 minutes

2. Active-Active

Eliminate failover time: RTO → seconds

3. Health Checks

Faster detection: Reduce detection time from minutes to seconds

4. Runbooks

Document procedures: Reduce decision time

5. Load Balancers

Automatic traffic routing: Reduce recovery time

Improving RPO

Techniques

1. Increase Replication Frequency

# MM2: Faster replication
producer.linger.ms = 0
consumer.fetch.min.bytes = 1

2. Synchronous Replication

acks = all
min.insync.replicas = 2

3. Reduce Checkpoint Interval

emit.checkpoints.interval.seconds = 10  # Every 10 seconds

4. Active-Active

Both clusters have data: RPO = 0

Monitoring RTO/RPO

Metrics

RTO Monitoring:

Time to detect failure (monitoring alert lag)
Time to execute failover (manual or automated)

RPO Monitoring:

Replication lag: kafka-consumer-groups.sh --describe --group __mirror_maker
Last checkpoint age: Check __checkpoint topic

Alerting:

If replication_lag > RPO_target:
  Alert "RPO at risk"

If failover_time > RTO_target:
  Alert "RTO exceeded"

Key Takeaways

RTO: Maximum acceptable downtime (detection + decision + recovery + verification).
RPO: Maximum acceptable data loss (replication lag + checkpoint interval).
Active-Passive: RTO = 10-30 min, RPO = replication lag.
Active-Active: RTO = seconds, RPO = 0 (eventual consistency).
Stretch Cluster: RTO = 0, RPO = 0 (synchronous replication).
Cost-Benefit: Calculate downtime cost vs infrastructure cost to justify architecture.
Monitoring: Track replication lag and failover time to ensure RTO/RPO targets met.

What's Next?

Next Section: 10.4 – Cluster Governance & Architecture Decisions

What We Covered in Section 10.3:

✅ Fault tolerance mechanisms (10.3.1)
✅ Broker failure recovery playbooks (10.3.2)
✅ KRaft vs ZooKeeper resilience (10.3.3)
✅ Backup and restore strategies (10.3.4)
✅ RTO-RPO modeling (10.3.5)

What's Coming:

❌ Topic governance and standards
❌ Partition sizing and rebalancing
❌ Quota and tenant governance

References:

Multi-cluster DR → 10.2.5
Backup strategies → 10.3.4