KafkaIQ

10.4.4 – Broker Placement and Topology

Module: M10 – Advanced Topics & Architecture
Section: 10.4 – Cluster Governance & Architecture Decisions
Goal: Design broker placement and rack awareness strategies

Introduction

Broker placement impacts availability, performance, and cost. This module covers placement strategies for cloud and on-premises deployments.

Rack Awareness

What Is Rack Awareness?

Purpose: Distribute partition replicas across failure domains (racks, availability zones).

Config (server.properties):

broker.rack=us-east-1a  # Availability zone or physical rack

How It Works:

  • Kafka ensures replicas of same partition are in different racks
  • If rack fails, partition remains available (replicas in other racks)

Example: Multi-AZ Deployment

Setup:

Broker 1: broker.rack=us-east-1a
Broker 2: broker.rack=us-east-1b
Broker 3: broker.rack=us-east-1c

Topic: orders, RF=3, 1 partition

Replica Placement:

Partition 0: Leader (Broker 1, AZ-A), Follower (Broker 2, AZ-B), Follower (Broker 3, AZ-C)

Result: Can tolerate full AZ failure (2 replicas in other AZs).

Cloud Placement Strategies

AWS Deployment

Architecture:

Region: us-east-1
  ├── AZ-A: Brokers 1-3
  ├── AZ-B: Brokers 4-6
  └── AZ-C: Brokers 7-9

Instance Types:

  • Compute-optimized: c5.2xlarge (8 vCPU, 16 GB RAM) for high-throughput
  • Storage-optimized: i3.4xlarge (16 vCPU, 122 GB RAM, 2x1.9TB NVMe) for high-storage

Storage:

  • EBS gp3: 3000 IOPS, 125 MB/sec (general purpose)
  • EBS io2: 64000 IOPS, 1000 MB/sec (high performance)
  • Instance store: Lowest latency, but ephemeral (data lost on instance failure)

Recommendation:

  • Use EBS gp3 for most workloads
  • Use instance store for ultra-low latency (but ensure RF≥3)

Multi-Region Deployment

Architecture:

Region A (us-east-1): Cluster A
Region B (us-west-2): Cluster B
  → MirrorMaker 2 replication

Use Case: DR, geo-distributed users.

Cost: Cross-region transfer = $0.02/GB.

On-Premises Placement

Physical Rack Placement

Architecture:

Rack 1: Brokers 1-3, Switch 1
Rack 2: Brokers 4-6, Switch 2
Rack 3: Brokers 7-9, Switch 3

Rack Awareness Config:

broker.rack=rack-1  # Brokers 1-3
broker.rack=rack-2  # Brokers 4-6
broker.rack=rack-3  # Brokers 7-9

Result: Tolerate rack failure (switch failure, power failure).

Broker Sizing

Sizing Formula

CPU:

CPU Cores = (Producer Throughput + Consumer Throughput + Replication Throughput) / 50 MB/sec per core

Example:
  Producers: 200 MB/sec
  Consumers: 400 MB/sec
  Replication: 100 MB/sec
  Total: 700 MB/sec
  CPU = 700 / 50 = 14 cores

Memory:

RAM = 8 GB (base) + Page Cache Size

Page Cache = 2× Peak Message Rate × Message Size × 30 seconds

Example:
  10k messages/sec × 10 KB × 30 sec = 3 GB
  RAM = 8 + (2 × 3) = 14 GB

Rule of Thumb: 32-64 GB RAM per broker (general purpose).

Disk:

Disk Size = Daily Throughput × Retention Days × Replication Factor

Example:
  100 GB/day × 7 days × 3 RF = 2.1 TB
  Provision: 3 TB (buffer for growth)

Network:

Network = (Producer + Consumer + Replication) × 1.5

Example:
  Producers: 200 MB/sec
  Consumers: 400 MB/sec
  Replication: 100 MB/sec
  Total: 700 MB/sec
  Network = 700 × 1.5 = 1050 MB/sec = 10 Gbps

Scaling Strategies

Vertical Scaling

Method: Upgrade broker instance size (more CPU, RAM, disk).

Pros:

  • ✅ Simple (no partition reassignment)

Cons:

  • ❌ Downtime (broker restart)
  • ❌ Limited by instance size (max CPU/RAM)

Use Case: Small clusters (< 10 brokers).

Horizontal Scaling

Method: Add more brokers to cluster.

Steps:

  1. Add new brokers
  2. Reassign partitions to new brokers
  3. Monitor replication (can take hours)

Pros:

  • ✅ No downtime
  • ✅ Unlimited scaling

Cons:

  • ❌ Complex (partition reassignment)

Use Case: Large clusters (> 10 brokers).

Key Takeaways

  1. Rack Awareness: Distribute replicas across AZs/racks for fault tolerance.
  2. Multi-AZ: 3 AZs, brokers in each, broker.rack config.
  3. AWS Storage: EBS gp3 (general), instance store (ultra-low latency).
  4. Broker Sizing: 14+ cores, 32-64 GB RAM, 3 TB disk, 10 Gbps network (general).
  5. Scaling: Vertical (upgrade instances), horizontal (add brokers).
  6. Multi-Region: Separate clusters, MirrorMaker 2 for replication.

What's Next?

Next Module: 10.4.5 – Upgrade and Maintenance Strategies

References:

  • Broker configs → M2: Brokers & Topics
  • Multi-datacenter → 10.2