10.4.3 – Quota and Tenant Governance

Module: M10 – Advanced Topics & Architecture
Section: 10.4 – Cluster Governance & Architecture Decisions
Goal: Implement quotas and multi-tenant governance

Introduction

Multi-tenant Kafka clusters require quotas to prevent resource monopolization and ensure fair resource allocation.

Kafka Quota Types

1. Producer Quotas

Config: Limit bytes/sec produced per client.

Example:

kafka-configs.sh --alter \
  --add-config 'producer_byte_rate=1048576' \  # 1 MB/sec
  --entity-type clients \
  --entity-name payment-service \
  --bootstrap-server localhost:9092

Use Case: Prevent runaway producers from overwhelming brokers.

2. Consumer Quotas

Config: Limit bytes/sec consumed per client.

Example:

kafka-configs.sh --alter \
  --add-config 'consumer_byte_rate=2097152' \  # 2 MB/sec
  --entity-type clients \
  --entity-name analytics-pipeline \
  --bootstrap-server localhost:9092

Use Case: Prevent consumer bandwidth exhaustion.

3. Request Rate Quotas

Config: Limit requests/sec per client.

Example:

kafka-configs.sh --alter \
  --add-config 'request_percentage=50' \  # 50% of broker thread time
  --entity-type clients \
  --entity-name batch-job \
  --bootstrap-server localhost:9092

Use Case: Prevent metadata storms (e.g., excessive topic list requests).

Multi-Tenant Strategies

Strategy 1: Shared Cluster with Quotas

Architecture:

Single Kafka Cluster
  ├── Team A (quota: 10 MB/sec)
  ├── Team B (quota: 20 MB/sec)
  └── Team C (quota: 5 MB/sec)

Pros:

✅ Resource efficiency (shared infrastructure)
✅ Lower operational cost

Cons:

❌ Noisy neighbor risk (one team impacts others)
❌ Security isolation limited (ACLs only)

Use Case: Internal teams, trusted tenants

Strategy 2: Cluster Per Tenant

Architecture:

Tenant A → Cluster A (dedicated)
Tenant B → Cluster B (dedicated)
Tenant C → Cluster C (dedicated)

Pros:

✅ Strong isolation (no noisy neighbors)
✅ Independent scaling
✅ Security isolation (network-level)

Cons:

❌ Higher cost (3x infrastructure)
❌ Operational overhead (manage multiple clusters)

Use Case: External customers, compliance requirements

Strategy 3: Topic Prefixes per Tenant

Architecture:

Shared Cluster
  ├── tenatA.orders, tenantA.payments
  ├── tenantB.orders, tenantB.payments
  └── tenantC.orders, tenantC.payments

Pros:

✅ Shared infrastructure
✅ Clear ownership (topic prefixes)

Cons:

❌ No resource isolation (quotas needed)

Use Case: SaaS platforms with many small tenants

Quota Enforcement

How Quotas Work

Mechanism: Brokers throttle clients exceeding quota by delaying responses.

Example:

Client produces at 2 MB/sec (quota = 1 MB/sec)
  → Broker delays response by 1 second
  → Client backs off, throughput drops to 1 MB/sec

Key Point: Quotas are soft limits (enforced via delay, not rejection).

Monitoring Quotas

Metrics:

kafka.server:type=ClientMetrics,client-id=*,name=byte-rate
kafka.server:type=ClientMetrics,client-id=*,name=throttle-time

Alert: If throttle-time > 0, client is being throttled (may need higher quota).

Tenant Isolation

ACL-Based Isolation

Example: Team A can only access teamA.* topics.

kafka-acls.sh --add \
  --allow-principal User:teamA \
  --operation Read --operation Write \
  --topic teamA.* \
  --bootstrap-server localhost:9092

Enforcement: Brokers reject unauthorized requests.

Network Isolation

Architecture:

Tenant A → VPC A → Kafka (private link)
Tenant B → VPC B → Kafka (private link)

Use Case: External customers (strong security requirements).

Quota Tuning

Setting Initial Quotas

Rule of Thumb:

Team Quota = Expected Peak Throughput × 1.5

Example:
  Team A: 10 MB/sec peak
  Quota = 10 × 1.5 = 15 MB/sec

Why 1.5x: Buffer for spikes, prevent unnecessary throttling.

Dynamic Quota Adjustment

Increase Quota:

kafka-configs.sh --alter \
  --add-config 'producer_byte_rate=20971520' \  # 20 MB/sec
  --entity-type clients \
  --entity-name teamA \
  --bootstrap-server localhost:9092

Takes Effect: Immediately (no restart needed).

Chargeback Model

Cost Allocation

Purpose: Charge teams for Kafka resource usage.

Metrics to Track:

Bytes produced/consumed
Partition count (allocated to team)
Retention (storage cost)

Example:

Team A:
  Produced: 1 TB/month
  Consumed: 2 TB/month
  Partitions: 100
  Storage: 5 TB

Cost = ($0.05/GB produced) + ($0.02/GB consumed) + ($10/partition/month) + ($0.02/GB storage)
     = $50 + $40 + $1,000 + $100
     = $1,190/month

Key Takeaways

Quota Types: Producer, consumer, request rate quotas (prevent resource monopolization).
Multi-Tenant Strategies: Shared cluster (quotas), cluster per tenant (isolation), topic prefixes.
Quota Enforcement: Soft limits (broker delays responses, clients back off).
Isolation: ACLs (topic-level), network (VPC-level).
Tuning: Set quota = peak throughput × 1.5, adjust dynamically.
Chargeback: Track bytes produced/consumed, partitions, storage for cost allocation.

What's Next?

Next Module: 10.4.4 – Broker Placement and Topology

References:

Quotas → M6: Operations
ACLs → M9: Kafka Security