KafkaIQ

6.1 Quotas, Throttling & Resource Management

Module Overview

Goal: Protect the cluster from abuse and ensure fair resource usage.

What You'll Learn:

  • Client quotas (producer, consumer, request)
  • Throughput throttling mechanisms
  • Request rate limits
  • Fair usage strategies
  • Multi-tenant cluster design
  • Quota configuration and monitoring
  • Handling quota violations

What's NOT Covered Here:

  • Performance tuning (covered in Module 5.1)
  • Dynamic configuration mechanics (covered in Module 3.1)
  • Monitoring details (covered in Module 4.x)

Key Insight: Quotas protect the cluster from noisy neighbors and ensure predictable performance for all applications.

1. Why Quotas Matter

The Noisy Neighbor Problem

SCENARIO: Shared Kafka cluster

Application A (well-behaved):
└─ Produces 10 MB/s consistently

Application B (misbehaving):
└─ Suddenly produces 1 GB/s (bug or misconfiguration)

WITHOUT QUOTAS:
├─ Application B saturates broker network
├─ Application A experiences latency spikes
├─ All consumers affected by backlog
└─ Cluster instability

WITH QUOTAS:
├─ Application B throttled to quota limit
├─ Application A unaffected
├─ Cluster remains stable
└─ Application B errors (alerting team to fix bug)

Use Cases for Quotas

1. MULTI-TENANCY:
   └─ Isolate teams/applications in shared cluster

2. COST CONTROL:
   └─ Prevent runaway costs from excessive usage

3. STABILITY:
   └─ Protect cluster from unexpected traffic spikes

4. FAIRNESS:
   └─ Ensure equal access to resources

5. CAPACITY MANAGEMENT:
   └─ Control resource usage within planned limits

2. Types of Quotas

Three Quota Types

1. PRODUCER QUOTAS:
   └─ Limit bytes produced per second

2. CONSUMER QUOTAS:
   └─ Limit bytes consumed per second

3. REQUEST QUOTAS:
   └─ Limit request percentage (CPU time)

Applied to:
├─ User (authenticated clients)
├─ Client ID
├─ User + Client ID combination
└─ Default (all unmatched clients)

3. Producer Quotas

What They Control

Producer Quota = Max bytes produced per second

Enforcement:
├─ Broker tracks bytes produced per user/client
├─ If quota exceeded: Broker delays responses
├─ Producer waits, sees increased latency
└─ Throttling transparent to producer

Example:
├─ Quota: 10 MB/s
├─ Producer sends 15 MB/s
├─ Broker throttles last 5 MB/s
└─ Producer experiences backpressure

Setting Producer Quotas

# Set producer quota for user
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name alice \
  --alter \
  --add-config producer_byte_rate=10485760

# 10 MB/s = 10 * 1024 * 1024 bytes/s

# Set producer quota for client ID
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type clients \
  --entity-name my-producer-app \
  --alter \
  --add-config producer_byte_rate=20971520

# Set default producer quota
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-default \
  --alter \
  --add-config producer_byte_rate=1048576

# 1 MB/s default for all users

Producer Quota Metrics

JMX Metrics:

kafka.server:type=Request,user={user},client-id={client}

Metrics:
├─ throttle-time: Time in milliseconds client was throttled
├─ byte-rate: Current produce byte rate
└─ request-rate: Current request rate

High throttle-time indicates:
└─ Client exceeding quota (expected behavior)

4. Consumer Quotas

What They Control

Consumer Quota = Max bytes consumed per second

Enforcement:
├─ Broker tracks bytes fetched per user/client
├─ If quota exceeded: Broker delays fetch responses
├─ Consumer experiences increased fetch latency
└─ Consumer lag may increase if processing fast

Example:
├─ Quota: 20 MB/s
├─ Consumer fetches 30 MB/s
├─ Broker throttles to 20 MB/s
└─ Consumer experiences delays

Setting Consumer Quotas

# Set consumer quota for user
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name bob \
  --alter \
  --add-config consumer_byte_rate=20971520

# 20 MB/s

# Set consumer quota for client ID
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type clients \
  --entity-name my-consumer-app \
  --alter \
  --add-config consumer_byte_rate=52428800

# 50 MB/s

# Set default consumer quota
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-default \
  --alter \
  --add-config consumer_byte_rate=2097152

# 2 MB/s default

5. Request Quotas

What They Control

Request Quota = Max request percentage (CPU time)

Measured as:
└─ Percentage of time broker's request handler threads spend on client's requests

Example:
├─ Request quota: 10%
├─ Broker has 8 request handler threads
├─ Client can use max 0.8 thread-equivalents
└─ Excessive requests are throttled

Use case:
└─ Limit clients that make many small requests
└─ Prevent request storms

Setting Request Quotas

# Set request quota (percentage)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name charlie \
  --alter \
  --add-config request_percentage=10

# Client can use max 10% of broker request handler time

# Set default request quota
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-default \
  --alter \
  --add-config request_percentage=5

# 5% default for all users

6. Quota Precedence

Entity Hierarchy

QUOTA LOOKUP ORDER (first match wins):

1. /config/users/<user>/clients/<client-id>
   └─ Most specific: Exact user + client ID

2. /config/users/<user>
   └─ User-specific quota

3. /config/clients/<client-id>
   └─ Client ID-specific quota

4. /config/users/<default>/clients/<client-id>
   └─ Default user + specific client ID

5. /config/users/<user>/clients/<default>
   └─ Specific user + default client ID

6. /config/users/<default>/clients/<default>
   └─ Default for both user and client ID

7. /config/users/<default>
   └─ Default user quota

8. /config/clients/<default>
   └─ Default client ID quota

9. No quota
   └─ Unlimited (not recommended)

Example Precedence

Configuration:
├─ User alice: producer_byte_rate = 10 MB/s
├─ Client ID my-app: producer_byte_rate = 20 MB/s
└─ Default user: producer_byte_rate = 1 MB/s

Scenarios:

Client: user=alice, client-id=my-app
└─ Quota: 10 MB/s (user alice quota)

Client: user=bob, client-id=my-app
└─ Quota: 20 MB/s (client-id my-app quota)

Client: user=bob, client-id=other-app
└─ Quota: 1 MB/s (default user quota)

Client: user=alice, client-id=other-app
└─ Quota: 10 MB/s (user alice quota)

7. Setting Combined Quotas

Multiple Quotas per Entity

# Set both producer and consumer quotas
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name dave \
  --alter \
  --add-config producer_byte_rate=10485760,consumer_byte_rate=20971520

# Set all three quota types
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name eve \
  --alter \
  --add-config producer_byte_rate=10485760,consumer_byte_rate=20971520,request_percentage=10

8. Viewing Quotas

List Current Quotas

# View user quotas
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name alice \
  --describe

# Output:
# Configs for user-principal 'alice' are:
#   producer_byte_rate=10485760 sensitive=false synonyms={DYNAMIC_USER_CONFIG:producer_byte_rate=10485760}

# View client quotas
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type clients \
  --entity-name my-app \
  --describe

# View all user quotas
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --describe

# View default quotas
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-default \
  --describe

9. Removing Quotas

Delete Quota Configurations

# Remove user quota (revert to default or no quota)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name alice \
  --alter \
  --delete-config producer_byte_rate

# Remove specific quota from combined config
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name bob \
  --alter \
  --delete-config consumer_byte_rate

# Remove all quotas for user
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users \
  --entity-name charlie \
  --alter \
  --delete-config producer_byte_rate,consumer_byte_rate,request_percentage

10. Monitoring Quotas

Key Metrics

JMX Metrics:

kafka.server:type=Request,user={user},client-id={client}

Key metrics:
├─ throttle-time-avg: Average throttle time in ms
├─ throttle-time-max: Max throttle time in ms
├─ byte-rate: Current byte rate
└─ request-rate: Current request rate

High throttle-time indicates:
└─ Client hitting quota limit (expected if quota configured)

Zero throttle-time indicates:
└─ Client under quota (not being throttled)

Client-Side Metrics

Producer Metrics:

produce-throttle-time-avg
produce-throttle-time-max

Consumer Metrics:

fetch-throttle-time-avg
fetch-throttle-time-max

Use case:
├─ Applications can monitor if they're being throttled
├─ Alert if throttle time > 0 (unexpected quota hit)
└─ Dashboard showing quota utilization

11. Multi-Tenant Cluster Design

Isolation Strategies

STRATEGY 1: Quota-Based Isolation (Soft)
├─ Single cluster, multiple tenants
├─ Each tenant has quota
├─ Shared resources, fair usage enforced
└─ Cost-effective, simple

STRATEGY 2: Topic-Level Isolation (Medium)
├─ Separate topics per tenant
├─ Quotas per tenant
├─ ACLs for access control
└─ Better isolation, more complex

STRATEGY 3: Cluster-Level Isolation (Hard)
├─ Separate cluster per tenant
├─ Complete isolation
├─ No noisy neighbor risk
└─ Higher cost, more management overhead

RECOMMENDATION:
└─ Start with Strategy 1 (quotas)
└─ Move to Strategy 2 if noisy neighbor issues persist
└─ Strategy 3 only for critical/high-value tenants

Example Multi-Tenant Configuration

CLUSTER WITH 3 TENANTS:

Tenant A (high priority):
├─ producer_byte_rate: 50 MB/s
├─ consumer_byte_rate: 100 MB/s
└─ Topics: tenantA.*

Tenant B (medium priority):
├─ producer_byte_rate: 20 MB/s
├─ consumer_byte_rate: 40 MB/s
└─ Topics: tenantB.*

Tenant C (low priority):
├─ producer_byte_rate: 10 MB/s
├─ consumer_byte_rate: 20 MB/s
└─ Topics: tenantC.*

Configuration:
# Set quotas per tenant user principal
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users --entity-name tenantA \
  --alter --add-config producer_byte_rate=52428800,consumer_byte_rate=104857600

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users --entity-name tenantB \
  --alter --add-config producer_byte_rate=20971520,consumer_byte_rate=41943040

kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type users --entity-name tenantC \
  --alter --add-config producer_byte_rate=10485760,consumer_byte_rate=20971520

12. Handling Quota Violations

What Happens When Quota Exceeded?

CLIENT PERSPECTIVE:

Producer:
├─ send() calls take longer to complete
├─ buffer fills up if sending faster than quota
├─ May hit max.block.ms timeout
└─ Error: TimeoutException (if buffer full too long)

Consumer:
├─ poll() calls take longer to return
├─ fetch.max.wait.ms is effectively extended
├─ Consumer lag may increase
└─ No error (just slower consumption)

BROKER PERSPECTIVE:
├─ Tracks bytes/requests per client
├─ Calculates delay needed to enforce quota
├─ Delays response to client
└─ Metrics show throttle-time

Responding to Quota Violations

INVESTIGATION:

1. Check if violation is expected
   └─ Is quota set correctly for this client?

2. Check client behavior
   ├─ Traffic spike (legitimate or bug?)
   ├─ Misconfigured batching
   └─ Runaway loop

3. Check quota utilization
   └─ Is client consistently hitting quota or occasional?

ACTIONS:

If quota too low:
└─ Increase quota for legitimate use case

If client misbehaving:
└─ Fix application bug (loop, misconfiguration)

If traffic spike expected:
└─ Temporarily increase quota
└─ Or scale cluster to handle traffic

13. Fair Usage Policies

Defining Fair Usage

FAIR USAGE POLICY EXAMPLE:

Default Quotas:
├─ Producer: 1 MB/s per application
├─ Consumer: 2 MB/s per application
├─ Request: 5% per application

Request Process:
├─ Teams can request higher quotas
├─ Justify business need
├─ Capacity planning review
└─ Approved by Platform team

Monitoring:
├─ Monthly quota utilization review
├─ Alert if consistently hitting quota
└─ Proactive capacity planning

Enforcement:
├─ Automatic throttling at quota limit
├─ Alert to team when throttled
└─ Temporary quota increase for urgent needs

14. Quota Best Practices

Configuration

✅ DO:
├─ Set default quotas for all users
├─ Set explicit quotas for known high-traffic apps
├─ Monitor quota utilization regularly
├─ Document quota policies
├─ Review quotas quarterly
└─ Test quota enforcement in non-production

❌ DON'T:
├─ Leave unlimited (no default quota)
├─ Set quotas too low (causes legitimate apps to fail)
├─ Set quotas too high (defeats purpose)
├─ Ignore quota violations (investigate causes)
└─ Change quotas without notification

Capacity Planning with Quotas

QUOTA-BASED CAPACITY PLANNING:

1. Define cluster capacity
   └─ Example: 500 MB/s total throughput

2. Reserve capacity for overhead
   └─ 30% headroom: 350 MB/s usable

3. Allocate to tenants via quotas
   ├─ Tenant A: 150 MB/s
   ├─ Tenant B: 100 MB/s
   ├─ Tenant C: 50 MB/s
   └─ Default: 50 MB/s for others
   └─ Total: 350 MB/s

4. Monitor actual usage
   └─ Adjust quotas or scale cluster as needed

BENEFIT:
└─ Predictable performance
└─ Clear capacity allocation

15. Advanced Quota Scenarios

Scenario 1: Burst Traffic

CHALLENGE:
└─ Application has occasional traffic spikes (10x normal)

OPTIONS:

Option 1: High steady quota
└─ Wasteful (unused most of time)

Option 2: Low quota + manual increase
└─ Operationally expensive

Option 3: Token bucket / burst allowance
└─ NOT supported natively in Kafka
└─ Workaround: Temporarily increase quota during known events

Option 4: Separate cluster for burst traffic
└─ Expensive but effective

RECOMMENDATION:
└─ Set quota for 90th percentile traffic
└─ Accept throttling during rare spikes
└─ Or scale cluster to handle spikes

Scenario 2: Gradual Rollout

PROCESS:

1. Deploy new application with low quota
   └─ producer_byte_rate: 1 MB/s

2. Monitor behavior for 24 hours
   └─ Ensure no runaway loops or bugs

3. Gradually increase quota
   ├─ Day 1: 1 MB/s
   ├─ Day 2: 5 MB/s
   ├─ Day 3: 10 MB/s
   └─ Day 4: 20 MB/s (target)

4. Monitor at each step
   └─ Catch issues before full load

BENEFIT:
└─ Safe rollout with controlled blast radius

16. Key Takeaways

You've learned:

✅ Quotas protect cluster from noisy neighbors and ensure fairness
✅ Three quota types: producer byte rate, consumer byte rate, request percentage
✅ Quotas applied to: user, client ID, or user + client ID combination
✅ Precedence: User + Client ID > User > Client ID > Default
✅ Throttling: Broker delays responses to enforce quotas
✅ Multi-tenancy: Quotas enable shared clusters with isolation
✅ Monitoring: Track throttle-time metrics to identify violations
✅ Fair usage: Define policies and allocate capacity via quotas
✅ Best practice: Set default quotas, monitor utilization, adjust as needed

17. Quota Configuration Checklist

INITIAL SETUP:

☑ Set default user quota (conservative)
☑ Set default client quota (if using client IDs)
☑ Document quota policy
☑ Configure monitoring for throttle-time

PER-APPLICATION:

☑ Identify expected throughput
☑ Set appropriate quota (with headroom)
☑ Test enforcement in non-production
☑ Monitor utilization after deployment
☑ Adjust quota based on actual usage

ONGOING:

☑ Review quota utilization monthly
☑ Adjust quotas for growth
☑ Investigate quota violations
☑ Update capacity plan
☑ Communicate quota changes

Summary

Quotas enable shared clusters with predictable performance:

Quota Management =
├─ Protect cluster from abuse
├─ Ensure fair resource allocation
├─ Enable multi-tenancy
├─ Provide predictable performance
└─ Support capacity planning

Good Quota Strategy =
├─ Set conservative defaults
├─ Explicit quotas for known apps
├─ Monitor utilization
├─ Adjust based on usage patterns
├─ Document and communicate policies
└─ Balance protection vs flexibility

Remember: Quotas are a powerful tool for cluster stability, but they're not a substitute for proper capacity planning. Scale the cluster when quotas consistently hit limits.

Next up: Module 7.1 — Coordination System Operations (ZooKeeper & KRaft) — Understand Kafka's coordination layer and the transition from ZooKeeper to KRaft.