KafkaIQ4.2 Client, Broker & Topic-Level Monitoring
Module Overview
Goal: Explain where problems surface and how to identify bottlenecks.
What You'll Learn:
- Producer metrics and monitoring
- Consumer metrics and lag monitoring
- Topic-level indicators
- Broker-level bottleneck identification
- Correlating metrics across layers
- Troubleshooting common scenarios
What's NOT Covered Here:
- Core metric definitions (covered in Module 4.1)
- Alerting configuration (covered in Module 4.3)
- Performance tuning solutions (covered in Module 5.1)
Key Insight: Problems rarely exist in isolation. Learning to correlate metrics across producers, brokers, topics, and consumers is key to effective troubleshooting.
1. The Monitoring Stack
Three Observation Layers
CLIENT LAYER (Producers & Consumers):
├─ Application metrics
├─ Client library metrics
├─ Connection health
└─ Error rates
↕ (produces/consumes)
BROKER LAYER:
├─ Request handling
├─ Disk I/O
├─ Network throughput
└─ Replication
↕ (serves)
TOPIC LAYER:
├─ Partition distribution
├─ Message rates
├─ Retention
└─ Consumer lag
2. Producer Monitoring
Key Producer Metrics
record-send-rate
Metric: record-send-rate
Type: Producer metric (client-side)
What it measures:
└─ Rate of records sent per second
Units: records/second
Interpretation:
├─ Steady: Normal operation
├─ Drop to 0: Producer stopped or paused
├─ Spike: Burst of data
└─ Declining: Backpressure or throttling
record-error-rate
Metric: record-error-rate
Type: Producer metric (client-side)
What it measures:
└─ Rate of failed send attempts
Units: records/second
Target: 0 or near-zero
High error rate causes:
├─ Network issues
├─ Broker capacity exceeded
├─ Topic doesn't exist
├─ Authorization failures
└─ Message too large
request-latency-avg / request-latency-max
Metric: request-latency-avg, request-latency-max
Type: Producer metric (client-side)
What it measures:
└─ Time from send() to ack received
Units: milliseconds
Targets:
├─ Average: < 50ms
├─ Max (p99): < 200ms
High latency causes:
├─ Broker overloaded
├─ Network congestion
├─ Replication lag (if acks=all)
└─ Large batches (linger.ms)
buffer-available-bytes
Metric: buffer-available-bytes
Type: Producer metric (client-side)
What it measures:
└─ Available space in producer buffer
Units: bytes
Interpretation:
├─ Near max: Buffer empty (healthy)
├─ Decreasing: Producer faster than broker
├─ Near 0: Buffer full (blocking)
└─ 0: Producer blocked, max.block.ms timeout risk
Producer Health Indicators
HEALTHY PRODUCER:
├─ record-send-rate: steady
├─ record-error-rate: 0
├─ request-latency-avg: < 50ms
├─ buffer-available-bytes: > 50% of max
└─ Compression-rate: > 0 (if enabled)
UNHEALTHY PRODUCER:
├─ record-send-rate: dropping
├─ record-error-rate: > 0
├─ request-latency-avg: > 200ms
├─ buffer-available-bytes: < 10% of max
└─ Errors in logs
3. Consumer Monitoring
Key Consumer Metrics
records-consumed-rate
Metric: records-consumed-rate
Type: Consumer metric (client-side)
What it measures:
└─ Rate of records consumed per second
Units: records/second
Interpretation:
├─ Steady: Normal consumption
├─ Drop to 0: Consumer paused, stopped, or no data
├─ Lower than expected: Consumer slow
└─ Spike: Catching up after lag
fetch-latency-avg
Metric: fetch-latency-avg
Type: Consumer metric (client-side)
What it measures:
└─ Time waiting for fetch response from broker
Units: milliseconds
Target: < 100ms
High latency causes:
├─ fetch.min.bytes waiting for data
├─ Broker overloaded
├─ Network issues
└─ Disk I/O bottleneck on broker
Consumer Lag (CRITICAL)
Metric: records-lag-max, records-lag
Type: Consumer metric (client-side)
What it measures:
└─ Number of records consumer is behind latest offset
Units: number of records
Interpretation:
├─ 0: Consumer caught up (ideal)
├─ < 1000: Acceptable lag (depends on traffic)
├─ > 10000: Consumer falling behind
├─ Growing: Consumer can't keep up (PROBLEM)
└─ Shrinking: Consumer catching up (RECOVERY)
Consumer lag formula:
Lag = (Log End Offset - Current Consumer Offset)
Example:
├─ Log End Offset: 1,000,000
├─ Consumer Offset: 995,000
└─ Lag: 5,000 records
Monitoring Consumer Lag
kafka-consumer-groups tool:
# View consumer group lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group \
--describe
# Output:
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
# my-consumer-group orders 0 1000000 1000000 0
# my-consumer-group orders 1 995000 1000000 5000 ← LAG!
Burrow (LinkedIn's lag monitoring):
Burrow provides:
├─ Consumer lag monitoring
├─ Lag trend analysis (increasing/decreasing)
├─ Consumer status (OK, WARNING, ERROR)
├─ API for integration
└─ Historical lag data
Status classification:
├─ OK: Consumer keeping up
├─ WARNING: Lag increasing but manageable
└─ ERROR: Consumer falling far behind
4. Topic-Level Monitoring
Partition Distribution
# Check partition distribution
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe \
--topic orders
# Look for:
├─ Balanced leader distribution
├─ All replicas in ISR
├─ No under-replicated partitions
└─ Even partition count per broker
Topic Throughput
Per-Topic Metrics (broker-side):
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=orders
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=orders
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=orders
Use cases:
├─ Identify hotspot topics
├─ Monitor critical topics separately
├─ Validate traffic patterns
└─ Capacity planning per topic
Topic Size on Disk
Monitoring topic size:
# Via metrics
kafka.log:type=Log,name=Size,topic=orders,partition=0
# Via command line
du -sh /var/kafka/data/orders-*
Trends to watch:
├─ Unexpected growth (retention issue?)
├─ Size not shrinking (compaction not working?)
├─ Imbalanced partition sizes
└─ Approaching disk capacity
5. Broker-Level Monitoring
CPU Usage
Metric: System CPU usage (OS-level)
Targets:
├─ Average: < 70%
├─ Peak: < 90%
High CPU causes:
├─ Compression/decompression
├─ Request handling
├─ Replication
├─ GC activity
└─ Under-provisioned cluster
Resolution:
├─ Add brokers (scale horizontally)
├─ Tune GC
├─ Optimize compression settings
└─ Check for slow consumers causing backpressure
Disk Usage
Metric: Disk space used/available (OS-level)
Targets:
├─ < 70% full (comfortable)
├─ < 80% full (monitor closely)
├─ > 80% full (add capacity soon)
└─ > 90% full (CRITICAL)
Monitoring:
├─ Total disk space
├─ Per log.dirs directory
├─ Growth rate
└─ Time to full (at current rate)
Network Bandwidth
Metrics: Network bytes in/out (OS-level)
Bottleneck indicators:
├─ Network utilization > 80%
├─ Packet drops
├─ TCP retransmissions
└─ High network processor busy percent
Resolution:
├─ Upgrade network (1 Gbps → 10 Gbps)
├─ Enable compression
├─ Add more brokers
└─ Review replica.fetch.max.bytes
Disk I/O
Metrics: Disk read/write IOPS, throughput, latency (OS-level)
Bottleneck indicators:
├─ Disk utilization > 80%
├─ High I/O wait percentage
├─ Slow LogFlushRateAndTimeMs
└─ Growing request queue size
Resolution:
├─ Use SSDs instead of HDDs
├─ Add more disks (JBOD)
├─ Increase log.dirs count
└─ Review flush settings
6. Correlating Metrics Across Layers
Scenario 1: Producer Slowness
SYMPTOMS:
├─ Producer: High request-latency-avg
├─ Producer: Buffer filling up
└─ Producer: record-send-rate decreasing
INVESTIGATION:
1. Check broker throughput
└─ Is BytesInPerSec unusually high?
2. Check broker latency
└─ Is TotalTimeMs (Produce) high?
3. Check broker resources
└─ CPU, disk I/O, network at limit?
4. Check replication lag
└─ Are followers slow (if acks=all)?
CONCLUSION:
├─ Broker overloaded → Scale cluster
├─ Replication slow → Add disk capacity
└─ Network saturated → Upgrade network
Scenario 2: Consumer Lag Growing
SYMPTOMS:
├─ Consumer: Lag increasing steadily
├─ Consumer: records-consumed-rate < MessagesInPerSec
└─ Broker: BytesOutPerSec high (consumers catching up)
INVESTIGATION:
1. Check consumer processing time
└─ Is application slow to process messages?
2. Check fetch settings
└─ Is fetch.min.bytes/max.wait.ms configured?
3. Check consumer thread count
└─ Is max.poll.records too high?
4. Check partition count
└─ Can you add more consumers for parallelism?
CONCLUSION:
├─ Consumer slow → Optimize app, add consumers
├─ Partition bottleneck → Increase partitions
├─ Network slow → Check consumer-broker connectivity
└─ Topic too fast → Review producer throttling
Scenario 3: Disk Space Running Out
SYMPTOMS:
├─ Disk usage > 85%
├─ Growth rate: 10 GB/hour
└─ Time to full: 12 hours
INVESTIGATION:
1. Check topic retention
└─ Is retention.ms set appropriately?
2. Check log cleanup
└─ Is log.retention.check.interval.ms working?
3. Check topic sizes
└─ Which topics are largest?
4. Check compaction (if applicable)
└─ Is min.cleanable.dirty.ratio appropriate?
CONCLUSION:
├─ Retention too long → Reduce retention.ms
├─ Cleanup not running → Check logs for errors
├─ Single topic huge → Review that topic's design
└─ Add disk capacity → Plan scaling
7. Common Bottleneck Patterns
Producer-Side Bottlenecks
INDICATORS:
├─ High record-error-rate
├─ High request-latency
├─ Buffer exhaustion
└─ Timeout errors
ROOT CAUSES:
├─ Broker capacity exceeded
├─ Network congestion
├─ Large messages (tune max.message.bytes)
├─ Batching too aggressive
└─ Serialization overhead
Broker-Side Bottlenecks
INDICATORS:
├─ High RequestQueueSize
├─ Low RequestHandlerAvgIdlePercent
├─ High disk I/O wait
└─ High CPU usage
ROOT CAUSES:
├─ Under-provisioned cluster
├─ Hotspot topic/partition
├─ Slow disks (use SSD)
├─ Too many partitions per broker
└─ Replication overhead
Consumer-Side Bottlenecks
INDICATORS:
├─ Growing consumer lag
├─ Low records-consumed-rate
├─ High fetch-latency
└─ Rebalance frequency
ROOT CAUSES:
├─ Slow message processing
├─ Too few consumers (add parallelism)
├─ fetch.min.bytes waiting
├─ Large message batches
└─ GC pauses
8. Monitoring Rebalances
Consumer Group Rebalance Metrics
Indicators of rebalancing:
├─ Consumer: rebalance-latency-avg
├─ Consumer: rebalance-latency-max
├─ Consumer: rebalance-rate-per-hour
Frequent rebalances indicate:
├─ session.timeout.ms too aggressive
├─ max.poll.interval.ms too short
├─ Consumer instances flapping
├─ Network instability
└─ Long processing times
9. Multi-Broker Monitoring
Cluster-Wide Views
AGGREGATE METRICS:
Throughput:
└─ Sum of BytesInPerSec across all brokers
Capacity:
└─ Lowest broker's remaining disk space
Health:
└─ Any broker with URP > 0?
Balance:
└─ Even distribution of leaders and partitions?
Use cases:
├─ Capacity planning
├─ Identify imbalanced brokers
├─ Detect single broker issues
└─ Cluster-wide health checks
10. Client ID Tracking
Using Client IDs for Monitoring
Benefits of client IDs:
├─ Identify which application is producer/consumer
├─ Track per-client metrics
├─ Apply per-client quotas
└─ Troubleshoot specific applications
Configuration:
# Producer
props.put("client.id", "order-service-producer-1");
# Consumer
props.put("client.id", "inventory-consumer-group");
Metrics filtered by client ID:
└─ All client metrics include client ID tag
└─ Broker metrics can show per-client stats
11. Lag vs Throughput Analysis
Understanding the Relationship
SCENARIO 1: Low lag, low throughput
├─ Consumer keeping up
├─ Producer sending slowly
└─ Normal low-traffic operation
SCENARIO 2: Low lag, high throughput
├─ Consumer keeping up
├─ Heavy traffic
└─ Healthy, well-scaled system
SCENARIO 3: High lag, low throughput
├─ Consumer not running or stuck
├─ Immediate investigation needed
└─ Data piling up
SCENARIO 4: High lag, high throughput
├─ Consumer trying to catch up
├─ Producer overwhelming consumer
└─ Need more consumer capacity
12. Key Takeaways
You've learned:
✅ Producer metrics track send rates, errors, latency, buffer usage
✅ Consumer lag is the most critical consumer metric (target: 0)
✅ Topic-level metrics show per-topic throughput and size
✅ Broker bottlenecks appear in CPU, disk I/O, or network saturation
✅ Correlating metrics across layers reveals root causes
✅ Common patterns help quickly diagnose issues
✅ Rebalances can indicate consumer group instability
✅ Client IDs enable per-application tracking
✅ Lag vs throughput analysis shows system health
13. Monitoring Checklist
DAILY CHECKS:
Producer Health:
☑ record-error-rate = 0
☑ request-latency < 100ms (p99)
☑ No buffer exhaustion
Consumer Health:
☑ Consumer lag < 1000 (or 0)
☑ Lag not growing
☑ records-consumed-rate steady
Broker Health:
☑ CPU < 70%
☑ Disk usage < 80%
☑ Network not saturated
☑ Request handlers > 50% idle
Topic Health:
☑ No under-replicated partitions
☑ Balanced partition distribution
☑ Expected throughput
Summary
Effective monitoring spans all layers:
Client → Broker → Topic Monitoring =
├─ Producers: Send rates, errors, latency
├─ Consumers: Lag, throughput, rebalances
├─ Brokers: Resources, requests, health
├─ Topics: Distribution, size, throughput
└─ Correlate across layers to find root cause
Good Monitoring =
├─ Track all three layers
├─ Understand metric relationships
├─ Recognize common patterns
├─ Act on trends, not just snapshots
└─ Monitor proactively, not reactively
Remember: Problems manifest across layers. A producer issue might be caused by a broker bottleneck. Consumer lag might stem from partition imbalance. Always correlate metrics to understand the full picture.
Next up: Module 4.3 — Alerting & Operational Dashboards — Turn metrics into actionable insights with effective alerts and visualization.