KafkaIQ

4.1 Core Kafka Metrics

Module Overview

Goal: Build a shared vocabulary for observability.

What You'll Learn:

  • Key Kafka metrics categories
  • Throughput metrics (bytes in/out, messages in)
  • Latency metrics (request latency, response time)
  • Request handler metrics
  • Disk and network metrics
  • Controller metrics
  • JVM and resource metrics
  • How to access and interpret metrics

What's NOT Covered Here:

  • Alerting strategy (covered in Module 4.3)
  • Dashboard design (covered in Module 4.3)
  • Client-specific metrics (covered in Module 4.2)
  • Performance tuning (covered in Module 5.1)

Key Insight: Metrics are the foundation of observability. Understanding what each metric means is critical for diagnosing issues and optimizing performance.

1. Kafka Metrics Architecture

How Kafka Exposes Metrics

KAFKA USES JMX (Java Management Extensions):

┌──────────────────────────────┐
│   Kafka Broker Process       │
│   ┌────────────────────────┐ │
│   │  JMX MBeans            │ │
│   │  (Metrics Registry)    │ │
│   └────────────────────────┘ │
│            ↓                  │
│   JMX Port: 9999 (default)   │
└──────────────────────────────┘
           ↓
   ┌──────────────────┐
   │ Monitoring Tools │
   │ - Prometheus     │
   │ - Grafana        │
   │ - Datadog        │
   │ - New Relic      │
   └──────────────────┘

Accessing Metrics

# Enable JMX on Kafka broker
# In kafka-server-start.sh or systemd service:
export JMX_PORT=9999
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false"

# Query metrics via JMX
jconsole localhost:9999

# Or use command-line tools
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec \
  --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi

2. Metric Categories

Six Core Categories

1. THROUGHPUT METRICS
   └─ Bytes in/out, messages in, requests per second

2. LATENCY METRICS
   └─ Request latency, response time, replication lag

3. RESOURCE METRICS
   └─ Disk usage, network bandwidth, CPU, memory

4. REPLICATION METRICS
   └─ Under-replicated partitions, ISR shrinks/expands

5. REQUEST METRICS
   └─ Request queue size, handler idle percent

6. CONTROLLER METRICS
   └─ Leader elections, partition count, active controllers

3. Throughput Metrics

Broker-Level Throughput

MessagesInPerSec

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

What it measures:
└─ Rate of messages produced to broker (all topics)

Units: messages/second

Interpretation:
├─ Spike: Sudden increase in producer traffic
├─ Drop: Producer issues or application down
└─ Steady: Normal operation

BytesInPerSec

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec

What it measures:
└─ Rate of bytes produced to broker (all topics)

Units: bytes/second

Interpretation:
├─ High: Heavy producer traffic
├─ Combined with MessagesInPerSec → average message size
└─ Monitor against network capacity

BytesOutPerSec

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec

What it measures:
└─ Rate of bytes consumed from broker (all topics)

Units: bytes/second

Interpretation:
├─ Should correlate with consumer activity
├─ Includes follower replication traffic
└─ Much higher than BytesInPerSec → consumer lag catchup

Per-Topic Throughput

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic={topic_name}

Available per-topic metrics:
├─ MessagesInPerSec (per topic)
├─ BytesInPerSec (per topic)
├─ BytesOutPerSec (per topic)
└─ TotalProduceRequestsPerSec (per topic)

Use case:
└─ Identify which topics are hotspots
└─ Monitor specific high-value topics

4. Latency Metrics

Request Latency

TotalTimeMs (Produce)

JMX Bean: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce

What it measures:
└─ Total time to process produce requests

Units: milliseconds

Components:
├─ Queue time: Waiting in request queue
├─ Local time: Leader writes to disk
├─ Remote time: Waiting for follower acks
└─ Response send time: Sending ack to producer

Target: < 100ms (p99)

TotalTimeMs (Fetch)

JMX Bean: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Fetch

What it measures:
└─ Total time to process fetch requests (consumers, followers)

Units: milliseconds

Target: < 50ms (p99)

High fetch latency causes:
├─ Disk I/O bottleneck
├─ Large fetch.min.bytes (consumers waiting)
└─ Network saturation

Replication Lag

ReplicaMaxLag

JMX Bean: kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica

What it measures:
└─ Maximum lag (in messages) for any follower replica

Units: number of messages

Interpretation:
├─ 0: All followers caught up (healthy)
├─ < 1000: Acceptable lag (depends on traffic)
├─ > 10000: Followers falling behind
└─ Increasing: Replication capacity issue

5. Resource Metrics

Disk Metrics

Log Size

JMX Bean: kafka.log:type=Log,name=Size,topic={topic},partition={partition}

What it measures:
└─ Size of log data on disk per partition

Units: bytes

Use case:
├─ Monitor disk space usage
├─ Plan capacity
└─ Verify retention policies working

LogFlushRateAndTimeMs

JMX Bean: kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs

What it measures:
└─ Rate and latency of log flushes to disk

Units: flushes/second, milliseconds

High flush time indicates:
└─ Disk I/O saturation
└─ Need for SSD or more disks

Network Metrics

NetworkProcessorAvgIdlePercent

JMX Bean: kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent

What it measures:
└─ Percentage of time network threads are idle

Units: percentage (0-100)

Interpretation:
├─ >80%: Healthy (threads mostly idle)
├─ 50-80%: Moderate load
├─ <50%: Network threads saturated
└─ <20%: Critical (need more network threads)

6. Replication Health Metrics

Under-Replicated Partitions

JMX Bean: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

What it measures:
└─ Number of partitions where ISR < replica set

Units: count

Target: 0

Non-zero indicates:
├─ Broker failure or slowness
├─ Network issues
├─ Disk I/O bottleneck
└─ Replication capacity exceeded

ISR Shrinks/Expands

JMX Bean: kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
JMX Bean: kafka.server:type=ReplicaManager,name=IsrExpandsPerSec

What it measures:
└─ Rate of ISR membership changes

Units: events/second

High shrink rate indicates:
├─ Followers frequently falling behind
├─ replica.lag.time.max.ms too aggressive
└─ Capacity issues

High expand rate (after shrinks) indicates:
└─ Recovery happening (good)

Offline Partitions

JMX Bean: kafka.controller:type=KafkaController,name=OfflinePartitionsCount

What it measures:
└─ Number of partitions with no leader

Units: count

Target: 0

Non-zero indicates:
├─ All ISR replicas down
├─ Unclean leader election disabled
└─ Critical issue requiring immediate attention

7. Request Handler Metrics

Request Queue Size

JMX Bean: kafka.network:type=RequestChannel,name=RequestQueueSize

What it measures:
└─ Number of requests waiting to be processed

Units: count

Interpretation:
├─ 0: No backlog (healthy)
├─ < 100: Acceptable
├─ > 500: Request handlers overloaded
└─ Growing: Need more I/O threads or capacity

Request Handler Idle Percent

JMX Bean: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent

What it measures:
└─ Percentage of time request handlers are idle

Units: percentage (0-100)

Interpretation:
├─ >80%: Handlers mostly idle (healthy)
├─ 50-80%: Moderate load
├─ <50%: Handlers saturated
└─ <20%: Critical (need more I/O threads)

8. Controller Metrics

Active Controller Count

JMX Bean: kafka.controller:type=KafkaController,name=ActiveControllerCount

What it measures:
└─ Whether this broker is the active controller

Values:
├─ 1: This broker is the controller
└─ 0: This broker is NOT the controller

Use case:
└─ Identify which broker is controller
└─ Monitor for controller failovers

Leader Election Rate

JMX Bean: kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs

What it measures:
└─ Rate and latency of leader elections

Units: elections/second, milliseconds

High rate indicates:
├─ Frequent broker failures
├─ Unstable cluster
└─ Network issues

High latency indicates:
├─ Many partitions
├─ Controller overloaded
└─ Coordination system slow

Partition Count

JMX Bean: kafka.controller:type=KafkaController,name=GlobalPartitionCount

What it measures:
└─ Total number of partitions across all topics

Units: count

Use case:
├─ Capacity planning
├─ Monitor against recommended limits (< 4000 per broker)
└─ Understand cluster scale

9. JVM Metrics

Heap Memory

JMX Bean: java.lang:type=Memory,name=HeapMemoryUsage

What it measures:
└─ JVM heap memory usage

Attributes:
├─ used: Currently used heap
├─ committed: Currently allocated heap
├─ max: Maximum heap size
└─ init: Initial heap size

Target:
└─ used < 70% of max (avoid GC thrashing)

Garbage Collection

JMX Bean: java.lang:type=GarbageCollector,name={collector_name}

Collectors:
├─ G1 Young Generation
├─ G1 Old Generation
└─ (varies by GC algorithm)

Metrics:
├─ CollectionCount: Number of GC events
├─ CollectionTime: Time spent in GC
└─ GC pause time per hour

Target:
└─ < 1% of time spent in GC
└─ Pause times < 100ms

10. Producer Metrics (Broker-Side)

Failed Produce Requests

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec

What it measures:
└─ Rate of failed produce requests

Units: requests/second

Causes:
├─ NOT_ENOUGH_REPLICAS (min.insync.replicas not met)
├─ MESSAGE_TOO_LARGE
├─ INVALID_TIMESTAMP
└─ Other validation failures

Target: 0 or near-zero

11. Consumer Metrics (Broker-Side)

Failed Fetch Requests

JMX Bean: kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec

What it measures:
└─ Rate of failed fetch requests

Units: requests/second

Causes:
├─ OFFSET_OUT_OF_RANGE (consumer lag too high)
├─ UNKNOWN_TOPIC_OR_PARTITION
├─ NOT_LEADER_FOR_PARTITION
└─ Transient errors

Target: 0 or near-zero

12. Key Metric Relationships

Throughput vs Latency

RELATIONSHIP:

High throughput + low latency:
└─ Healthy cluster, well-tuned

High throughput + high latency:
└─ Cluster saturated, need capacity

Low throughput + high latency:
└─ Bottleneck somewhere (disk, network, CPU)

Low throughput + low latency:
└─ Normal (light load)

ISR vs Under-Replication

RELATIONSHIP:

ISR shrinks ↑ + Under-replicated partitions ↑:
└─ Replication falling behind (capacity issue)

ISR shrinks ↑ + Under-replicated partitions = 0:
└─ Temporary fluctuations (transient issues)

ISR expands ↑:
└─ Recovery from previous issues

13. Metrics Collection Best Practices

Polling Frequency

RECOMMENDED INTERVALS:

High-frequency (10-30 seconds):
├─ Throughput (MessagesInPerSec, BytesInPerSec)
├─ Request latency
├─ Under-replicated partitions
└─ Offline partitions

Medium-frequency (1 minute):
├─ Resource usage (CPU, memory, disk)
├─ Network metrics
├─ Request handler idle percent
└─ GC metrics

Low-frequency (5 minutes):
├─ Disk space
├─ Partition count
└─ Topic-level metrics

Retention

RECOMMENDED RETENTION:

Raw metrics:
└─ 7-14 days (high resolution)

Aggregated metrics:
├─ 1-minute rollups: 90 days
├─ 5-minute rollups: 1 year
└─ 1-hour rollups: 2+ years

Use case:
├─ Recent data: Troubleshooting
├─ Historical data: Capacity planning, trend analysis

14. Metric Export Options

Prometheus

# JMX Exporter for Prometheus
# Download jmx_prometheus_javaagent.jar

# Start Kafka with exporter
export KAFKA_OPTS="-javaagent:/path/to/jmx_prometheus_javaagent.jar=7071:/path/to/config.yml"

# Prometheus scrapes metrics from http://broker:7071/metrics

Grafana

# Use Prometheus as data source
# Import Kafka dashboard (ID: 721 or custom)

Key panels:
├─ Throughput over time
├─ Request latency (p50, p95, p99)
├─ Under-replicated partitions
├─ Disk usage per broker
└─ GC pause time

Commercial Tools

INTEGRATION OPTIONS:

Datadog:
└─ Native Kafka integration via JMX

New Relic:
└─ JMX integration or agent

Confluent Control Center:
└─ Built-in monitoring (Confluent Platform)

LinkedIn Cruise Control:
└─ Rebalancing + monitoring

15. Key Takeaways

You've learned:

✅ Kafka exposes metrics via JMX
✅ Six core categories: throughput, latency, resources, replication, requests, controller
✅ Throughput metrics measure bytes/messages in/out per second
✅ Latency metrics measure request processing time
✅ Under-replicated partitions is critical health indicator (target: 0)
✅ Request handler idle percent shows processing capacity
✅ Controller metrics track cluster-wide state
✅ JVM metrics (heap, GC) indicate broker health
✅ Metrics relationships help diagnose issues
✅ Collection frequency balances granularity vs cost

16. Essential Metrics Checklist

MUST MONITOR:

Cluster Health:
☑ Under-replicated partitions (target: 0)
☑ Offline partitions (target: 0)
☑ Active controller count (exactly 1 in cluster)

Performance:
☑ Request latency (p99 < 100ms)
☑ Throughput (bytes in/out per sec)
☑ Network processor idle percent (> 50%)

Resources:
☑ Disk usage per broker (< 70%)
☑ JVM heap usage (< 70%)
☑ GC pause time (< 100ms)

Replication:
☑ ISR shrinks/expands rate
☑ Replica lag

Summary

Metrics are your eyes into Kafka:

Core Metrics =
├─ Throughput (how much data)
├─ Latency (how fast)
├─ Resources (capacity remaining)
├─ Replication (health status)
├─ Requests (queue depth)
└─ Controller (cluster state)

Good Monitoring =
├─ Collect the right metrics
├─ At the right frequency
├─ Store with sufficient retention
├─ Visualize effectively
└─ Alert on critical thresholds

Remember: Metrics tell the story of your cluster's health. Learn to read them, and you'll diagnose issues before they become outages.

Next up: Module 4.2 — Client, Broker & Topic-Level Monitoring — Learn where problems surface and how to identify bottlenecks.