8.1 – Monitoring & Metrics
Overview
This module is the authoritative guide to monitoring and metrics for Kafka Connect clusters. By the end, you will understand:
- Key JMX metrics for workers, connectors, and tasks.
- Connector health indicators and failure signals.
- Consumer lag monitoring for sink connectors.
- Operational dashboards and alerting strategies.
All Connect monitoring concepts live here. Other modules (e.g., performance tuning, error handling) will reference this when discussing observability, but will not re-explain these concepts.
Monitoring Architecture
What to Monitor
┌──────────────────────────────────────┐
│ Kafka Connect Cluster │
│ │
│ Worker Metrics │
│ ├─ Connector count │
│ ├─ Task count │
│ ├─ Task failures │
│ └─ Rebalance frequency │
│ │
│ Connector Metrics │
│ ├─ Status (running, failed) │
│ ├─ Task status │
│ └─ Error logs │
│ │
│ Task Metrics │
│ ├─ Throughput (records/sec) │
│ ├─ Latency (batch time) │
│ ├─ Offset lag (sink only) │
│ └─ Error count │
│ │
└──────────────────────────────────────┘
Monitoring Stack
Industry Standard:
- JMX Exporter: Export Connect JMX metrics to Prometheus.
- Prometheus: Scrape and store metrics.
- Grafana: Visualize metrics and create dashboards.
- Alertmanager: Send alerts (Slack, PagerDuty, email).
Worker Metrics (JMX)
Key Metrics
1. Connector Count
Metric: kafka.connect:type=connect-worker-metrics,connector={connector-name}
Attribute: connector-count
Purpose: Total number of connectors running on this worker.
Alert: If connector count drops unexpectedly (connector deleted or failed).
2. Task Count
Metric: kafka.connect:type=connect-worker-metrics
Attribute: task-count
Purpose: Total number of tasks running on this worker.
Alert: If task count is 0 (all tasks failed or worker idle).
3. Rebalance Time
Metric: kafka.connect:type=connect-worker-rebalance-metrics
Attribute: rebalance-avg-time-ms, rebalance-max-time-ms
Purpose: Average and max time for worker rebalancing.
Alert: If rebalance time > 10 seconds (indicates cluster instability).
4. Task Failures
Metric: kafka.connect:type=connect-worker-metrics
Attribute: task-error-total-count
Purpose: Total number of task failures across all connectors.
Alert: If task failures > 0 (investigate connector logs).
Example: Query JMX Metrics via JConsole
Connect to Worker:
jconsole localhost:9999
Navigate to MBeans → kafka.connect → connect-worker-metrics
View Attributes: connector-count, task-count, task-error-total-count
Connector-Level Metrics
REST API: Connector Status
Endpoint: GET /connectors/{name}/status
Example:
curl http://localhost:8083/connectors/jdbc-source/status | jq
Response:
{
"name": "jdbc-source",
"connector": {
"state": "RUNNING",
"worker_id": "worker1:8083"
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "worker1:8083"
},
{
"id": 1,
"state": "FAILED",
"worker_id": "worker2:8083",
"trace": "org.apache.kafka.connect.errors.ConnectException: Failed to connect to database..."
}
],
"type": "source"
}
Key Fields
| Field | Values | Meaning |
|---|---|---|
connector.state |
RUNNING, PAUSED, FAILED |
Connector status |
tasks[].state |
RUNNING, FAILED |
Task status |
tasks[].trace |
Stack trace | Error details (if failed) |
Health Checks
Prometheus Exporter Pattern:
import requests
def check_connector_health(connector_name):
response = requests.get(f"http://localhost:8083/connectors/{connector_name}/status")
status = response.json()
if status["connector"]["state"] != "RUNNING":
return {"status": "unhealthy", "reason": "Connector not running"}
for task in status["tasks"]:
if task["state"] != "RUNNING":
return {"status": "unhealthy", "reason": f"Task {task['id']} failed"}
return {"status": "healthy"}
Alert: If any connector or task state is not RUNNING, send alert.
Task-Level Metrics (JMX)
Source Connector Metrics
1. Poll Rate
Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: source-record-poll-rate
Purpose: Records polled per second from the external system.
Expected: > 100 records/sec (depends on workload).
Alert: If poll rate = 0 for > 5 minutes (connector stalled).
2. Write Rate
Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: source-record-write-rate
Purpose: Records written to Kafka per second.
Expected: Should match poll-rate (if no transformations drop records).
Alert: If write rate < poll rate (records being dropped).
3. Poll Batch Time
Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: poll-batch-avg-time-ms
Purpose: Average time to poll a batch from the external system.
Expected: < 100ms (depends on external system).
Alert: If > 1000ms (external system slow or overloaded).
Sink Connector Metrics
1. Read Rate
Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: sink-record-read-rate
Purpose: Records read from Kafka per second.
Expected: > 100 records/sec (depends on workload).
Alert: If read rate = 0 for > 5 minutes (consumer lag or connector stalled).
2. Send Rate
Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: sink-record-send-rate
Purpose: Records written to external system per second.
Expected: Should match read-rate (if no errors).
Alert: If send rate < read rate (external system slow or errors).
3. Put Batch Time
Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: put-batch-avg-time-ms
Purpose: Average time to write a batch to the external system.
Expected: < 500ms (depends on external system).
Alert: If > 5000ms (external system slow, network issues, or large batches).
4. Offset Commit Success Rate
Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: offset-commit-success-percentage
Purpose: Percentage of successful offset commits.
Expected: 100%.
Alert: If < 99% (offset commit failures, investigate logs).
Consumer Lag Monitoring (Sink Connectors)
What is Consumer Lag?
Consumer lag is the difference between the latest offset in a Kafka partition and the last committed offset by the sink connector.
Formula: Lag = Latest Offset - Committed Offset
Monitoring Lag
Using Kafka CLI
kafka-consumer-groups --bootstrap-server localhost:9092 \
--describe --group connect-my-sink-connector
Output:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
connect-my-sink-connector orders 0 5000 5100 100
connect-my-sink-connector orders 1 4500 4500 0
Interpretation:
- Partition 0: Lag = 100 (sink is 100 records behind).
- Partition 1: Lag = 0 (sink is caught up).
Using JMX Metrics
Metric: kafka.consumer:type=consumer-fetch-manager-metrics,client-id={client-id}
Attribute: records-lag-max
Purpose: Maximum lag across all partitions.
Alert: If lag > 10,000 records for > 10 minutes (sink is falling behind).
Failure Signals and Alerting
Critical Alerts
1. Connector State = FAILED
Query:
curl http://localhost:8083/connectors/{name}/status | jq '.connector.state'
Alert: If state = FAILED, send PagerDuty alert.
Action: Investigate logs, check external system connectivity, restart connector.
2. Task State = FAILED
Query:
curl http://localhost:8083/connectors/{name}/status | jq '.tasks[].state'
Alert: If any task state = FAILED, send Slack alert.
Action: Check task logs, identify root cause (network, schema mismatch, etc.).
3. High Consumer Lag (Sink)
Metric: records-lag-max (JMX)
Threshold: Lag > 10,000 records for > 10 minutes.
Alert: Send Slack alert.
Action: Increase tasks.max, optimize sink writes, scale external system.
4. DLQ Topic Has Records
Query:
kafka-run-class kafka.tools.GetOffsetShell \
--broker-list localhost:9092 \
--topic connect-dlq
Alert: If DLQ topic offset > 0, send alert.
Action: Inspect DLQ records, identify root cause (schema mismatch, malformed data).
5. High Task Failure Rate
Metric: task-error-total-count (JMX)
Threshold: > 10 failures/hour.
Alert: Send email alert.
Action: Investigate connector logs, check error tolerance configs.
Operational Dashboards (Grafana)
Example: Connect Cluster Dashboard
Panels:
Connector Count (Gauge)
- Query:
kafka_connect_connect_worker_metrics_connector_count - Threshold: > 0 (green), = 0 (red)
- Query:
Task Count (Gauge)
- Query:
kafka_connect_connect_worker_metrics_task_count - Threshold: > 0 (green), = 0 (red)
- Query:
Source Throughput (Graph)
- Query:
sum(rate(kafka_connect_source_task_metrics_source_record_write_rate[5m])) - Label: "Records/sec"
- Query:
Sink Throughput (Graph)
- Query:
sum(rate(kafka_connect_sink_task_metrics_sink_record_send_rate[5m])) - Label: "Records/sec"
- Query:
Consumer Lag (Graph)
- Query:
kafka_consumer_fetch_manager_metrics_records_lag_max - Threshold: > 10,000 (yellow), > 50,000 (red)
- Query:
Task Failures (Counter)
- Query:
sum(kafka_connect_connect_worker_metrics_task_error_total_count) - Threshold: > 0 (alert)
- Query:
DLQ Records (Counter)
- Query:
kafka_log_log_size{topic="connect-dlq"} - Threshold: > 0 (alert)
- Query:
Logging Best Practices
Log Levels
Worker Config (log4j.properties):
log4j.logger.org.apache.kafka.connect=INFO
log4j.logger.io.confluent.connect=INFO
Increase for Debugging:
log4j.logger.org.apache.kafka.connect=DEBUG
Key Log Patterns
Connector Startup
[2023-10-27 10:00:00,123] INFO Starting connector jdbc-source (org.apache.kafka.connect.runtime.WorkerConnector)
Task Assignment
[2023-10-27 10:00:05,456] INFO Assigned task jdbc-source-0 to worker worker1:8083 (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
Error
[2023-10-27 10:05:30,789] ERROR Failed to poll records from database (com.example.JdbcSourceTask)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to database
at com.example.JdbcSourceTask.poll(JdbcSourceTask.java:123)
Offset Commit
[2023-10-27 10:10:00,000] INFO Committing offsets for task jdbc-source-0 (org.apache.kafka.connect.runtime.WorkerSourceTask)
Structured Logging for Connect
For production deployments, use structured logging (JSON format) to make logs machine-readable and easier to query in centralized logging systems (ELK, Splunk, CloudWatch, etc.).
Example: JSON Log Format
Log4j2 Configuration (log4j2.xml):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<JsonLayout compact="true" eventEol="true">
<KeyValuePair key="timestamp" value="$${date:yyyy-MM-dd'T'HH:mm:ss.SSSZ}"/>
<KeyValuePair key="level" value="$${level}"/>
<KeyValuePair key="logger" value="$${logger}"/>
<KeyValuePair key="message" value="$${message}"/>
<KeyValuePair key="thread" value="$${thread}"/>
</JsonLayout>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
Example JSON Log Output:
{
"timestamp": "2023-10-27T10:00:00.123Z",
"level": "INFO",
"logger": "org.apache.kafka.connect.runtime.WorkerConnector",
"message": "Starting connector jdbc-source",
"thread": "connect-worker-1",
"connector_name": "jdbc-source",
"worker_id": "worker1:8083"
}
Benefits:
- Queryable: Filter logs by
connector_name,level,worker_id. - Centralized: Easy to ingest into ELK, Splunk, or CloudWatch.
- Alerting: Trigger alerts based on error patterns (e.g.,
level=ERROR AND connector_name=jdbc-source).
JMX Configuration for Kafka Connect
To expose JMX metrics for monitoring, configure the JVM when starting Connect workers.
Step 1: Enable JMX on Worker Startup
Standalone Mode:
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=localhost"
bin/connect-standalone.sh config/connect-standalone.properties config/jdbc-source.properties
Distributed Mode:
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=<worker_hostname>"
bin/connect-distributed.sh config/connect-distributed.properties
Security Note: For production, enable JMX authentication and SSL/TLS:
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.password.file=/etc/kafka/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/kafka/jmxremote.access \
-Dcom.sun.management.jmxremote.ssl=true
Step 2: Install JMX Exporter (Prometheus)
Download JMX Exporter:
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar
Create JMX Exporter Config (jmx-exporter-config.yml):
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
- pattern: "kafka.connect<type=(.+), client-id=(.+)><>(.+):"
name: kafka_connect_$1_$3
labels:
client_id: "$2"
- pattern: "kafka.connect<type=(.+)><>(.+):"
name: kafka_connect_$1_$2
Start Worker with JMX Exporter:
export KAFKA_OPTS="-javaagent:/path/to/jmx_prometheus_javaagent-0.19.0.jar=7071:/path/to/jmx-exporter-config.yml"
bin/connect-distributed.sh config/connect-distributed.properties
Verify Metrics:
curl http://localhost:7071/metrics | grep kafka_connect
Prometheus Scrape Config (prometheus.yml):
scrape_configs:
- job_name: 'kafka-connect'
static_configs:
- targets: ['localhost:7071']
Log Rotation with Logback
For production deployments, use Logback for advanced log rotation and compression.
Logback Configuration (logback.xml):
<configuration>
<!-- Console Appender -->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<!-- Rolling File Appender -->
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/kafka/connect.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- Daily rollover -->
<fileNamePattern>/var/log/kafka/connect-%d{yyyy-MM-dd}.log.gz</fileNamePattern>
<!-- Keep 30 days of history -->
<maxHistory>30</maxHistory>
<!-- Max total log size: 10GB -->
<totalSizeCap>10GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<!-- Root Logger -->
<root level="INFO">
<appender-ref ref="STDOUT"/>
<appender-ref ref="FILE"/>
</root>
<!-- Connect-specific loggers -->
<logger name="org.apache.kafka.connect" level="INFO"/>
<logger name="io.confluent.connect" level="INFO"/>
<!-- Increase verbosity for debugging -->
<!-- <logger name="org.apache.kafka.connect" level="DEBUG"/> -->
</configuration>
Benefits:
- Daily Rotation: Logs are rotated daily (e.g.,
connect-2023-10-27.log.gz). - Automatic Compression: Old logs are gzipped to save disk space.
- Retention Policy: Keep only 30 days of logs.
- Size Cap: Prevent disk from filling up (10GB limit).
Apply Configuration:
export KAFKA_LOG4J_OPTS="-Dlogback.configurationFile=/path/to/logback.xml"
bin/connect-distributed.sh config/connect-distributed.properties
Monitoring Architecture Diagram
Here's a complete monitoring architecture for production Kafka Connect deployments:
┌─────────────────────────────────────────────────────────────┐
│ Kafka Connect Cluster │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Worker1 │ │ Worker2 │ │ Worker3 │ │
│ │ :8083 │ │ :8083 │ │ :8083 │ │
│ │ JMX:9999│ │ JMX:9999│ │ JMX:9999│ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
└───────┼─────────────┼─────────────┼────────────────────────┘
│ │ │
├─────────────┴─────────────┘
│ JMX Metrics (Port 7071 via JMX Exporter)
▼
┌─────────────────────────────────────────────────────────────┐
│ Prometheus │
│ - Scrapes metrics from JMX Exporter │
│ - Stores time-series data │
│ - Query: PromQL │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Grafana │
│ - Visualizes metrics via dashboards │
│ - Alerts on threshold breaches │
│ - Panels: Throughput, Lag, Failures, CPU/Memory │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Alertmanager │
│ - Receives alerts from Grafana/Prometheus │
│ - Routes to Slack, PagerDuty, Email │
│ - Examples: │
│ • Connector Failed → Slack #alerts │
│ • Consumer Lag > 50K → PagerDuty │
│ • Task Error Rate > 10/hr → Email │
└─────────────────────────────────────────────────────────────┘
┌────────────────────────────────┐
│ Centralized Logging (ELK) │
│ - Worker logs ingested │
│ - Query: Kibana │
│ - Alerts: ElastAlert │
└────────────────────────────────┘
Components:
- JMX Exporter: Exposes Connect JMX metrics as Prometheus metrics.
- Prometheus: Scrapes and stores metrics, evaluates alert rules.
- Grafana: Visualizes metrics, sends alerts.
- Alertmanager: Routes alerts to communication channels.
- ELK (Elasticsearch, Logstash, Kibana): Centralized log aggregation and search.
Troubleshooting Workflow for Connect Failures
Use this step-by-step workflow to troubleshoot Kafka Connect issues systematically.
Step 1: Check Connector Status
curl http://localhost:8083/connectors/jdbc-source/status | jq
Possible States:
- RUNNING: Connector and all tasks are healthy.
- FAILED: Connector or tasks have failed (check
tracefield for error). - PAUSED: Connector is paused (intentional).
Action: If FAILED, note the error message and proceed to Step 2.
Step 2: Review Worker Logs
Location: /var/log/kafka/connect.log (or configured path)
What to Look For:
ERRORmessages related to the failed connector/task.ConnectException,SQLException,TimeoutException.
Example Error:
[2023-10-27 10:05:30,789] ERROR Failed to poll records from database (com.example.JdbcSourceTask)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to database
at com.example.JdbcSourceTask.poll(JdbcSourceTask.java:123)
Action: Identify root cause (e.g., database unreachable, credentials invalid).
Step 3: Test External System Connectivity
For Databases:
psql -h localhost -U myuser -d mydb -c "SELECT 1;"
For REST APIs:
curl https://api.example.com/health
For S3:
aws s3 ls s3://my-bucket/
Action: If connectivity fails, fix network/firewall/credentials issues.
Step 4: Check JMX Metrics
Query relevant JMX metrics to identify performance bottlenecks:
# Task error count
curl http://localhost:7071/metrics | grep task_error_total_count
# Source record write rate
curl http://localhost:7071/metrics | grep source_record_write_rate
# Consumer lag (sink connectors)
curl http://localhost:7071/metrics | grep records_lag_max
Action: If throughput is low or lag is high, investigate performance (see Module 5.1).
Step 5: Verify Kafka Topic Health
List Topics:
kafka-topics --bootstrap-server localhost:9092 --list
Describe Topic (check partitions, replication):
kafka-topics --bootstrap-server localhost:9092 --describe --topic my-topic
Consume from Topic (verify data is flowing):
kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --from-beginning --max-messages 10
Action: If topic is missing or has wrong configuration, recreate or update topic.
Step 6: Restart Connector (Last Resort)
If all else fails, restart the connector:
curl -X POST http://localhost:8083/connectors/jdbc-source/restart
When to Restart:
- Transient network issues.
- External system was temporarily unavailable.
- Config changes require restart.
Warning: Restarting does not fix root causes (e.g., invalid credentials, schema mismatch).
Troubleshooting Checklist
Use this checklist for systematic debugging:
- Check connector status via REST API (
/connectors/{name}/status) - Review worker logs for ERROR messages
- Test external system connectivity (database, API, S3, etc.)
- Check JMX metrics (throughput, lag, error count)
- Verify Kafka topic exists and has data
- Check Schema Registry (if using Avro/JSON Schema)
- Verify worker resource usage (CPU, memory, disk)
- Review connector configuration (use
/config/validate) - Check network/firewall rules (if in containerized/cloud environment)
- Restart connector (if all else fails)
Key Takeaways
- Worker metrics: Connector count, task count, rebalance time, task failures.
- Connector metrics: Status (REST API), connector/task state (RUNNING, FAILED).
- Task metrics: Throughput (poll/write rate for source, read/send rate for sink), latency (batch time).
- Consumer lag: Monitor for sink connectors (if lag > 10K for > 10min, alert).
- Critical alerts: Connector/task failures, high lag, DLQ records, high error rate.
- Monitoring stack: JMX Exporter → Prometheus → Grafana → Alertmanager.
What's Next?
- Module 9.1 covers security best practices for Kafka Connect deployments.
- Module 5.1 discusses performance tuning (use metrics to identify bottlenecks).
- Module 6.1 explains error handling (use metrics to detect failures).
Further Reading
End of Module 8.1