KafkaIQ

8.1 – Monitoring & Metrics


Overview

This module is the authoritative guide to monitoring and metrics for Kafka Connect clusters. By the end, you will understand:

  • Key JMX metrics for workers, connectors, and tasks.
  • Connector health indicators and failure signals.
  • Consumer lag monitoring for sink connectors.
  • Operational dashboards and alerting strategies.

All Connect monitoring concepts live here. Other modules (e.g., performance tuning, error handling) will reference this when discussing observability, but will not re-explain these concepts.


Monitoring Architecture

What to Monitor

┌──────────────────────────────────────┐
│        Kafka Connect Cluster         │
│                                      │
│  Worker Metrics                      │
│    ├─ Connector count                │
│    ├─ Task count                     │
│    ├─ Task failures                  │
│    └─ Rebalance frequency            │
│                                      │
│  Connector Metrics                   │
│    ├─ Status (running, failed)       │
│    ├─ Task status                    │
│    └─ Error logs                     │
│                                      │
│  Task Metrics                        │
│    ├─ Throughput (records/sec)       │
│    ├─ Latency (batch time)           │
│    ├─ Offset lag (sink only)         │
│    └─ Error count                    │
│                                      │
└──────────────────────────────────────┘

Monitoring Stack

Industry Standard:

  • JMX Exporter: Export Connect JMX metrics to Prometheus.
  • Prometheus: Scrape and store metrics.
  • Grafana: Visualize metrics and create dashboards.
  • Alertmanager: Send alerts (Slack, PagerDuty, email).

Worker Metrics (JMX)

Key Metrics

1. Connector Count

Metric: kafka.connect:type=connect-worker-metrics,connector={connector-name}
Attribute: connector-count

Purpose: Total number of connectors running on this worker.

Alert: If connector count drops unexpectedly (connector deleted or failed).


2. Task Count

Metric: kafka.connect:type=connect-worker-metrics
Attribute: task-count

Purpose: Total number of tasks running on this worker.

Alert: If task count is 0 (all tasks failed or worker idle).


3. Rebalance Time

Metric: kafka.connect:type=connect-worker-rebalance-metrics
Attribute: rebalance-avg-time-ms, rebalance-max-time-ms

Purpose: Average and max time for worker rebalancing.

Alert: If rebalance time > 10 seconds (indicates cluster instability).


4. Task Failures

Metric: kafka.connect:type=connect-worker-metrics
Attribute: task-error-total-count

Purpose: Total number of task failures across all connectors.

Alert: If task failures > 0 (investigate connector logs).


Example: Query JMX Metrics via JConsole

Connect to Worker:

jconsole localhost:9999

Navigate to MBeans → kafka.connect → connect-worker-metrics

View Attributes: connector-count, task-count, task-error-total-count


Connector-Level Metrics

REST API: Connector Status

Endpoint: GET /connectors/{name}/status

Example:

curl http://localhost:8083/connectors/jdbc-source/status | jq

Response:

{
  "name": "jdbc-source",
  "connector": {
    "state": "RUNNING",
    "worker_id": "worker1:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "worker1:8083"
    },
    {
      "id": 1,
      "state": "FAILED",
      "worker_id": "worker2:8083",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Failed to connect to database..."
    }
  ],
  "type": "source"
}

Key Fields

Field Values Meaning
connector.state RUNNING, PAUSED, FAILED Connector status
tasks[].state RUNNING, FAILED Task status
tasks[].trace Stack trace Error details (if failed)

Health Checks

Prometheus Exporter Pattern:

import requests

def check_connector_health(connector_name):
    response = requests.get(f"http://localhost:8083/connectors/{connector_name}/status")
    status = response.json()
    
    if status["connector"]["state"] != "RUNNING":
        return {"status": "unhealthy", "reason": "Connector not running"}
    
    for task in status["tasks"]:
        if task["state"] != "RUNNING":
            return {"status": "unhealthy", "reason": f"Task {task['id']} failed"}
    
    return {"status": "healthy"}

Alert: If any connector or task state is not RUNNING, send alert.


Task-Level Metrics (JMX)

Source Connector Metrics

1. Poll Rate

Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: source-record-poll-rate

Purpose: Records polled per second from the external system.

Expected: > 100 records/sec (depends on workload).

Alert: If poll rate = 0 for > 5 minutes (connector stalled).


2. Write Rate

Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: source-record-write-rate

Purpose: Records written to Kafka per second.

Expected: Should match poll-rate (if no transformations drop records).

Alert: If write rate < poll rate (records being dropped).


3. Poll Batch Time

Metric: kafka.connect:type=source-task-metrics,connector={connector-name},task={task-id}
Attribute: poll-batch-avg-time-ms

Purpose: Average time to poll a batch from the external system.

Expected: < 100ms (depends on external system).

Alert: If > 1000ms (external system slow or overloaded).


Sink Connector Metrics

1. Read Rate

Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: sink-record-read-rate

Purpose: Records read from Kafka per second.

Expected: > 100 records/sec (depends on workload).

Alert: If read rate = 0 for > 5 minutes (consumer lag or connector stalled).


2. Send Rate

Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: sink-record-send-rate

Purpose: Records written to external system per second.

Expected: Should match read-rate (if no errors).

Alert: If send rate < read rate (external system slow or errors).


3. Put Batch Time

Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: put-batch-avg-time-ms

Purpose: Average time to write a batch to the external system.

Expected: < 500ms (depends on external system).

Alert: If > 5000ms (external system slow, network issues, or large batches).


4. Offset Commit Success Rate

Metric: kafka.connect:type=sink-task-metrics,connector={connector-name},task={task-id}
Attribute: offset-commit-success-percentage

Purpose: Percentage of successful offset commits.

Expected: 100%.

Alert: If < 99% (offset commit failures, investigate logs).


Consumer Lag Monitoring (Sink Connectors)

What is Consumer Lag?

Consumer lag is the difference between the latest offset in a Kafka partition and the last committed offset by the sink connector.

Formula: Lag = Latest Offset - Committed Offset

Monitoring Lag

Using Kafka CLI

kafka-consumer-groups --bootstrap-server localhost:9092 \
  --describe --group connect-my-sink-connector

Output:

GROUP                          TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
connect-my-sink-connector      orders      0          5000            5100            100
connect-my-sink-connector      orders      1          4500            4500            0

Interpretation:

  • Partition 0: Lag = 100 (sink is 100 records behind).
  • Partition 1: Lag = 0 (sink is caught up).

Using JMX Metrics

Metric: kafka.consumer:type=consumer-fetch-manager-metrics,client-id={client-id}
Attribute: records-lag-max

Purpose: Maximum lag across all partitions.

Alert: If lag > 10,000 records for > 10 minutes (sink is falling behind).


Failure Signals and Alerting

Critical Alerts

1. Connector State = FAILED

Query:

curl http://localhost:8083/connectors/{name}/status | jq '.connector.state'

Alert: If state = FAILED, send PagerDuty alert.

Action: Investigate logs, check external system connectivity, restart connector.


2. Task State = FAILED

Query:

curl http://localhost:8083/connectors/{name}/status | jq '.tasks[].state'

Alert: If any task state = FAILED, send Slack alert.

Action: Check task logs, identify root cause (network, schema mismatch, etc.).


3. High Consumer Lag (Sink)

Metric: records-lag-max (JMX)

Threshold: Lag > 10,000 records for > 10 minutes.

Alert: Send Slack alert.

Action: Increase tasks.max, optimize sink writes, scale external system.


4. DLQ Topic Has Records

Query:

kafka-run-class kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 \
  --topic connect-dlq

Alert: If DLQ topic offset > 0, send alert.

Action: Inspect DLQ records, identify root cause (schema mismatch, malformed data).


5. High Task Failure Rate

Metric: task-error-total-count (JMX)

Threshold: > 10 failures/hour.

Alert: Send email alert.

Action: Investigate connector logs, check error tolerance configs.


Operational Dashboards (Grafana)

Example: Connect Cluster Dashboard

Panels:

  1. Connector Count (Gauge)

    • Query: kafka_connect_connect_worker_metrics_connector_count
    • Threshold: > 0 (green), = 0 (red)
  2. Task Count (Gauge)

    • Query: kafka_connect_connect_worker_metrics_task_count
    • Threshold: > 0 (green), = 0 (red)
  3. Source Throughput (Graph)

    • Query: sum(rate(kafka_connect_source_task_metrics_source_record_write_rate[5m]))
    • Label: "Records/sec"
  4. Sink Throughput (Graph)

    • Query: sum(rate(kafka_connect_sink_task_metrics_sink_record_send_rate[5m]))
    • Label: "Records/sec"
  5. Consumer Lag (Graph)

    • Query: kafka_consumer_fetch_manager_metrics_records_lag_max
    • Threshold: > 10,000 (yellow), > 50,000 (red)
  6. Task Failures (Counter)

    • Query: sum(kafka_connect_connect_worker_metrics_task_error_total_count)
    • Threshold: > 0 (alert)
  7. DLQ Records (Counter)

    • Query: kafka_log_log_size{topic="connect-dlq"}
    • Threshold: > 0 (alert)

Logging Best Practices

Log Levels

Worker Config (log4j.properties):

log4j.logger.org.apache.kafka.connect=INFO
log4j.logger.io.confluent.connect=INFO

Increase for Debugging:

log4j.logger.org.apache.kafka.connect=DEBUG

Key Log Patterns

Connector Startup

[2023-10-27 10:00:00,123] INFO Starting connector jdbc-source (org.apache.kafka.connect.runtime.WorkerConnector)

Task Assignment

[2023-10-27 10:00:05,456] INFO Assigned task jdbc-source-0 to worker worker1:8083 (org.apache.kafka.connect.runtime.distributed.DistributedHerder)

Error

[2023-10-27 10:05:30,789] ERROR Failed to poll records from database (com.example.JdbcSourceTask)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to database
    at com.example.JdbcSourceTask.poll(JdbcSourceTask.java:123)

Offset Commit

[2023-10-27 10:10:00,000] INFO Committing offsets for task jdbc-source-0 (org.apache.kafka.connect.runtime.WorkerSourceTask)

Structured Logging for Connect

For production deployments, use structured logging (JSON format) to make logs machine-readable and easier to query in centralized logging systems (ELK, Splunk, CloudWatch, etc.).

Example: JSON Log Format

Log4j2 Configuration (log4j2.xml):

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <JsonLayout compact="true" eventEol="true">
                <KeyValuePair key="timestamp" value="$${date:yyyy-MM-dd'T'HH:mm:ss.SSSZ}"/>
                <KeyValuePair key="level" value="$${level}"/>
                <KeyValuePair key="logger" value="$${logger}"/>
                <KeyValuePair key="message" value="$${message}"/>
                <KeyValuePair key="thread" value="$${thread}"/>
            </JsonLayout>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>

Example JSON Log Output:

{
  "timestamp": "2023-10-27T10:00:00.123Z",
  "level": "INFO",
  "logger": "org.apache.kafka.connect.runtime.WorkerConnector",
  "message": "Starting connector jdbc-source",
  "thread": "connect-worker-1",
  "connector_name": "jdbc-source",
  "worker_id": "worker1:8083"
}

Benefits:

  • Queryable: Filter logs by connector_name, level, worker_id.
  • Centralized: Easy to ingest into ELK, Splunk, or CloudWatch.
  • Alerting: Trigger alerts based on error patterns (e.g., level=ERROR AND connector_name=jdbc-source).

JMX Configuration for Kafka Connect

To expose JMX metrics for monitoring, configure the JVM when starting Connect workers.

Step 1: Enable JMX on Worker Startup

Standalone Mode:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.rmi.server.hostname=localhost"

bin/connect-standalone.sh config/connect-standalone.properties config/jdbc-source.properties

Distributed Mode:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.rmi.server.hostname=<worker_hostname>"

bin/connect-distributed.sh config/connect-distributed.properties

Security Note: For production, enable JMX authentication and SSL/TLS:

-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.password.file=/etc/kafka/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/kafka/jmxremote.access \
-Dcom.sun.management.jmxremote.ssl=true

Step 2: Install JMX Exporter (Prometheus)

Download JMX Exporter:

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar

Create JMX Exporter Config (jmx-exporter-config.yml):

lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  - pattern: "kafka.connect<type=(.+), client-id=(.+)><>(.+):"
    name: kafka_connect_$1_$3
    labels:
      client_id: "$2"
  - pattern: "kafka.connect<type=(.+)><>(.+):"
    name: kafka_connect_$1_$2

Start Worker with JMX Exporter:

export KAFKA_OPTS="-javaagent:/path/to/jmx_prometheus_javaagent-0.19.0.jar=7071:/path/to/jmx-exporter-config.yml"
bin/connect-distributed.sh config/connect-distributed.properties

Verify Metrics:

curl http://localhost:7071/metrics | grep kafka_connect

Prometheus Scrape Config (prometheus.yml):

scrape_configs:
  - job_name: 'kafka-connect'
    static_configs:
      - targets: ['localhost:7071']

Log Rotation with Logback

For production deployments, use Logback for advanced log rotation and compression.

Logback Configuration (logback.xml):

<configuration>
    <!-- Console Appender -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Rolling File Appender -->
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/kafka/connect.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- Daily rollover -->
            <fileNamePattern>/var/log/kafka/connect-%d{yyyy-MM-dd}.log.gz</fileNamePattern>
            <!-- Keep 30 days of history -->
            <maxHistory>30</maxHistory>
            <!-- Max total log size: 10GB -->
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Root Logger -->
    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="FILE"/>
    </root>

    <!-- Connect-specific loggers -->
    <logger name="org.apache.kafka.connect" level="INFO"/>
    <logger name="io.confluent.connect" level="INFO"/>
    
    <!-- Increase verbosity for debugging -->
    <!-- <logger name="org.apache.kafka.connect" level="DEBUG"/> -->
</configuration>

Benefits:

  • Daily Rotation: Logs are rotated daily (e.g., connect-2023-10-27.log.gz).
  • Automatic Compression: Old logs are gzipped to save disk space.
  • Retention Policy: Keep only 30 days of logs.
  • Size Cap: Prevent disk from filling up (10GB limit).

Apply Configuration:

export KAFKA_LOG4J_OPTS="-Dlogback.configurationFile=/path/to/logback.xml"
bin/connect-distributed.sh config/connect-distributed.properties

Monitoring Architecture Diagram

Here's a complete monitoring architecture for production Kafka Connect deployments:

┌─────────────────────────────────────────────────────────────┐
│                Kafka Connect Cluster                        │
│                                                               │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐                   │
│  │ Worker1 │   │ Worker2 │   │ Worker3 │                   │
│  │ :8083   │   │ :8083   │   │ :8083   │                   │
│  │ JMX:9999│   │ JMX:9999│   │ JMX:9999│                   │
│  └────┬────┘   └────┬────┘   └────┬────┘                   │
└───────┼─────────────┼─────────────┼────────────────────────┘
        │             │             │
        ├─────────────┴─────────────┘
        │ JMX Metrics (Port 7071 via JMX Exporter)
        ▼
┌─────────────────────────────────────────────────────────────┐
│                      Prometheus                             │
│  - Scrapes metrics from JMX Exporter                        │
│  - Stores time-series data                                  │
│  - Query: PromQL                                            │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                      Grafana                                │
│  - Visualizes metrics via dashboards                        │
│  - Alerts on threshold breaches                             │
│  - Panels: Throughput, Lag, Failures, CPU/Memory            │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                  Alertmanager                               │
│  - Receives alerts from Grafana/Prometheus                  │
│  - Routes to Slack, PagerDuty, Email                        │
│  - Examples:                                                │
│    • Connector Failed → Slack #alerts                       │
│    • Consumer Lag > 50K → PagerDuty                         │
│    • Task Error Rate > 10/hr → Email                        │
└─────────────────────────────────────────────────────────────┘

            ┌────────────────────────────────┐
            │  Centralized Logging (ELK)     │
            │  - Worker logs ingested        │
            │  - Query: Kibana               │
            │  - Alerts: ElastAlert          │
            └────────────────────────────────┘

Components:

  1. JMX Exporter: Exposes Connect JMX metrics as Prometheus metrics.
  2. Prometheus: Scrapes and stores metrics, evaluates alert rules.
  3. Grafana: Visualizes metrics, sends alerts.
  4. Alertmanager: Routes alerts to communication channels.
  5. ELK (Elasticsearch, Logstash, Kibana): Centralized log aggregation and search.

Troubleshooting Workflow for Connect Failures

Use this step-by-step workflow to troubleshoot Kafka Connect issues systematically.

Step 1: Check Connector Status

curl http://localhost:8083/connectors/jdbc-source/status | jq

Possible States:

  • RUNNING: Connector and all tasks are healthy.
  • FAILED: Connector or tasks have failed (check trace field for error).
  • PAUSED: Connector is paused (intentional).

Action: If FAILED, note the error message and proceed to Step 2.


Step 2: Review Worker Logs

Location: /var/log/kafka/connect.log (or configured path)

What to Look For:

  • ERROR messages related to the failed connector/task.
  • ConnectException, SQLException, TimeoutException.

Example Error:

[2023-10-27 10:05:30,789] ERROR Failed to poll records from database (com.example.JdbcSourceTask)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to database
    at com.example.JdbcSourceTask.poll(JdbcSourceTask.java:123)

Action: Identify root cause (e.g., database unreachable, credentials invalid).


Step 3: Test External System Connectivity

For Databases:

psql -h localhost -U myuser -d mydb -c "SELECT 1;"

For REST APIs:

curl https://api.example.com/health

For S3:

aws s3 ls s3://my-bucket/

Action: If connectivity fails, fix network/firewall/credentials issues.


Step 4: Check JMX Metrics

Query relevant JMX metrics to identify performance bottlenecks:

# Task error count
curl http://localhost:7071/metrics | grep task_error_total_count

# Source record write rate
curl http://localhost:7071/metrics | grep source_record_write_rate

# Consumer lag (sink connectors)
curl http://localhost:7071/metrics | grep records_lag_max

Action: If throughput is low or lag is high, investigate performance (see Module 5.1).


Step 5: Verify Kafka Topic Health

List Topics:

kafka-topics --bootstrap-server localhost:9092 --list

Describe Topic (check partitions, replication):

kafka-topics --bootstrap-server localhost:9092 --describe --topic my-topic

Consume from Topic (verify data is flowing):

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --from-beginning --max-messages 10

Action: If topic is missing or has wrong configuration, recreate or update topic.


Step 6: Restart Connector (Last Resort)

If all else fails, restart the connector:

curl -X POST http://localhost:8083/connectors/jdbc-source/restart

When to Restart:

  • Transient network issues.
  • External system was temporarily unavailable.
  • Config changes require restart.

Warning: Restarting does not fix root causes (e.g., invalid credentials, schema mismatch).


Troubleshooting Checklist

Use this checklist for systematic debugging:

  • Check connector status via REST API (/connectors/{name}/status)
  • Review worker logs for ERROR messages
  • Test external system connectivity (database, API, S3, etc.)
  • Check JMX metrics (throughput, lag, error count)
  • Verify Kafka topic exists and has data
  • Check Schema Registry (if using Avro/JSON Schema)
  • Verify worker resource usage (CPU, memory, disk)
  • Review connector configuration (use /config/validate)
  • Check network/firewall rules (if in containerized/cloud environment)
  • Restart connector (if all else fails)

Key Takeaways

  1. Worker metrics: Connector count, task count, rebalance time, task failures.
  2. Connector metrics: Status (REST API), connector/task state (RUNNING, FAILED).
  3. Task metrics: Throughput (poll/write rate for source, read/send rate for sink), latency (batch time).
  4. Consumer lag: Monitor for sink connectors (if lag > 10K for > 10min, alert).
  5. Critical alerts: Connector/task failures, high lag, DLQ records, high error rate.
  6. Monitoring stack: JMX Exporter → Prometheus → Grafana → Alertmanager.

What's Next?

  • Module 9.1 covers security best practices for Kafka Connect deployments.
  • Module 5.1 discusses performance tuning (use metrics to identify bottlenecks).
  • Module 6.1 explains error handling (use metrics to detect failures).

Further Reading


End of Module 8.1