KafkaIQ

4.3 Alerting & Operational Dashboards

Module Overview

Goal: Turn metrics into actions through effective alerting and visualization.

What You'll Learn:

  • Alerting principles and best practices
  • Signal vs noise in alerts
  • SLA-based alert design
  • Common Kafka alerts and thresholds
  • Dashboard structure and design
  • Alert fatigue prevention
  • Runbook integration

What's NOT Covered Here:

  • Specific monitoring tool configuration (Prometheus, Grafana, Datadog)
  • Metric definitions (covered in Module 4.1)
  • Performance tuning actions (covered in Module 5.1)

Key Insight: Good alerts wake you up for real problems. Bad alerts train you to ignore them. The goal is actionable alerting with minimal false positives.

1. Alerting Principles

The Purpose of Alerts

ALERTS SHOULD:
├─ Notify of actual problems (not potential future issues)
├─ Require human action
├─ Be actionable (clear next steps)
├─ Fire rarely (not constantly)
└─ Provide context for diagnosis

ALERTS SHOULD NOT:
├─ Fire for expected behavior
├─ Be information-only (use dashboards instead)
├─ Require deep analysis to understand
├─ Fire so often they're ignored
└─ Lack clear remediation steps

Alert Severity Levels

CRITICAL (P1) - Immediate Action Required:
├─ Service outage or imminent failure
├─ Data loss occurring
├─ SLA breach in progress
└─ Examples: Offline partitions, cluster unavailable

HIGH (P2) - Action Required Soon:
├─ Degraded performance
├─ Capacity warnings
├─ Approaching SLA breach
└─ Examples: High lag, disk > 85%, high latency

MEDIUM (P3) - Investigation Needed:
├─ Anomalies detected
├─ Trends moving wrong direction
├─ Minor issues
└─ Examples: Rebalances frequent, ISR shrinks

LOW (P4) - Informational:
├─ FYI notifications
├─ Trend alerts
├─ Non-urgent issues
└─ Examples: Forecast alerts, capacity planning

2. Alert Design Patterns

Threshold-Based Alerts

SIMPLE THRESHOLD:

if (metric > threshold) {
    fire_alert()
}

Example:
if (under_replicated_partitions > 0) {
    alert("Cluster has under-replicated partitions")
}

Pros:
└─ Simple to understand and implement

Cons:
├─ Doesn't account for trends
├─ Can be noisy if metric fluctuates
└─ No context on severity

Rate of Change Alerts

TREND-BASED:

if (metric_increase_rate > threshold) {
    fire_alert()
}

Example:
if (consumer_lag_growth > 1000 msgs/min for 5 min) {
    alert("Consumer lag growing rapidly")
}

Pros:
├─ Catches problems early
└─ Less noisy than absolute thresholds

Cons:
└─ Requires historical data

Composite Alerts

MULTIPLE CONDITIONS:

if (condition_A AND condition_B) {
    fire_alert()
}

Example:
if (disk_usage > 80% AND growth_rate > 5GB/hour) {
    alert("Disk will be full in < 24 hours")
}

Pros:
├─ More accurate (fewer false positives)
└─ Provides context

Cons:
└─ More complex to configure

3. Common Kafka Alerts

Critical Alerts (P1)

Offline Partitions

Alert Name: Offline Partitions Detected
Condition: OfflinePartitionsCount > 0
Duration: Immediate
Severity: CRITICAL

Why critical:
└─ Data unavailable to producers/consumers
└─ Potential data loss

Action:
1. Identify affected topics and partitions
2. Check broker health (all ISR replicas down?)
3. Restart failed brokers if needed
4. Consider unclean leader election (data loss risk)
5. Escalate if not resolved in 15 minutes

Runbook: /runbooks/offline-partitions.md

Cluster Unavailable

Alert Name: Kafka Cluster Unreachable
Condition: All brokers unresponsive OR controller missing
Duration: 1 minute
Severity: CRITICAL

Why critical:
└─ Complete service outage

Action:
1. Check if ZooKeeper/KRaft is running
2. Check network connectivity to brokers
3. Check if brokers are running (systemctl status kafka)
4. Review broker logs for crash cause
5. Escalate immediately

Runbook: /runbooks/cluster-unavailable.md

High Priority Alerts (P2)

Under-Replicated Partitions

Alert Name: Under-Replicated Partitions
Condition: UnderReplicatedPartitions > 0 for 10 minutes
Duration: 10 minutes (sustained)
Severity: HIGH

Why important:
└─ Reduced fault tolerance
└─ Risk of data loss if leader fails

Action:
1. Identify which brokers have under-replicated partitions
2. Check broker health (CPU, disk I/O, network)
3. Check replica lag metrics
4. If broker slow, investigate resource bottleneck
5. If broker down, restart or decommission

Runbook: /runbooks/under-replicated-partitions.md

High Consumer Lag

Alert Name: Consumer Lag Exceeds Threshold
Condition: consumer_lag > 10000 for consumer_group AND growing
Duration: 5 minutes
Severity: HIGH

Why important:
└─ Consumers falling behind
└─ May miss SLA for data freshness

Action:
1. Check consumer application health
2. Review consumer processing time
3. Check if partitions balanced across consumers
4. Consider adding more consumers
5. Review producer traffic (sudden spike?)

Runbook: /runbooks/consumer-lag.md

Disk Space Critical

Alert Name: Broker Disk Space Critical
Condition: disk_usage > 85%
Duration: 5 minutes
Severity: HIGH

Why important:
└─ Risk of disk full (broker crashes)

Action:
1. Check disk growth rate
2. Estimate time until full
3. Review topic retention settings
4. Consider emergency retention reduction
5. Add disk capacity if needed
6. Plan broker scaling

Runbook: /runbooks/disk-space-critical.md

Medium Priority Alerts (P3)

High Request Latency

Alert Name: Request Latency High
Condition: request_latency_p99 > 200ms for 10 minutes
Duration: 10 minutes
Severity: MEDIUM

Why important:
└─ Performance degradation

Action:
1. Check broker resource usage (CPU, disk, network)
2. Review request queue size
3. Check if specific topic is hotspot
4. Identify slow producers/consumers
5. Consider scaling if persistent

Runbook: /runbooks/high-latency.md

Frequent Rebalances

Alert Name: Frequent Consumer Rebalances
Condition: rebalance_rate > 6 per hour for consumer_group
Duration: 1 hour
Severity: MEDIUM

Why important:
└─ Indicates consumer instability
└─ Processing pauses during rebalances

Action:
1. Check consumer logs for rebalance causes
2. Review session.timeout.ms and max.poll.interval.ms
3. Check if consumers crashing/restarting
4. Review processing time vs max.poll.interval.ms
5. Tune consumer configs if needed

Runbook: /runbooks/frequent-rebalances.md

4. Signal vs Noise

Avoiding Alert Fatigue

COMMON CAUSES OF ALERT FATIGUE:

1. Alerts firing too frequently
   └─ Solution: Increase thresholds or duration

2. Alerts for expected behavior
   └─ Solution: Adjust thresholds for normal variance

3. Alerts with no action needed
   └─ Solution: Convert to dashboard metric

4. Duplicate alerts
   └─ Solution: Consolidate related alerts

5. Alerts during maintenance
   └─ Solution: Implement maintenance windows

Alert Tuning Process

ITERATIVE TUNING:

1. Initial Alert Setup
   └─ Conservative thresholds (fewer false negatives)

2. Monitor Alert Frequency
   ├─ Track true positives vs false positives
   └─ If false positive rate > 20%, adjust

3. Adjust Thresholds
   ├─ Increase threshold if too noisy
   ├─ Add duration requirement
   └─ Consider time-of-day adjustments

4. Validate
   └─ Did adjustment reduce noise without missing issues?

5. Repeat
   └─ Continuous improvement based on operational experience

5. SLA-Based Alerting

Defining SLAs

EXAMPLE KAFKA SLAs:

Availability:
└─ 99.9% uptime (< 8.76 hours downtime per year)

Latency:
├─ p50: < 10ms
├─ p95: < 50ms
└─ p99: < 100ms

Durability:
└─ Zero data loss (with acks=all, RF=3)

Throughput:
└─ Support 100 MB/s sustained write throughput

Consumer Lag:
└─ Lag < 1 minute (time-based, depends on traffic)

Alert on SLA Violations

SLA-BASED ALERTS:

Alert Name: Latency SLA Breach
Condition: request_latency_p99 > 100ms for 15 minutes
Severity: HIGH
Why: Breaching latency SLA

Alert Name: Availability SLA Risk
Condition: OfflinePartitionsCount > 0
Severity: CRITICAL
Why: Violating availability SLA

Alert Name: Consumer Lag SLA Breach
Condition: consumer_lag_seconds > 60
Severity: HIGH
Why: Breaching freshness SLA

6. Dashboard Design

Dashboard Hierarchy

LEVEL 1: EXECUTIVE DASHBOARD (High-Level Health)
├─ Cluster availability (uptime)
├─ Total throughput (messages/sec)
├─ Active alerts count
├─ Consumer lag summary
└─ Capacity utilization

LEVEL 2: OPERATIONAL DASHBOARD (Day-to-Day Monitoring)
├─ Broker health (CPU, disk, network)
├─ Topic throughput
├─ Consumer group lag
├─ Under-replicated partitions
├─ Request latency
└─ Recent alerts

LEVEL 3: DETAILED DASHBOARD (Deep Dive / Troubleshooting)
├─ Per-broker metrics
├─ Per-topic metrics
├─ Per-consumer-group metrics
├─ Replication metrics
├─ GC metrics
└─ Detailed timeseries

Essential Dashboard Panels

Cluster Health Panel

KPIs:
├─ Under-replicated partitions (target: 0)
├─ Offline partitions (target: 0)
├─ Active controller count (target: 1)
├─ Broker count (target: configured count)
└─ Status: Green/Yellow/Red

Visualization: Single stat + gauge
Update frequency: 30 seconds

Throughput Panel

Metrics:
├─ MessagesInPerSec (cluster-wide)
├─ BytesInPerSec (cluster-wide)
└─ BytesOutPerSec (cluster-wide)

Visualization: Line chart over time (last 24 hours)
Aggregation: Sum across all brokers
Update frequency: 30 seconds

Consumer Lag Panel

Metrics:
└─ Consumer lag per consumer group

Visualization:
├─ Table: consumer group, topic, partition, lag
├─ Heatmap: Color-coded by lag severity
└─ Time series: Lag trend over time

Thresholds:
├─ Green: lag < 1000
├─ Yellow: lag 1000-10000
└─ Red: lag > 10000

Update frequency: 1 minute

Resource Utilization Panel

Metrics:
├─ CPU usage per broker
├─ Disk usage per broker
├─ Network throughput per broker
└─ JVM heap usage per broker

Visualization: Multi-line chart (one line per broker)
Thresholds:
├─ Green: < 70%
├─ Yellow: 70-85%
└─ Red: > 85%

Update frequency: 1 minute

Request Latency Panel

Metrics:
├─ Request latency p50
├─ Request latency p95
├─ Request latency p99

Visualization: Line chart with percentile bands
SLA line: 100ms (p99)

Update frequency: 30 seconds

7. Dashboard Best Practices

Design Principles

✅ DO:
├─ Most important metrics at top
├─ Use color consistently (green/yellow/red)
├─ Show trends over time (not just current value)
├─ Include SLA thresholds as reference lines
├─ Group related metrics together
├─ Use appropriate time ranges (1h, 24h, 7d)
└─ Provide drill-down links

❌ DON'T:
├─ Overcrowd dashboards (< 12 panels per view)
├─ Use inconsistent color schemes
├─ Show metrics without context
├─ Forget to add units (MB/s, ms, %)
├─ Use 3D charts or pie charts for time series
└─ Ignore mobile/small screen layouts

8. Alert Routing and Escalation

Alert Routing Strategy

ROUTING BY SEVERITY:

CRITICAL (P1):
├─ PagerDuty: Immediate page
├─ Slack: #kafka-critical channel
├─ Email: oncall@company.com
└─ Escalate after 15 min if not acknowledged

HIGH (P2):
├─ Slack: #kafka-alerts channel
├─ Email: kafka-oncall@company.com
└─ Escalate to page if not resolved in 1 hour

MEDIUM (P3):
├─ Slack: #kafka-alerts channel
└─ Email: Daily digest

LOW (P4):
├─ Dashboard only
└─ Weekly report

Escalation Policy

TIER 1 (Initial Response): Kafka Ops Team
├─ Acknowledge alert within 5 minutes
├─ Begin investigation
└─ Escalate to Tier 2 if needed

TIER 2 (Expert Team): Senior Kafka Engineers
├─ Deep troubleshooting
├─ Complex remediation
└─ Escalate to Tier 3 if needed

TIER 3 (Engineering Leadership):
├─ Major outages
├─ Architectural decisions
└─ Vendor escalation if needed

9. Runbook Integration

Runbook Structure

RUNBOOK TEMPLATE:

# Alert: [Alert Name]

## Severity
[CRITICAL/HIGH/MEDIUM/LOW]

## Description
What this alert means

## Impact
What's affected and why it matters

## Investigation Steps
1. Check metric X
2. Review logs for Y
3. Validate Z

## Common Causes
- Cause 1 (frequency: X%)
- Cause 2 (frequency: Y%)
- Cause 3 (frequency: Z%)

## Remediation
### Quick Fix (temporary)
- Action 1
- Action 2

### Permanent Fix
- Action 1
- Action 2

## Verification
How to verify issue is resolved

## Prevention
How to prevent in future

## Related Alerts
- Alert A
- Alert B

## Contacts
- Primary: @oncall-team
- Escalation: @senior-engineers

10. Maintenance Windows

Silencing Alerts During Maintenance

MAINTENANCE WINDOW PROCESS:

1. Schedule Maintenance
   └─ Announce in advance (24-48 hours)

2. Enable Maintenance Mode
   ├─ Silence non-critical alerts
   ├─ Keep critical alerts (offline partitions)
   └─ Add dashboard annotation

3. Perform Maintenance
   └─ Follow runbook procedures

4. Disable Maintenance Mode
   └─ Re-enable all alerts

5. Validate
   └─ Check for missed issues during window

11. Alert Performance Metrics

Measuring Alert Quality

KEY METRICS:

1. Mean Time to Detect (MTTD)
   └─ Time from issue start to alert fired
   Target: < 2 minutes

2. Mean Time to Acknowledge (MTTA)
   └─ Time from alert fired to acknowledged
   Target: < 5 minutes

3. Mean Time to Resolve (MTTR)
   └─ Time from alert fired to resolved
   Target: Varies by severity (15 min to 4 hours)

4. False Positive Rate
   └─ Percentage of alerts that weren't real issues
   Target: < 10%

5. Alert Coverage
   └─ Percentage of real issues caught by alerts
   Target: > 95%

12. Sample Alert Configurations

Prometheus Alert Example

groups:
- name: kafka_critical
  interval: 30s
  rules:
  - alert: OfflinePartitions
    expr: kafka_controller_kafkacontroller_offline_partitions_count > 0
    for: 1m
    labels:
      severity: critical
      component: kafka
    annotations:
      summary: "Kafka has offline partitions"
      description: "Cluster {{ $labels.cluster }} has {{ $value }} offline partitions"
      runbook: "https://runbooks.company.com/kafka/offline-partitions"

  - alert: UnderReplicatedPartitions
    expr: kafka_server_replicamanager_under_replicated_partitions > 0
    for: 10m
    labels:
      severity: high
      component: kafka
    annotations:
      summary: "Under-replicated partitions detected"
      description: "Broker {{ $labels.instance }} has {{ $value }} under-replicated partitions"
      runbook: "https://runbooks.company.com/kafka/under-replicated-partitions"

13. Key Takeaways

You've learned:

✅ Good alerts are actionable, require human intervention, and fire rarely
✅ Severity levels (P1-P4) determine urgency and routing
✅ Alert patterns include threshold, rate-of-change, and composite
✅ Critical Kafka alerts: offline partitions, under-replication, disk full, high lag
✅ Signal vs noise is managed through tuning and duration requirements
✅ SLA-based alerts focus on business impact
✅ Dashboard hierarchy serves different audiences (exec, ops, troubleshooting)
✅ Runbooks provide step-by-step remediation guidance
✅ Alert quality is measured by MTTD, MTTA, MTTR, false positive rate

14. Alerting Checklist

ALERT CONFIGURATION:

☑ Clear name and description
☑ Appropriate severity level
☑ Threshold based on operational experience
☑ Duration requirement (avoid transient spikes)
☑ Routing to correct team/channel
☑ Runbook link included
☑ Tested and validated
☑ Reviewed quarterly

DASHBOARD DESIGN:

☑ Most important metrics visible without scrolling
☑ Color-coded health indicators
☑ Trend charts with appropriate time ranges
☑ SLA thresholds marked
☑ Drill-down capabilities
☑ Mobile-friendly layout
☑ Regular review and updates

Summary

Effective alerting turns metrics into action:

Good Alerting =
├─ Fires only for real problems
├─ Provides clear context
├─ Includes remediation steps
├─ Routes to right team
├─ Escalates appropriately
└─ Minimizes false positives

Good Dashboards =
├─ Show what matters most
├─ Provide context and trends
├─ Enable quick diagnosis
├─ Serve multiple audiences
└─ Updated and maintained

Together =
├─ Proactive problem detection
├─ Fast incident response
├─ Reduced downtime
├─ Better operational visibility
└─ Data-driven decisions

Remember: Alerts should wake you up for fires, not smoke. Dashboards should tell the story at a glance. Both require continuous tuning based on operational experience.

Next up: Module 5.1 — Performance Tuning & Capacity Planning — Optimize your Kafka cluster for maximum efficiency and plan for growth.