10.4.3 – Monitoring Schema Registry

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.4 – Schema Registry Operations
Goal: Monitor Schema Registry health, performance, and operational metrics

Introduction

Effective monitoring ensures Schema Registry availability, catches issues early, and provides visibility into schema operations.

Why This Module Matters: Schema Registry downtime = all producers/consumers fail. Proactive monitoring prevents outages.

Key Metrics to Monitor

1. JMX Metrics

Schema Registry exposes JMX metrics for monitoring.

Enable JMX:

# schema-registry.properties
jmx.port=9999

2. HTTP Health Check

Endpoint: GET /

curl http://localhost:8081/

Response (Healthy):

{}

Response (Unhealthy):

Connection refused / 500 Internal Server Error

Monitoring Setup:

# Prometheus health check
curl -f http://localhost:8081/ || exit 1

Critical Metrics

Schema Registration Rate

Metric: kafka.schema.registry:type=registered-count

What it measures: Number of schemas registered per second

Alert Threshold: Sudden spike may indicate:

Schema evolution activity
Misconfigured producer (re-registering same schema)
Schema Registry restart (cache warming)

Prometheus Query:

rate(schema_registry_registered_count_total[5m])

Schema Retrieval Latency

Metric: kafka.schema.registry:type=request-latency-ms

What it measures: Time to retrieve schema from registry

Alert Threshold: P99 > 100ms

Why it matters: High latency = consumer/producer delays

Troubleshooting:

Check _schemas topic replication lag
Verify Schema Registry heap size
Check disk I/O

Cache Hit Rate

Metric: kafka.schema.registry:type=cache-hit-rate

What it measures: Percentage of schema requests served from cache

Alert Threshold: < 95%

Why it matters: Low cache hit rate = excessive network calls

Troubleshooting:

Increase cache size: max.schemas.per.subject=10000
Check if producers are registering too many schemas

Primary Instance Status

Metric: kafka.schema.registry:type=master-slave-role

Values:

1: Primary (master)
0: Standby (slave)

Alert: No primary instance detected

Why it matters: Only primary accepts write requests

_schemas Topic Lag

Metric: kafka.schema.registry:type=consumer-lag

What it measures: Consumer lag on _schemas topic

Alert Threshold: > 1000 messages

Why it matters: High lag = stale schema cache on standby instances

Troubleshooting:

Check Kafka broker health
Verify _schemas topic replication
Check Schema Registry consumer group status

Operational Metrics

Schema Count

REST API:

curl http://localhost:8081/subjects | jq 'length'

Dashboard Metric: Total number of subjects

Version Count per Subject

REST API:

curl http://localhost:8081/subjects/users-value/versions | jq 'length'

Dashboard: Track schema evolution activity

Compatibility Check Failures

Metric: kafka.schema.registry:type=compatibility-check-fail-count

What it measures: Number of rejected schema registrations

Alert Threshold: > 10/hour

Why it matters: Indicates:

Breaking schema changes attempted
Compatibility mode misconfigured
Developer education needed

Monitoring Stack Setup

Prometheus Configuration

prometheus.yml:

scrape_configs:
  - job_name: 'schema-registry'
    static_configs:
      - targets: ['schema-registry-1:9999', 'schema-registry-2:9999']
    metrics_path: /metrics

Grafana Dashboard

Key Panels:

Schema Registry Health
- Uptime
- Primary/Standby status
- HTTP response time
Schema Activity
- Registrations per minute
- Compatibility check failures
- Top subjects by version count
Performance
- Request latency (P50, P95, P99)
- Cache hit rate
- _schemas topic lag
Capacity
- Heap usage
- Number of subjects
- Total schema versions

Alert Rules

Prometheus Alerts:

groups:
  - name: schema-registry
    rules:
      - alert: SchemaRegistryDown
        expr: up{job="schema-registry"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Schema Registry instance down"
          
      - alert: HighCompatibilityCheckFailures
        expr: rate(compatibility_check_fail_count[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High rate of compatibility check failures"
          
      - alert: LowCacheHitRate
        expr: cache_hit_rate < 0.95
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Schema Registry cache hit rate below 95%"
          
      - alert: NoP primaryInstance
        expr: count(master_slave_role == 1) == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "No primary Schema Registry instance"

Logging Best Practices

Enable DEBUG Logging

log4j.properties:

log4j.rootLogger=INFO, stdout

# Enable debug for schema operations
log4j.logger.io.confluent.kafka.schemaregistry=DEBUG

# Log compatibility check failures
log4j.logger.io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry=INFO

Key Log Messages

Schema Registration:

INFO  Registering new schema: subject=users-value, schema={...}
INFO  Schema registered with ID=42

Compatibility Check Failure:

ERROR Compatibility check failed: subject=users-value, error=Reader field email not found

Primary Election:

INFO  This instance is now the primary

Health Check Script

#!/bin/bash

SCHEMA_REGISTRY="http://localhost:8081"

# 1. Check HTTP connectivity
if ! curl -f -s "$SCHEMA_REGISTRY/" > /dev/null; then
  echo "❌ Schema Registry unreachable"
  exit 1
fi

# 2. Check primary status
MODE=$(curl -s "$SCHEMA_REGISTRY/mode" | jq -r '.mode')
if [ "$MODE" != "READWRITE" ]; then
  echo "⚠️  Schema Registry in $MODE mode"
fi

# 3. Check subject count
SUBJECT_COUNT=$(curl -s "$SCHEMA_REGISTRY/subjects" | jq 'length')
echo "✅ Subjects: $SUBJECT_COUNT"

# 4. Test schema registration (compatibility check only)
TEST_SCHEMA='{"schema":"{\"type\":\"record\",\"name\":\"Test\",\"fields\":[]}"}'
COMPAT_TEST=$(curl -s -X POST "$SCHEMA_REGISTRY/compatibility/subjects/test-subject/versions/latest" \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d "$TEST_SCHEMA")

if echo $COMPAT_TEST | jq -e '.is_compatible' > /dev/null; then
  echo "✅ Schema compatibility check working"
else
  echo "✅ Schema Registry API working"
fi

echo "✅ Schema Registry healthy"

Troubleshooting Commands

Check Schema Registry Logs

tail -f /var/log/schema-registry/schema-registry.log

Check _schemas Topic

kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic _schemas \
  --from-beginning

Verify Primary Instance

curl http://schema-registry-1:8081/mode
curl http://schema-registry-2:8081/mode
curl http://schema-registry-3:8081/mode

# Only one should return "READWRITE"

Check Consumer Group Lag

kafka-consumer-groups --bootstrap-server localhost:9092 \
  --group schema-registry \
  --describe

Key Takeaways

Health Check: GET / endpoint for HTTP health monitoring.
Critical Metrics: Registration rate, retrieval latency, cache hit rate, primary status.
Cache Hit Rate: Should be > 95% for optimal performance.
_schemas Topic Lag: Monitor consumer lag on internal topic.
Primary Instance: Only one instance should be primary (master).
Compatibility Failures: Track rejected schema registrations.
Prometheus: Expose JMX metrics via JMX exporter.
Grafana: Build dashboard for schema registry health/performance.
Alerts: Set up alerts for downtime, low cache hit rate, no primary.
Logging: Enable DEBUG logging for troubleshooting schema issues.

What's Next?

Next Module: 10.4.4 – Troubleshooting Schema Issues

What We Covered:

✅ Monitoring metrics and JMX
✅ Health checks and alerts
✅ Logging best practices
✅ Troubleshooting commands

What's Coming:

❌ Common schema issues
❌ Troubleshooting workflows
❌ Production incident response

References:

REST API → 10.4.2: Schema Registry REST API
Troubleshooting → 10.4.4: Troubleshooting Schema Issues
Operations → 10.4: Schema Registry Operations