KafkaIQ

10.4.1 – Topic Governance and Standards

Module: M10 – Advanced Topics & Architecture
Section: 10.4 – Cluster Governance & Architecture Decisions
Goal: Establish topic naming, lifecycle, and governance standards

Introduction

As Kafka clusters grow, topic proliferation becomes a problem. This module covers governance strategies to manage topics at scale.

Topic Naming Standards

Naming Convention Template

Format: <domain>.<entity>.<version>.<environment>

Examples:

payments.order-completed.v1.prod
analytics.user-events.v2.dev
inventory.stock-update.v1.staging

Benefits:

  • Searchable (grep for "payments.*")
  • Versioned (schema evolution)
  • Environment isolation (prod vs dev)

Naming Rules

Required:

  • Lowercase only (avoid MyTopic)
  • Hyphen separators (avoid my_topic or myTopic)
  • No special characters (only a-z, 0-9, -, .)

Forbidden:

  • Generic names (data, events, topic1)
  • Underscores (conflicts with internal topics like __consumer_offsets)
  • Capital letters (case-sensitive issues)

Topic Lifecycle Management

Lifecycle Stages

1. Proposal

  • Team proposes topic with business justification
  • Includes: name, partition count, replication factor, retention

2. Approval

  • Platform team reviews for compliance with standards
  • Checks: naming convention, resource allocation, security requirements

3. Creation

  • Automated via CI/CD or self-service portal
  • Topic created with approved configs

4. Active

  • Monitored for usage, performance, compliance
  • Periodic reviews (quarterly)

5. Deprecation

  • Mark as deprecated (convention: DEPRECATED-<original-name>)
  • Consumers migrated to new topic
  • Grace period (e.g., 90 days)

6. Deletion

  • Topic deleted after grace period
  • Archived to S3 if needed for compliance

Self-Service Topic Creation

Portal/API Approach

Architecture:

Developer → Web Portal/API → Validation → Approval → Kafka Admin API → Topic Created

Validation Checks:

  • Naming convention compliance
  • Partition count within limits
  • Replication factor ≥ 2
  • Retention within bounds (not infinite)
  • Security: ACLs specified

Example API Request:

POST /api/topics
{
  "name": "payments.refund-requests.v1.prod",
  "partitions": 12,
  "replication_factor": 3,
  "retention_ms": 604800000,  // 7 days
  "owner_team": "payments-team",
  "justification": "New refund processing pipeline"
}

Auto-Approval Criteria:

  • ✅ Standard topic (< 50 partitions, < 7 days retention)
  • ✅ Naming compliant
  • ✅ Team has quota available

Manual Approval Required:

  • ❌ > 50 partitions
  • ❌ > 30 days retention
  • ❌ New high-throughput service

Topic Metadata Registry

Centralized Catalog

Purpose: Document all topics, owners, schemas, consumers.

Schema:

Topic: payments.order-completed.v1.prod
Owner: payments-team
Description: Order completion events for downstream services
Schema: Avro (Schema Registry ID: 12345)
Consumers: [billing-service, analytics-pipeline]
Created: 2024-01-15
Last Modified: 2024-03-20
Status: Active

Tools:

  • Confluent Control Center: Topic catalog with schema integration
  • Kafka UI (open-source): Topic browser with metadata
  • Custom portal: Internal topic registry

Topic Quota Management

Partition Quota

Problem: Unbounded topic creation → cluster overload.

Solution: Partition quota per team.

Example:

Team: payments-team
Quota: 500 partitions
Current Usage: 350 partitions
Available: 150 partitions

Enforcement: Self-service portal rejects requests exceeding quota.

Storage Quota

Problem: Infinite retention → disk exhaustion.

Solution: Storage quota per team.

Example:

Team: analytics-team
Quota: 10 TB
Current Usage: 7.5 TB
Available: 2.5 TB

Enforcement: Auto-scale retention downward if quota exceeded (with alerts).

Schema Governance

Schema Registry Integration

Requirement: All topics must register schema.

Benefits:

  • Schema evolution tracking
  • Compatibility enforcement (backward, forward, full)
  • Consumer protection (no breaking changes)

Example:

# Register schema for topic
kafka-avro-console-producer \
  --topic payments.order-completed.v1.prod \
  --property value.schema='{"type":"record","name":"Order","fields":[{"name":"order_id","type":"string"}]}'

Schema Evolution Policy

Allowed:

  • ✅ Add optional fields (backward compatible)
  • ✅ Remove optional fields (forward compatible)

Forbidden:

  • ❌ Remove required fields
  • ❌ Change field type

Enforcement: Schema Registry compatibility mode

compatibility = BACKWARD  # New consumers can read old messages

Topic Deprecation Policy

Deprecation Process

Step 1: Mark Deprecated

  • Add prefix: DEPRECATED-payments.old-orders.v1.prod
  • Update metadata registry

Step 2: Notify Consumers

  • Email all consumer teams (from registry)
  • Grace period: 90 days

Step 3: Monitor Usage

  • Track consumer lag (if zero, safe to delete)

Step 4: Delete

  • After grace period, delete topic
  • Archive to S3 if compliance required

Key Takeaways

  1. Naming Standards: <domain>.<entity>.<version>.<environment> (lowercase, hyphens).
  2. Lifecycle: Proposal → Approval → Creation → Active → Deprecation → Deletion.
  3. Self-Service: Portal/API with validation and auto-approval for standard topics.
  4. Topic Registry: Centralized catalog (owner, schema, consumers, status).
  5. Quotas: Partition and storage quotas per team (prevent overload).
  6. Schema Governance: Mandatory schema registration, compatibility enforcement.
  7. Deprecation: 90-day grace period, archive if needed.

What's Next?

Next Module: 10.4.2 – Partition Sizing and Rebalancing Strategy

References:

  • Topic configs → M2: Brokers & Topics
  • Schema Registry → M8: Kafka Connect