KafkaIQ

10.3.4 – Real-World Evolution Scenarios

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.3 – Schema Evolution & Compatibility
Goal: Apply schema evolution principles to real-world production scenarios

Introduction

This module presents real-world schema evolution scenarios, complete with solutions, timelines, and lessons learned from production systems.

Why This Module Matters: Learning from real-world examples helps you avoid common pitfalls and make better schema evolution decisions.

Scenario 1: Adding User Email (Simple Evolution)

Context

System: User registration service
Initial Schema: User ID and username only
Requirement: Add optional email field for marketing campaigns
Timeline: 2 weeks

v1 Schema (Initial)

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.users",
  "fields": [
    {"name": "userId", "type": "long"},
    {"name": "username", "type": "string"}
  ]
}

v2 Schema (Add Email)

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.users",
  "fields": [
    {"name": "userId", "type": "long"},
    {"name": "username", "type": "string"},
    {
      "name": "email",
      "type": ["null", "string"],
      "default": null,
      "doc": "User email address. Added v2 (2025-02-01)"
    }
  ]
}

Migration Plan

Week 1:

  1. Test v2 schema compatibility locally
  2. Register v2 schema in dev/staging
  3. Deploy v2 consumers (can handle v1 and v2 data)
  4. Verify consumers work correctly

Week 2:

  1. Deploy v2 producers (start writing email field)
  2. Monitor for errors
  3. Validate email data in downstream systems

Result

✅ Zero downtime
✅ Backward compatible (old consumers ignore email)
✅ Forward compatible (new consumers handle missing email)

Scenario 2: Renaming Field (Breaking Change)

Context

System: E-commerce order processing
Initial Schema: Field named userId
Requirement: Rename to customerId for consistency
Timeline: 6 months (dual-write period)

Phase 1: Add New Field (Month 1)

v2 Schema:

{
  "fields": [
    {"name": "userId", "type": "long", "doc": "DEPRECATED: Use customerId"},
    {"name": "customerId", "type": "long"}
  ]
}

Producer Code:

Order order = Order.newBuilder()
    .setUserId(123L)        // Write both for compatibility
    .setCustomerId(123L)
    .build();

Phase 2: Migrate Consumers (Months 2-4)

Consumer Code (Old):

long id = order.getUserId();

Consumer Code (New):

// Prefer customerId, fallback to userId
long id = order.getCustomerId() != null ? 
    order.getCustomerId() : order.getUserId();

Track migration progress:

SELECT service_name, uses_customer_id 
FROM consumer_migration_tracker
WHERE migrated = false;

Phase 3: Make userId Optional (Month 5)

v3 Schema:

{
  "fields": [
    {
      "name": "userId",
      "type": ["null", "long"],
      "default": null,
      "doc": "DEPRECATED: Use customerId. Will be removed in v4."
    },
    {"name": "customerId", "type": "long"}
  ]
}

Producer Code:

Order order = Order.newBuilder()
    .setCustomerId(123L)  // Only write customerId now
    .build();

Phase 4: Remove userId (Month 6)

v4 Schema:

{
  "fields": [
    {"name": "customerId", "type": "long"}
  ]
}

Validation: Ensure zero consumers read userId before deploying v4.

Lessons Learned

  • Timeline: 6 months was necessary due to 20+ consuming services
  • Monitoring: Tracked migration progress via dashboard
  • Rollback: Kept dual-write for extra month as safety buffer
  • Communication: Weekly updates to consuming teams

Scenario 3: Type Change (Int → Long)

Context

System: Payment transaction processing
Initial Schema: transactionId as int (32-bit)
Problem: Hit 2 billion transaction limit
Requirement: Promote to long (64-bit)
Timeline: 1 week (type promotion is safe)

v1 Schema

{"name": "transactionId", "type": "int"}

Range: -2,147,483,648 to 2,147,483,647
Problem: Running out of IDs!

v2 Schema (Promote to Long)

{"name": "transactionId", "type": "long"}

Range: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807

Why It's Safe

Avro Type Promotion Rules:

  • ✅ int → long is allowed
  • ✅ v2 consumer reads v1 data: int auto-promoted to long
  • ✅ v1 consumer reads v2 data: Works if value < 2^31

Deployment:

  1. Register v2 schema (compatibility check passes)
  2. Deploy producers (write long values)
  3. Deploy consumers (read long values)

Caution

⚠️ Old consumers (v1) will fail if new producers write values > 2^31!

Solution: Coordinate deployment to upgrade all consumers within 1 week.

Result

✅ Seamless migration
✅ No data loss
✅ Future-proof (won't hit limit for decades)

Scenario 4: Complex Nested Schema Evolution

Context

System: Order management system
Initial Schema: Flat order structure
Requirement: Add shipping address (nested object)
Timeline: 3 weeks

v1 Schema (Flat)

{
  "name": "Order",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId", "type": "long"},
    {"name": "total", "type": "double"}
  ]
}

v2 Schema (Add Nested Address)

{
  "name": "Order",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId", "type": "long"},
    {"name": "total", "type": "double"},
    {
      "name": "shippingAddress",
      "type": ["null", {
        "type": "record",
        "name": "Address",
        "fields": [
          {"name": "street", "type": "string"},
          {"name": "city", "type": "string"},
          {"name": "zipCode", "type": "string"},
          {"name": "country", "type": "string"}
        ]
      }],
      "default": null
    }
  ]
}

Migration Strategy

Week 1: Register v2 schema, deploy consumers
Week 2: Deploy producers writing address data
Week 3: Validate address data in warehouse

Consumer Code:

Order order = consumer.poll().value();
Address address = order.getShippingAddress();

if (address != null) {
    String street = address.getStreet();
    String city = address.getCity();
    // Process address
} else {
    // Old order without address
    logger.info("Order {} has no shipping address", order.getOrderId());
}

Result

✅ Backward compatible (old orders work without address)
✅ Nested schema handled gracefully
✅ Type safety preserved

Scenario 5: Multi-Topic Migration

Context

System: Fraud detection pipeline
Initial: Single transactions topic
Requirement: Split into approved-transactions and flagged-transactions
Timeline: 2 months

Initial Architecture

Producers → transactions topic → Fraud Service → Database

Target Architecture

Producers → transactions topic → Fraud Service → approved-transactions topic
                                                → flagged-transactions topic

Phase 1: Create New Topics (Week 1)

kafka-topics.sh --create --topic approved-transactions \
  --partitions 20 --replication-factor 3

kafka-topics.sh --create --topic flagged-transactions \
  --partitions 10 --replication-factor 3

Phase 2: Dual-Write (Weeks 2-6)

Fraud Service Code:

if (transaction.isFraudulent()) {
    // Write to both old and new topics
    producer.send(new ProducerRecord<>("transactions", transaction));
    producer.send(new ProducerRecord<>("flagged-transactions", transaction));
} else {
    producer.send(new ProducerRecord<>("transactions", transaction));
    producer.send(new ProducerRecord<>("approved-transactions", transaction));
}

Phase 3: Migrate Consumers (Weeks 7-8)

Old Consumer:

consumer.subscribe(Collections.singletonList("transactions"));
// Processes all transactions

New Consumers:

// Consumer 1: Approved transactions only
consumer.subscribe(Collections.singletonList("approved-transactions"));

// Consumer 2: Flagged transactions only
consumer.subscribe(Collections.singletonList("flagged-transactions"));

Phase 4: Stop Writing to Old Topic (Week 9)

// Only write to new topics
if (transaction.isFraudulent()) {
    producer.send(new ProducerRecord<>("flagged-transactions", transaction));
} else {
    producer.send(new ProducerRecord<>("approved-transactions", transaction));
}

Phase 5: Decommission Old Topic (Week 12)

# After retention period (30 days)
kafka-topics.sh --delete --topic transactions

Result

✅ Zero data loss
✅ Independent scaling (approved vs flagged topics)
✅ Better observability (separate metrics per topic)

Key Takeaways

  1. Simple Evolution: Add optional fields with defaults (2-week timeline).

  2. Rename Field: Requires dual-write period (6-month timeline).

  3. Type Promotion: int → long is safe, deploy quickly (1-week timeline).

  4. Nested Schema: Use unions with null for optional nested records.

  5. Multi-Topic Migration: Dual-write, migrate consumers, decommission old topic (2-month timeline).

  6. Communication: Weekly updates to consuming teams critical for complex migrations.

  7. Monitoring: Track migration progress via dashboards and alerts.

  8. Rollback Plan: Keep dual-write period longer than needed as safety buffer.

  9. Validation: Test compatibility locally before production deployment.

  10. Documentation: Document all schema changes with dates and reasoning.

What's Next?

Next Module: 10.4 – Schema Registry Operations

What We Covered:

  • ✅ Real-world evolution scenarios
  • ✅ Migration timelines
  • ✅ Lessons learned
  • ✅ Complex migrations

What's Coming:

  • ❌ Subject naming strategies
  • ❌ Schema Registry REST API
  • ❌ Monitoring and troubleshooting

References:

  • Schema evolution → 10.3.1: Schema Evolution Concepts
  • Compatibility modes → 10.3.2: Compatibility Types
  • Versioning rules → 10.3.3: Versioning Rules and Migration Strategies