10.3.1 – Schema Evolution Concepts

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.3 – Schema Evolution & Compatibility
Goal: Understand the fundamentals of schema evolution and why it's critical for long-lived data systems

Introduction

Schema evolution is the ability to change a schema over time while maintaining compatibility with existing producers and consumers.

Why This Module Matters: In production systems, schemas evolve continuously. Understanding evolution rules prevents production outages caused by incompatible schema changes.

Why Schema Evolution Matters

The Problem: Static Schemas Don't Scale

Scenario: E-commerce platform with 20 microservices consuming orders topic.

Day 1: Schema looks like this:

{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId", "type": "long"},
    {"name": "total", "type": "double"}
  ]
}

Day 90: Product team needs to add currency field:

{
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "userId", "type": "long"},
    {"name": "total", "type": "double"},
    {"name": "currency", "type": "string"}  // NEW!
  ]
}

Without Schema Evolution:

Deploy new producer (v2) → Writes messages with "currency" field
                            ↓
Old consumers (v1) crash → Can't handle "currency" field
                            ↓
Production outage → Rollback required

With Schema Evolution:

1. Register schema v2 with default value for "currency"
2. Schema Registry validates compatibility
3. If compatible: Approve
4. Old consumers (v1) still work (ignore new field)
5. New consumers (v2) work with both old and new messages
6. Zero downtime! ✅

Schema Evolution Basics

What Can Change?

Change Type	Example	Backward Compatible?	Forward Compatible?
Add optional field	Add `email: string` with `default: null`	✅	✅
Delete field	Remove `middleName`	✅	❌
Rename field	`userId` → `customerId`	❌	❌
Change field type	`age: int` → `age: string`	❌	❌
Promote type	`int` → `long`	✅	✅

Evolution Timeline

v1: Schema with fields [A, B, C]
  ↓
v2: Add field D (with default)  ← Backward compatible
  ↓
v3: Add field E (with default)  ← Backward compatible
  ↓
v4: Delete field C              ← NOT forward compatible
  ↓
v5: Rename field B → B2         ← BREAKING CHANGE ❌

The Three Evolution Patterns

1. Backward Compatibility

Definition: New schema can read data written by old schema.

Use Case: Upgrade consumers first, then producers.

Example:

v1 Producer writes: {orderId: "123", total: 99.99}
                     ↓
v2 Consumer reads:  {orderId: "123", total: 99.99, currency: "USD" (default)}

Allowed Changes:

✅ Add field with default value
✅ Delete field

Deployment Order:

1. Deploy new consumers (v2) → Can handle v1 and v2 data
2. Deploy new producers (v2)  → Start writing v2 data

2. Forward Compatibility

Definition: Old schema can read data written by new schema.

Use Case: Upgrade producers first, then consumers.

Example:

v2 Producer writes: {orderId: "123", total: 99.99, currency: "USD"}
                     ↓
v1 Consumer reads:  {orderId: "123", total: 99.99} (ignores currency)

Allowed Changes:

✅ Add field (old consumer ignores it)
✅ Delete field with default value (old consumer uses default)

Deployment Order:

1. Deploy new producers (v2) → Start writing v2 data with new fields
2. Deploy new consumers (v2) → Can now use new fields

3. Full Compatibility

Definition: Both backward AND forward compatible.

Use Case: Deploy producers and consumers in any order.

Example:

v1 ↔ v2: Can read each other's data in any direction

Allowed Changes:

✅ Add optional field (with default)
✅ Delete field with default value

Deployment Order:

Can deploy in any order:
  - Producers first OR consumers first
  - No coordination needed ✅

Default Values: The Key to Evolution

Without Default Values

// v1
{
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "total", "type": "double"}
  ]
}

// v2 (add required field) ❌ BREAKS BACKWARD COMPATIBILITY
{
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "total", "type": "double"},
    {"name": "currency", "type": "string"}  // No default!
  ]
}

Problem:

v2 Consumer reads v1 data (no "currency" field)
  ↓
Deserialization error: Missing required field "currency"
  ↓
Consumer crashes ❌

With Default Values

// v2 (add optional field with default) ✅ BACKWARD COMPATIBLE
{
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "total", "type": "double"},
    {"name": "currency", "type": ["null", "string"], "default": null}
  ]
}

Result:

v2 Consumer reads v1 data (no "currency" field)
  ↓
Deserializer uses default value: currency = null
  ↓
Consumer works ✅

Union Types for Optional Fields

Avro Union Syntax

Making a field optional:

{
  "name": "email",
  "type": ["null", "string"],  // Union: null OR string
  "default": null
}

Order matters:

// ✅ Correct: default must match first type
{"type": ["null", "string"], "default": null}

// ❌ Wrong: default doesn't match first type
{"type": ["string", "null"], "default": null}  // Incompatible!

Multiple Union Types

{
  "name": "value",
  "type": ["null", "int", "string"],
  "default": null
}

Java Handling:

Object value = record.get("value");
if (value == null) {
    // Handle null
} else if (value instanceof Integer) {
    // Handle int
} else if (value instanceof String) {
    // Handle string
}

Type Promotion

Type promotion allows changing a field type to a "wider" type without breaking compatibility.

Allowed Promotions

From	To	Example
int	long, float, double	`age: int` → `age: long`
long	float, double	`timestamp: long` → `timestamp: double`
float	double	`price: float` → `price: double`
string	bytes	`data: string` → `data: bytes`
bytes	string	`data: bytes` → `data: string`

Example: int → long

v1 Schema:

{"name": "userId", "type": "int"}

v2 Schema (promote to long):

{"name": "userId", "type": "long"}

Compatibility:

✅ v2 Consumer reads v1 data: int value auto-promoted to long
✅ v1 Consumer reads v2 data: long value fits in int range (if < 2^31)

Caution: v1 consumer will fail if v2 writes long value > 2^31!

Breaking Changes

1. Rename Field

// v1
{"name": "userId", "type": "long"}

// v2 (rename) ❌ BREAKING
{"name": "customerId", "type": "long"}

Why?:

Old consumer expects "userId" → Not found in v2 data
New consumer expects "customerId" → Not found in v1 data

Workaround: Use aliases (Avro feature):

{
  "name": "customerId",
  "type": "long",
  "aliases": ["userId"]  // ✅ Backward compatible
}

2. Remove Field Without Default

// v1
{
  "fields": [
    {"name": "userId", "type": "long"},
    {"name": "email", "type": "string"}
  ]
}

// v2 (remove email) ❌ NOT FORWARD COMPATIBLE
{
  "fields": [
    {"name": "userId", "type": "long"}
  ]
}

Problem:

v1 Consumer (expects "email") reads v2 data (no "email")
  ↓
Deserialization error ❌

Solution: Keep field but mark as deprecated (add default):

{"name": "email", "type": ["null", "string"], "default": null, "doc": "DEPRECATED"}

3. Change Field Type (Non-Promotable)

// v1
{"name": "age", "type": "int"}

// v2 (change to string) ❌ BREAKING
{"name": "age", "type": "string"}

Why?: int and string are incompatible types.

Workaround: Add new field with new type:

{
  "fields": [
    {"name": "age", "type": "int", "doc": "DEPRECATED"},
    {"name": "ageString", "type": "string"}
  ]
}

Evolution Best Practices

1. Always Provide Default Values

// ❌ Bad: No default
{"name": "newField", "type": "string"}

// ✅ Good: Has default
{"name": "newField", "type": ["null", "string"], "default": null}

2. Never Rename Fields

// ❌ Bad: Rename
v1: {"name": "userId", ...}
v2: {"name": "customerId", ...}

// ✅ Good: Use aliases
v2: {"name": "customerId", "aliases": ["userId"], ...}

3. Never Remove Required Fields

// ❌ Bad: Remove required field
v1: {"name": "email", "type": "string"}
v2: (deleted)

// ✅ Good: Make optional, mark deprecated
v2: {"name": "email", "type": ["null", "string"], "default": null, "doc": "DEPRECATED"}

4. Test Schema Changes Before Deployment

# Test compatibility before registering
curl -X POST http://localhost:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[...]}"}'

# Response: {"is_compatible": true}

5. Use Semantic Versioning for Schemas

v1.0.0: Initial schema
v1.1.0: Add optional field (backward compatible)
v1.2.0: Add another optional field (backward compatible)
v2.0.0: Breaking change (removed field, renamed field)

Key Takeaways

Schema Evolution: Changing schemas over time while maintaining compatibility.
Three Patterns: Backward (new reads old), Forward (old reads new), Full (both).
Default Values: Critical for backward compatibility when adding fields.
Union Types: Enable optional fields via ["null", "type"] syntax.
Type Promotion: int→long, long→double, float→double are allowed.
Breaking Changes: Rename field, remove required field, change non-promotable type.
Deployment Order: Backward (consumers first), Forward (producers first), Full (any order).
Best Practices: Always provide defaults, never rename/remove required fields, test before deployment.
Aliases: Avro feature to handle field renames without breaking compatibility.
Schema Registry: Validates all changes against compatibility rules before accepting.

What's Next?

Next Module: 10.3.2 – Compatibility Types (Backward/Forward/Full)

What We Covered:

✅ Schema evolution fundamentals
✅ Backward, forward, full compatibility
✅ Default values and union types
✅ Type promotion
✅ Breaking changes

What's Coming:

❌ Schema Registry compatibility modes
❌ Configuring compatibility levels
❌ Versioning rules

References:

Avro fundamentals → 10.2.1: Avro Fundamentals
Compatibility modes → 10.3.2: Compatibility Types
Real-world scenarios → 10.3.4: Real-World Evolution Scenarios