10.3.2 – Compatibility Types (Backward/Forward/Full)

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.3 – Schema Evolution & Compatibility
Goal: Master Schema Registry compatibility modes and how to configure them for your use case

Introduction

Schema Registry enforces compatibility modes to prevent breaking schema changes. Understanding these modes is critical for managing schema evolution in production.

Why This Module Matters: Choosing the wrong compatibility mode can either block safe changes or allow dangerous ones. This module teaches you how to configure Schema Registry correctly.

The Seven Compatibility Modes

Schema Registry supports 7 compatibility modes:

Mode	Direction	Transitive?	Description
BACKWARD	New → Old	No	New schema can read old data (last version only)
BACKWARD_TRANSITIVE	New → Old	Yes	New schema can read ALL old versions
FORWARD	Old → New	No	Old schema can read new data (last version only)
FORWARD_TRANSITIVE	Old → New	Yes	Old schema can read ALL new versions
FULL	Both	No	Backward + Forward (last version only)
FULL_TRANSITIVE	Both	Yes	Backward + Forward (ALL versions)
NONE	-	-	No compatibility checks (dangerous!)

1. BACKWARD Compatibility (Default)

Definition

New schema (v2) can read data written by old schema (v1).

Validation: Schema Registry checks if v2 can deserialize messages written by v1.

Allowed Changes

✅ Add field with default value:

// v1
{"fields": [{"name": "userId", "type": "long"}]}

// v2 (add field)
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": ["null", "string"], "default": null}
]}

✅ Delete field:

// v1
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "tempField", "type": "string"}
]}

// v2 (remove tempField)
{"fields": [{"name": "userId", "type": "long"}]}

Forbidden Changes

❌ Add required field (no default):

// v2
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": "string"}  // No default!
]}

Why?: v2 consumer can't deserialize v1 data (missing email field).

❌ Rename field:

// v1: "userId"
// v2: "customerId"

Why?: v2 consumer expects customerId, but v1 data has userId.

Use Case

Upgrade consumers before producers:

1. Deploy v2 consumers → Can read v1 and v2 data
2. Deploy v2 producers → Start writing v2 data

Example: Add new analytics field to user events. Upgrade analytics service (consumer) first to handle new field.

Configuration

# Set BACKWARD compatibility for subject
curl -X PUT http://localhost:8081/config/users-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

2. BACKWARD_TRANSITIVE

Definition

New schema (v5) can read data written by ALL previous versions (v1, v2, v3, v4).

Validation: Schema Registry checks if v5 can deserialize messages from v1, v2, v3, and v4.

Example

v1: {userId}
v2: {userId, email}
v3: {userId, email, phone}
v4: {userId, email, phone, address}
v5: {userId, email, phone, address, country}

BACKWARD:            v5 must read v4 ✅
BACKWARD_TRANSITIVE: v5 must read v1, v2, v3, v4 ✅

Use Case

Long-lived topics with strict compatibility:

Consumers may process old messages (e.g., reprocessing last 90 days)
Must ensure v5 consumer can deserialize messages from any version

Example: Financial transactions with 7-year retention. New consumer must handle schema from 7 years ago.

Configuration

curl -X PUT http://localhost:8081/config/transactions-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

3. FORWARD Compatibility

Definition

Old schema (v1) can read data written by new schema (v2).

Validation: Schema Registry checks if v1 consumer can deserialize messages written by v2 producer.

Allowed Changes

✅ Add field (old consumer ignores it):

// v1
{"fields": [{"name": "userId", "type": "long"}]}

// v2 (add field)
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": ["null", "string"], "default": null}
]}

// v1 consumer reads v2 data → Ignores "email" field ✅

✅ Delete field with default:

// v1
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "tempField", "type": ["null", "string"], "default": null}
]}

// v2 (remove tempField)
{"fields": [{"name": "userId", "type": "long"}]}

// v1 consumer reads v2 data → Uses default for missing "tempField" ✅

Forbidden Changes

❌ Delete required field:

// v1
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": "string"}
]}

// v2 (remove email)
{"fields": [{"name": "userId", "type": "long"}]}

Why?: v1 consumer expects email, but v2 data doesn't have it.

Use Case

Upgrade producers before consumers:

1. Deploy v2 producers → Start writing v2 data with new fields
2. Wait (old consumers still work, ignore new fields)
3. Deploy v2 consumers → Can now use new fields

Example: Add experimental feature flag to events. Producers start writing it immediately, but old consumers don't crash.

Configuration

curl -X PUT http://localhost:8081/config/events-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FORWARD"}'

4. FORWARD_TRANSITIVE

Definition

All old schemas (v1, v2, v3) can read data written by new schema (v4).

Validation: Schema Registry checks if v1, v2, and v3 can deserialize v4 messages.

Example

v4 adds new field "country"

FORWARD:            v3 must read v4 ✅
FORWARD_TRANSITIVE: v1, v2, v3 must ALL read v4 ✅

Use Case

Long deployment windows with rolling upgrades:

Multiple consumer versions running simultaneously
Ensure all old consumers can handle new data

Example: 50 microservices consuming same topic, deployed over 1 week. Producers upgrade first.

Configuration

curl -X PUT http://localhost:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FORWARD_TRANSITIVE"}'

5. FULL Compatibility

Definition

Both BACKWARD AND FORWARD compatible (with last version only).

Validation:

New schema can read old data (BACKWARD)
Old schema can read new data (FORWARD)

Allowed Changes

✅ Only: Add optional field with default:

// v1
{"fields": [{"name": "userId", "type": "long"}]}

// v2 (add optional field)
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": ["null", "string"], "default": null}
]}

Why it works:

BACKWARD: v2 consumer reads v1 data → Uses default for email ✅
FORWARD: v1 consumer reads v2 data → Ignores email ✅

❌ Forbidden: Delete field:

// v1
{"fields": [
  {"name": "userId", "type": "long"},
  {"name": "email", "type": ["null", "string"], "default": null}
]}

// v2 (remove email)
{"fields": [{"name": "userId", "type": "long"}]}

Why?: FORWARD breaks (v1 consumer expects email but v2 data doesn't have it).

Use Case

Deploy producers and consumers in any order:

No coordination needed
Maximum flexibility

Example: Microservices team with independent deployment schedules.

Configuration (Default)

curl -X PUT http://localhost:8081/config/users-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL"}'

6. FULL_TRANSITIVE

Definition

Both BACKWARD_TRANSITIVE AND FORWARD_TRANSITIVE.

Validation:

New schema can read ALL old versions (BACKWARD_TRANSITIVE)
ALL old schemas can read new data (FORWARD_TRANSITIVE)

Example

v1 → v2 → v3 → v4 → v5

FULL_TRANSITIVE ensures:
- v5 can read v1, v2, v3, v4 ✅
- v1, v2, v3, v4 can read v5 ✅

Use Case

Strictest compatibility for critical systems:

Financial transactions
Regulatory compliance
Long-term data retention

Example: Healthcare records with 50-year retention. Any version must read any other version.

Configuration

curl -X PUT http://localhost:8081/config/medical-records-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

7. NONE (No Validation)

Definition

No compatibility checks. Any schema change is allowed.

Validation: None. Schema Registry accepts all schemas.

Use Case

❌ Never use in production!

✅ Only for:

Development/testing environments
Rapid prototyping
One-time data migrations

Configuration

curl -X PUT http://localhost:8081/config/dev-topic-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "NONE"}'

Configuring Compatibility Levels

Global Default

# Set global default compatibility
curl -X PUT http://localhost:8081/config \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

Per-Subject Override

# Override for specific subject
curl -X PUT http://localhost:8081/config/users-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

Precedence: Subject-level overrides global default.

Check Current Compatibility

# Global
curl http://localhost:8081/config

# Subject-specific
curl http://localhost:8081/config/users-value

Testing Compatibility

Before Registering Schema

# Test if new schema is compatible
curl -X POST http://localhost:8081/compatibility/subjects/users-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"userId\",\"type\":\"long\"},{\"name\":\"email\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
  }'

Response (Compatible):

{"is_compatible": true}

Response (Incompatible):

{
  "is_compatible": false,
  "messages": ["Reader field email not found in writer schema"]
}

Compatibility Mode Decision Matrix

Scenario	Recommended Mode	Reason
Default choice	BACKWARD	Works for most use cases
Long-lived topics (90+ days retention)	BACKWARD_TRANSITIVE	Old consumers may reprocess old data
Rolling deployments (producers first)	FORWARD	Producers upgrade before consumers
Independent deployments	FULL	No coordination needed
Critical data (financial, healthcare)	FULL_TRANSITIVE	Maximum safety
Dev/test only	NONE	Rapid iteration

Real-World Examples

Example 1: E-Commerce Orders (BACKWARD_TRANSITIVE)

Scenario: Orders retained for 1 year for analytics reprocessing.

Configuration:

curl -X PUT http://localhost:8081/config/orders-value \
  -d '{"compatibility": "BACKWARD_TRANSITIVE"}'

Why?: Analytics service may reprocess orders from 12 months ago. New schema must handle all old versions.

Example 2: User Events (FULL)

Scenario: 20 microservices consuming user events, deployed independently.

Configuration:

curl -X PUT http://localhost:8081/config/user-events-value \
  -d '{"compatibility": "FULL"}'

Why?: No deployment coordination. Producers and consumers upgrade independently.

Example 3: IoT Telemetry (FORWARD)

Scenario: 1 million IoT devices sending telemetry. Can't upgrade all devices instantly.

Configuration:

curl -X PUT http://localhost:8081/config/telemetry-value \
  -d '{"compatibility": "FORWARD"}'

Why?: New backend (consumer) deploys first. Old devices (producers) continue working.

Key Takeaways

7 Compatibility Modes: BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, NONE.
BACKWARD: New schema reads old data. Upgrade consumers first.
FORWARD: Old schema reads new data. Upgrade producers first.
FULL: Both directions. Deploy in any order.
TRANSITIVE: Checks all versions, not just last one.
Default: BACKWARD (safest for most use cases).
Configuration: Set globally or per-subject.
Testing: Use /compatibility/ endpoint before registering schema.
Production: Never use NONE in production.
Decision: Choose based on deployment pattern and data retention.

What's Next?

Next Module: 10.3.3 – Versioning Rules and Migration Strategies

What We Covered:

✅ 7 compatibility modes
✅ Configuration methods
✅ Testing compatibility
✅ Decision matrix

What's Coming:

❌ Versioning best practices
❌ Migration strategies
❌ Handling breaking changes

References:

Schema evolution → 10.3.1: Schema Evolution Concepts
Real-world scenarios → 10.3.4: Real-World Evolution Scenarios
Operations → 10.4: Schema Registry Operations