10.1.4 – Schema Registry Architecture & Flow

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.1 – Schema Registry Foundations
Goal: Understand Schema Registry's internal architecture, storage, and message flow

Introduction

Schema Registry is not part of Kafka brokers — it's a separate service with its own architecture, storage mechanism, and deployment considerations.

Why This Module Matters: Understanding how Schema Registry works internally is critical for production deployment, troubleshooting, and performance tuning.

High-Level Architecture

┌───────────────────────────────────────────────────────────┐
│                   Kafka Cluster                           │
│   ┌─────────────────────────────────────────────┐        │
│   │   Topic: _schemas                            │        │
│   │   - Single partition                         │        │
│   │   - Compacted                                │        │
│   │   - Stores all schema definitions            │        │
│   └─────────────────────────────────────────────┘        │
└───────────────────────────────────────────────────────────┘
                         ↑
                         │ (reads/writes schema state)
                         │
┌───────────────────────────────────────────────────────────┐
│           Schema Registry Cluster (HA Setup)              │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│   │   Instance 1 │  │   Instance 2 │  │   Instance 3 │  │
│   │   (primary)  │  │   (standby)  │  │   (standby)  │  │
│   │              │  │              │  │              │  │
│   │   REST API   │  │   REST API   │  │   REST API   │  │
│   │   :8081      │  │   :8081      │  │   :8081      │  │
│   └──────────────┘  └──────────────┘  └──────────────┘  │
│          ↑                                                │
│          │ (in-memory cache of schemas)                  │
└───────────────────────────────────────────────────────────┘
         ↑                     ↑
         │                     │
┌─────────────┐         ┌─────────────┐
│  Producer   │         │  Consumer   │
│  (Avro      │         │  (Avro      │
│  Serializer)│         │  Deserializer)│
└─────────────┘         └─────────────┘

Core Components

1. Schema Registry Server

Technology: JVM-based application (Java)
Port: 8081 (default)
Deployment: Standalone service (not embedded in brokers)

Key Characteristics:

Stateless (all state in Kafka topic _schemas)
Can run multiple instances for high availability
Uses primary-standby model (single-primary writes)
Caches schemas in memory for fast reads

2. Internal Kafka Topic: `_schemas`

Purpose: Persistent storage for all schemas and metadata.

Configuration:

Topic Name:         _schemas (customizable via kafkastore.topic)
Partitions:         1 (single partition only!)
Replication Factor: 3 (recommended)
Cleanup Policy:     compact
Retention:          infinite

Why Single Partition?

Ensures total ordering of schema registrations
Simplifies primary election
Avoids consistency issues across partitions

What's Stored:

Key: "users-value:1"
Value: {
  "schema": "{\"type\":\"record\",...}",
  "schemaType": "AVRO",
  "references": [],
  "metadata": {...}
}

Detailed Configuration Properties:

# schema-registry.properties
kafkastore.topic=_schemas  # Default value (customizable)
kafkastore.topic.replication.factor=3  # Should match your Kafka topology

Storage Format:

Key: Schema subject + version (e.g., "users-value:1")
Value: Schema definition (AVRO/JSON/PROTOBUF) + metadata (ID, compatibility settings)
Compacted: Retains only the latest version of each schema while preserving history
Replicated: Across brokers for fault tolerance

Immutability Principle:

Schemas are never modified once written
Only new versions are added
This ensures consumers can always deserialize old messages

Backup Strategy:

# Mirror _schemas topic to another cluster for disaster recovery
kafka-mirror-maker --consumer.config source.properties \
  --producer.config target.properties \
  --whitelist="_schemas"

Monitoring Command:

# Check topic health
kafka-topics --bootstrap-server localhost:9092 --describe --topic _schemas

Troubleshooting:

If Schema Registry fails to start, verify:

Topic exists and is accessible

Configured replication factor matches available brokers

ACLs allow Schema Registry full access to _schemas topic

3. Primary-Standby Mode

Write Path (Single-Primary):

Producer → Registers schema → Primary Schema Registry
                                      ↓
                          Writes to _schemas topic
                                      ↓
                          Standby instances consume update

Read Path (All Instances):

Consumer → Retrieves schema → Any Schema Registry instance
                                      ↓
                              Returns from local cache

Leader Election:

Schema Registry uses ZooKeeper/Kafka for leader election
Only primary instance accepts write requests (POST)
Standby instances reject writes with HTTP 422

4. Secondary Storage & Cache Layers

In-Memory Cache (Primary Layer):

Purpose: Avoid Kafka reads on every serialization/deserialization.

Cache Structure:

Map<Integer, Schema> schemaIdCache;      // schema_id → schema
Map<String, Integer> schemaCache;        // schema_hash → schema_id
Map<String, List<Integer>> subjectCache; // subject → [v1_id, v2_id, ...]

Cache Invalidation: Synchronized via _schemas topic consumption.

Historical Note: Zookeeper Storage (Deprecated):

In older versions, Schema Registry used ZooKeeper for:

Compatibility settings (if not using Kafka-native storage)
Schema ID sequencing
Stored in /schema-registry path
Key properties: kafkastore.topic, master.eligibility, host.name

Modern Approach:

Zookeeper dependency is phased out in newer Kafka versions (KRaft mode)
Kafka-native metadata storage using __cluster_metadata or KRaft mode
The _schemas topic is the source of truth — all registries read from it to stay in sync

Key Notes:

All Schema Registry instances read from _schemas topic to stay in sync
Schemas are immutable — once written, never modified (only new versions added)
Backup strategy: Mirror _schemas topic to another cluster for disaster recovery

Message Registration Flow (Producer Side)

Step-by-Step: First Time Registration

1. Producer creates Avro record
   ↓
2. KafkaAvroSerializer.serialize() called
   ↓
3. Serializer computes schema hash
   ↓
4. Check local cache: Is schema_id known for this hash?
   ↓
   NO → Continue to step 5
   ↓
5. POST /subjects/<subject>/versions
   ↓
6. Schema Registry (Primary):
   a. Validates schema syntax
   b. Checks compatibility with previous versions
   c. If compatible:
      - Assigns new schema_id (e.g., 42)
      - Writes to _schemas topic
      - Updates local cache
      - Returns {"id": 42}
   d. If incompatible:
      - Returns HTTP 409 Conflict
   ↓
7. Serializer caches schema_id = 42
   ↓
8. Serializer prepends [0x00][0x00 0x00 0x00 0x2A] to message
                               ↑
                      schema_id = 42 (hex: 0x2A)
   ↓
9. Message sent to Kafka broker

Step-by-Step: Subsequent Messages (Cached)

1. Producer creates Avro record with same schema
   ↓
2. Serializer computes schema hash
   ↓
3. Check local cache: schema_id = 42 (HIT!)
   ↓
4. Prepend [0x00][0x00 0x00 0x00 0x2A] to message
   ↓
5. Send to Kafka (no Schema Registry call)

Performance: Cached path is 100x faster (no network call).

Schema Retrieval Flow (Consumer Side)

Step-by-Step: First Time Retrieval

1. Consumer receives message: [0x00][0x00 0x00 0x00 0x2A][data...]
   ↓
2. KafkaAvroDeserializer.deserialize() called
   ↓
3. Extract schema_id from bytes 1-4: schema_id = 42
   ↓
4. Check local cache: Is schema known for id=42?
   ↓
   NO → Continue to step 5
   ↓
5. GET /schemas/ids/42
   ↓
6. Schema Registry (Any Instance):
   a. Check local cache: schema_id=42
   b. If cached: Return schema from memory
   c. If not cached: Read from _schemas topic
   d. Return schema JSON
   ↓
7. Deserializer caches schema for id=42
   ↓
8. Deserializer decodes binary data using schema
   ↓
9. Return GenericRecord/SpecificRecord to consumer

Step-by-Step: Subsequent Messages (Cached)

1. Consumer receives message with schema_id = 42
   ↓
2. Check local cache: schema found (HIT!)
   ↓
3. Decode binary data using cached schema
   ↓
4. Return record to consumer (no Schema Registry call)

REST API Endpoints

Schema Registration

POST /subjects/{subject}/versions

curl -X POST http://localhost:8081/subjects/users-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[...]}"
  }'

Response:

{"id": 42}

Schema Retrieval by ID

GET /schemas/ids/{id}

Retrieve schema by global schema ID.

curl http://localhost:8081/schemas/ids/42

Response:

{
  "schema": "{\"type\":\"record\",\"name\":\"User\",...}"
}

List All Subjects

GET /subjects

curl http://localhost:8081/subjects

Response:

["users-value", "orders-value", "payments-key"]

List Versions for Subject

GET /subjects/{subject}/versions

curl http://localhost:8081/subjects/users-value/versions

Response:

[1, 2, 3]

Get Specific Version

GET /subjects/{subject}/versions/{version}

curl http://localhost:8081/subjects/users-value/versions/2

Response:

{
  "subject": "users-value",
  "version": 2,
  "id": 42,
  "schema": "{\"type\":\"record\",...}"
}

Check Compatibility

POST /compatibility/subjects/{subject}/versions/{version}

Test if new schema is compatible with specified version.

curl -X POST http://localhost:8081/compatibility/subjects/users-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{
    "schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[...]}"
  }'

Response:

{"is_compatible": true}

High Availability Setup

Multi-Instance Deployment

Production Setup (3 instances):

┌───────────────────────────────────────────────────┐
│  Load Balancer (HAProxy / NGINX)                 │
│  http://schema-registry.company.com:8081         │
└───────────────────────────────────────────────────┘
         ↓                 ↓                ↓
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ Instance 1   │  │ Instance 2   │  │ Instance 3   │
│ (primary)    │  │ (standby)    │  │ (standby)    │
│ host1:8081   │  │ host2:8081   │  │ host3:8081   │
└──────────────┘  └──────────────┘  └──────────────┘

Behavior:

Reads: Load-balanced across all instances
Writes: Automatically routed to primary (standbys return 422)

Configuration for HA

schema-registry.properties:

# Kafka connection
kafkastore.bootstrap.servers=broker1:9092,broker2:9092,broker3:9092
kafkastore.topic=_schemas
kafkastore.topic.replication.factor=3

# Schema Registry cluster
schema.registry.group.id=schema-registry-cluster

# ZooKeeper (for leader election, deprecated in newer versions)
kafkastore.connection.url=zk1:2181,zk2:2181,zk3:2181

# Or use Kafka-based coordination (preferred)
leader.eligibility=true

Failover Behavior

Scenario: Primary instance crashes

1. Primary (host1) crashes
   ↓
2. ZooKeeper detects failure
   ↓
3. New leader election triggered
   ↓
4. Instance 2 becomes primary
   ↓
5. Writes now succeed on Instance 2
   ↓
6. Total downtime: ~2-5 seconds

Result: Automatic failover with minimal downtime.

Performance Characteristics

Caching Impact

Without Cache (hypothetical):

1 million messages/sec × 2 Schema Registry calls (serialize + deserialize)
= 2 million HTTP calls/sec ❌ Impossible!

With Cache:

First message: 1 HTTP call (cache miss)
Next 999,999 messages: 0 HTTP calls (cache hit)

Result: 99.9999% cache hit rate ✅

Typical Latency

Operation	Latency (P50)	Latency (P99)
Cache Hit (serialize/deserialize)	0.1 ms	0.5 ms
Cache Miss (first time)	10 ms	50 ms
Schema Registration	20 ms	100 ms

Recommendation: Warm up cache in producer/consumer initialization.

Storage Considerations

`_schemas` Topic Size

Example Calculation:

1,000 subjects (topics)
× 10 versions per subject
× 5 KB per schema
= 50 MB total

With replication factor 3:
= 150 MB total storage

Result: Schema Registry storage is negligible compared to actual Kafka data.

Retention Configuration

# _schemas topic (DO NOT DELETE!)
retention.ms=-1          # Infinite retention
cleanup.policy=compact   # Keep latest version per key
min.compaction.lag.ms=0  # Compact immediately

Why Infinite Retention?

Consumers need to deserialize old messages from long-retained topics
Deleting schemas breaks deserialization permanently

Security Integration

Authentication

Basic Authentication:

# Schema Registry server
authentication.method=BASIC
authentication.roles=admin,developer,readonly
authentication.realm=SchemaRegistry-Props

SASL/SCRAM:

kafkastore.security.protocol=SASL_SSL
kafkastore.sasl.mechanism=SCRAM-SHA-256
kafkastore.sasl.jaas.config=...

Authorization

Schema Registry ACLs:

# Allow user "alice" to read schemas
curl -X POST http://localhost:8081/acls \
  -u admin:password \
  -d '{
    "username": "alice",
    "permission": "READ",
    "resource": "Subject:users-*"
  }'

Auto-Registration of Schemas

Configuration: `auto.register.schemas`

The auto.register.schemas setting controls automatic schema registration:

When Enabled (Default):

Producers automatically register new schemas on first use
Convenient for development and prototyping
Schema ID assigned automatically

When Disabled:

Schemas must be explicitly registered before use (via REST API or CLI)
Prevents accidental schema registration
Better control in production environments

Configuration Example:

// Producer configuration
props.put("auto.register.schemas", "false");  // Disable auto-registration

Workflow Impact:

auto.register.schemas = true:
  Producer → New Schema → Auto-register → Get ID → Serialize → Send

auto.register.schemas = false:
  Producer → New Schema → Check registry → Error (schema not found)
  Developer → Manually register via API → Producer succeeds

Best Practice:

✅ Enable for development: Fast iteration, no manual steps
✅ Disable for production: Explicit control after schema stabilization
✅ Use CI/CD: Register schemas as part of deployment pipeline

Affects:

Schema ID assignment workflow
Compatibility checking behavior
Error handling in producers

Key Takeaways

Separate Service: Schema Registry is not part of Kafka brokers.
_schemas Topic: Single-partition, compacted Kafka topic stores all schemas.
Primary-Standby: Writes go to primary, reads can hit any instance.
In-Memory Cache: Both Schema Registry and clients cache schemas locally.
Registration Flow: Producer → Serialize → Check cache → POST to registry → Cache ID → Send to Kafka.
Retrieval Flow: Consumer → Receive message → Extract schema ID → GET from registry → Cache schema → Deserialize.
REST API: Standard HTTP endpoints for registration, retrieval, compatibility checking.
High Availability: Deploy 3+ instances with load balancer for failover.
Performance: 99.9999% cache hit rate makes Schema Registry calls negligible.
Storage: Minimal overhead (~MB per 1000 subjects).

What's Next?

Next Module: 10.2 – Avro & Strongly Typed Messaging

What We Covered:

✅ Schema Registry architecture
✅ Internal storage (_schemas topic)
✅ Registration and retrieval flows
✅ REST API endpoints
✅ High availability setup

What's Coming:

❌ Avro serialization deep dive
❌ SpecificRecord vs GenericRecord
❌ Code generation

References:

Compatibility checking → 10.3.2: Compatibility Types
Monitoring → 10.4.3: Monitoring Schema Registry
Troubleshooting → 10.4.4: Troubleshooting Schema Issues

10.1.4 – Schema Registry Architecture & Flow

Introduction

High-Level Architecture

Core Components

1. Schema Registry Server

2. Internal Kafka Topic: _schemas

3. Primary-Standby Mode

4. Secondary Storage & Cache Layers

Message Registration Flow (Producer Side)

Step-by-Step: First Time Registration

Step-by-Step: Subsequent Messages (Cached)

Schema Retrieval Flow (Consumer Side)

Step-by-Step: First Time Retrieval

Step-by-Step: Subsequent Messages (Cached)

REST API Endpoints

Schema Registration

Schema Retrieval by ID

List All Subjects

List Versions for Subject

Get Specific Version

Check Compatibility

High Availability Setup

Multi-Instance Deployment

Configuration for HA

Failover Behavior

Performance Characteristics

Caching Impact

Typical Latency

Storage Considerations

_schemas Topic Size

Retention Configuration

Security Integration

Authentication

Authorization

Auto-Registration of Schemas

Configuration: auto.register.schemas

Key Takeaways

What's Next?

2. Internal Kafka Topic: `_schemas`

`_schemas` Topic Size

Configuration: `auto.register.schemas`