10.2.1 – Avro Fundamentals

Module: Section 10 – Kafka Ecosystem & Tools
Section: 10.2 – Avro & Strongly Typed Messaging
Goal: Master Apache Avro schema definition, data types, and encoding basics

Introduction

Apache Avro is a binary serialization format designed for high-throughput data pipelines. It provides compact encoding, rich type support, and schema evolution capabilities.

Why This Module Matters: Avro is the default serialization format in the Kafka ecosystem (Confluent Platform, Kafka Streams, Kafka Connect).

What Is Avro?

Definition

Avro is a data serialization system with:

1. Binary encoding: Compact, fast serialization
2. Schema definition: JSON-based schema language
3. Schema evolution: Backward/forward compatibility
4. Code generation: Generate Java/Python/C# classes
5. Self-describing: Schema embedded in files (Avro container format)

Avro vs JSON Comparison

JSON Message:

{
  "userId": 123,
  "username": "alice",
  "email": "alice@example.com",
  "age": 30,
  "isActive": true
}

Size: 98 bytes

Avro Message:

[Schema ID: 5] [Binary: 123, "alice", "alice@example.com", 30, true]

Size: 5 bytes (header) + 35 bytes (data) = 40 bytes

Savings: 59% reduction ✅

Avro Schema Definition

Basic Structure

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.avro",
  "doc": "User profile schema",
  "fields": [
    {
      "name": "userId",
      "type": "long",
      "doc": "Unique user identifier"
    },
    {
      "name": "username",
      "type": "string"
    }
  ]
}

Required Fields

Field Definition

{
  "name": "email",
  "type": ["null", "string"],
  "default": null,
  "doc": "User email address (optional)"
}

Avro Data Types

Primitive Types

Complex Types

1. Record (Struct/Object)

{
  "type": "record",
  "name": "Address",
  "fields": [
    {"name": "street", "type": "string"},
    {"name": "city", "type": "string"},
    {"name": "zipCode", "type": "string"}
  ]
}

Usage: Group related fields together.

2. Enum (Enumeration)

{
  "type": "enum",
  "name": "OrderStatus",
  "symbols": ["PENDING", "CONFIRMED", "SHIPPED", "DELIVERED", "CANCELLED"]
}

Usage: Restrict field to predefined values.

3. Array (List)

{
  "name": "tags",
  "type": {
    "type": "array",
    "items": "string"
  }
}

Example Data: ["vip", "premium", "verified"]

4. Map (Dictionary)

{
  "name": "metadata",
  "type": {
    "type": "map",
    "values": "string"
  }
}

Example Data: {"source": "mobile", "campaign": "summer2025"}

5. Union (Multiple Possible Types)

{
  "name": "phoneNumber",
  "type": ["null", "string"],
  "default": null
}

Explanation:

• Field can be null OR string
• Order matters: ["null", "string"] makes field optional
• Default value must match first type (null)

6. Fixed (Fixed-Length Bytes)

{
  "type": "fixed",
  "name": "MD5Hash",
  "size": 16
}

Usage: UUIDs, hashes, binary identifiers.

Logical Types

Logical types add semantic meaning to primitive types:

Example: Timestamp Field

{
  "name": "createdAt",
  "type": "long",
  "logicalType": "timestamp-millis"
}

Java:

long epochMs = 1738407000000L;
Instant timestamp = Instant.ofEpochMilli(epochMs);

Comprehensive Logical Types Guide

What Are Logical Types?

Logical types annotate a base Avro primitive type to specify its intended meaning. They provide semantic meaning while maintaining binary compatibility.

All Supported Logical Types:

• Decimal
• UUID
• Date
• Time (millisecond + microsecond precision)
• Timestamp (millisecond + microsecond precision)
• Local timestamp (millisecond + microsecond precision)
• Duration

Date Example (Days Since Unix Epoch)

Based on int type (days since 1970-01-01):

{
  "name": "birthday",
  "type": {
    "type": "int",
    "logicalType": "date"
  },
  "doc": "Date of birth"
}

Example Value: 19358 (represents 2023-01-15)
Java Type: java.time.LocalDate

Time Examples (Milliseconds or Microseconds)

Time-millis (uses int, milliseconds since midnight):

{
  "name": "login_time",
  "type": {
    "type": "int",
    "logicalType": "time-millis"
  }
}

Time-micros (uses long, microseconds since midnight):

{
  "name": "precise_time",
  "type": {
    "type": "long",
    "logicalType": "time-micros"
  }
}

Example: 43200000 (time-millis) = 12:00:00.000 PM
Java Type: java.time.LocalTime

Timestamp Examples (Milliseconds or Microseconds)

Timestamp-millis (uses long, milliseconds since Unix epoch):

{
  "name": "created_at",
  "type": {
    "type": "long",
    "logicalType": "timestamp-millis"
  }
}

Timestamp-micros (uses long, microseconds since Unix epoch):

{
  "name": "precise_created_at",
  "type": {
    "type": "long",
    "logicalType": "timestamp-micros"
  }
}

Example: 1738407000000 (timestamp-millis) = 2025-02-01 12:30:00 UTC
Java Type: java.time.Instant

Decimal Example (Arbitrary Precision)

Uses bytes or fixed with scale (decimal digits) and precision (total digits):

{
  "name": "price",
  "type": {
    "type": "bytes",
    "logicalType": "decimal",
    "precision": 10,
    "scale": 2
  },
  "doc": "Product price in USD"
}

Example: Store 123.45 with precision=10 (total digits), scale=2 (decimals)
Java Type: java.math.BigDecimal

Configuration:

• precision: Total number of digits (e.g., 10)
• scale: Digits after decimal point (e.g., 2)

UUID Example (Universally Unique Identifier)

Represented as a string with uuid logical type:

{
  "name": "user_id",
  "type": {
    "type": "string",
    "logicalType": "uuid"
  }
}

Example: "550e8400-e29b-41d4-a716-446655440000"
Java Type: java.util.UUID

Duration Example (Time Duration)

Represented as fixed 12-byte array (months, days, milliseconds):

{
  "name": "rental_period",
  "type": {
    "type": "fixed",
    "name": "Duration",
    "size": 12,
    "logicalType": "duration"
  }
}

Java Type: java.time.Duration

Logical Types Quick Reference

Complete Schema Example

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.ecommerce.events",
  "doc": "E-commerce order event",
  "fields": [
    {
      "name": "orderId",
      "type": "string",
      "doc": "Unique order identifier"
    },
    {
      "name": "userId",
      "type": "long"
    },
    {
      "name": "items",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "LineItem",
          "fields": [
            {"name": "productId", "type": "string"},
            {"name": "quantity", "type": "int"},
            {"name": "pricePerUnit", "type": "double"}
          ]
        }
      }
    },
    {
      "name": "totalAmount",
      "type": "double"
    },
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "OrderStatus",
        "symbols": ["PENDING", "CONFIRMED", "SHIPPED", "DELIVERED"]
      }
    },
    {
      "name": "shippingAddress",
      "type": ["null", {
        "type": "record",
        "name": "Address",
        "fields": [
          {"name": "street", "type": "string"},
          {"name": "city", "type": "string"},
          {"name": "country", "type": "string"},
          {"name": "zipCode", "type": "string"}
        ]
      }],
      "default": null
    },
    {
      "name": "createdAt",
      "type": "long",
      "logicalType": "timestamp-millis"
    },
    {
      "name": "metadata",
      "type": {
        "type": "map",
        "values": "string"
      },
      "default": {}
    }
  ]
}

Avro Binary Encoding

How Avro Encodes Data

Key Principle: Schema is not embedded in message (unlike JSON).

Encoding Strategy:

1. No field names in binary data
2. No type tags (schema defines types)
3. Variable-length encoding for integers (ZigZag + Varint)
4. Length-prefixed strings and bytes

Encoding Example

Schema:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "active", "type": "boolean"}
  ]
}

Data:

{"id": 123, "name": "Alice", "active": true}

Binary Encoding:

[0xF6 0x01]          → id = 123 (Varint-encoded)
[0x0A]               → name length = 10
[0x41 0x6C ... 0x65] → "Alice" (UTF-8)
[0x01]               → active = true

Total Size: ~15 bytes (vs 45 bytes for JSON)

Varint Encoding

Why?: Efficiently encode small integers.

How It Works:

• Most significant bit (MSB) = continuation flag
• 7 bits per byte for data
• Small values use fewer bytes

Schema Evolution Rules

Backward Compatibility

Old consumers can read new data.

✅ Allowed Changes:

• Add field with default value
• Delete field (consumers ignore it)
• Promote type (e.g., int → long)

❌ Forbidden Changes:

• Remove field without default
• Rename field
• Change field type (non-promotable)

Forward Compatibility

New consumers can read old data.

✅ Allowed Changes:

• Add field (new consumers handle missing field)
• Delete field with default

❌ Forbidden Changes:

• Add required field (no default)
• Change field type

Full Compatibility

Both backward AND forward compatible.

✅ Allowed Changes:

• Add optional field (with default)
• Delete field with default

Avro vs Protobuf vs JSON

Key Takeaways

1. Avro is a binary serialization format optimized for Kafka.

2. Schema: Defined in JSON format with rich type system.

3. Primitive Types: int, long, float, double, boolean, string, bytes, null.

4. Complex Types: record, enum, array, map, union, fixed.

5. Logical Types: Semantic types like timestamp-millis, date, uuid, decimal.

6. Binary Encoding: No field names, variable-length integers, 50-70% smaller than JSON.

7. Schema Evolution: Supports backward, forward, and full compatibility.

8. Unions: Enable optional fields via ["null", "string"] syntax.

9. Default Values: Critical for backward compatibility (add fields safely).

10. Kafka Native: Best-in-class support in Confluent ecosystem.

What's Next?

Next Module: 10.2.2 – Avro Serialization with Kafka

What We Covered:

• ✅ Avro fundamentals
• ✅ Schema definition syntax
• ✅ Data types (primitive, complex, logical)
• ✅ Binary encoding
• ✅ Evolution rules

What's Coming:

• ❌ KafkaAvroSerializer/Deserializer
• ❌ Schema Registry integration
• ❌ Producer/consumer code examples

References:

• Schema Registry → 10.1: Schema Registry Foundations
• Schema evolution → 10.3: Schema Evolution & Compatibility
• Code generation → 10.2.3: SpecificRecord vs GenericRecord