KafkaIQ

1.1 Introduction & Producer Mental Model

Module Overview

Goal: Build a correct mental model of Kafka Producers without diving into tuning, internals, or reliability guarantees.

What You'll Learn:

  • What a Kafka Producer is and its role in the ecosystem
  • The high-level flow: Producer → Topic → Partition → Broker
  • Record structure and components
  • The send lifecycle at a conceptual level
  • When (not how) partition selection happens

What's NOT Covered Here:

  • Configuration deep dives (covered in Module 2.1)
  • Reliability guarantees (covered in Module 5)
  • Internal mechanics like RecordAccumulator (covered in Module 7.1)

Prerequisites & Setup

Before you start, you'll need the Kafka client library in your project.

Maven

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.7.0</version>
</dependency>

Gradle

implementation 'org.apache.kafka:kafka-clients:3.7.0'

Note: Use the latest stable version compatible with your Kafka cluster version.

1. What is a Kafka Producer?

Definition

A Kafka Producer is a client application that:

  • Publishes records (messages) to Kafka topics
  • Decides which partition to send each record to
  • Handles the network communication with Kafka brokers
  • Reports success or failure of each send operation

Think of a producer as a specialized mail carrier that:

  • Takes your letters (records)
  • Determines which mailbox (partition) to deliver them to
  • Ensures delivery and reports back the result

Producer's Core Responsibilities

┌─────────────────────────────────────────────────┐
│           PRODUCER RESPONSIBILITIES             │
├─────────────────────────────────────────────────┤
│ 1. Serialize data into bytes                    │
│ 2. Determine target partition                   │
│ 3. Batch records for efficiency                 │
│ 4. Compress data (optional)                     │
│ 5. Send to appropriate broker                   │
│ 6. Handle responses and errors                  │
└─────────────────────────────────────────────────┘

2. The Producer → Broker Flow (Mental Model)

High-Level Architecture

┌──────────────────┐
│   APPLICATION    │
│                  │
│  producer.send() │
└────────┬─────────┘
         │
         ▼
┌─────────────────────────────────────────────────┐
│           KAFKA PRODUCER CLIENT                 │
│  ┌────────────┐  ┌────────────┐  ┌───────────┐ │
│  │ Serializer │→│ Partitioner│→│  Batcher  │ │
│  └────────────┘  └────────────┘  └───────────┘ │
└────────┬─────────────────────────────────────────┘
         │ Network
         ▼
┌─────────────────────────────────────────────────┐
│              KAFKA CLUSTER                      │
│                                                 │
│  Topic: "orders"                                │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐     │
│  │Partition0│  │Partition1│  │Partition2│     │
│  │ Broker 1 │  │ Broker 2 │  │ Broker 3 │     │
│  └──────────┘  └──────────┘  └──────────┘     │
└─────────────────────────────────────────────────┘

The Journey of a Record

  1. Application creates a record with key, value, and optional metadata
  2. Producer serializes the key and value into bytes
  3. Producer determines which partition to send to
  4. Producer batches the record with others going to the same partition
  5. Producer sends the batch to the broker leader for that partition
  6. Broker writes the record to the partition log
  7. Broker responds with success or error
  8. Producer invokes callback (if provided) with the result

3. Understanding Records

Record Structure

A Kafka ProducerRecord contains:

ProducerRecord<K, V> {
    String topic;              // Required: target topic
    Integer partition;         // Optional: specific partition
    Long timestamp;            // Optional: event timestamp
    K key;                     // Optional: record key
    V value;                   // Required: record value
    Iterable<Header> headers;  // Optional: metadata headers
}

Components Explained

Topic (Required)

  • The logical channel where records are published
  • Must exist before sending (or auto-creation must be enabled)
  • Example: "user-events", "orders", "logs"

Value (Required)

  • The actual message content/payload
  • Can be any type (serialized to bytes)
  • Examples: JSON object, Avro record, plain string, binary data

Key (Optional but Important)

  • Used for partition selection
  • Enables ordering guarantees
  • Common use cases:
    • userId for user events
    • orderId for order updates
    • deviceId for IoT data

Key Insight: Records with the same key always go to the same partition (given same partition count).

Partition (Optional)

  • Manually specify target partition
  • Overrides partitioner logic
  • Use sparingly — usually let the partitioner decide

Timestamp (Optional)

  • Event time (when the event occurred)
  • Defaults to producer's current time
  • Used for time-based processing and retention

Headers (Optional)

  • Key-value metadata pairs attached to each message
  • Important: Headers are not part of the key/value payload
  • Headers travel with the record and are visible to consumers
  • Headers do NOT affect partitioning or ordering

Common Use Cases:

  • Correlation IDs / distributed tracing
  • Schema version hints
  • Routing information or feature flags
  • Metadata (source system, timestamp, user ID)

Example:

ProducerRecord<String, String> record = 
    new ProducerRecord<>("orders", "order-123", "order-data");

// Add headers
record.headers().add("trace-id", "abc-123".getBytes());
record.headers().add("source-system", "payment-service".getBytes());
record.headers().add("schema-version", "v2".getBytes());

producer.send(record);

Key Points:

  • Headers are lightweight (don't abuse them)
  • Not indexed or searchable by Kafka
  • Consumer can read headers without deserializing the value
  • Useful for routing decisions in stream processing

4. Topics, Partitions, and Distribution

Topics: Logical Organization

A topic is a logical category or feed name:

  • Similar to a database table or message queue
  • Divided into partitions for scalability
  • Example: payment-transactions topic

Partitions: Physical Distribution

Each topic consists of one or more partitions:

Topic: "user-events" (3 partitions)

Partition 0: [msg1][msg5][msg9][msg13]...  ← Broker 1
Partition 1: [msg2][msg6][msg10][msg14]... ← Broker 2
Partition 2: [msg3][msg7][msg11][msg15]... ← Broker 3

Key Characteristics:

  • Each partition is an ordered, immutable sequence of records
  • Each record in a partition gets a unique offset (position)
  • Partitions enable parallelism (multiple consumers can read different partitions)
  • Partitions provide fault tolerance (replicated across brokers)

Why Multiple Partitions?

  1. Scalability: Distribute data across multiple brokers
  2. Throughput: Producers can write to multiple partitions in parallel
  3. Consumer parallelism: Each consumer in a group reads from different partitions
  4. Fault tolerance: Partition replicas on different brokers

5. The Send Lifecycle (High-Level)

Synchronous vs Asynchronous Sends

Fire-and-Forget (Async, No Callback)

producer.send(record); // Don't wait for response
  • Fastest, but no delivery guarantee confirmation
  • Use for non-critical data where some loss is acceptable

Asynchronous with Callback

producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.println("Sent to partition: " + metadata.partition());
    } else {
        System.err.println("Send failed: " + exception.getMessage());
    }
});
  • Non-blocking, efficient, with delivery confirmation
  • Recommended for most use cases

Synchronous Send

RecordMetadata metadata = producer.send(record).get(); // Blocks
  • Blocks until broker acknowledges
  • Slowest, but provides immediate success/failure
  • Use sparingly (e.g., critical single record)

6. Partition Selection (When, Not How)

When Does Partition Selection Happen?

The producer determines the target partition before sending, based on:

  1. Manual partition specified? → Use that partition
  2. Key provided? → Hash the key to determine partition
  3. No key? → Use round-robin or sticky partitioning
┌─────────────────────────────────────────┐
│      Partition Selection Decision       │
├─────────────────────────────────────────┤
│                                         │
│  Partition number provided?             │
│         ├─ YES → Use that partition    │
│         └─ NO ↓                        │
│                                         │
│  Key provided?                          │
│         ├─ YES → Hash(key) % numPartitions│
│         └─ NO → Round-robin/Sticky     │
│                                         │
└─────────────────────────────────────────┘

Important: We'll cover the how and strategies in Module 4.1 (Partitioning Strategy).

7. Complete Working Example

Full Java Producer Example

Here's a complete, runnable example that demonstrates the core concepts:

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class OrderProducer {
    public static void main(String[] args) {
        // 1. Configuration
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // 2. Create producer
        Producer<String, String> producer = new KafkaProducer<>(props);

        // 3. Create and send record
        String topic = "orders";
        String key = "order-123";
        String value = "{\"userId\":42, \"item\":\"Bagel\", \"amount\":4.50}";

        ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
        
        // Send with callback
        producer.send(record, (metadata, exception) -> {
            if (exception == null) {
                System.out.println("Sent to topic: " + metadata.topic() +
                                   ", partition: " + metadata.partition() +
                                   ", offset: " + metadata.offset());
            } else {
                System.err.println("Error: " + exception.getMessage());
            }
        });

        // 4. Close producer
        producer.close();
    }
}

Output:

Sent to topic: orders, partition: 2, offset: 1543

What This Example Shows:

  1. Configuration: Minimal required settings (bootstrap servers, serializers)
  2. Record Structure: Topic, key, value
  3. Async Send: Non-blocking with callback
  4. Metadata: Partition and offset returned on success
  5. Error Handling: Exception handling in callback
  6. Cleanup: Proper producer shutdown

Key Components:

  • ProducerConfig.BOOTSTRAP_SERVERS_CONFIG: Where to connect
  • ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG: How to serialize keys
  • ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG: How to serialize values
  • ProducerRecord: The message container (topic, key, value)
  • Callback: Handles success/failure asynchronously

8. Producer States and Lifecycle

Producer Lifecycle

CREATE → CONFIGURE → INITIALIZE → ACTIVE → CLOSING → CLOSED
   │         │            │          │         │         │
   │         │            │          │         │         └─ Resources freed
   │         │            │          │         └─ Flush pending records
   │         │            │          └─ Can send records
   │         │            └─ Connected to brokers
   │         └─ Set properties (bootstrap, serializers)
   └─ new KafkaProducer<>(properties)

Basic Lifecycle Example

// 1. CREATE & CONFIGURE
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// 2. INITIALIZE
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// 3. ACTIVE - Send records
ProducerRecord<String, String> record = 
    new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);

// 4. CLOSING - Clean shutdown
producer.close();

9. Mental Model: Key Takeaways

The Producer's Job

APPLICATION DATA
      ↓
 [Serialize to bytes]
      ↓
 [Choose partition]
      ↓
 [Batch with other records]
      ↓
 [Send to broker]
      ↓
 [Wait for acknowledgment]
      ↓
 [Return success/failure]

Three Levels of Abstraction

  1. User Level: Call producer.send(record)
  2. Producer Level: Serialize, partition, batch, send
  3. Broker Level: Write to log, replicate, acknowledge

You control level 1.
The producer handles level 2.
The broker handles level 3.

Questions You Should Be Able to Answer

✅ What is a Kafka Producer?
✅ What are the components of a ProducerRecord?
✅ What's the difference between sync and async sends?
✅ Why do topics have multiple partitions?
✅ When does partition selection happen?
✅ What is an offset?

Questions We Haven't Answered Yet

❌ How do I tune producer performance? → Module 8
❌ How do I ensure exactly-once delivery? → Module 5
❌ What happens inside the producer? → Module 7.1
❌ How do I handle failures? → Module 6.1

10. Simple Working Example

Minimal Producer Code

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class SimpleProducer {
    
    public static void main(String[] args) {
        // Configuration
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", 
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", 
            "org.apache.kafka.common.serialization.StringSerializer");
        
        // Create producer
        KafkaProducer<String, String> producer = new KafkaProducer<>(props);
        
        try {
            // Create record
            ProducerRecord<String, String> record = 
                new ProducerRecord<>(
                    "my-topic",           // topic
                    "user-123",           // key
                    "User signed up"      // value
                );
            
            // Send asynchronously with callback
            producer.send(record, (metadata, exception) -> {
                if (exception == null) {
                    System.out.printf(
                        "Record sent to partition %d with offset %d%n",
                        metadata.partition(),
                        metadata.offset()
                    );
                } else {
                    System.err.println("Error sending record: " + exception);
                }
            });
            
        } finally {
            // Clean shutdown
            producer.close();
        }
    }
}

What This Does

  1. Configures connection to Kafka cluster
  2. Creates a producer instance
  3. Builds a record with topic, key, and value
  4. Sends the record asynchronously
  5. Logs the result via callback
  6. Closes producer cleanly

11. Common Beginner Misconceptions

❌ Misconception 1: "Producer sends directly to a partition file"

Reality: Producer sends to the broker that is the leader for that partition. The broker handles writing to disk.

❌ Misconception 2: "One producer per topic"

Reality: One producer can send to multiple topics. Topic is specified per-record, not per-producer.

❌ Misconception 3: "Records are sent one at a time"

Reality: Producer batches multiple records for efficiency (transparent to you).

❌ Misconception 4: "Key and value must be strings"

Reality: They can be any type — you choose the serializer. Common types: String, JSON, Avro, Protobuf.

❌ Misconception 5: "send() blocks until broker receives it"

Reality: send() is asynchronous by default — it returns immediately. Use .get() to block.

12. Frequently Misunderstood Producer Behaviors

These are subtle behaviors that even experienced developers sometimes miss:

🔹 send() is Asynchronous by Default

producer.send(record); // Returns immediately!
// Record NOT yet sent to broker

// To wait for confirmation:
producer.send(record).get(); // Now it blocks

Key Point: send() adds the record to a buffer and returns a Future<RecordMetadata>. The actual sending happens in a background thread.

🔹 Ordering is Per-Partition, Not Per-Topic

// These may NOT arrive in order at broker:
producer.send(new ProducerRecord<>("topic", null, "msg1")); // → Partition 0
producer.send(new ProducerRecord<>("topic", null, "msg2")); // → Partition 1
producer.send(new ProducerRecord<>("topic", null, "msg3")); // → Partition 0

// But these WILL be ordered:
producer.send(new ProducerRecord<>("topic", "key-A", "msg1")); // → Same partition
producer.send(new ProducerRecord<>("topic", "key-A", "msg2")); // → Same partition
producer.send(new ProducerRecord<>("topic", "key-A", "msg3")); // → Same partition

Kafka only guarantees ordering within a single partition.

🔹 Same Key → Same Partition (But Not Same Offset)

// Both go to same partition
producer.send(new ProducerRecord<>("topic", "user-123", "login"));
producer.send(new ProducerRecord<>("topic", "user-123", "logout"));

// But they get different offsets: 100, 101, 102, etc.

Same key means same partition, not same offset.

🔹 Callbacks Run on I/O Thread

producer.send(record, (metadata, exception) -> {
    // ⚠️ This runs in producer's I/O thread!
    // Keep this fast — don't block with heavy operations
    System.out.println("Offset: " + metadata.offset());
});

Avoid in callbacks:

  • Database calls
  • HTTP requests
  • Heavy computation
  • Blocking operations

OK in callbacks:

  • Logging
  • Updating metrics
  • Simple error handling

Summary

What We Covered

✅ Producer definition and responsibilities
✅ Record structure: topic, key, value, headers, timestamp
✅ Producer → Broker flow at a high level
✅ Topics and partitions: logical vs physical organization
✅ Send lifecycle: fire-and-forget, async, sync
✅ Partition selection: when it happens (not how)
✅ Producer lifecycle: create → configure → send → close

What's Next

  • Module 2.1: Producer APIs & Core Configuration
    Learn the essential configs and what they control

  • Module 3.1: Serialization & Data Modeling
    How to correctly serialize your data

  • Module 4.1: Partitioning Strategy & Message Ordering
    Deep dive into partition selection and ordering guarantees

Key Mental Model

┌─────────────────────────────────────────────────────┐
│  "A Producer is a smart client that decides         │
│   WHERE to send data and reports the result."       │
│                                                      │
│   WHERE = Which partition                           │
│   HOW = Configuration (next module)                 │
│   WHAT = Serialization (module 3.1)                 │
│   RELIABILITY = Guarantees (module 5)               │
└─────────────────────────────────────────────────────┘

After this module, you should be able to explain what a producer does and write a basic working producer. The next modules will teach you how to configure, optimize, and operate producers in production.

Next Module: 2.1 - Producer APIs & Core Configuration →