1.1 – Kafka Connect Fundamentals & Architecture

Overview

This module establishes the foundational mental model for Kafka Connect as an integration framework. By the end of this module, you will understand what Connect is, why it exists, and how its runtime architecture enables scalable, reliable data integration with Kafka.

What is Kafka Connect?

Definition

Kafka Connect is a distributed, scalable, and fault-tolerant framework for streaming data between Apache Kafka and external systems—databases, message queues, file systems, cloud services, and more.

Key Characteristics

• Integration Framework: Not a general-purpose stream processor, but a specialized tool for data movement in and out of Kafka.
• Declarative Configuration: Connectors are defined via JSON/properties files, not custom code (in most cases).
• Pluggable: Hundreds of community and commercial connectors exist for popular systems (PostgreSQL, MySQL, S3, Elasticsearch, etc.).
• Operational Simplicity: Connect handles offset management, error handling, and scaling automatically.

Why Kafka Connect Exists

The Problem with Custom Producers/Consumers

Before Connect, integrating external systems with Kafka required:

1. Writing Custom Code: Developers had to implement producers/consumers for every integration.
2. Reinventing the Wheel: Offset management, error handling, retries, and scaling logic were duplicated across projects.
3. Operational Overhead: Each custom integration was a new service to deploy, monitor, and maintain.
4. Lack of Standardization: No common patterns for configuration, monitoring, or failure handling.

How Connect Solves This

Kafka Connect provides:

• Reusable Connectors: Write (or use existing) connectors once, configure them for different use cases.
• Built-in Offset Management: Source connectors store offsets in Kafka; sink connectors use Kafka consumer offsets.
• Fault Tolerance: Distributed mode ensures tasks are reassigned if workers fail.
• Standardized REST API: Deploy, configure, and manage connectors via HTTP.
• Centralized Monitoring: Common metrics and health checks for all connectors.

Analogy: Kafka Connect is like a universal adapter for Kafka—instead of building custom cables (producers/consumers) for every device (external system), you plug in a pre-built adapter (connector).

Kafka Connect Runtime Architecture

High-Level Components

┌─────────────────────────────────────────────────────────┐
│                 Kafka Connect Cluster                   │
│                                                           │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐        │
│  │  Worker 1  │  │  Worker 2  │  │  Worker 3  │        │
│  │            │  │            │  │            │        │
│  │  Task 1.1  │  │  Task 1.2  │  │  Task 2.1  │        │
│  │  Task 2.2  │  │  Task 3.1  │  │            │        │
│  └────────────┘  └────────────┘  └────────────┘        │
│         │                 │                 │            │
└─────────┼─────────────────┼─────────────────┼──────────┘
          │                 │                 │
          ▼                 ▼                 ▼
    ┌─────────────────────────────────────────────┐
    │           Kafka Cluster (Topics)            │
    └─────────────────────────────────────────────┘
          │                                   ▲
          ▼                                   │
   External System                     External System
   (e.g., Database)                    (e.g., S3)

Core Concepts

1. Workers

• Definition: JVM processes that execute connector tasks.
• Responsibilities:
  • Host and run connectors and tasks.
  • Expose REST API for connector management.
  • Coordinate task assignment (in distributed mode).
  • Store and manage connector/task state.

Analogy: Workers are like servers in a cluster—each worker can run multiple tasks, and they coordinate to balance work.

2. Connectors

• Definition: Logical jobs that define what data to move and how to move it.
• Responsibilities:
  • Define configuration (e.g., database connection, table names).
  • Split work into tasks for parallel execution.
  • Monitor and manage task lifecycle.

Types:

• Source Connectors: Pull data from external systems into Kafka.
• Sink Connectors: Push data from Kafka into external systems.

Analogy: A connector is like a job description—it defines the work, but doesn't do the work itself. Tasks are the actual workers.

3. Tasks

• Definition: Units of parallelism that perform the actual data movement.
• Responsibilities:
  • Read/write data from/to external systems.
  • Produce/consume records to/from Kafka.
  • Report progress and offsets.

Key Point: A single connector can spawn multiple tasks to parallelize data movement (e.g., one task per database table partition).

Data Flow Through Kafka Connect

Source Connector Flow

External System
    │
    ├─ poll() ───► Source Task 1 ───► Producer ───► Kafka Topic (Partition 0)
    │
    └─ poll() ───► Source Task 2 ───► Producer ───► Kafka Topic (Partition 1)

Steps:

1. Poll External System: Each source task polls the external system for new data (e.g., database query, file read).
2. Transform to SourceRecord: Tasks convert external data into Connect's internal SourceRecord format.
3. Produce to Kafka: Tasks use an internal Kafka producer to write records to Kafka topics.
4. Commit Offsets: Source offsets (e.g., database row ID, file position) are stored in a Kafka topic (connect-offsets in distributed mode).

Sink Connector Flow

Kafka Topic (Partition 0) ───► Consumer ───► Sink Task 1 ───► put() ───► External System
                                                      │
Kafka Topic (Partition 1) ───► Consumer ───► Sink Task 2 ───┘

Steps:

1. Consume from Kafka: Each sink task consumes records from assigned Kafka partitions.
2. Transform to External Format: Tasks convert SinkRecord objects into the external system's format.
3. Write to External System: Tasks batch-write data to the external system (e.g., database INSERT, S3 upload).
4. Commit Consumer Offsets: Kafka consumer offsets are committed after successful writes, ensuring at-least-once delivery.

REST API Role

Purpose

The Connect REST API is the primary interface for managing connectors. It allows operators to:

• Deploy Connectors: Submit connector configurations.
• Monitor Status: Check connector and task health.
• Update Configuration: Modify running connectors without downtime.
• Pause/Resume: Control connector execution.
• Delete Connectors: Remove connectors and their tasks.

Example: Creating a Connector

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "jdbc-source-connector",
    "config": {
      "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
      "tasks.max": "3",
      "connection.url": "jdbc:postgresql://localhost:5432/mydb",
      "mode": "incrementing",
      "incrementing.column.name": "id",
      "topic.prefix": "postgres-"
    }
  }'

Key Endpoints

Note: The REST API is hosted by each worker, but in distributed mode, any worker can handle requests (state is shared via Kafka internal topics).

Workers, Tasks, and Connectors: A Concrete Example

Scenario: PostgreSQL to Kafka (Source)

Setup:

• 1 Source Connector: jdbc-source-connector
• 3 Tasks (configured via tasks.max=3)
• 3 Workers in distributed mode

Execution:

1. Connector Creation: Operator submits connector config via REST API to Worker 1.
2. Task Generation: The connector plugin determines how to split work (e.g., one task per database table or partition).
3. Task Assignment: The Connect cluster assigns tasks to workers:
   • Worker 1: Task 1 (reads from users table)
   • Worker 2: Task 2 (reads from orders table)
   • Worker 3: Task 3 (reads from products table)
4. Data Flow: Each task polls its table, produces records to Kafka, and commits source offsets.
5. Fault Tolerance: If Worker 2 fails, Tasks 2's work is reassigned to Worker 1 or 3.

Kafka Connect vs. Custom Producers/Consumers

When to Use Connect:

• Moving data from databases, files, or cloud services into Kafka.
• Standard integrations where a connector already exists.
• Teams want operational simplicity and less code to maintain.

When to Use Custom Code:

• Complex business logic that doesn't fit the connector model.
• Real-time event enrichment or stateful transformations (use Kafka Streams instead).
• Latency-sensitive applications requiring fine-grained control.

Deployment Modes (Brief Introduction)

Kafka Connect supports two deployment modes:

1. Standalone Mode: Single worker process, no fault tolerance. Used for development/testing.
2. Distributed Mode: Multi-worker cluster, fault-tolerant, scalable. Used in production.

Note: Deployment modes are covered in detail in Module 3.2 – Standalone vs Distributed Mode. Here, we only introduce the concept.

Built-in Connectors

Kafka Connect ships with several built-in connectors out-of-the-box. These are included in the Apache Kafka distribution and are primarily used for testing, demos, and simple file-based operations.

Available Built-in Connectors

Note: For production use cases (e.g., JDBC, S3, Elasticsearch), you'll typically use external plugins from Confluent Hub or other sources.

Popular External Connectors

While built-in connectors are useful for testing, most real-world integrations require external connectors. Below is a curated list of popular connectors with links to official documentation.

Database Connectors

• JDBC Source & Sink Connector
  Enables integration with any relational database supporting JDBC, allowing data to flow into and out of Kafka topics.

Debezium CDC Connectors (Change Data Capture)

• MySQL Source Connector
  Captures real-time changes from MySQL databases and streams them into Kafka.

• PostgreSQL Source Connector
  Monitors PostgreSQL databases for changes and streams them into Kafka.

• MongoDB Source Connector
  Tracks changes in MongoDB collections and streams them into Kafka.

Cloud & Storage Connectors

• Amazon S3 Sink Connector
  Streams data from Kafka topics into Amazon S3 buckets, supporting various data formats (JSON, Avro, Parquet).

• HDFS Sink Connector
  Writes data from Kafka topics to Hadoop Distributed File System (HDFS), facilitating big data storage solutions.

SFTP Connectors

• SFTP Source Connector
  Reads files from a remote SFTP server and pushes their contents into Kafka topics.

• SFTP Sink Connector
  Writes Kafka topic data as files onto an SFTP server.

Search & Analytics Connectors

• Elasticsearch Sink Connector
  Streams data from Kafka topics into Elasticsearch, enabling real-time search and analytics capabilities.

Messaging & Monitoring Connectors

• JMS Source Connector
  Integrates with JMS-compliant messaging systems, allowing messages to be consumed into Kafka.

• Datadog Metrics Sink Connector
  Exports metrics from Kafka topics to Datadog for monitoring and observability.

Pro Tip: Check Confluent Hub for the full catalog of 200+ community and commercial connectors.

Key Takeaways

1. Kafka Connect is an integration framework for moving data in/out of Kafka without writing custom code.
2. Workers host connectors and tasks; Connectors define jobs; Tasks do the actual work.
3. Source connectors pull data into Kafka; Sink connectors push data out.
4. REST API is the primary interface for deploying and managing connectors.
5. Connect automates offset management, scaling, error handling, and fault tolerance.
6. Use Connect for standard integrations; use custom code for complex logic.

Quick Start: Running Connect with Docker

For students who want to get hands-on experience immediately, here's a minimal Docker Compose setup to run Kafka Connect locally.

Docker Compose Example

version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  kafka-connect:
    image: confluentinc/cp-kafka-connect:7.5.0
    depends_on:
      - kafka
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_GROUP_ID: connect-cluster
      CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"

Start the Stack

docker-compose up -d

Verify Connect is Running

curl http://localhost:8083/

Expected Output:

{
  "version": "7.5.0",
  "commit": "...",
  "kafka_cluster_id": "..."
}

Create a Test Connector

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "file-source-test",
    "config": {
      "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
      "tasks.max": "1",
      "file": "/tmp/test.txt",
      "topic": "connect-test"
    }
  }'

Note: A complete hands-on Docker lab with PostgreSQL integration is available in the Appendix: Docker Deployment Guide.

What's Next?

• Module 2.1 explains Source vs. Sink connector semantics in detail.
• Module 3 covers configuration, deployment modes, and scaling.
• Module 7 teaches how to develop custom connectors.
• Appendix A provides a complete hands-on Docker lab with PostgreSQL.

Further Reading

• Kafka Connect Documentation
• Confluent Hub (connector repository)
• KIP-382: MirrorMaker 2.0 (Connect-based replication)

End of Module 1.1