1.5 – Installation and Running Kafka

Module: M1 – Kafka Fundamentals
Goal: Hands-on setup of Kafka for local development

Introduction

This module covers installing and running Kafka for local development and testing.

Important: This is a local dev setup, not production deployment. Production deployment is covered in M6: Operations.

Prerequisites

Hardware Requirements

Kafka is a resource-intensive application. Proper hardware sizing prevents operational issues down the line.

Development / Testing

Key Recommendations:

• Use SSD: Kafka is I/O intensive. SSDs provide 10-100x better performance than HDDs.
• Page Cache: Kafka relies on OS page cache. More RAM = more cache = better read performance.

Production

Production Considerations:

Storage Sizing:

Disk Space = (message rate) × (avg message size) × (retention time) × (replication factor)

Example:
  1000 msg/sec × 1 KB × 7 days × 3 (RF=3) = 1.8 TB per topic

Disk I/O:

• Multiple disks across different controllers improve I/O performance significantly.
• RAID 10 (mirroring + striping) balances performance and redundancy.
• Avoid RAID 5/6 (parity overhead kills write performance).

Network:

• Dedicated, low-latency network with high bandwidth.
• Replication factor 3 triples network traffic (leader → 2 followers).
• At 100 MB/sec write throughput, replication consumes 200 MB/sec network bandwidth.

Required Ports

Kafka and its dependencies require specific network ports. Ensure these ports are open in your firewall configuration.

Firewall Configuration:

Development (single machine):

# No firewall rules needed (localhost only)

Production (multi-broker cluster):

# Allow Kafka client port from application servers
sudo ufw allow from 10.0.1.0/24 to any port 9092

# Allow inter-broker communication
sudo ufw allow from 10.0.2.0/24 to any port 9093

# Allow ZooKeeper from Kafka brokers only
sudo ufw allow from 10.0.2.0/24 to any port 2181

AWS Security Group Example:

Inbound Rules:
  - Port 9092: Source = Application security group
  - Port 9093: Source = Kafka broker security group (self)
  - Port 2181: Source = Kafka broker + ZK security group

Pre-Installation Checklist

Use this checklist before installing Kafka:

• Java 11+ installed and accessible (java -version works)
• Sufficient disk space available (10 GB+ for dev, 500 GB+ for production)
• Required ports are open (9092, 9093, 2181) in firewall
• Network connectivity verified between machines (for multi-node)
• User account created for running Kafka (recommended: non-root user kafka)
• NTP/time synchronization enabled (critical for distributed systems)
• Kafka binary distribution downloaded (from Apache Kafka website)

Create Kafka User (production best practice):

# Create dedicated user (non-root)
sudo useradd -r -s /bin/false kafka

# Create Kafka directories
sudo mkdir -p /opt/kafka /var/kafka/data /var/log/kafka
sudo chown -R kafka:kafka /opt/kafka /var/kafka /var/log/kafka

Verify Time Synchronization:

# Check NTP status
timedatectl status

# If not synchronized, enable NTP
sudo timedatectl set-ntp true

Why Time Sync Matters: Kafka uses timestamps for retention, log rolling, and replication. Clock skew between brokers causes inconsistencies.

System Requirements

Check Java Installation

Check Java version:

java -version

Expected output:

openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

If Java is not installed:

macOS (Homebrew):

brew install openjdk@17

Linux (Ubuntu):

sudo apt update
sudo apt install openjdk-17-jdk

Windows:

• Download JDK from Adoptium
• Install and add to PATH

Download Kafka

Option 1: Download from Apache Kafka

Step 1: Go to https://kafka.apache.org/downloads

Step 2: Download the latest binary (not source):

kafka_2.13-3.6.0.tgz  (Scala 2.13, Kafka 3.6.0)

Step 3: Extract the archive:

tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0

Option 2: Homebrew (macOS)

brew install kafka

Kafka files location:

/usr/local/opt/kafka/

Option 3: Docker (Recommended for Quick Start)

Create docker-compose.yml:

version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    hostname: kafka
    container_name: kafka
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

Start Kafka:

docker-compose up -d

Verify Kafka is running:

docker logs kafka

KRaft vs ZooKeeper Mode

Kafka can run in two modes:

Key Difference: KRaft removes dependency on ZooKeeper (simpler, faster).

Recommendation: Use KRaft mode for new setups.

KRaft Mode (Recommended)

Kafka 3.3+ supports KRaft mode (no ZooKeeper required).

Benefits:

• ✅ Simpler setup (no ZooKeeper)
• ✅ Faster metadata operations
• ✅ Better scalability (10,000+ partitions)
• ✅ Faster leader election (< 1 second)

ZooKeeper Mode (Legacy)

Kafka < 3.0 requires ZooKeeper.

Why ZooKeeper?: Stores cluster metadata (brokers, topics, partitions, ISR).

Note: ZooKeeper mode is deprecated in Kafka 3.0+ and will be removed in Kafka 4.0.

Running Kafka in KRaft Mode (Local, Single Broker)

Step 1: Generate Cluster ID

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
echo $KAFKA_CLUSTER_ID

Output:

MkU3OEVBNTcwNTJENDM2Qk

Step 2: Format Storage Directory

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Output:

Formatting /tmp/kraft-combined-logs with metadata.version 3.6-IV2.

What This Does: Initializes Kafka's metadata log (like initializing a database).

Step 3: Start Kafka Server

bin/kafka-server-start.sh config/kraft/server.properties

Expected output:

[2023-01-15 10:30:00,123] INFO Kafka Server started (kafka.server.KafkaServer)
[2023-01-15 10:30:00,456] INFO [BrokerToControllerChannelManager broker=1] Started (kafka.server.BrokerToControllerChannelManager)

Kafka is now running on localhost:9092.

Step 4: Verify Kafka is Running

Check logs:

tail -f logs/server.log

Create a test topic:

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092

List topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Expected output:

test

Success! Kafka is running in KRaft mode.

Running Kafka in ZooKeeper Mode (Legacy)

Step 1: Start ZooKeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Expected output:

[2023-01-15 10:30:00,123] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

ZooKeeper is now running on port 2181.

Step 2: Start Kafka Broker

Open a new terminal:

bin/kafka-server-start.sh config/server.properties

Expected output:

[2023-01-15 10:30:05,789] INFO Kafka Server started (kafka.server.KafkaServer)

Kafka is now running on localhost:9092.

Step 3: Verify Kafka is Running

Create a test topic:

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

List topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Expected output:

test

Success! Kafka is running in ZooKeeper mode.

Running a Multi-Broker Cluster (Local)

Why Multi-Broker?

Single broker = No fault tolerance (if broker crashes, Kafka is down).

Multi-broker = Fault tolerance (if one broker crashes, others take over).

Step 1: Create Broker Configurations

Copy default config:

cp config/kraft/server.properties config/kraft/server-1.properties
cp config/kraft/server.properties config/kraft/server-2.properties
cp config/kraft/server.properties config/kraft/server-3.properties

Step 2: Edit Broker Configs

Broker 1 (server-1.properties):

node.id=1
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
log.dirs=/tmp/kraft-broker-1-logs

Broker 2 (server-2.properties):

node.id=2
listeners=PLAINTEXT://localhost:9192,CONTROLLER://localhost:9193
log.dirs=/tmp/kraft-broker-2-logs

Broker 3 (server-3.properties):

node.id=3
listeners=PLAINTEXT://localhost:9292,CONTROLLER://localhost:9293
log.dirs=/tmp/kraft-broker-3-logs

Key Changes:

• node.id: Unique broker ID
• listeners: Different ports for each broker
• log.dirs: Separate data directories

Step 3: Format Storage for All Brokers

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server-1.properties
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server-2.properties
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server-3.properties

Step 4: Start All Brokers

Terminal 1:

bin/kafka-server-start.sh config/kraft/server-1.properties

Terminal 2:

bin/kafka-server-start.sh config/kraft/server-2.properties

Terminal 3:

bin/kafka-server-start.sh config/kraft/server-3.properties

Step 5: Verify Cluster

Create topic with RF=3:

bin/kafka-topics.sh --create --topic orders \
  --partitions 10 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

Describe topic:

bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Expected output:

Topic: orders   Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
Topic: orders   Partition: 1    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
Topic: orders   Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
...

Success! You have a 3-broker cluster with replication.

Multi-Node Cluster Setup (Production)

A multi-node cluster provides fault tolerance, scalability, and is necessary for production environments. This section assumes three machines for a typical production setup.

Prerequisites for Multi-Node Setup

• 3 or more machines with Java 11+ installed
• All machines on the same network with low latency (< 1ms preferred)
• SSH access between machines (for management)
• Synchronized clocks across all machines (NTP required)
• Firewall rules configured (ports 9092, 9093, 2181, 2888, 3888)

Why Clock Synchronization Matters: Kafka uses timestamps for retention, log rolling, and replication. Clock skew causes data inconsistencies.

Architecture Overview

┌─────────────────────────────────────────────────────┐
│          ZooKeeper Ensemble (Quorum)                │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │   ZK 1   │  │   ZK 2   │  │   ZK 3   │         │
│  │ (Server1)│  │ (Server2)│  │ (Server3)│         │
│  └─────┬────┘  └─────┬────┘  └─────┬────┘         │
└────────┼─────────────┼─────────────┼───────────────┘
         │             │             │
┌────────▼─────────────▼─────────────▼───────────────┐
│              Kafka Broker Cluster                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │ Broker 1 │  │ Broker 2 │  │ Broker 3 │         │
│  │ (Node 1) │  │ (Node 2) │  │ (Node 3) │         │
│  └──────────┘  └──────────┘  └──────────┘         │
└─────────────────────────────────────────────────────┘

Network Layout Example:

Node 1: 192.168.1.10 (ZooKeeper Server ID: 1, Broker ID: 1)
Node 2: 192.168.1.11 (ZooKeeper Server ID: 2, Broker ID: 2)
Node 3: 192.168.1.12 (ZooKeeper Server ID: 3, Broker ID: 3)

Step 1: Set Up ZooKeeper Ensemble

ZooKeeper requires an odd number of nodes for quorum (3, 5, or 7 typical). This setup uses 3 nodes.

Why Odd Number?: ZooKeeper requires a majority (quorum) to function. With 3 nodes, you can tolerate 1 failure. With 5 nodes, you can tolerate 2 failures.

On Each Machine: Download and Extract Kafka

# Download Kafka (includes ZooKeeper)
cd /opt
sudo wget https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz
sudo tar -xzf kafka_2.13-3.7.0.tgz
sudo ln -s kafka_2.13-3.7.0 kafka
sudo chown -R kafka:kafka /opt/kafka

Configure ZooKeeper on Node 1 (192.168.1.10)

Create ZooKeeper configuration:

sudo mkdir -p /var/zookeeper/data
sudo chown -R kafka:kafka /var/zookeeper

cat > /opt/kafka/config/zookeeper.properties << 'EOF'
# Data directory
dataDir=/var/zookeeper/data

# Client port
clientPort=2181

# Ensemble configuration
server.1=192.168.1.10:2888:3888
server.2=192.168.1.11:2888:3888
server.3=192.168.1.12:2888:3888

# Performance tuning
tickTime=2000
initLimit=10
syncLimit=5
maxClientCnxns=0
EOF

Create myid file (identifies this ZooKeeper server):

echo "1" | sudo tee /var/zookeeper/data/myid

Configure ZooKeeper on Node 2 (192.168.1.11)

Same configuration as Node 1, but different myid:

sudo mkdir -p /var/zookeeper/data
sudo chown -R kafka:kafka /var/zookeeper

# Copy the same zookeeper.properties from Node 1
# ...

# Create myid with server ID 2
echo "2" | sudo tee /var/zookeeper/data/myid

Configure ZooKeeper on Node 3 (192.168.1.12)

Same configuration as Node 1, but different myid:

echo "3" | sudo tee /var/zookeeper/data/myid

Start ZooKeeper on All Nodes

On each node:

# Start ZooKeeper in background
cd /opt/kafka
sudo -u kafka bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

# Verify startup
sleep 3
tail -50 logs/zookeeper.out

Verify ZooKeeper Ensemble

Test on each node:

# Check if ZooKeeper is responsive
echo "ruok" | nc localhost 2181
# Expected output: imok

# Check server status (leader or follower)
echo "stat" | nc localhost 2181 | grep Mode
# Expected output: Mode: leader (on one node)
#                  Mode: follower (on two nodes)

Ensemble Health:

• 1 leader (handles writes)
• 2 followers (replicate from leader)

Test Failover: Stop ZooKeeper on one node. The ensemble should remain functional (2 out of 3 = quorum).

Step 2: Configure Kafka Brokers for Multi-Node Cluster

Configure Broker 1 (192.168.1.10)

sudo mkdir -p /var/kafka/data
sudo chown -R kafka:kafka /var/kafka

cat > /opt/kafka/config/server.properties << 'EOF'
######################### Broker Identity #########################
broker.id=1

######################### Listeners #########################
# Internal listener (inter-broker communication)
listeners=PLAINTEXT://192.168.1.10:9092

# Advertised listener (how clients connect)
advertised.listeners=PLAINTEXT://192.168.1.10:9092

######################### ZooKeeper #########################
zookeeper.connect=192.168.1.10:2181,192.168.1.11:2181,192.168.1.12:2181

######################### Log Directories #########################
log.dirs=/var/kafka/data

######################### Replication #########################
default.replication.factor=3
min.insync.replicas=2
num.replica.fetchers=4
replica.fetch.max.bytes=1048576

######################### Performance #########################
num.network.threads=8
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

######################### Retention #########################
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

######################### Recovery #########################
num.recovery.threads.per.data.dir=4

######################### ZooKeeper Timeouts #########################
zookeeper.session.timeout.ms=6000
zookeeper.connection.timeout.ms=6000
EOF

Configure Broker 2 (192.168.1.11)

Same configuration as Broker 1, but change:

broker.id=2
listeners=PLAINTEXT://192.168.1.11:9092
advertised.listeners=PLAINTEXT://192.168.1.11:9092

Configure Broker 3 (192.168.1.12)

Same configuration as Broker 1, but change:

broker.id=3
listeners=PLAINTEXT://192.168.1.12:9092
advertised.listeners=PLAINTEXT://192.168.1.12:9092

Step 3: Start All Kafka Brokers

On each node:

cd /opt/kafka
sudo -u kafka bin/kafka-server-start.sh -daemon config/server.properties

# Wait for startup
sleep 5

# Verify broker is running
tail -50 logs/server.log | grep "started (kafka.server.KafkaServer)"

Step 4: Verify Multi-Node Cluster

From any broker machine:

# Check broker connectivity
bin/kafka-broker-api-versions.sh \
  --bootstrap-server 192.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9092

# List topics
bin/kafka-topics.sh --list \
  --bootstrap-server 192.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9092

Create a replicated topic to test the cluster:

bin/kafka-topics.sh --create \
  --topic replicated-topic \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server 192.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9092

# Describe the topic to see replica distribution
bin/kafka-topics.sh --describe \
  --topic replicated-topic \
  --bootstrap-server 192.168.1.10:9092,192.168.1.11:9092,192.168.1.12:9092

Expected output:

Topic: replicated-topic PartitionCount: 6   ReplicationFactor: 3
Topic: replicated-topic Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
Topic: replicated-topic Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
Topic: replicated-topic Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
Topic: replicated-topic Partition: 3  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
Topic: replicated-topic Partition: 4  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
Topic: replicated-topic Partition: 5  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2

Interpretation:

• Replicas are distributed across all 3 brokers (1, 2, 3)
• Each partition has a leader (handles reads/writes)
• ISR (In-Sync Replicas) shows all replicas are caught up

Success! You have a production-ready, fault-tolerant Kafka cluster.

Step 5: Test Fault Tolerance

Simulate broker failure:

# Stop Broker 3 on Node 3
sudo -u kafka bin/kafka-server-stop.sh

Verify cluster still works:

# Describe topic again
bin/kafka-topics.sh --describe \
  --topic replicated-topic \
  --bootstrap-server 192.168.1.10:9092,192.168.1.11:9092

Expected:

• Partitions with Leader=3 now have new leaders (1 or 2)
• ISR no longer includes Broker 3
• Cluster is still functional (can read/write)

Restart Broker 3:

sudo -u kafka bin/kafka-server-start.sh -daemon config/server.properties

After restart: Broker 3 rejoins ISR, leadership may rebalance.

Docker Compose Setup (Multi-Broker)

Create docker-compose.yml:

version: '3.8'
services:
  kafka-1:
    image: confluentinc/cp-kafka:7.5.0
    hostname: kafka-1
    container_name: kafka-1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-1:29092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://kafka-1:29092,CONTROLLER://kafka-1:29093'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-logs'
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

  kafka-2:
    image: confluentinc/cp-kafka:7.5.0
    hostname: kafka-2
    container_name: kafka-2
    ports:
      - "9093:9093"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-2:29092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://kafka-2:29092,CONTROLLER://kafka-2:29093'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-logs'
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

  kafka-3:
    image: confluentinc/cp-kafka:7.5.0
    hostname: kafka-3
    container_name: kafka-3
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-3:29092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://kafka-3:29092,CONTROLLER://kafka-3:29093'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-logs'
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

Start cluster:

docker-compose up -d

Verify cluster:

docker exec -it kafka-1 kafka-topics --create --topic test --bootstrap-server kafka-1:29092 --replication-factor 3 --partitions 3

Configuration Files

Key Configuration Files

Important Broker Configs (High-Level)

config/kraft/server.properties:

# Broker ID (must be unique per broker)
node.id=1

# Listeners (how clients connect)
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

# Data directory (where Kafka stores events)
log.dirs=/tmp/kraft-combined-logs

# Cluster ID (unique per cluster)
# Generated via: bin/kafka-storage.sh random-uuid

Note: Deep config tuning is covered in M2: Brokers & Topics and M6: Operations.

Stopping Kafka

Stop Kafka Broker

KRaft mode:

bin/kafka-server-stop.sh

ZooKeeper mode (stop broker first, then ZooKeeper):

bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh

Docker

docker-compose down

Troubleshooting

Issue 1: Port Already in Use

Error:

java.net.BindException: Address already in use

Solution: Check if Kafka is already running.

lsof -i :9092  # Find process using port 9092
kill -9 <PID>   # Kill process

Issue 2: Java Not Found

Error:

Error: JAVA_HOME is not set

Solution: Set JAVA_HOME environment variable.

macOS/Linux:

export JAVA_HOME=$(/usr/libexec/java_home -v 17)

Windows:

set JAVA_HOME=C:\Program Files\Java\jdk-17

Issue 3: Kafka Won't Start (KRaft Mode)

Error:

Failed to load metadata log

Solution: Format storage directory.

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Issue 4: Out of Memory

Error:

java.lang.OutOfMemoryError: Java heap space

Solution: Increase JVM heap size.

Edit bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"

Verifying Cluster Health and Performance

Regular health checks and performance tests ensure your cluster operates optimally.

Basic Health Checks

1. Check Broker Connectivity

# Test connection to a broker
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Expected output:

localhost:9092 (id: 1 rack: null) -> (
    Produce(0): 0 to 9 [usable: 9],
    Fetch(1): 0 to 13 [usable: 13],
    ...
)

Success: Broker is responsive and advertising supported API versions.

2. Check Under-Replicated Partitions

# Check for partitions missing replicas
bin/kafka-topics.sh --describe \
  --bootstrap-server localhost:9092 \
  --under-replicated-partitions

Healthy Cluster: No output (all partitions fully replicated).

Unhealthy Cluster:

Topic: orders  Partition: 2  Leader: 1  Replicas: 1,2,3  Isr: 1,2

Interpretation: Partition 2 has 3 replicas, but only 2 are in-sync (Broker 3 is lagging or down).

3. Check Broker Logs

# Monitor broker log in real-time
tail -f /opt/kafka/logs/server.log

# Search for errors
grep ERROR /opt/kafka/logs/server.log

# Check recent startup/shutdown messages
tail -50 /opt/kafka/logs/server.log

Warning Signs:

• ERROR or FATAL messages
• OutOfMemoryError
• Connection refused or Connection reset
• ISR shrinks (indicates replica lag)

Monitoring with JMX

Kafka exposes metrics via JMX (Java Management Extensions) for monitoring tools.

Enable JMX Monitoring

Edit Kafka startup script or set environment variable:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Djava.rmi.server.hostname=localhost \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false"

# Start Kafka with JMX enabled
bin/kafka-server-start.sh config/server.properties

JMX Port: 9999 (default)

Key Metrics to Monitor

Access JMX Metrics

Using JConsole (GUI):

jconsole localhost:9999

Using Command Line (jmxterm):

java -jar jmxterm.jar -l localhost:9999

Production Monitoring: Use Prometheus + JMX Exporter or tools like Datadog, New Relic, Grafana.

Performance Testing

Performance tests validate that your cluster can handle expected load.

Producer Performance Test

Test producer throughput:

bin/kafka-producer-perf-test.sh \
  --topic perf-test \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props \
    bootstrap.servers=localhost:9092 \
    acks=all \
    compression.type=snappy

Parameters:

• --num-records 1000000: Send 1 million messages
• --record-size 1024: Each message is 1 KB
• --throughput -1: Unlimited (send as fast as possible)
• acks=all: Wait for all replicas (strongest durability)
• compression.type=snappy: Enable compression

Expected Output:

500000 records sent, 99800.4 records/sec (97.46 MB/sec), 5.2 ms avg latency, 150.0 ms max latency.
1000000 records sent, 100050.025013 records/sec (97.70 MB/sec), 4.8 ms avg latency, 180.0 ms max latency, 3 ms 50th, 15 ms 95th, 100 ms 99th, 150 ms 99.9th.

Interpretation:

• Throughput: ~100,000 msg/sec (100 MB/sec)
• Latency: Median (p50) = 3ms, p99 = 100ms
• Bottleneck: If throughput < 50,000 msg/sec on modern hardware, investigate broker CPU, disk I/O, network.

Consumer Performance Test

Test consumer throughput:

bin/kafka-consumer-perf-test.sh \
  --broker-list localhost:9092 \
  --topic perf-test \
  --messages 1000000 \
  --threads 1

Expected Output:

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-01-15 10:30:00:000, 2023-01-15 10:30:05:123, 976.56, 190.12, 1000000, 194893.79, 1234, 3889, 251.20, 257194.44

Interpretation:

• Throughput: ~195,000 msg/sec (190 MB/sec)
• Fetch Time: 3.9 seconds (time spent reading data)
• Rebalance Time: 1.2 seconds (time spent rebalancing partitions)

Benefit: Consumer throughput is typically 2-3x producer throughput due to zero-copy optimization.

End-to-End Latency Test

Test latency from produce to consume:

bin/kafka-run-class.sh kafka.tools.EndToEndLatency \
  localhost:9092 \
  perf-test \
  10000 \
  all \
  1024

Parameters:

• localhost:9092: Broker address
• perf-test: Topic name
• 10000: Number of messages
• all: Producer acks setting
• 1024: Message size (bytes)

Expected Output:

Avg latency: 5.2 ms
Percentiles: 50th = 3 ms, 99th = 20 ms, 99.9th = 50 ms

Interpretation:

• p50 (median): 3ms (half of messages complete within 3ms)
• p99: 20ms (99% of messages complete within 20ms)
• p99.9: 50ms (outliers, possibly due to GC pauses)

Production Target: p99 latency < 50ms for real-time applications.

Health Check Checklist

Use this checklist to verify cluster health:

• All brokers are running and responsive (kafka-broker-api-versions.sh)
• No ERROR or FATAL messages in recent broker logs
• All partitions have leaders (no offline partitions)
• No under-replicated partitions (--under-replicated-partitions)
• Producer/consumer tests complete successfully (throughput meets expectations)
• No unusual latency (p99 latency < 50ms for real-time apps)
• Disk usage on log directories is growing as expected (not full, not static)
• Consumer group lag is acceptable (lag < retention window)
• ZooKeeper ensemble is healthy (all nodes responsive, if using ZooKeeper mode)
• Network connectivity between brokers is stable (< 1ms latency)
• Exactly 1 active controller (ActiveControllerCount=1)

Common Performance Issues

Issue 1: Low Producer Throughput (< 10,000 msg/sec)

Possible Causes:

• Small batch size → Increase batch.size (e.g., 64 KB)
• No compression → Enable compression.type=snappy
• acks=all with slow replicas → Check replication lag
• Network bottleneck → Use 10 Gbps network

Issue 2: High Consumer Lag

Possible Causes:

• Consumer processing too slow → Add more consumers (scale out)
• Too few partitions → Create topic with more partitions
• Consumer rebalancing frequently → Increase session.timeout.ms

Issue 3: Brokers Under-Replicated

Possible Causes:

• Broker down or network issue → Check broker logs, network
• Disk I/O saturation → Use faster disks (SSD), add more brokers
• Replication throttled → Check replica.fetch.max.bytes

Key Takeaways

1. Prerequisites: Java 11+ required.

2. KRaft Mode (Recommended): Kafka 3.3+ runs without ZooKeeper (simpler, faster).

3. ZooKeeper Mode (Legacy): Kafka < 3.0 requires ZooKeeper (deprecated).

4. Local Setup: Download Kafka, format storage, start broker.

5. Multi-Broker Cluster: Run 3+ brokers for fault tolerance.

6. Docker: Use Docker Compose for quick local setup.

7. Configuration: node.id, listeners, log.dirs are key configs.

8. Stopping Kafka: Use kafka-server-stop.sh.

What's Next?

Next Module: 1.6 – Basic Operations and CLI Walkthrough

What We Covered:

• ✅ Installation (download, Java setup)
• ✅ KRaft vs ZooKeeper mode
• ✅ Running Kafka (single broker, multi-broker)
• ✅ Docker Compose setup
• ✅ Configuration files (high-level)
• ✅ Troubleshooting

What We Didn't Cover (Coming in Later Modules):

• ❌ Production deployment → M6: Operations
• ❌ Cluster sizing and scaling → M6: Operations
• ❌ Security setup (TLS, SASL) → M9: Security
• ❌ Performance tuning → M6: Operations

Installation and Running Kafka: Complete ✅