KafkaIQ

1.6 – Basic Operations and CLI Walkthrough

Module: M1 – Kafka Fundamentals
Goal: First contact with Kafka command-line tools

Introduction

This module covers basic Kafka operations using command-line tools.

Important: This is basic CLI familiarity for learning and testing, not admin mastery. Deep CLI operations and admin best practices are covered in M6: Operations & Administration.

Kafka CLI Tools Overview

Kafka provides command-line tools for managing topics, producing/consuming messages, and inspecting the cluster.

Key Tools:

Tool Purpose
kafka-topics.sh Create, list, describe, delete topics
kafka-console-producer.sh Send messages to a topic
kafka-console-consumer.sh Read messages from a topic
kafka-console-consumer.sh --group Read with consumer group
kafka-configs.sh View and alter configurations
kafka-consumer-groups.sh Manage consumer groups
kafka-broker-api-versions.sh Check broker versions and APIs

Location: All CLI tools are in bin/ directory (e.g., bin/kafka-topics.sh).

Topic Management

Create a Topic

Command:

kafka-topics.sh --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Explanation:

  • --topic orders: Topic name
  • --partitions 3: Create 3 partitions
  • --replication-factor 1: Each partition has 1 copy (no replication)
  • --bootstrap-server localhost:9092: Kafka broker address

Output:

Created topic orders.

List Topics

Command:

kafka-topics.sh --list --bootstrap-server localhost:9092

Output:

orders
payments
user-activity

Describe a Topic

Command:

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Output:

Topic: orders   PartitionCount: 3       ReplicationFactor: 1    Configs: 
        Topic: orders   Partition: 0    Leader: 1       Replicas: 1     Isr: 1
        Topic: orders   Partition: 1    Leader: 1       Replicas: 1     Isr: 1
        Topic: orders   Partition: 2    Leader: 1       Replicas: 1     Isr: 1

Explanation:

  • PartitionCount: 3 partitions
  • ReplicationFactor: 1 (no replication)
  • Leader: Broker 1 is leader for all partitions
  • Replicas: Partition is stored on Broker 1
  • Isr: In-Sync Replicas (only Broker 1)

Delete a Topic

Command:

kafka-topics.sh --delete --topic orders --bootstrap-server localhost:9092

Output:

Topic orders is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

Important: Topic deletion is asynchronous (may take a few seconds).

Verify deletion:

kafka-topics.sh --list --bootstrap-server localhost:9092

Producing Messages

Produce Messages (Interactive)

Command:

kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

Prompt:

>

Type messages (press Enter after each):

>order-1
>order-2
>order-3

Exit: Press Ctrl+C.

What Happened: 3 messages were sent to topic orders.

Produce Messages with Keys

Command:

kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --property "parse.key=true" \
  --property "key.separator=:"

Type messages (format: key:value):

>user-123:order-1
>user-456:order-2
>user-123:order-3

Exit: Press Ctrl+C.

What Happened: Messages were sent with keys (e.g., user-123). Events with the same key go to the same partition.

Produce Messages from File

Create a file messages.txt:

order-1
order-2
order-3

Command:

kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  < messages.txt

What Happened: All messages from messages.txt were sent to Kafka.

Consuming Messages

Consume Messages (From End)

Command:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

Behavior: Waits for new messages (doesn't read old messages).

Output (when new messages arrive):

order-1
order-2
order-3

Exit: Press Ctrl+C.

Consume Messages (From Beginning)

Command:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --from-beginning

Behavior: Reads all messages from the topic (starting at offset 0).

Output:

order-1
order-2
order-3
(... all historical messages ...)

Consume Messages with Keys

Command:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --property "print.key=true" \
  --property "key.separator=:"

Output:

user-123:order-1
user-456:order-2
user-123:order-3

Explanation: Shows key:value for each message.

Consume Messages with Metadata

Command:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --property "print.key=true" \
  --property "print.offset=true" \
  --property "print.partition=true" \
  --property "print.timestamp=true"

Output:

Partition:0     Offset:0        Timestamp:1673784123456 Key:user-123    Value:order-1
Partition:1     Offset:0        Timestamp:1673784123789 Key:user-456    Value:order-2
Partition:0     Offset:1        Timestamp:1673784124012 Key:user-123    Value:order-3

Explanation: Shows partition, offset, timestamp, key, and value for each message.

Consumer Groups

Consume with Consumer Group

Command:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group payment-processors

What Happened:

  • Consumer joined consumer group payment-processors
  • Offset is tracked per consumer group
  • If you restart, consumer resumes from last committed offset

List Consumer Groups

Command:

kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

Output:

payment-processors
analytics
fraud-detection

Describe Consumer Group

Command:

kafka-consumer-groups.sh \
  --describe \
  --group payment-processors \
  --bootstrap-server localhost:9092

Output:

GROUP               TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID    HOST           CLIENT-ID
payment-processors  orders    0          10              12              2    consumer-1-... /127.0.0.1     consumer-1
payment-processors  orders    1          8               10              2    consumer-1-... /127.0.0.1     consumer-1
payment-processors  orders    2          15              15              0    consumer-1-... /127.0.0.1     consumer-1

Explanation:

  • CURRENT-OFFSET: Consumer's current position (offset 10 for partition 0)
  • LOG-END-OFFSET: Latest offset in partition (offset 12 for partition 0)
  • LAG: Number of unread messages (2 for partition 0)
  • CONSUMER-ID: Consumer instance handling the partition

Key Insight: LAG = LOG-END-OFFSET - CURRENT-OFFSET. LAG shows how far behind the consumer is.

Reset Consumer Group Offset

Reset to beginning:

kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group payment-processors \
  --topic orders \
  --reset-offsets \
  --to-earliest \
  --execute

Output:

GROUP               TOPIC     PARTITION  NEW-OFFSET
payment-processors  orders    0          0
payment-processors  orders    1          0
payment-processors  orders    2          0

What Happened: Consumer group offset was reset to 0 (start of log). Next consumer will re-read all messages.

Other Reset Options:

  • --to-latest: Reset to end of log (skip all messages)
  • --to-offset 100: Reset to specific offset
  • --shift-by -10: Move offset backward by 10

Configuration Management

View Topic Configuration

Command:

kafka-configs.sh \
  --describe \
  --entity-type topics \
  --entity-name orders \
  --bootstrap-server localhost:9092

Output:

Dynamic configs for topic orders are:
  retention.ms=604800000
  segment.bytes=1073741824
  compression.type=producer

Explanation: Shows topic-level configurations (retention, segment size, compression).

Alter Topic Configuration

Set retention to 1 day:

kafka-configs.sh \
  --alter \
  --entity-type topics \
  --entity-name orders \
  --add-config retention.ms=86400000 \
  --bootstrap-server localhost:9092

Output:

Completed updating config for topic orders.

Verify:

kafka-configs.sh \
  --describe \
  --entity-type topics \
  --entity-name orders \
  --bootstrap-server localhost:9092

Output:

Dynamic configs for topic orders are:
  retention.ms=86400000

View Broker Configuration

Command:

kafka-configs.sh \
  --describe \
  --entity-type brokers \
  --entity-name 1 \
  --bootstrap-server localhost:9092

Output:

Dynamic configs for broker 1 are:

Note: Broker-level config changes are covered in M6: Operations.

Inspecting the Cluster

Check Broker Version

Command:

kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Output:

localhost:9092 (id: 1 rack: null) -> (
        Produce(0): 0 to 9 [usable: 9],
        Fetch(1): 0 to 13 [usable: 13],
        ListOffsets(2): 0 to 7 [usable: 7],
        ...
)

Explanation: Shows Kafka broker ID and supported API versions.

List Cluster Metadata

Use kafka-metadata-tool.sh (KRaft mode):

kafka-metadata-tool.sh \
  --snapshot /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log \
  --print

Output: Shows cluster metadata (brokers, topics, partitions).

Note: This is advanced usage. Typically not needed for basic operations.

Monitoring with CLI

Check Under-Replicated Partitions

Command:

kafka-topics.sh \
  --describe \
  --under-replicated-partitions \
  --bootstrap-server localhost:9092

Output (if any partitions are under-replicated):

Topic: orders   Partition: 2    Leader: 1       Replicas: 1,2,3 Isr: 1,2

Explanation: Partition 2 has 3 replicas, but only 2 are in-sync (Broker 3 is lagging).

If output is empty: All partitions are fully replicated (healthy).

Check Unavailable Partitions

Command:

kafka-topics.sh \
  --describe \
  --unavailable-partitions \
  --bootstrap-server localhost:9092

Output (if any partitions have no leader):

Topic: payments  Partition: 0    Leader: none    Replicas: 1,2,3 Isr:

Explanation: Partition 0 has no leader (all replicas are down). Critical issue.

If output is empty: All partitions have leaders (healthy).

Complete Example: End-to-End Workflow

Let's walk through a complete workflow: create topic, produce messages, consume messages.

Step 1: Create Topic

kafka-topics.sh --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Output:

Created topic orders.

Step 2: Describe Topic

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Output:

Topic: orders   PartitionCount: 3       ReplicationFactor: 1    Configs: 
        Topic: orders   Partition: 0    Leader: 1       Replicas: 1     Isr: 1
        Topic: orders   Partition: 1    Leader: 1       Replicas: 1     Isr: 1
        Topic: orders   Partition: 2    Leader: 1       Replicas: 1     Isr: 1

Step 3: Produce Messages

Terminal 1 (Producer):

kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --property "parse.key=true" \
  --property "key.separator=:"

Type messages:

>user-123:order-1
>user-456:order-2
>user-123:order-3
>user-789:order-4

Exit: Press Ctrl+C.

Step 4: Consume Messages (Consumer Group 1)

Terminal 2 (Consumer Group: payment-processors):

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group payment-processors \
  --from-beginning \
  --property "print.key=true"

Output:

user-123:order-1
user-456:order-2
user-123:order-3
user-789:order-4

Keep terminal open.

Step 5: Consume Messages (Consumer Group 2)

Terminal 3 (Consumer Group: analytics):

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group analytics \
  --from-beginning \
  --property "print.key=true"

Output:

user-123:order-1
user-456:order-2
user-123:order-3
user-789:order-4

Key Insight: Both consumer groups read the same messages (independent offset tracking).

Step 6: Check Consumer Group Lag

Terminal 4:

kafka-consumer-groups.sh \
  --describe \
  --group payment-processors \
  --bootstrap-server localhost:9092

Output:

GROUP               TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
payment-processors  orders    0          2               2               0
payment-processors  orders    1          1               1               0
payment-processors  orders    2          1               1               0

Explanation: Consumer has read all messages (LAG = 0).

Step 7: Produce More Messages

Terminal 1 (Producer, same as before):

>user-123:order-5
>user-456:order-6

Observe:

  • Terminal 2 (payment-processors): Immediately sees new messages
  • Terminal 3 (analytics): Immediately sees new messages

Key Insight: Consumers poll for new messages in real-time.

Step 8: Check Lag Again

Terminal 4:

kafka-consumer-groups.sh \
  --describe \
  --group payment-processors \
  --bootstrap-server localhost:9092

Output:

GROUP               TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
payment-processors  orders    0          4               4               0
payment-processors  orders    1          2               2               0
payment-processors  orders    2          1               1               0

Explanation: Consumer is still caught up (LAG = 0).

CLI Best Practices

1. Use --bootstrap-server (Not --zookeeper)

Good:

kafka-topics.sh --list --bootstrap-server localhost:9092

Deprecated (Kafka 2.5+):

kafka-topics.sh --list --zookeeper localhost:2181

Why: ZooKeeper mode is deprecated. Always use --bootstrap-server.

2. Always Describe Topics Before Producing

Check partition count before producing:

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Why: Understand partition count (affects parallelism) and replication factor (affects durability).

3. Use Consumer Groups for Testing

Always specify --group:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --group test-group

Why: Tracks offset. If consumer restarts, it resumes from last offset (doesn't re-read all messages).

4. Monitor Consumer Lag

Regularly check lag:

kafka-consumer-groups.sh \
  --describe \
  --group payment-processors \
  --bootstrap-server localhost:9092

Why: High lag indicates consumer is falling behind (may need more consumers or optimization).

5. Use --from-beginning for Debugging

When debugging, read all messages:

kafka-console-consumer.sh \
  --topic orders \
  --bootstrap-server localhost:9092 \
  --from-beginning

Why: Ensures you see all events (useful for debugging).

Key Takeaways

  1. Topic Management: Create, list, describe, delete topics with kafka-topics.sh.

  2. Producing Messages: Use kafka-console-producer.sh to send messages (with or without keys).

  3. Consuming Messages: Use kafka-console-consumer.sh to read messages (from beginning or end).

  4. Consumer Groups: Track offset per group. Check lag with kafka-consumer-groups.sh.

  5. Configuration: View/alter topic configs with kafka-configs.sh.

  6. Monitoring: Check under-replicated partitions, unavailable partitions, broker versions.

  7. Best Practices: Use --bootstrap-server, always specify consumer group, monitor lag.

What's Next?

Next Module: M2 – Brokers & Topics (Deep Dive)

What We Covered:

  • ✅ Topic management (create, list, describe, delete)
  • ✅ Producing messages (console producer, with keys)
  • ✅ Consuming messages (console consumer, from beginning)
  • ✅ Consumer groups (describe, reset offsets)
  • ✅ Configuration management (view, alter)
  • ✅ Basic monitoring (under-replicated, unavailable partitions)

What We Didn't Cover (Coming in Later Modules):

  • ❌ Advanced topic configs and tuning → M2: Brokers & Topics
  • ❌ Producer API (Java, Python, Go) → M3: Producers
  • ❌ Consumer API (Java, Python, Go) → M4: Consumers
  • ❌ Advanced operations (partition reassignment, rebalancing) → M6: Operations
  • ❌ Cluster monitoring and alerting → M6: Operations

Congratulations! You've completed Kafka Fundamentals. You now understand:

  • ✅ What Kafka is and why it exists
  • ✅ Event-driven architecture and Kafka's design
  • ✅ Core concepts (topics, partitions, offsets, brokers, consumer groups)
  • ✅ Kafka APIs (Producer, Consumer, Admin, Streams, Connect)
  • ✅ How to install and run Kafka
  • ✅ Basic CLI operations

You're ready to dive into:

  • M2: Brokers & Topics (deep dive into storage, replication, partitioning)
  • M3: Producers (writing data to Kafka at scale)
  • M4: Consumers (reading data from Kafka at scale)

Basic Operations and CLI Walkthrough: Complete ✅

Kafka Fundamentals Module: Complete ✅✅✅