KafkaIQ1.1 Kafka Cluster Architecture & Administration Fundamentals
Module Overview
Goal: Establish the operator's mental model of Kafka.
What You'll Learn:
- What makes up a Kafka cluster
- Brokers, controllers, and metadata
- Logical vs physical architecture
- Control plane vs data plane
- Admin responsibilities vs application responsibilities
- The operator's perspective on Kafka
What's NOT Covered Here:
- Topic configurations (covered in Module 2.2)
- Performance tuning (covered in Module 5.1)
- Monitoring details (covered in Module 4.x)
- Specific operational tasks (covered in subsequent modules)
Key Insight: After this module, you'll understand what you are operating — not how to tune it. This is the foundation for all operational work.
1. What is a Kafka Cluster?
Definition
A Kafka Cluster is:
├─ A group of Kafka brokers working together
├─ Sharing metadata and coordination
├─ Appearing as a single logical system to clients
└─ Providing fault tolerance and scalability
From an operator's perspective:
Cluster = Multiple Servers + Shared State + Coordination
Minimal Cluster
Minimum viable Kafka cluster:
┌──────────────┐
│ Broker 1 │
│ (id: 0) │
└──────────────┘
Single broker = minimal cluster
└─ Good for: Development, testing
└─ Not for: Production (no fault tolerance)
Production Cluster
Typical production setup:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Broker 1 │ │ Broker 2 │ │ Broker 3 │
│ (id: 0) │ │ (id: 1) │ │ (id: 2) │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
└──────────────────┴──────────────────┘
│
Coordination Layer
(ZooKeeper or KRaft)
Key characteristics:
- 3+ brokers (fault tolerance)
- Shared coordination (metadata, leadership)
- Independent but coordinated (each broker is autonomous)
2. Core Components of a Kafka Cluster
The Five Key Components
1. BROKERS
└─ The servers that store and serve data
2. CONTROLLER
└─ The broker responsible for cluster-wide decisions
3. METADATA
└─ Cluster state (topics, partitions, configs)
4. COORDINATION SYSTEM
└─ ZooKeeper (legacy) or KRaft (modern)
5. DATA (Topics & Partitions)
└─ The actual messages stored on disk
Component Relationships
┌─────────────────────────────────────────────────┐
│ KAFKA CLUSTER │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Broker 1 │ │ Broker 2 │ │ Broker 3 │ │
│ │ │ │ │ │ │ │
│ │ Data + │ │ Data + │ │ Data + │ │
│ │ Metadata │ │ Metadata │ │ Metadata │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │
│ └─────────────┴─────────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ CONTROLLER │ │
│ │ (One Active Broker) │ │
│ └───────────────────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ COORDINATION SYSTEM │ │
│ │ (ZooKeeper / KRaft) │ │
│ └───────────────────────┘ │
└─────────────────────────────────────────────────┘
3. Brokers: The Workhorses
What is a Broker?
A Kafka broker is a server process that:
├─ Stores topic partitions on disk
├─ Handles client requests (produce, fetch)
├─ Manages replication
├─ Participates in cluster coordination
└─ Reports metrics and health
Broker Identification
Each broker has a unique ID:
broker.id=0 → Broker 0
broker.id=1 → Broker 1
broker.id=2 → Broker 2
Key properties:
├─ Unique per cluster
├─ Stable across restarts
├─ Used for partition assignment
└─ Referenced in metadata
Important: Broker IDs are permanent identifiers. Once assigned, they should not change.
Broker Responsibilities
DATA PLANE (Serving Client Traffic):
├─ Accept produce requests
├─ Serve fetch requests
├─ Replicate data between brokers
└─ Write/read from disk
CONTROL PLANE (Cluster Management):
├─ Report health to controller
├─ Participate in leader elections
├─ Sync metadata changes
└─ Handle reassignments
Broker as a State Machine
BROKER LIFECYCLE:
1. Startup
├─ Load config
├─ Register with coordination system
└─ Report available partitions
2. Running
├─ Serve client requests
├─ Replicate data
└─ Send heartbeats
3. Shutdown
├─ Stop accepting new requests
├─ Complete in-flight requests
└─ Flush data to disk
4. The Controller: Cluster Brain
What is the Controller?
The Controller is:
├─ One special broker elected as "leader"
├─ Responsible for cluster-wide decisions
├─ Manages partition leadership
├─ Handles broker join/leave events
└─ Propagates metadata changes
Key Insight: The controller is not a separate process — it's a role that one of the existing brokers takes on.
Controller Election
HOW CONTROLLER IS ELECTED:
ZooKeeper Mode:
├─ All brokers race to create ephemeral node in ZK
├─ First one wins → becomes controller
├─ If controller fails, node deleted → new election
└─ Other brokers watch for changes
KRaft Mode:
├─ Raft consensus among controller quorum
├─ Majority vote elects leader
├─ More stable, no ZooKeeper dependency
└─ Covered in detail in Module 7.1
Controller Responsibilities
CONTROLLER DUTIES:
1. Partition Leadership Management
└─ Decides which broker is leader for each partition
2. Broker Lifecycle Events
├─ Broker joins → assign partitions
├─ Broker fails → reassign leaders
└─ Broker leaves → rebalance partitions
3. Topic Operations
├─ Create topics
├─ Delete topics
└─ Change partition count
4. Metadata Propagation
└─ Distribute metadata updates to all brokers
5. ISR Management
└─ Track in-sync replica sets
Controller Failover
WHAT HAPPENS IF CONTROLLER FAILS?
1. Controller broker crashes
↓
2. ZK/KRaft detects failure
↓
3. New controller election
↓
4. New controller reads state
↓
5. Cluster operations resume
Duration: Seconds (typically < 10s)
Impact during failover:
├─ Data plane: UNAFFECTED (brokers keep serving)
├─ Control plane: PAUSED (no admin operations)
└─ Client traffic: CONTINUES normally
5. Metadata: The Shared Truth
What is Metadata?
Metadata = Cluster configuration and state
Includes:
├─ Topic definitions (name, partition count, replication factor)
├─ Partition assignments (which brokers host which partitions)
├─ Partition leaders (which broker is leader for each partition)
├─ ISR status (in-sync replicas per partition)
├─ Broker configs
├─ ACLs and security configs
└─ Dynamic configurations
Metadata Storage
ZooKeeper Mode (Legacy):
┌──────────────────────────────────────┐
│ Metadata stored in ZooKeeper │
│ ├─ /brokers/topics/... │
│ ├─ /brokers/ids/... │
│ ├─ /controller │
│ └─ /config/... │
└──────────────────────────────────────┘
↓ (cached locally)
┌──────────────────────────────────────┐
│ Each broker maintains local cache │
└──────────────────────────────────────┘
KRaft Mode (Modern):
┌──────────────────────────────────────┐
│ Metadata stored in Kafka topic │
│ __cluster_metadata │
│ ├─ Replicated via Raft consensus │
│ └─ No external dependency │
└──────────────────────────────────────┘
Metadata Consistency
Critical Rule:
All brokers must have eventually consistent metadata.
Metadata Update Flow:
1. Admin operation (create topic)
↓
2. Controller receives request
↓
3. Controller updates metadata store
↓
4. Controller sends UpdateMetadata to all brokers
↓
5. Brokers update local cache
↓
6. Cluster state is consistent
Consistency guarantee: All brokers converge to same state (eventual consistency).
6. Logical vs Physical Architecture
Physical Architecture
What's actually running on your servers:
Physical View (3-node cluster):
Server 1 (10.0.1.10) Server 2 (10.0.1.11) Server 3 (10.0.1.12)
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ kafka-server │ │ kafka-server │ │ kafka-server │
│ PID: 12345 │ │ PID: 12346 │ │ PID: 12347 │
│ Port: 9092 │ │ Port: 9092 │ │ Port: 9092 │
│ Disk: /var/kafka │ │ Disk: /var/kafka │ │ Disk: /var/kafka │
│ RAM: 32 GB │ │ RAM: 32 GB │ │ RAM: 32 GB │
│ CPU: 8 cores │ │ CPU: 8 cores │ │ CPU: 8 cores │
└──────────────────┘ └──────────────────┘ └──────────────────┘
Logical Architecture
How clients and operators think about Kafka:
Logical View:
KAFKA CLUSTER
┌────────────────────────────────────────────────┐
│ │
│ Topics: │
│ ├─ orders (12 partitions, RF=3) │
│ ├─ payments (6 partitions, RF=3) │
│ └─ users (3 partitions, RF=3) │
│ │
│ Brokers: │
│ ├─ Broker 0 (Leader for P0, P3, P6...) │
│ ├─ Broker 1 (Leader for P1, P4, P7...) │
│ └─ Broker 2 (Leader for P2, P5, P8...) │
│ │
│ Controller: Broker 1 │
└────────────────────────────────────────────────┘
The Abstraction Layers
CLIENT VIEW:
"I'm writing to 'orders' topic"
↓
LOGICAL VIEW:
"Orders topic has 12 partitions distributed across brokers"
↓
PHYSICAL VIEW:
"Partition 0 is on disk at /var/kafka/orders-0/"
As an operator, you work with both:
- Logical: Topic design, partition counts, replication
- Physical: Disk usage, network throughput, server resources
7. Control Plane vs Data Plane
Two Planes of Operation
CONTROL PLANE (Cluster Management):
├─ Low frequency, high importance
├─ Examples:
│ ├─ Create topic
│ ├─ Add broker
│ ├─ Elect partition leader
│ └─ Change configuration
└─ Coordinated via Controller
DATA PLANE (Message Traffic):
├─ High frequency, performance-critical
├─ Examples:
│ ├─ Producer sends message
│ ├─ Consumer fetches messages
│ ├─ Follower replicates data
│ └─ Read/write to disk
└─ Direct broker-to-client communication
Separation of Concerns
KAFKA CLUSTER
┌──────────────────────────────────────────────┐
│ │
│ CONTROL PLANE (Metadata & Coordination) │
│ ┌──────────────────────────────────────┐ │
│ │ Controller │ │
│ │ ├─ Partition assignment │ │
│ │ ├─ Leader election │ │
│ │ └─ Metadata updates │ │
│ └──────────────────────────────────────┘ │
│ │
│ ───────────────────────────────────────── │
│ │
│ DATA PLANE (Message Traffic) │
│ ┌──────────────────────────────────────┐ │
│ │ All Brokers │ │
│ │ ├─ Accept produce requests │ │
│ │ ├─ Serve fetch requests │ │
│ │ ├─ Replicate data │ │
│ │ └─ Disk I/O │ │
│ └──────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────┘
Key Insight:
Control plane issues (controller failure) do NOT impact data plane (message traffic continues).
Operational Implications
DATA PLANE OPERATIONS:
├─ Must be fast (microsecond latency)
├─ Must scale horizontally
├─ Optimized for throughput
└─ Monitoring: focus on latency, throughput
CONTROL PLANE OPERATIONS:
├─ Can be slower (millisecond latency)
├─ Coordination required
├─ Consistency critical
└─ Monitoring: focus on availability, correctness
8. Admin Responsibilities vs Application Responsibilities
Clear Boundary
ADMIN (Operator) Responsibilities:
├─ Cluster health
├─ Broker operations
├─ Topic creation/deletion
├─ Partition management
├─ Performance tuning
├─ Security configuration
├─ Monitoring & alerting
├─ Capacity planning
└─ Disaster recovery
APPLICATION (Developer) Responsibilities:
├─ Producer configuration
├─ Consumer configuration
├─ Message schema design
├─ Partition key selection
├─ Error handling
├─ Client-side performance
├─ Idempotency logic
└─ Application monitoring
The Operational Contract
┌────────────────────────────────────────────┐
│ ADMINS PROVIDE: │
│ ├─ Available cluster │
│ ├─ Sufficient capacity │
│ ├─ Proper security │
│ └─ SLA guarantees │
└────────────────────────────────────────────┘
↕
┌────────────────────────────────────────────┐
│ APPLICATIONS MUST: │
│ ├─ Use topics correctly │
│ ├─ Handle errors gracefully │
│ ├─ Respect quotas │
│ └─ Monitor their own metrics │
└────────────────────────────────────────────┘
Who Does What? (Examples)
| Task | Admin | App Team |
|---|---|---|
| Create topic | ✅ | ❌ |
| Choose partition key | ❌ | ✅ |
| Set replication factor | ✅ | ❌ |
| Handle deserialization errors | ❌ | ✅ |
| Add broker to cluster | ✅ | ❌ |
| Implement retry logic | ❌ | ✅ |
| Monitor disk usage | ✅ | ❌ |
| Monitor consumer lag | Both | ✅ (primary) |
| Tune broker configs | ✅ | ❌ |
| Tune producer batching | ❌ | ✅ |
Shared Responsibilities
Some areas require COLLABORATION:
1. Topic Design
├─ Admin: Partition count, replication, retention
└─ App: Naming, schema, key selection
2. Performance
├─ Admin: Broker tuning, capacity
└─ App: Client tuning, batching
3. Monitoring
├─ Admin: Cluster-level metrics
└─ App: Application-level metrics
4. Security
├─ Admin: ACLs, TLS, SASL config
└─ App: Credential management, compliance
9. Kafka Admin Tools & Commands
Command-Line Tools Reference
Kafka includes powerful built-in tools for cluster administration:
CORE ADMIN TOOLS:
┌──────────────────────────────┬─────────────────────────────────────────────┐
│ Tool │ Purpose │
├──────────────────────────────┼─────────────────────────────────────────────┤
│ kafka-topics.sh │ Create, list, describe, delete topics │
│ kafka-configs.sh │ Alter configs for topics, brokers, users │
│ kafka-broker-api-versions.sh │ Check broker versions and API compatibility │
│ kafka-reassign-partitions.sh │ Reassign partitions across brokers │
│ kafka-consumer-groups.sh │ Manage and monitor consumer groups │
│ kafka-log-dirs.sh │ Query log directory info per broker │
│ kafka-acls.sh │ Manage access control lists (ACLs) │
│ kafka-leader-election.sh │ Trigger preferred leader election │
│ kafka-metadata.sh │ Query cluster metadata (KRaft mode) │
└──────────────────────────────┴─────────────────────────────────────────────┘
🧠 Note: These tools communicate with the Kafka Controller via the Admin API. Most operations require
--bootstrap-server(modern) rather than--zookeeper(deprecated).
Tool Categories
1. Topic Management:
# Create topic
kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic orders --partitions 3 --replication-factor 3
# List topics
kafka-topics.sh --bootstrap-server localhost:9092 --list
# Describe topic
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --topic orders
# Delete topic
kafka-topics.sh --bootstrap-server localhost:9092 \
--delete --topic orders
2. Configuration Management:
# Alter topic config
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders \
--alter --add-config retention.ms=86400000
# View configs
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders --describe
# Delete config override
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics --entity-name orders \
--alter --delete-config retention.ms
3. Consumer Group Management:
# List consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
# Describe group (view lag)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group --describe
# Reset offsets
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group --topic orders \
--reset-offsets --to-earliest --execute
4. Security & ACLs:
# Grant read permission on topic
kafka-acls.sh --bootstrap-server localhost:9092 \
--add --allow-principal User:alice \
--operation Read --topic orders
# List ACLs
kafka-acls.sh --bootstrap-server localhost:9092 --list
🔒 Security Note:
kafka-acls.shis used for authentication and authorization. ACL management requires security configuration (SASL/SSL) to be enabled. Detailed ACL and security operations are covered in the Kafka Security course module.
5. Partition Management:
# Reassign partitions (advanced)
kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
--reassignment-json-file reassignment.json --execute
# Trigger preferred leader election
kafka-leader-election.sh --bootstrap-server localhost:9092 \
--election-type PREFERRED --all-topic-partitions
Admin API (Programmatic Access)
For automation and integration, use the Java Admin API:
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient adminClient = AdminClient.create(props);
// Example operations covered in Module 3.1 (Dynamic Configuration)
10. The Operator's Mental Model
How to Think Like a Kafka Operator
CLUSTER = DISTRIBUTED SYSTEM
Key principles:
1. Failures are normal
└─ Design for fault tolerance
2. Coordination is tricky
└─ Understand controller role
3. Data is replicated
└─ Replication = availability + durability
4. State is distributed
└─ No single point of failure
5. Scaling is horizontal
└─ Add brokers, not bigger machines
The Operational Feedback Loop
1. OBSERVE
└─ Collect metrics, logs, alerts
2. DIAGNOSE
└─ Identify bottlenecks, failures
3. ACT
└─ Add capacity, tune configs, fix issues
4. VALIDATE
└─ Verify improvements
5. REPEAT
└─ Continuous improvement
Common Operational Tasks
DAILY:
├─ Check cluster health
├─ Monitor disk usage
├─ Review alerts
└─ Check under-replicated partitions
WEEKLY:
├─ Review performance trends
├─ Plan capacity changes
├─ Update configurations
└─ Security reviews
MONTHLY:
├─ Capacity planning
├─ Version upgrades
├─ Disaster recovery drills
└─ Documentation updates
AS NEEDED:
├─ Add/remove brokers
├─ Create/delete topics
├─ Handle incidents
└─ Performance tuning
11. Cluster Topology Patterns
Single Cluster
┌─────────────────────────────────────┐
│ KAFKA CLUSTER (datacenter-1) │
│ ├─ Broker 1 │
│ ├─ Broker 2 │
│ ├─ Broker 3 │
│ └─ All producers & consumers │
└─────────────────────────────────────┘
Use case:
└─ Single datacenter, simple setup
Multi-Cluster (Active-Active)
┌──────────────────────────┐ ┌──────────────────────────┐
│ CLUSTER A (US-East) │ │ CLUSTER B (US-West) │
│ ├─ Broker 1 │ │ ├─ Broker 1 │
│ ├─ Broker 2 │ │ ├─ Broker 2 │
│ └─ Broker 3 │ │ └─ Broker 3 │
└──────────────────────────┘ └──────────────────────────┘
↕ MirrorMaker 2 ↕
(async replication)
Use case:
└─ Multi-region, disaster recovery, data locality
Multi-Cluster (Aggregate)
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Cluster A │ │ Cluster B │ │ Cluster C │
│ (Region 1) │ │ (Region 2) │ │ (Region 3) │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└────────────────┴────────────────┘
↓
┌─────────────────┐
│ Central Cluster │
│ (Aggregate) │
└─────────────────┘
Use case:
└─ Collect data from edge clusters to central for analytics
12. Architectural Best Practices
Broker Sizing
PRODUCTION RECOMMENDATIONS:
Minimum:
└─ 3 brokers (fault tolerance)
Typical:
└─ 5-10 brokers (medium workload)
Large:
└─ 20-50 brokers (high throughput)
Limits:
└─ 100+ brokers (coordination overhead increases)
Rule of thumb:
└─ Add brokers for throughput, not for single-topic performance
Replication Factor
REPLICATION FACTOR GUIDELINES:
RF=1:
└─ No fault tolerance (use only for non-critical data)
RF=2:
└─ Minimal fault tolerance (survives 1 broker failure)
RF=3: ✅ RECOMMENDED
└─ Good fault tolerance (survives 2 broker failures)
└─ Industry standard for production
RF=4+:
└─ High availability requirements (survives 3+ failures)
└─ Higher cost (more storage, network)
Partition Count
PARTITION COUNT CONSIDERATIONS:
Too few:
└─ Low parallelism (bottleneck)
Too many:
└─ Coordination overhead, slow leader elections
Sweet spot:
└─ 1000-4000 partitions per broker (total across all topics)
Formula (rough):
partitions = (target throughput) / (partition throughput)
Example:
├─ Target: 100 MB/s
├─ Single partition: 10 MB/s
└─ Partitions needed: 10
13. Key Takeaways
You've learned:
✅ Kafka cluster = group of brokers + coordination + shared metadata
✅ Brokers handle data plane (messages) and participate in control plane
✅ Controller is elected leader responsible for cluster-wide decisions
✅ Metadata is the shared truth about cluster state
✅ Logical architecture (topics/partitions) abstracts physical infrastructure
✅ Control plane (admin) is separate from data plane (message traffic)
✅ Admin responsibilities focus on cluster health, not application logic
✅ Operator mental model treats Kafka as distributed, fault-tolerant system
14. Questions You Should Be Able to Answer
After this module, you should be able to answer:
✅ What components make up a Kafka cluster?
✅ What is the controller and what does it do?
✅ What's the difference between control plane and data plane?
✅ What is metadata and where is it stored?
✅ What are admin responsibilities vs application responsibilities?
✅ What's the minimum number of brokers for production?
✅ What happens when the controller fails?
✅ How do brokers differ from the controller?
15. What's Next?
Now that you understand cluster architecture, the next modules will teach you how to operate it:
- Module 2.1: Broker Lifecycle Management — Start, stop, add, remove brokers safely
- Module 2.2: Topic Design & Configuration — Create topics with proper settings
- Module 2.3: Partitions, Replication & ISR — Understand replication health
- Module 4.x: Monitoring & Observability — Observe cluster health
- Module 5.1: Performance Tuning — Optimize cluster performance
Summary
The Operator's View:
Kafka Cluster =
├─ Brokers (store & serve data)
├─ Controller (manages cluster)
├─ Metadata (shared state)
├─ Coordination (ZK or KRaft)
└─ Data plane vs control plane
Your job as operator:
├─ Keep cluster healthy
├─ Ensure capacity
├─ Monitor performance
├─ Handle failures gracefully
└─ Enable applications to succeed
Remember: You're operating a distributed system — plan for failures, monitor actively, and scale horizontally.
Next up: Module 2.1 — Broker Lifecycle Management — Learn how to safely start, stop, and manage brokers.