KafkaIQ

topic: learn-kafka · 175 records

Learn Apache Kafka, free.

175 modules covering everything from your first topic to replication internals, Kafka Streams, Connect and production operations. No account needed to read.

section 1 · 6 modules

Kafka Fundamentals

What Kafka is, why it exists, and the core vocabulary: events, topics, partitions, brokers, producers and consumers.

  1. What Is Kafka and Why It Exists
  2. Event Streaming and Kafka Architecture
  3. Core Concepts and Building Blocks
  4. Kafka APIs and Client Landscape
  5. Installation and Running Kafka
  6. Basic Operations and CLI Walkthrough

section 2 · 14 modules

Brokers & Topics

How brokers store data: topics, partitions, segments, retention, compaction and the broker vs topic configuration model.

  1. Broker Fundamentals and Cluster Role
  2. Topics, Partitions & Storage Model
    1. Topics and Partitions Explained
    2. Log Structure and Message Storage
    3. Segments and Physical Storage Layout
  3. Replication and Fault Tolerance (Conceptual Model)
  4. Log Management & Cleanup Policies
    1. Retention Policies
    2. Log Compaction
    3. Cleanup Tradeoffs and Guarantees
  5. Designing Topics and Partition Strategy
  6. Configuration Model: Broker vs Topic
    1. Broker Level Configurations
    2. Topic Level Configurations
    3. Config Precedence and Overrides
  7. Dynamic Configuration Management
  8. CLI Management for Brokers and Topics

section 3 · 13 modules

Producers

Writing to Kafka well: producer configuration, serialization, partitioning, delivery semantics, idempotence, batching and tuning.

  1. Introduction and Producer Mental Model
  2. Producer APIs and Core Configuration
  3. Serialization and Data Modeling
  4. Partitioning Strategy and Message Ordering
  5. Reliability
    1. Delivery Semantics Explained
    2. Idempotent Producer
    3. Exactly Once Semantics (Producer Side)
  6. Error Handling and Failure Scenarios
  7. Producer Internals
  8. Performance & Tuning
    1. Batching and Buffering
    2. Latency vs Throughput Tuning
    3. Compression Strategies
  9. Practical Producer Patterns and Real World Scenarios

section 4 · 13 modules

Consumers

Reading from Kafka: consumer groups, rebalancing, offset management, delivery semantics and consumer tuning.

  1. Introduction and Consumer Mental Model
  2. Consumer Groups and Coordination
  3. Broker Side Partition and Leader Management
  4. Partition Assignment Strategies
  5. Rebalancing and Group Stability
  6. Offset Management & Delivery Semantics
    1. Offsets Explained
    2. Commit Strategies
    3. Consumer Delivery Semantics
  7. Consumer Lifecycle and Poll Loop Internals
  8. Configuration and Performance Tuning
  9. Error Handling and Resilience
  10. Monitoring, Lag and Observability
  11. Practical Consumer Patterns and Real World Scenarios

section 5 · 17 modules

Kafka Streams

Stream processing with Kafka Streams: topologies, stateless and stateful operations, windows, state stores and fault tolerance.

  1. Introduction to Kafka Streams and Processing Model
  2. Core Abstractions KStream, KTable and GlobalKTable
  3. Building a Kafka Streams Application
  4. Stateless Operations
  5. Stateful Processing & Windows
    1. Stateful Operations
    2. Windowing Concepts and Types
    3. Windowed Aggregations
  6. State Stores & Fault Tolerance
    1. State Stores Explained
    2. Changelogs and Standby Replicas
    3. Restore and Recovery
  7. Topology Design and Internal Topics
  8. Joins in Kafka Streams
  9. Exactly Once Processing in Kafka Streams
  10. Error Handling and Failure Scenarios
  11. Interactive Queries and Testing
  12. Monitoring and Operating Kafka Streams Applications
  13. Advanced Configuration and Choosing the Right Tool

section 6 · 11 modules

Operations

Running Kafka in production: cluster setup, broker and topic management, monitoring, metrics and capacity planning.

  1. Kafka Cluster Architecture and Administration Fundamentals
  2. Broker & Topic Management
    1. Broker Lifecycle Management
    2. Topic Design and Configuration
    3. Partitions, Replication and ISR Management
  3. Dynamic Configuration and Runtime Changes
  4. Monitoring, Metrics & Observability
    1. Core Kafka Metrics
    2. Client, Broker and Topic Level Monitoring
    3. Alerting and Operational Dashboards
  5. Performance Tuning and Capacity Planning
  6. Quotas, Throttling and Resource Management
  7. Coordination System Operations (ZooKeeper and KRaft)

section 7 · 16 modules

Internal Mechanics

Under the hood: log structure, replication, ISR, leader election, the controller and KRaft metadata.

  1. End to End Message Flow and Core Architecture
  2. Topics, Partitions & Log Structure
    1. Topic and Partition Internals
    2. Log Segments, Indexes and Storage Layout
    3. Compaction and Retention Mechanics
  3. Replication & Durability Protocol
    1. Replication Protocol Explained
    2. ISR and High Watermark Mechanics
    3. Follower Fetching and Data Consistency
  4. Leader Election & Failover Mechanics
    1. Partition Leadership
    2. Leader Election Algorithms
    3. Failover Scenarios and Edge Cases
  5. Cluster Metadata & Coordination
    1. Metadata Model and Cluster State
    2. ZooKeeper Coordination (Legacy)
    3. KRaft Metadata Quorum
  6. The Kafka Controller Internals
  7. Broker Lifecycle and State Transitions
  8. Transaction and Exactly Once Internals (Broker Side)

section 8 · 14 modules

Kafka Connect

Moving data in and out with Kafka Connect: workers, connectors, converters, SMTs and custom connector development.

  1. Kafka Connect Fundamentals and Architecture
  2. Connector Model Source vs Sink
  3. Connector Configuration & Deployment
    1. Connector Configuration Basics
    2. Standalone vs Distributed Mode
    3. Task Parallelism and Scaling
  4. Single Message Transforms (SMTs)
  5. Performance Tuning and Throughput Optimization
  6. Error Handling, Retries and Dead Letter Queues
  7. Custom Connector Development
    1. Connector Development Model
    2. Source Connector Implementation
    3. Sink Connector Implementation
    4. Testing and Packaging Connectors
  8. Monitoring and Metrics
  9. Security in Kafka Connect

section 9 · 4 modules

Security

Securing a cluster: TLS encryption, SASL authentication, ACL authorization and security operations.

  1. Kafka Security Fundamentals and Threat Model
  2. Encryption & Transport Security
    1. TLS SSL Encryption Between Clients and Brokers
    2. Inter Broker Encryption
    3. Certificates, Keystores and Truststores

section 10 · 25 modules

Advanced Topics

Architecture-level topics: transactions and exactly-once, multi-datacenter replication, high availability, governance and performance.

  1. Graceful Lifecycle and Testing Strategies
  2. Transactions and Exactly Once Semantics
    1. Transaction Protocol and Coordinator
    2. Producer Transactions
    3. Consumer Isolation Levels
    4. End to End EOS Pipelines
    5. EOS Anti Patterns and Debugging
    6. EOS Performance Tradeoffs
  3. Multi Datacenter Replication and Geo Architecture
    1. Multi Cluster Design Models
    2. MirrorMaker and MM2 Architecture
    3. Active Passive vs Active Active
    4. Replication Consistency Challenges
    5. Geo DR Patterns
  4. High Availability and Disaster Recovery
    1. Fault Tolerance Mechanisms Review
    2. Broker Failure Recovery Playbooks
    3. KRaft vs ZooKeeper Resilience
    4. Backup and Restore Strategies
    5. RTO RPO Modeling
  5. Cluster Governance and Architecture Decisions
    1. Topic Governance and Standards
    2. Partition Sizing and Rebalancing Strategy
    3. Quota and Tenant Governance
    4. Broker Placement and Topology
    5. Upgrade and Maintenance Strategies
  6. Advanced Performance Tuning Layer
    1. JVM Tuning for Kafka
    2. OS and Disk Tuning
    3. Network Stack Optimization

section 11 · 42 modules

Ecosystem Tools

The wider ecosystem: Schema Registry and Avro, schema evolution, ksqlDB and the Kafka REST Proxy.

  1. Schema Registry Foundations
    1. What Is Schema Registry
    2. Why Schemas Matter in Event Systems
    3. Schema Formats Overview
    4. Schema Registry Architecture and Flow
  2. Avro and Strongly Typed Messaging
    1. Avro Fundamentals
    2. Avro Serialization with Kafka
    3. SpecificRecord vs GenericRecord
    4. Strongly Typed Producers and Consumers
  3. Schema Evolution and Compatibility
    1. Schema Evolution Concepts
    2. Compatibility Types
    3. Versioning Rules and Migration Strategies
    4. Real World Evolution Scenarios
  4. Schema Registry Operations
    1. Subject Naming Strategies
    2. Schema Registry REST API
    3. Monitoring Schema Registry
    4. Troubleshooting Schema Issues
  5. ksqlDB Fundamentals
    1. Introduction to ksqlDB
    2. Streams vs Tables in ksqlDB
    3. ksqlDB Architecture
  6. Querying and Processing
    1. Creating Streams and Tables
    2. Transformations and Filters
    3. Aggregations and Windows
    4. Stateful Processing in ksqlDB
  7. Joins and Enrichment
    1. Stream Stream Joins
    2. Stream Table Joins
    3. Enrichment Patterns
    4. Materialized Views
  8. ksqlDB in Production
    1. Deployment Models
    2. Scaling and Performance
    3. Fault Tolerance
    4. Operational Best Practices
  9. Kafka REST Proxy
    1. REST Proxy Fundamentals
    2. When to Use REST vs Native Clients
    3. Producing Messages via REST
    4. Consuming Messages via REST
    5. Serialization and Schema Registry with REST
  10. REST Proxy Operations and Ecosystem
    1. REST Proxy Deployment and Configuration
    2. Security Considerations
    3. Troubleshooting REST Proxy
    4. Kafka UI and Monitoring Tools
    5. Connectors and Integration Tools Overview
    6. Ecosystem Tooling Landscape