6.1 – Broker-Level Configurations

What are Broker-Level Configurations?

Broker-level configurations are settings that apply to the entire broker (and serve as defaults for all topics hosted on that broker).

Where Configured: server.properties file (or ZooKeeper/KRaft metadata)

Scope: Applies to:

• All topics on the broker (unless overridden at topic level)
• Broker behavior (network, disk, replication, etc.)
• Cluster-wide defaults

Configuration File: server.properties

Location: /config/server.properties (Kafka distribution)

Example:

# Broker Identity
broker.id=1
listeners=PLAINTEXT://:9092

# Log Storage
log.dirs=/var/lib/kafka-logs

# Retention Defaults
log.retention.hours=168
log.retention.bytes=-1

# Segment Defaults
log.segment.bytes=1073741824
log.segment.ms=604800000

# Replication Defaults
default.replication.factor=3
min.insync.replicas=2

# Network
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400

Core Broker Configurations

1. Broker Identity

Why broker.id Matters:

• Controller uses broker ID to track which broker is responsible for which partitions
• If broker restarts with same ID, it rejoins with same partition assignments
• If broker restarts with different ID, Kafka treats it as a new broker (triggers rebalance)

2. Network Listeners

Example (multiple listeners):

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://kafka1.internal:9092,SSL://kafka1.external:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

Use Case: Internal clients use PLAINTEXT (port 9092), external clients use SSL (port 9093).

3. Storage Configuration

Example (multiple disks):

log.dirs=/mnt/disk1/kafka,/mnt/disk2/kafka,/mnt/disk3/kafka

Behavior: Kafka distributes partitions across directories (JBOD).

Recommendation: Use multiple disks for parallelism (not RAID).

4. Retention Defaults (Broker-Wide)

These apply to all topics unless overridden at the topic level.

Precedence: log.retention.ms > log.retention.minutes > log.retention.hours

Example:

log.retention.ms=604800000  # 7 days
log.retention.bytes=-1       # Infinite size

5. Segment Defaults

Why These Matter:

• Smaller segments = finer-grained retention (but more files)
• Larger segments = fewer files (but coarser-grained retention)

6. Cleanup Policy Defaults

Example:

log.cleanup.policy=delete
log.cleaner.enable=true
log.cleaner.threads=2  # Use 2 threads for compaction

7. Replication Defaults

Replication Fetch Configurations

These configs control how follower replicas sync data from leaders:

How Replication Fetch Works:

Follower (Broker 2)                Leader (Broker 1)
       │                                  │
       ├──────Fetch Request───────────────▶│
       │  (partition: orders-0,            │
       │   fetchOffset: 1000,              │
       │   maxBytes: 1048576)              │
       │                                   │
       │                          ┌────────▼────────┐
       │                          │ Read from log   │
       │                          │ (offsets 1000+) │
       │                          └────────┬────────┘
       │                                   │
       │◀──────Fetch Response───────────────┤
       │  (records: 1000-1500,             │
       │   size: 800 KB)                   │
       │                                   │
       ▼                                   │
┌──────────────────┐                      │
│ Write to local   │                      │
│ replica log      │                      │
└──────────────────┘                      │

Key Point: Replication is follower-driven (followers actively fetch from leaders, not push-based).

Tuning Replication Performance

1. Increase num.replica.fetchers for High-Throughput Topics

Problem: Single replication thread can become a bottleneck.

Solution: Increase threads (especially if you have many partitions):

num.replica.fetchers=4   # Use 4 threads per broker

Benefit: Each thread can fetch different partitions in parallel.

Recommendation:

• Small clusters (< 100 partitions): num.replica.fetchers=1-2
• Large clusters (100-1000 partitions): num.replica.fetchers=4-8
• Very large clusters (1000+ partitions): num.replica.fetchers=8-16

2. Increase replica.fetch.max.bytes for Large Messages

Problem: Default 1 MB may be too small if messages are large.

Solution: Increase fetch size:

replica.fetch.max.bytes=10485760  # 10 MB

Benefit: Fewer fetch requests needed (higher throughput).

Trade-off: More memory usage per fetch thread.

3. Adjust replica.lag.time.max.ms for Network Latency

Problem: Default 30s may be too aggressive for high-latency networks (cross-region).

Solution: Increase timeout:

replica.lag.time.max.ms=60000  # 60 seconds

Use Case: Cross-datacenter replication (high latency).

Warning: Longer timeout means slower detection of failed replicas.

4. Tune replica.fetch.wait.max.ms for Low-Latency Replication

Problem: Default 500ms wait time adds latency to replication.

Solution: Reduce wait time:

replica.fetch.wait.max.ms=100  # 100ms

Benefit: Faster replication (lower lag).

Trade-off: More CPU usage (more frequent fetch requests).

Production Recommendation:

default.replication.factor=3
min.insync.replicas=2
num.replica.fetchers=4             # Parallel replication
replica.fetch.max.bytes=1048576    # 1 MB (default, adjust if needed)
replica.lag.time.max.ms=30000      # 30 seconds

Why min.insync.replicas=2?

With RF=3 and min.insync.replicas=2:

• At least 2 replicas must be in-sync for writes to succeed
• Can tolerate 1 broker failure (2 replicas still available)
• Prevents data loss (message is on ≥2 brokers before ack)

7.5. Topic Auto-Creation Configuration

What is Auto-Creation?

Kafka can automatically create topics when:

• A producer starts writing messages to a non-existent topic
• A consumer starts reading messages from a non-existent topic
• Any client requests metadata for a non-existent topic

Example:

// Producer writes to non-existent topic "orders"
producer.send(new ProducerRecord<>("orders", "key", "value"));

// If auto.create.topics.enable=true:
// Kafka automatically creates topic "orders" with:
// - Partitions: num.partitions (default: 1)
// - Replication Factor: default.replication.factor (default: 1)

The Auto-Creation Problem

Default values are dangerous for production:

auto.create.topics.enable=true    # Enabled by default
num.partitions=1                  # Only 1 partition!
default.replication.factor=1      # No fault tolerance!

Result: Auto-created topics have:

• ❌ 1 partition → No parallelism (bottleneck)
• ❌ RF=1 → No fault tolerance (broker crash = data loss)
• ❌ No custom configs → Uses broker defaults (may not match requirements)

Example Failure Scenario:

1. Developer writes code: producer.send(new ProducerRecord<>("analytics-events", ...))
2. Kafka auto-creates topic "analytics-events" (1 partition, RF=1)
3. Production traffic hits 1000 msg/sec (bottleneck on single partition)
4. Broker hosting partition crashes → all data lost (RF=1)

Production Best Practice: Disable Auto-Creation

Recommended Config:

auto.create.topics.enable=false   # Disable auto-creation

Why Disable?

• Prevents accidental under-configured topics (1 partition, RF=1)
• Forces explicit topic creation with proper settings
• Avoids production surprises (typos in topic names create unwanted topics)

Alternative: Set Better Defaults

If you must keep auto-creation enabled:

auto.create.topics.enable=true
num.partitions=10                 # Better default (some parallelism)
default.replication.factor=3      # Production-ready RF

Trade-off: Still risky (topics created without custom configs like retention, compaction).

How to Handle Auto-Creation in Different Environments

Checking if a Topic Was Auto-Created

View topic configs:

kafka-topics --describe --topic analytics-events --bootstrap-server localhost:9092

Output:

Topic: analytics-events
PartitionCount: 1        ← Warning: Only 1 partition (likely auto-created)
ReplicationFactor: 1     ← Warning: No fault tolerance (likely auto-created)
Configs: 

If you see 1 partition and RF=1, the topic was likely auto-created with defaults.

Fixing Auto-Created Topics

You cannot directly change partition count or RF after creation, but you can:

Option 1: Increase Partitions (breaks key-to-partition mapping):

kafka-topics --alter --topic analytics-events --partitions 10 --bootstrap-server localhost:9092

Option 2: Delete and Recreate (data loss):

kafka-topics --delete --topic analytics-events --bootstrap-server localhost:9092
kafka-topics --create --topic analytics-events --partitions 10 --replication-factor 3 --bootstrap-server localhost:9092

Option 3: Create New Topic and Migrate (safest):

kafka-topics --create --topic analytics-events-v2 --partitions 10 --replication-factor 3 --bootstrap-server localhost:9092
# Dual-write to both topics, migrate consumers, delete old topic

Monitoring Auto-Created Topics

Alert on topics with RF=1:

kafka-topics --describe --bootstrap-server localhost:9092 | grep "ReplicationFactor: 1"

Recommendation: Set up alerts for any topic with RF < 3 in production.

8. Request Handling

Tuning:

• High throughput: Increase num.io.threads (e.g., 16)
• Many clients: Increase num.network.threads (e.g., 8)

9. Controller Configuration

Why controlled.shutdown.enable=true?

Graceful shutdown:

1. Broker transfers partition leadership to other brokers
2. Broker flushes logs to disk
3. Broker exits cleanly

Without controlled shutdown:

• Broker exits immediately (clients see connection errors)
• Controller must detect failure and elect new leaders (5-10 seconds of unavailability)

10. ZooKeeper Configuration (Legacy)

Note: For KRaft mode (Kafka 3.0+), these configs are not needed (no ZooKeeper dependency).

11. KRaft Configuration (Kafka 3.0+)

Example (KRaft mode, combined broker + controller):

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093

Broker Configuration Best Practices

1. Use Multiple log.dirs (JBOD)

Why: Parallelism (distribute partitions across disks), no RAID overhead.

Example:

log.dirs=/mnt/disk1/kafka,/mnt/disk2/kafka,/mnt/disk3/kafka

Benefit: If one disk fails, only partitions on that disk are lost (Kafka replication handles it).

2. Set default.replication.factor=3

Why: RF=1 = no fault tolerance. RF=2 = risky (split-brain). RF=3 = good balance.

Example:

default.replication.factor=3

Note: This only affects auto-created topics. Explicitly created topics use their own RF.

3. Set min.insync.replicas=2

Why: Prevents data loss when using acks=all (ensures message is on ≥2 brokers).

Example:

min.insync.replicas=2

Combination:

default.replication.factor=3
min.insync.replicas=2

Result: Can tolerate 1 broker failure (2 replicas remain, writes succeed).

4. Enable controlled.shutdown.enable=true

Why: Graceful shutdown minimizes client impact (no connection errors).

Example:

controlled.shutdown.enable=true

5. Tune num.io.threads for Throughput

Why: More threads = more parallelism for request processing.

Default: 8 (good for most workloads)

High Throughput: 16-32

Example:

num.io.threads=16

6. Monitor Disk Usage

Configs to Watch:

• log.retention.ms (how long data is kept)
• log.retention.bytes (max size per partition)

Rule: Set retention ≤ 70% of disk capacity (leave room for growth).

Example:

Disk: 1 TB
Usable: 700 GB (70%)

If throughput = 10 GB/day:
Max retention = 700 GB / 10 GB/day = 70 days

Common Broker Configurations (Full Example)

Production Broker Config (server.properties):

############################# Broker Identity #############################
broker.id=1
broker.rack=us-east-1a

############################# Network #############################
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka1.example.com:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT

num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
queued.max.requests=500

############################# Storage #############################
log.dirs=/mnt/disk1/kafka,/mnt/disk2/kafka,/mnt/disk3/kafka

############################# Retention Defaults #############################
log.retention.ms=604800000  # 7 days
log.retention.bytes=-1

log.segment.bytes=1073741824  # 1 GB
log.segment.ms=604800000      # 7 days

############################# Cleanup #############################
log.cleanup.policy=delete
log.cleaner.enable=true
log.cleaner.threads=2

############################# Replication #############################
default.replication.factor=3
min.insync.replicas=2
replica.lag.time.max.ms=10000
num.replica.fetchers=4

############################# Controller #############################
controlled.shutdown.enable=true
controlled.shutdown.max.retries=3

############################# ZooKeeper (Legacy) #############################
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
zookeeper.connection.timeout.ms=18000
zookeeper.session.timeout.ms=18000

When to Change Broker Configs

Common Scenarios:

Viewing Current Broker Configs

Check Effective Config

kafka-broker-api-versions --bootstrap-server localhost:9092 --describe

Or (via JMX):

# View broker config via JMX
jconsole <broker-host>:9999

Or (via ZooKeeper, legacy):

kafka-configs --zookeeper localhost:2181 --entity-type brokers --entity-name 1 --describe

Key Takeaways

1. Broker Configs = Cluster-Wide Defaults: Broker configs apply to all topics (unless overridden at topic level).

2. Core Configs: Broker ID, listeners, storage (log.dirs), retention, replication, request handling.

3. Production Defaults: RF=3, min.insync.replicas=2, controlled shutdown enabled.

4. JBOD: Use multiple log.dirs for parallelism (no RAID needed).

5. Tuning: Increase num.io.threads for throughput, num.network.threads for many clients.

6. Controlled Shutdown: Always enable for graceful broker restarts (minimizes client impact).

7. Retention Planning: Set retention ≤ 70% of disk capacity (leave room for growth).

8. KRaft vs ZooKeeper: KRaft mode eliminates ZooKeeper configs (Kafka 3.0+).

What's Next?

Next Module: 6.2 – Topic-Level Configurations

What We Covered:

• ✅ Broker-level configuration overview
• ✅ Core broker configs (identity, network, storage, retention, replication)
• ✅ Best practices (JBOD, RF=3, min.insync.replicas=2, controlled shutdown)
• ✅ Tuning for throughput and many clients
• ✅ Production broker config example

What We Didn't Cover (Explicitly Deferred):

• ❌ Topic-level configuration overrides → Module 6.2
• ❌ Config precedence rules → Module 6.3
• ❌ Dynamic config updates → Module 7.1
• ❌ Performance tuning deep dive → M6: Operations

Broker-Level Configurations: Complete ✅