2.2 – Inter-Broker Encryption

Overview

This module is the authoritative source for securing broker-to-broker communication in Kafka. By the end, you will understand:

Why inter-broker encryption matters
How replication and controller traffic flows between brokers
How to configure inter-broker TLS encryption
Mixed listener setups for phased rollouts

This module covers inter-broker encryption ONLY. Client-broker encryption is covered in Module 2.1. Inter-broker authentication is covered in Module 3.

All inter-broker TLS concepts live here. Other modules will reference this but not re-explain it.

Why Inter-Broker Encryption Matters

What is Inter-Broker Communication?

Kafka brokers communicate with each other for:

Replication: Followers fetch data from leaders to maintain replicas.
Controller Operations: The controller sends LeaderAndIsr, UpdateMetadata requests to brokers.
Metadata Exchange: Brokers share cluster state (e.g., broker registration, topic configs).
Partition Reassignment: Brokers coordinate during manual partition moves.

All of this traffic is unencrypted by default.

Threat: Replication Traffic Snooping

Scenario:

An attacker gains access to the internal network (e.g., compromised VM in the same AWS VPC).
The attacker runs a packet sniffer (tcpdump, Wireshark) on the network.
The attacker captures all inter-broker replication traffic.
Result: The attacker can read ALL topics, even if client-broker traffic is encrypted.

┌─────────────┐                  ┌─────────────┐
│  Broker 1   │                  │  Broker 2   │
│  (Leader)   │─────────────────►│ (Follower)  │
│             │  Replication     │             │
│  Topic: PII │  (Unencrypted)   │  Topic: PII │
└─────────────┘                  └─────────────┘
       ▲
       │ Network Sniffer (Attacker)
       │ Captures ALL replicated data

Without inter-broker encryption: Even if clients use TLS, replication traffic is exposed.

With inter-broker encryption: Replication traffic is encrypted, protecting data from internal network attackers.

Threat: Rogue Broker Attack

Scenario:

An attacker compromises a machine in the Kafka network.
The attacker starts a rogue Kafka broker and adds it to the cluster.
The rogue broker requests replication from legitimate brokers.
Result: The rogue broker receives all topic data.

Without inter-broker authentication: Any broker can join the cluster and replicate data.

With inter-broker mutual TLS (mTLS): Brokers verify each other's identity using certificates, preventing rogue brokers.

Inter-Broker Communication Flow

Replication Traffic (Leader → Follower)

┌─────────────────────────────────────────────────────────┐
│              Kafka Cluster (3 Brokers)                  │
│                                                           │
│  Topic: orders, Partition 0                              │
│                                                           │
│  ┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│  │  Broker 1   │      │  Broker 2   │      │  Broker 3   │
│  │  (Leader)   │─────►│ (Follower)  │      │ (Follower)  │
│  │             │  1   │             │      │             │
│  │  Log:       │      │  Log:       │      │  Log:       │
│  │  [msg1,2,3] │      │  [msg1,2,3] │      │  [msg1,2,3] │
│  └─────────────┘      └─────────────┘      └─────────────┘
│         │                                          ▲
│         │                                          │
│         └──────────────────────────────────────────┘
│                       2 (Replication)
│
└─────────────────────────────────────────────────────────┘

Steps:

Producer writes to Broker 1 (leader for partition 0).
Broker 1 (leader) writes message to local log.
Broker 2 and 3 (followers) fetch new messages from Broker 1 via inter-broker replication.
Followers write to local log and acknowledge to leader.

Risk: Step 3 (replication) travels over the network in plaintext by default.

Controller Traffic (Controller → Brokers)

┌─────────────────────────────────────────────────────────┐
│              Kafka Cluster (3 Brokers)                  │
│                                                           │
│  ┌─────────────┐                                        │
│  │  Broker 1   │                                        │
│  │ (Controller)│                                        │
│  │             │                                        │
│  └──────┬──────┘                                        │
│         │ LeaderAndIsr, UpdateMetadata                  │
│         │                                                │
│         ├────────────────┬────────────────────┐        │
│         ▼                ▼                    ▼        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │
│  │  Broker 1   │  │  Broker 2   │  │  Broker 3   │  │
│  │             │  │             │  │             │  │
│  └─────────────┘  └─────────────┘  └─────────────┘  │
│                                                           │
└─────────────────────────────────────────────────────────┘

Controller Responsibilities:

Sends LeaderAndIsr requests (e.g., when a leader fails, controller elects a new leader).
Sends UpdateMetadata requests (e.g., when topic configs change).

Risk: Controller traffic contains sensitive cluster metadata (e.g., broker IDs, partition assignments).

Configuring Inter-Broker TLS Encryption

Approach: Separate Listener for Inter-Broker Traffic

Kafka allows you to define separate listeners for:

Client traffic (external producers/consumers)
Inter-broker traffic (replication, controller)

This allows phased rollouts: encrypt inter-broker traffic first, then client traffic.

Configuration Example: SSL-Only Inter-Broker

Broker Configuration (server.properties):

# Define three listeners:
# - PLAINTEXT for internal clients (port 9092)
# - SSL for external clients (port 9093)
# - SSL_INTERNAL for inter-broker (port 9094)
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093,SSL_INTERNAL://0.0.0.0:9094

# Map listener names to security protocols
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SSL_INTERNAL:SSL

# Advertised listeners (clients/brokers use these)
advertised.listeners=PLAINTEXT://kafka1.example.com:9092,SSL://kafka1.example.com:9093,SSL_INTERNAL://kafka1-internal.example.com:9094

# Inter-broker listener (use SSL_INTERNAL)
inter.broker.listener.name=SSL_INTERNAL

# SSL configuration (applies to SSL and SSL_INTERNAL listeners)
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

# Optional: Use separate keystore for inter-broker traffic
# ssl.keystore.location=/var/private/ssl/kafka.inter-broker.keystore.jks

Key Parameters:

Parameter	Purpose
`listeners`	Define all listeners (client + inter-broker)
`listener.security.protocol.map`	Map listener names to protocols (PLAINTEXT, SSL, SASL_SSL)
`advertised.listeners`	Advertise listener addresses to clients and brokers
`inter.broker.listener.name`	Specify which listener is used for inter-broker communication
`ssl.keystore.location`	Keystore for broker's private key + certificate

Legacy Configuration: `security.inter.broker.protocol`

Note for Kafka < 0.10.x: Older Kafka versions (pre-0.10.0) used security.inter.broker.protocol instead of inter.broker.listener.name to configure inter-broker security.

Legacy Config (Kafka < 0.10.0):

# Old way (single protocol for all inter-broker communication)
security.inter.broker.protocol=SSL

# SSL configuration (applies to inter-broker traffic)
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

Modern Config (Kafka 0.10.0+):

# New way (supports multiple listeners with different protocols)
listeners=PLAINTEXT://0.0.0.0:9092,SSL_INTERNAL://0.0.0.0:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL_INTERNAL:SSL
inter.broker.listener.name=SSL_INTERNAL

# SSL configuration
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

Why the Change?

The legacy security.inter.broker.protocol approach had limitations:

Single Protocol Only: All inter-broker communication had to use the same protocol (e.g., SSL). You couldn't have different security for client vs inter-broker traffic.
No Listener Separation: Couldn't use different ports or network interfaces for client vs inter-broker traffic.
Limited Flexibility: Couldn't support mixed security during migrations (e.g., PLAINTEXT for some brokers, SSL for others).

The modern inter.broker.listener.name approach enables:

Multiple Listeners: Define separate listeners for clients, inter-broker, and external access.
Independent Security: Client traffic can use SASL_SSL, inter-broker can use SSL, external can use PLAINTEXT (for gradual rollout).
Network Segmentation: Inter-broker traffic can use internal network (e.g., 10.0.0.0/8), client traffic uses public network.

Migration Path (Legacy → Modern):

Step 1: Add new listener alongside legacy config:

# Keep legacy config (for backward compatibility)
security.inter.broker.protocol=SSL

# Add new listener (for future)
listeners=PLAINTEXT://0.0.0.0:9092,SSL_INTERNAL://0.0.0.0:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL_INTERNAL:SSL

Step 2: Test new listener manually:

openssl s_client -connect broker1:9094

Step 3: Switch to new listener:

# Remove legacy config
# security.inter.broker.protocol=SSL

# Use modern config
inter.broker.listener.name=SSL_INTERNAL

Step 4: Rolling restart brokers.

Recommendation: Always use inter.broker.listener.name in new deployments (Kafka 0.10.0+). Only use security.inter.broker.protocol if you're stuck on Kafka < 0.10.0 (which is EOL and unsupported).

Compatibility Note: If you're upgrading from Kafka < 0.10.0 to 0.10.0+, the legacy config still works (for backward compatibility), but you should migrate to the modern approach for better flexibility.

Why Use a Separate Inter-Broker Listener?

Benefits:

Phased Rollout: Encrypt inter-broker traffic first, then client traffic (or vice versa).
Network Segmentation: Use internal hostnames (e.g., kafka1-internal.example.com) for inter-broker, external hostnames for clients.
Performance Tuning: Apply different security configs for internal vs external traffic (e.g., stronger ciphers for external).

Example Use Case:

Internal network: Use weaker (but faster) ciphers for inter-broker traffic.
External network: Use stronger ciphers for client traffic (internet-facing).

Configuring Inter-Broker Mutual TLS (mTLS) Authentication

Problem: TLS encryption alone doesn't verify broker identity. A rogue broker can still join the cluster.

Solution: Enable mutual TLS (mTLS), where brokers authenticate each other using certificates.

Configuration: mTLS for Inter-Broker

Broker Configuration (server.properties):

# Enable mTLS for inter-broker listener
inter.broker.listener.name=SSL_INTERNAL

# Require client authentication (mTLS)
listener.name.ssl_internal.ssl.client.auth=required

# Truststore for verifying other brokers' certificates
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=kafka123

# Keystore for this broker's certificate
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

Key Parameter:

Parameter	Value	Purpose
`listener.name.ssl_internal.ssl.client.auth`	`required`	Forces brokers to present a certificate for authentication
`ssl.truststore.location`	Path to truststore	Contains CA certificates for verifying other brokers

How mTLS Works for Inter-Broker

Broker 1 (leader) presents its certificate to Broker 2 (follower).
Broker 2 verifies Broker 1's certificate using its truststore.
Broker 2 presents its certificate to Broker 1.
Broker 1 verifies Broker 2's certificate using its truststore.
If both verifications succeed, brokers establish an encrypted session.
If either fails, the connection is rejected.

Result: Only brokers with valid certificates can join the cluster.

Mixed Listener Setups (Phased Rollout)

Scenario 1: Encrypt Inter-Broker Only (Client Traffic Remains Plaintext)

Use Case: Internal network is trusted, but want to protect replication traffic.

Configuration:

listeners=PLAINTEXT://0.0.0.0:9092,SSL_INTERNAL://0.0.0.0:9094
advertised.listeners=PLAINTEXT://kafka1.example.com:9092,SSL_INTERNAL://kafka1-internal.example.com:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL_INTERNAL:SSL
inter.broker.listener.name=SSL_INTERNAL

ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

Result:

✅ Inter-broker traffic encrypted (replication, controller)
❌ Client traffic plaintext (producers, consumers)

Scenario 2: Encrypt Both Client and Inter-Broker

Use Case: Production deployment with full encryption.

Configuration:

listeners=SSL://0.0.0.0:9093,SSL_INTERNAL://0.0.0.0:9094
advertised.listeners=SSL://kafka1.example.com:9093,SSL_INTERNAL://kafka1-internal.example.com:9094
listener.security.protocol.map=SSL:SSL,SSL_INTERNAL:SSL
inter.broker.listener.name=SSL_INTERNAL

ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=kafka123
ssl.key.password=kafka123

Result:

✅ Inter-broker traffic encrypted
✅ Client traffic encrypted

Scenario 3: Gradual Migration (PLAINTEXT → SSL for Inter-Broker)

Step 1: Add SSL_INTERNAL listener alongside existing PLAINTEXT inter-broker listener.

listeners=PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://0.0.0.0:9091,SSL_INTERNAL://0.0.0.0:9094
inter.broker.listener.name=PLAINTEXT_INTERNAL

Step 2: Test SSL_INTERNAL connectivity manually:

openssl s_client -connect kafka1-internal.example.com:9094

Step 3: Switch inter-broker listener to SSL_INTERNAL:

inter.broker.listener.name=SSL_INTERNAL

Step 4: Rolling restart brokers (one at a time).

Step 5: Remove PLAINTEXT_INTERNAL listener after migration:

listeners=PLAINTEXT://0.0.0.0:9092,SSL_INTERNAL://0.0.0.0:9094

Performance Impact of Inter-Broker Encryption

Replication Throughput

Without TLS: Replication is CPU-light (network-bound).

With TLS: Replication becomes CPU-bound due to encryption/decryption.

Typical Impact:

Replication Lag: Increases by 10-20% under high load.
CPU Usage: Increases by 15-25% on broker processes.

Mitigation:

Use CPUs with AES-NI (hardware-accelerated AES encryption).
Use TLS 1.3 (faster than TLS 1.2).
Monitor replication lag: kafka.server:type=FetcherLagMetrics,name=ConsumerLag

Replication Lag Monitoring

JMX Metric:

kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=Replica,topic=<topic>,partition=<partition>

Alerting Rule (Prometheus):

- alert: HighReplicationLag
  expr: kafka_server_fetcherlagmetrics_consumerlag > 10000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Replication lag > 10K messages for 5 minutes"

Action: If replication lag increases significantly after enabling TLS, consider adding more brokers or upgrading CPUs.

Certificate Management for Inter-Broker mTLS

Single CA Approach (Recommended)

Setup:

Create a Certificate Authority (CA) for the cluster.
Issue a certificate for each broker, signed by the CA.
Each broker's truststore contains the CA certificate.

Benefits:

Easy to add new brokers (just issue a new cert from the CA).
Easy to rotate certificates (brokers trust the CA, not individual certs).

Example Workflow:

# 1. Create CA
openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650

# 2. Generate broker keystore and cert signing request (CSR)
keytool -keystore kafka.broker1.keystore.jks -alias broker1 -keyalg RSA -genkey

# 3. Export CSR
keytool -keystore kafka.broker1.keystore.jks -alias broker1 -certreq -file broker1-csr

# 4. Sign CSR with CA
openssl x509 -req -CA ca-cert -CAkey ca-key -in broker1-csr -out broker1-signed -days 365 -CAcreateserial

# 5. Import CA cert into broker keystore
keytool -keystore kafka.broker1.keystore.jks -alias CARoot -import -file ca-cert

# 6. Import signed cert into broker keystore
keytool -keystore kafka.broker1.keystore.jks -alias broker1 -import -file broker1-signed

# 7. Import CA cert into broker truststore
keytool -keystore kafka.broker.truststore.jks -alias CARoot -import -file ca-cert

Result: All brokers trust each other because they're all signed by the same CA.

Multi-CA Approach (Enterprise)

Use Case: Multi-datacenter deployments where each datacenter has its own CA.

Setup:

Truststore contains certificates from all CAs.
Brokers can authenticate brokers from other datacenters.

Example:

Datacenter 1 CA: dc1-ca-cert
Datacenter 2 CA: dc2-ca-cert
Truststore: Contains both dc1-ca-cert and dc2-ca-cert

Troubleshooting Inter-Broker TLS

Issue 1: "SSLHandshakeException" in Broker Logs

Symptom:

[2023-10-27 10:00:00,123] ERROR [ReplicaFetcher broker=1 partition=orders-0] Failed to fetch from broker-2:9094 (org.apache.kafka.clients.FetchSessionHandler)
javax.net.ssl.SSLHandshakeException: PKIX path validation failed

Cause: Broker 1's truststore doesn't contain Broker 2's CA certificate.

Fix: Import Broker 2's CA cert into Broker 1's truststore.

Issue 2: Under-Replicated Partitions After Enabling TLS

Symptom:

kafka-topics --describe --topic orders --bootstrap-server localhost:9092
Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1  ← Missing 2 and 3

Cause: Followers (Broker 2, 3) can't fetch from leader (Broker 1) due to TLS misconfiguration.

Fix:

Check inter-broker listener is configured: inter.broker.listener.name=SSL_INTERNAL
Verify keystore/truststore paths are correct.
Test inter-broker connectivity:

openssl s_client -connect broker1-internal:9094 -showcerts

Issue 3: High CPU After Enabling Inter-Broker TLS

Symptom: Broker CPU jumps from 30% to 80% after enabling TLS.

Cause: TLS encryption/decryption is CPU-intensive.