2.3 – Certificates, Keystores & Truststores

Overview

This module provides the practical, hands-on guide to managing certificates, keystores, and truststores for Kafka TLS encryption. By the end, you will understand:

• What certificates, keystores, and truststores are
• How certificate chains and trust hierarchies work
• Certificate generation workflows (self-signed vs CA-signed)
• Certificate rotation strategies for zero-downtime updates

This module is 100% practical. TLS theory is covered in Module 2.1. This module focuses on how to actually create, manage, and rotate certificates.

All certificate management concepts live here. Other modules reference this but don't re-explain certificate workflows.

Certificates, Keystores, and Truststores: Definitions

Certificate

A certificate is a digital document that binds a public key to an identity (e.g., a broker's hostname).

Contents:

• Public Key: Used to encrypt data sent to the certificate owner.
• Identity Information: Common Name (CN), Organization (O), etc.
• Issuer: Who signed the certificate (CA or self-signed).
• Validity Period: Not valid before/after dates.
• Signature: Cryptographic signature from the issuer (proves authenticity).

File Formats:

• PEM: Text format (-----BEGIN CERTIFICATE-----)
• DER: Binary format
• JKS: Java KeyStore format (contains private key + cert)

Keystore

A keystore is a secure container that holds:

• Private Key: The broker's secret key (never shared).
• Certificate: The broker's public certificate.
• (Optional) Certificate Chain: Intermediate CA certs leading to root CA.

Purpose: The broker uses the keystore to prove its identity during TLS handshakes.

Analogy: A keystore is like a passport (contains your identity + photo) that proves who you are.

File Format: Typically .jks (Java KeyStore) or .p12 (PKCS#12).

Truststore

A truststore is a secure container that holds:

• CA Certificates: Public certificates of trusted Certificate Authorities.
• (Optional) Broker Certificates: Specific broker certs (if not using a CA).

Purpose: The client/broker uses the truststore to verify the authenticity of the remote party's certificate.

Analogy: A truststore is like a list of trusted passport issuers (e.g., only trust passports issued by the U.S. State Department).

File Format: Typically .jks or .p12.

Keystore vs. Truststore Comparison

Certificate Chains and Trust Hierarchies

Certificate Chain

A certificate chain is a sequence of certificates where each certificate is signed by the next:

Root CA Certificate (self-signed, trusted by clients)
    │
    └─► Intermediate CA Certificate (signed by Root CA)
            │
            └─► Broker Certificate (signed by Intermediate CA)

Why Use a Chain?

• Security: If the Intermediate CA is compromised, revoke it without invalidating the Root CA.
• Scalability: Issue many broker certs from Intermediate CA without using Root CA frequently.

Example: Broker Certificate Chain

Broker Keystore Contains:

1. broker-cert.pem: Broker's certificate (CN=kafka1.example.com)
2. intermediate-ca-cert.pem: Intermediate CA certificate
3. root-ca-cert.pem: Root CA certificate

Client Truststore Contains:

• root-ca-cert.pem: Root CA certificate (client trusts this)

Trust Validation Flow:

1. Broker presents broker-cert to client.
2. Client checks: Is broker-cert signed by intermediate-ca-cert?
3. Client checks: Is intermediate-ca-cert signed by root-ca-cert?
4. Client checks: Is root-ca-cert in my truststore? ✅ Yes → Trust established.

Self-Signed vs. CA-Signed Certificates

Self-Signed Certificates

Definition: A certificate where the issuer and subject are the same (e.g., broker signs its own certificate).

Pros:

• Free: No cost for certificate issuance.
• Fast: Generate instantly with keytool or openssl.
• Good for Testing: Ideal for local development/testing.

Cons:

• Not Trusted by Default: Clients must manually import the self-signed cert into their truststore.
• Manual Management: No automated renewal, expiration tracking.
• Not Suitable for Production: No revocation mechanism if private key is compromised.

Use Case: Local development, testing, internal PoC.

CA-Signed Certificates

Definition: A certificate signed by a trusted Certificate Authority (e.g., Let's Encrypt, DigiCert, internal CA).

Pros:

• Trusted by Default: Clients already trust the CA (if using public CA like Let's Encrypt).
• Automated Renewal: Tools like Cert-Manager (Kubernetes) or ACME (Let's Encrypt) automate renewal.
• Revocation Support: Can revoke certificates via CRL or OCSP if compromised.

Cons:

• Cost: Public CAs charge per certificate (except Let's Encrypt, which is free).
• Complexity: Requires CA infrastructure (if using internal CA).

Use Case: Production deployments, especially internet-facing or multi-tenant clusters.

Certificate Generation Workflows

Workflow 1: Self-Signed Certificate (Testing)

Step 1: Generate Keystore with Self-Signed Certificate

keytool -keystore kafka.server.keystore.jks \
  -alias kafka-broker \
  -keyalg RSA \
  -keysize 2048 \
  -validity 365 \
  -genkey \
  -storepass kafka123 \
  -keypass kafka123 \
  -dname "CN=kafka1.example.com, OU=Engineering, O=Example Inc, L=San Francisco, ST=CA, C=US"

Output: kafka.server.keystore.jks (contains private key + self-signed cert)

Step 2: Export Certificate from Keystore

keytool -exportcert \
  -alias kafka-broker \
  -keystore kafka.server.keystore.jks \
  -rfc \
  -file kafka-broker-cert.pem \
  -storepass kafka123

Output: kafka-broker-cert.pem (broker's certificate in PEM format)

Step 3: Create Client Truststore and Import Broker Cert

keytool -importcert \
  -alias kafka-broker \
  -file kafka-broker-cert.pem \
  -keystore kafka.client.truststore.jks \
  -storepass client123 \
  -noprompt

Output: kafka.client.truststore.jks (contains broker's cert for trust validation)

Workflow 2: CA-Signed Certificate (Production)

Step 1: Generate Private Key and Certificate Signing Request (CSR)

# Generate private key
openssl genrsa -out kafka-broker.key 2048

# Generate CSR
openssl req -new \
  -key kafka-broker.key \
  -out kafka-broker.csr \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example Inc/OU=Engineering/CN=kafka1.example.com"

Output:

• kafka-broker.key: Private key (keep secure!)
• kafka-broker.csr: Certificate Signing Request (send to CA)

Step 2: Submit CSR to Certificate Authority

Option A: Public CA (Let's Encrypt)

Use Certbot (automated):

certbot certonly --standalone -d kafka1.example.com

Output (in /etc/letsencrypt/live/kafka1.example.com/):

• privkey.pem: Private key
• fullchain.pem: Certificate + intermediate CA cert
• chain.pem: Intermediate CA cert
• cert.pem: Certificate only

Option B: Internal CA

Submit kafka-broker.csr to your internal CA (e.g., HashiCorp Vault, AWS Private CA).

Example with OpenSSL (self-hosted CA):

# CA signs the CSR
openssl x509 -req \
  -in kafka-broker.csr \
  -CA ca-cert.pem \
  -CAkey ca-key.pem \
  -CAcreateserial \
  -out kafka-broker-signed.crt \
  -days 365 \
  -sha256

Output: kafka-broker-signed.crt (CA-signed certificate)

Step 3: Create Keystore from Private Key + Signed Certificate

# Convert to PKCS#12 format (Java-compatible)
openssl pkcs12 -export \
  -in kafka-broker-signed.crt \
  -inkey kafka-broker.key \
  -out kafka.server.keystore.p12 \
  -name kafka-broker \
  -password pass:kafka123

# (Optional) Convert PKCS#12 to JKS
keytool -importkeystore \
  -srckeystore kafka.server.keystore.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass kafka123 \
  -destkeystore kafka.server.keystore.jks \
  -deststoretype JKS \
  -deststorepass kafka123

Output: kafka.server.keystore.jks (contains private key + CA-signed cert)

Step 4: Create Client Truststore with CA Certificate

keytool -importcert \
  -alias ca-root \
  -file ca-cert.pem \
  -keystore kafka.client.truststore.jks \
  -storepass client123 \
  -noprompt

Output: kafka.client.truststore.jks (contains CA cert for trust validation)

Key Point: Clients only need the CA certificate in their truststore, not individual broker certificates.

Certificate Rotation Strategies

Why Rotate Certificates?

Reasons:

1. Expiration: Certificates have a validity period (e.g., 90 days for Let's Encrypt, 1 year for DigiCert).
2. Security Best Practice: Limit blast radius if a private key is compromised.
3. Compliance: Some regulations (PCI-DSS) require annual certificate rotation.

Strategy 1: Rolling Restart (Downtime Minimal)

Scenario: Replace expiring certificate across all brokers.

Steps:

1. Generate New Certificate for each broker (following Workflow 2).
2. Update Keystore on Broker 1.
3. Restart Broker 1 (new cert takes effect).
4. Wait for Broker 1 to Catch Up (check kafka-topics --describe for ISR).
5. Repeat for Broker 2, 3, …

Downtime: None (cluster remains available during rolling restart).

Caveat: Clients may see brief connection resets when brokers restart.

Strategy 2: Dual Keystore (Zero Downtime)

Scenario: Support both old and new certificates simultaneously during transition.

Kafka Limitation: Kafka does NOT support multiple certificates per listener.

Workaround: Use a load balancer (e.g., HAProxy, Envoy) that terminates TLS and presents new cert to clients while forwarding to brokers.

Use Case: Large-scale deployments where even brief connection resets are unacceptable.

Strategy 3: Let's Encrypt Automated Renewal

Setup: Use Certbot with a cron job to renew certificates before expiration.

Cron Job (runs daily):

0 2 * * * certbot renew --quiet --deploy-hook "systemctl restart kafka"

Behavior:

• Certbot checks if cert is expiring within 30 days.
• If yes, renew cert and restart Kafka (via deploy hook).
• If no, do nothing.

Result: Certificates auto-renew with minimal manual intervention.

Common Certificate Issues and Troubleshooting

Issue 1: "PKIX path validation failed"

Symptom:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Cause: Client's truststore doesn't contain the broker's CA certificate.

Fix:

# Import CA cert into client truststore
keytool -importcert -alias ca-root -file ca-cert.pem \
  -keystore kafka.client.truststore.jks -storepass client123 -noprompt

Issue 2: "Certificate has expired"

Symptom:

sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Cause: Broker's certificate has expired.

Fix: Generate a new certificate and update keystore (see Workflow 2).

Check Expiration:

keytool -list -v -keystore kafka.server.keystore.jks -storepass kafka123 | grep "Valid from"

Expected Output:

Valid from: Fri Oct 27 10:00:00 PDT 2023 until: Sat Oct 26 10:00:00 PDT 2024

Issue 3: "Hostname verification failed"

Symptom:

java.security.cert.CertificateException: No subject alternative names matching IP address 10.0.1.5 found

Cause: Broker's certificate CN or SAN doesn't match the hostname/IP in bootstrap.servers.

Fix Option 1: Regenerate cert with correct CN/SAN:

openssl req -new -key kafka-broker.key -out kafka-broker.csr \
  -subj "/CN=kafka1.example.com" \
  -addext "subjectAltName=DNS:kafka1.example.com,IP:10.0.1.5"

Fix Option 2: Disable hostname verification (NOT recommended for production):

ssl.endpoint.identification.algorithm=

Issue 4: "Keystore was tampered with, or password was incorrect"

Symptom:

java.io.IOException: Keystore was tampered with, or password was incorrect

Cause: Incorrect keystore password in ssl.keystore.password.

Fix: Verify password:

keytool -list -keystore kafka.server.keystore.jks -storepass <password>

If password is correct, keystore may be corrupted. Regenerate keystore.

Certificate Management Best Practices

1. Use CA-Signed Certificates in Production

Why: Clients automatically trust public CAs, no manual truststore distribution.

How: Use Let's Encrypt (free, automated) or enterprise CA (DigiCert, GlobalSign).

2. Set Appropriate Validity Periods

Why Short Validity: Limits exposure if private key is compromised.

3. Use Separate Certificates per Broker

Why: If one broker's key is compromised, you only need to revoke one cert, not all brokers.

How: Generate unique CSR per broker with broker-specific CN (e.g., kafka1.example.com, kafka2.example.com).

4. Store Private Keys Securely

Best Practices:

• File Permissions: chmod 600 on keystore files (only Kafka user can read).
• Encryption: Store keystores in encrypted volumes (e.g., AWS EBS encryption, LUKS).
• Secret Management: Use HashiCorp Vault, AWS Secrets Manager to store keystore passwords.

5. Automate Certificate Renewal

Tools:

• Let's Encrypt + Certbot: Free, automated renewal for public-facing brokers.
• Cert-Manager (Kubernetes): Automates cert issuance/renewal for K8s-hosted Kafka.
• HashiCorp Vault: Centralized certificate issuance with auto-renewal.

Why: Manual renewal is error-prone and causes outages.

6. Monitor Certificate Expiration

JMX Metric (custom, requires instrumentation):

long daysUntilExpiry = (cert.getNotAfter().getTime() - System.currentTimeMillis()) / (1000 * 60 * 60 * 24);

Alerting Rule (Prometheus):

- alert: CertificateExpiringSoon
  expr: kafka_cert_days_until_expiry < 30
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate expires in < 30 days"

7. Use Subject Alternative Names (SAN) for Multi-Host Certs

Scenario: Broker accessible via multiple hostnames/IPs.

Example:

openssl req -new -key kafka-broker.key -out kafka-broker.csr \
  -subj "/CN=kafka1.example.com" \
  -addext "subjectAltName=DNS:kafka1.example.com,DNS:kafka1-internal.local,IP:10.0.1.5"

Result: Certificate is valid for:

• kafka1.example.com
• kafka1-internal.local
• 10.0.1.5

Quick SSL Testing with Confluent CLI and Docker

For rapid local testing, use Confluent's Docker images with pre-configured SSL to avoid manual certificate generation.

Option 1: Confluent CLI (Local Services)

Install Confluent CLI:

curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest
export PATH=$(pwd)/bin:$PATH

Start Kafka with SSL Enabled:

confluent local services start

What This Does:

• Starts ZooKeeper, Kafka, Schema Registry, Connect, ksqlDB with SSL pre-configured.
• Auto-generates self-signed certificates.
• Stores configs in ~/.confluent/.

Verify SSL:

# Check broker is listening on SSL port
netstat -tuln | grep 9093

# Produce test message with SSL
kafka-console-producer --bootstrap-server localhost:9093 \
  --topic test \
  --producer.config ~/.confluent/kafka.properties

Benefit: Zero manual certificate setup for local testing.

Option 2: Docker Compose with Pre-Configured SSL

Create docker-compose.yml:

version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - "9093:9093"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: SSL://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: SSL://localhost:9093
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: key_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: truststore_creds
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./secrets:/etc/kafka/secrets

Generate Certificates (one-time setup):

mkdir secrets
cd secrets

# Generate keystore with self-signed cert
keytool -keystore kafka.server.keystore.jks -alias localhost \
  -keyalg RSA -genkey -storepass kafka123 -keypass kafka123 \
  -dname "CN=localhost"

# Export cert
keytool -exportcert -alias localhost -keystore kafka.server.keystore.jks \
  -rfc -file kafka-cert.pem -storepass kafka123

# Create truststore
keytool -importcert -alias localhost -file kafka-cert.pem \
  -keystore kafka.server.truststore.jks -storepass kafka123 -noprompt

# Create credential files
echo "kafka123" > keystore_creds
echo "kafka123" > key_creds
echo "kafka123" > truststore_creds

cd ..

Start Stack:

docker-compose up -d

Test SSL Connection:

# Create client properties
cat > client-ssl.properties <<EOF
security.protocol=SSL
ssl.truststore.location=./secrets/kafka.server.truststore.jks
ssl.truststore.password=kafka123
EOF

# Produce test message
docker exec -it kafka kafka-console-producer \
  --bootstrap-server localhost:9093 \
  --topic test \
  --producer.config /etc/kafka/secrets/../client-ssl.properties

Benefit: Reusable local SSL setup for testing. No need to regenerate certs for each test.

Option 3: Confluent's cp-demo (Full Security Demo)

Clone Confluent Demo:

git clone https://github.com/confluentinc/cp-demo
cd cp-demo

Start Demo (includes TLS + SASL + ACLs + RBAC):

./scripts/start.sh

What This Provides:

• Full Kafka cluster with TLS encryption
• SASL/PLAIN authentication
• ACLs pre-configured
• RBAC (Confluent Platform feature)
• Schema Registry, Connect, ksqlDB all secured

Use Case: Learn how all security layers work together without manual setup.

Access Control Center (Web UI):

http://localhost:9021

Benefit: See a production-like secure Kafka setup in < 5 minutes.

Comparison: Manual Setup vs Confluent Tools

Recommendation: Use Confluent CLI or Docker Compose for local dev/testing. Use manual setup for production to ensure you understand every step.

Key Takeaways

1. Keystore: Contains private key + certificate (broker's identity).
2. Truststore: Contains CA certificates (validates others' identities).
3. Self-signed certs: Good for testing, NOT for production.
4. CA-signed certs: Required for production, enables automated trust.
5. Certificate chain: Root CA → Intermediate CA → Broker Cert.
6. Rotation: Rolling restart with new certs, minimal downtime.
7. Expiration monitoring: Alert 30 days before expiry.
8. Best practices: Use CA-signed certs, automate renewal, secure private keys, use short validity periods (90 days).

What's Next?

• Module 3.1: SASL Overview & Handshake Flow (add authentication to TLS encryption)
• Module 3.3: Client Authentication Configuration (configure producers/consumers for SASL)
• Module 5.3: Secrets & Credential Management (securely store keystore passwords)
• Module 6.1: Kafka Security End-to-End (compose TLS, SASL, ACLs)

Further Reading

• Kafka TLS Documentation
• Let's Encrypt
• OpenSSL Certificate Management
• Java keytool Documentation

End of Module 2.3