Kafka Configuration

This section provides comprehensive configuration reference for Apache Kafka deployments, covering server-side settings, topic configuration, security, and operational parameters.

---

Configuration Overview

Kafka configuration operates at multiple levels, each with different scopes and modification methods:

Level	Scope	Persistence	Modification
Static	Broker	server.properties	Requires restart
Dynamic Broker	Broker/Cluster	Metadata	No restart
Topic	Single topic	Metadata	Immediate
Client	Connection	Client config	Per-connection

---

Configuration Sections

Broker Configuration

Server-side configuration for Kafka brokers:

• Node identity and KRaft/ZooKeeper settings
• Listeners and network configuration
• Storage and retention settings
• Replication and ISR management
• Threading and performance tuning
• Dynamic configuration updates

Topic Configuration

Per-topic settings that override broker defaults:

• Retention policies (time and size)
• Cleanup policies (delete and compact)
• Compression settings
• Replication and ISR requirements
• Segment configuration
• Message size limits

ACL Configuration

Access Control List management for authorization:

• Enabling authorization
• Resource types and operations
• Managing ACLs with kafka-acls.sh
• Common ACL patterns for producers, consumers, and streams
• Wildcard and prefix patterns
• Host-based restrictions

Quotas

Client resource quotas for rate limiting:

• Producer and consumer byte rate quotas
• Request percentage quotas
• User, client ID, and combined quotas
• Quota resolution and precedence
• Multi-tenant quota patterns
• Monitoring throttling

Tiered Storage

Remote storage configuration for long-term retention:

• Enabling tiered storage
• S3, GCS, and Azure Blob backends
• Local vs remote retention settings
• Performance tuning
• Monitoring and troubleshooting

System Properties

JVM and environment configuration:

• JAAS and security properties
• SSL/TLS debugging
• JVM memory and GC settings
• JMX configuration
• Logging configuration
• Environment variables

Configuration Providers

External secret management integration:

• Built-in providers (file, environment, directory)
• HashiCorp Vault integration
• AWS Secrets Manager
• Azure Key Vault
• GCP Secret Manager
• Custom provider implementation

---

Configuration Methods

Static Configuration

Edit server.properties and restart the broker:

server.properties

broker.id=1
log.dirs=/var/kafka-logs
num.partitions=3

Dynamic Configuration

Use kafka-configs.sh for runtime changes:

# Broker-level dynamic config
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config log.retention.ms=86400000

# Cluster-wide default
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-default \
  --alter \
  --add-config log.retention.ms=86400000

# Topic-level config
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type topics \
  --entity-name events \
  --alter \
  --add-config retention.ms=172800000

Viewing Configuration

# View broker configuration
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --describe

# View topic configuration
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type topics \
  --entity-name events \
  --describe

# View all dynamic configs
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --describe --all

---

Configuration Files

Standard File Locations

File	Purpose
server.properties	Broker configuration
jvm.options	JVM settings
log4j.properties	Logging configuration
jaas.conf	JAAS security configuration
connect-distributed.properties	Kafka Connect workers

Environment-Specific Configuration

/etc/kafka/
├── server.properties           # Core broker config
├── jvm.options                  # JVM settings
├── log4j.properties            # Logging
├── jaas.conf                   # JAAS config
├── ssl/
│   ├── kafka.keystore.jks     # Broker keystore
│   └── kafka.truststore.jks   # Trust store
└── secrets/
    └── credentials             # Sensitive values

---

Production Configuration Checklist

Reliability

Setting	Location	Recommended
min.insync.replicas	Broker/Topic	2 (with RF=3)
default.replication.factor	Broker	3
unclean.leader.election.enable	Broker/Topic	false
acks	Producer	all
enable.idempotence	Producer	true

Security

Setting	Location	Recommended
auto.create.topics.enable	Broker	false
allow.everyone.if.no.acl.found	Broker	false
ssl.client.auth	Broker	required
Security protocol	Listeners	SASL_SSL

Performance

Setting	Location	Recommended
num.io.threads	Broker	2× CPU cores
num.network.threads	Broker	2-3 per listener
compression.type	Producer/Topic	lz4 or zstd
batch.size	Producer	65536-131072

Operations

Setting	Location	Recommended
log.retention.hours	Broker/Topic	Based on requirements
JMX enabled	JVM options	Yes
GC logging	JVM options	Enabled
Quotas	Dynamic config	Set for tenants

---

Related Documentation

• Operations Overview - Operations guide
• CLI Tools - Command-line tools
• Monitoring - Metrics and alerting
• Security - Security architecture
• Performance - Performance tuning