Kafka Broker Configuration

Broker configuration controls the behavior of Kafka broker nodes. Configuration is defined in server.properties and through dynamic configuration updates.

---

Configuration Categories

Category	Description	Dynamic
Node Identity	Broker and controller identifiers	❌
Listeners	Network endpoints and protocols	❌
Storage	Log directories and retention	Partial
Replication	Replica fetching and ISR management	Partial
Network	Threading and buffer sizes	✅
Security	TLS, SASL, and authorization	❌

---

Node Identity

KRaft Mode (Kafka 3.0+)

# Unique node identifier (required)
node.id=1

# Node roles: broker, controller, or both
process.roles=broker,controller

# Controller quorum voters (id@host:port)
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093

# Controller listener name
controller.listener.names=CONTROLLER

Setting	Description	Required
node.id	Unique identifier for this node	✅
process.roles	Comma-separated roles: broker, controller, or both	✅
controller.quorum.voters	List of controller voters	✅
controller.listener.names	Listener name for controller communication	✅

ZooKeeper Mode (Legacy)

# Broker ID (must be unique in cluster)
broker.id=1

# ZooKeeper connection
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181/kafka
zookeeper.connection.timeout.ms=18000
zookeeper.session.timeout.ms=18000

ZooKeeper Deprecation

ZooKeeper mode is deprecated as of Kafka 3.5 and will be removed in Kafka 4.0. New deployments should use KRaft mode.

---

Listeners

Listeners define network endpoints where brokers accept client connections.

Listener Configuration

# Define listeners (protocol://host:port)
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093,SASL_SSL://0.0.0.0:9094

# Advertised listeners (what clients see)
advertised.listeners=PLAINTEXT://broker1.example.com:9092,SSL://broker1.example.com:9093

# Map listener names to security protocols
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL,CONTROLLER:PLAINTEXT

# Inter-broker communication listener
inter.broker.listener.name=PLAINTEXT

# Controller listener (KRaft only)
controller.listener.names=CONTROLLER

Listener Types

Protocol	Encryption	Authentication	Use Case
PLAINTEXT	❌	❌	Development, internal networks
SSL	✅	Optional (mTLS)	Encrypted communication
SASL_PLAINTEXT	❌	✅	Authentication without encryption
SASL_SSL	✅	✅	Production recommended

Multi-Network Configuration

# Separate internal and external listeners
listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093

# Different advertised addresses per listener
advertised.listeners=INTERNAL://broker1.internal:9092,EXTERNAL://broker1.public.example.com:9093

# Security protocol mapping
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SASL_SSL

# Use internal listener for inter-broker communication
inter.broker.listener.name=INTERNAL

---

Storage Configuration

Log Directories

# Log directory (single)
log.dir=/var/kafka-logs

# Multiple log directories (comma-separated)
log.dirs=/data1/kafka-logs,/data2/kafka-logs,/data3/kafka-logs

Multiple Log Directories

Using multiple log directories across different disks improves I/O throughput. Kafka distributes partitions across directories.

Segment Configuration

# Segment file size (default: 1GB)
log.segment.bytes=1073741824

# Time before segment is rolled (default: 7 days)
log.roll.hours=168
log.roll.ms=604800000

# Index file size
log.index.size.max.bytes=10485760

# Index interval (bytes between index entries)
log.index.interval.bytes=4096

Retention Configuration

# Time-based retention (default: 7 days)
log.retention.hours=168
log.retention.minutes=10080
log.retention.ms=604800000

# Size-based retention per partition (default: unlimited)
log.retention.bytes=-1

# Check interval for retention policy
log.retention.check.interval.ms=300000

# Delete or compact
log.cleanup.policy=delete

Setting	Default	Dynamic	Description
log.retention.ms	604800000	✅	Retention time in ms
log.retention.bytes	-1	✅	Max bytes per partition
log.segment.bytes	1073741824	✅	Segment file size
log.cleanup.policy	delete	✅	delete or compact

Log Compaction

# Enable compaction for specific topics via topic config
# Broker-level compaction settings:

# Compaction threads
log.cleaner.threads=1

# Compaction buffer size
log.cleaner.dedupe.buffer.size=134217728

# I/O buffer size per cleaner thread
log.cleaner.io.buffer.size=524288

# I/O throughput limit (bytes/sec, -1 = unlimited)
log.cleaner.io.max.bytes.per.second=-1

# Minimum ratio of dirty log to total log to trigger compaction
log.cleaner.min.cleanable.ratio=0.5

# Minimum time message remains uncompacted
log.cleaner.min.compaction.lag.ms=0

# Maximum time message remains uncompacted
log.cleaner.max.compaction.lag.ms=9223372036854775807

# Delete retention for tombstones
log.cleaner.delete.retention.ms=86400000

---

Replication Configuration

# Default replication factor for auto-created topics
default.replication.factor=3

# Minimum ISR for writes when acks=all
min.insync.replicas=2

# Replica fetcher threads
num.replica.fetchers=1

# Replica fetch settings
replica.fetch.max.bytes=1048576
replica.fetch.min.bytes=1
replica.fetch.wait.max.ms=500

# Replica lag
replica.lag.time.max.ms=30000

# Allow unclean leader election
unclean.leader.election.enable=false

Setting	Default	Recommended	Description
default.replication.factor	1	3	RF for auto-created topics
min.insync.replicas	1	2	Minimum ISR for acks=all
unclean.leader.election.enable	false	false	Allow out-of-sync replica as leader
replica.lag.time.max.ms	30000	30000	Max lag before removing from ISR

Unclean Leader Election

Setting unclean.leader.election.enable=true may result in data loss. An out-of-sync replica becoming leader means committed messages on the previous leader may be lost.

---

Network Configuration

Thread Pool Sizing

# Network threads (handle connections)
num.network.threads=3

# I/O threads (handle requests)
num.io.threads=8

# Background threads
background.threads=10

# Request handler threads
num.recovery.threads.per.data.dir=1

Sizing Guidelines:

Setting	Formula	Notes
num.network.threads	2-3 per listener	Handles connection I/O
num.io.threads	2x CPU cores	Handles request processing
num.recovery.threads.per.data.dir	1-2 per disk	Parallel log recovery on startup

Buffer Sizes

# Socket buffers
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400

# Maximum request size
socket.request.max.bytes=104857600

# Request queue
queued.max.requests=500

Request Processing

# Request timeout
request.timeout.ms=30000

# Maximum time request can wait in queue
queued.max.request.bytes=104857600

# Connections per IP
max.connections.per.ip=100
max.connections.per.ip.overrides=10.0.0.1:200

---

Topic Defaults

These settings apply to topics when not overridden at the topic level.

# Default partition count
num.partitions=3

# Auto-create topics
auto.create.topics.enable=false

# Delete topics
delete.topic.enable=true

# Compression
compression.type=producer

# Maximum message size
message.max.bytes=1048588

# Replication settings
default.replication.factor=3
min.insync.replicas=2

Auto-Create Topics

In production, auto.create.topics.enable should be false. Auto-created topics use default settings and bypass governance controls.

---

Dynamic Configuration

Certain broker settings can be modified without restart.

Configuration Precedence

When a config is defined at multiple levels, precedence order:

1. Dynamic per-broker config stored in metadata log
2. Dynamic cluster-wide default config stored in metadata log
3. Static broker config from server.properties
4. Kafka default value

Update Modes

Mode	Description	Scope
read-only	Requires broker restart	Static only
per-broker	Updated dynamically per broker	Individual broker
cluster-wide	Updated as cluster-wide default	All brokers

Applying Dynamic Configuration

# Per-broker configuration
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config num.io.threads=16

# Cluster-wide default
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-default \
  --alter \
  --add-config log.retention.ms=86400000

# View configuration
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --describe

# Delete dynamic config (revert to static/default)
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --delete-config num.io.threads

# Update log level dynamically
kafka-configs.sh --bootstrap-server kafka:9092 \
  --broker-logger 1 \
  --alter \
  --add-config org.apache.kafka.server.quota=DEBUG

Thread Pool Update Restrictions

Thread pool sizes can only be updated within a restricted range to ensure graceful handling:

• Minimum: currentSize / 2
• Maximum: currentSize × 2

This applies to: num.network.threads, num.io.threads, num.replica.fetchers, num.recovery.threads.per.data.dir, log.cleaner.threads, background.threads

Dynamically Configurable Settings

Setting	Scope	Description
log.cleaner.threads	Broker	Compaction threads
log.cleaner.io.buffer.size	Broker	Cleaner buffer
log.cleaner.io.max.bytes.per.second	Broker	Cleaner I/O limit
log.retention.ms	Broker/Cluster	Retention time
log.retention.bytes	Broker/Cluster	Retention size
message.max.bytes	Broker/Cluster	Max message size
num.io.threads	Broker	I/O thread count
num.network.threads	Broker	Network thread count
num.replica.fetchers	Broker	Replica fetcher count
ssl.keystore.location	Broker	SSL keystore path
ssl.keystore.password	Broker	SSL keystore password

Dynamic SSL Keystore Update

SSL keystores can be updated without restart for certificate rotation:

# Update keystore for specific listener
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config 'listener.name.sasl_ssl.ssl.keystore.location=/path/to/new.keystore.jks,listener.name.sasl_ssl.ssl.keystore.password=newpassword'

Requirements:

• New certificate must be signed by the same CA as the old certificate
• For inter-broker listener, new keystore must be trusted by existing truststore
• Config name must be prefixed with listener.name.{listenerName}.

Dynamic Listener Management

Listeners can be added or removed without restart:

# Add new listener
kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type brokers \
  --entity-name 1 \
  --alter \
  --add-config 'listeners=PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:9094,advertised.listeners=PLAINTEXT://broker1:9092,EXTERNAL://external.example.com:9094'

Inter-Broker Listener

The inter-broker listener cannot be changed dynamically. Changing inter.broker.listener.name requires a rolling restart.

---

JVM and OS Tuning

Recommended JVM Options

# jvm.options or KAFKA_HEAP_OPTS

# Heap size (typically 6-8GB max)
-Xms6g
-Xmx6g

# GC settings (G1GC recommended)
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35

# GC logging
-Xlog:gc*:file=/var/log/kafka/gc.log:time,level,tags:filecount=10,filesize=100M

OS Tuning

# File descriptors (per process)
ulimit -n 100000

# sysctl settings
vm.swappiness=1
vm.dirty_background_ratio=5
vm.dirty_ratio=60
net.core.wmem_default=131072
net.core.rmem_default=131072
net.ipv4.tcp_wmem=4096 65536 2048000
net.ipv4.tcp_rmem=4096 65536 2048000

---

Production Checklist

Category	Setting	Recommended Value
Reliability	min.insync.replicas	2 (with RF=3)
	unclean.leader.election.enable	false
	default.replication.factor	3
Security	auto.create.topics.enable	false
	allow.everyone.if.no.acl.found	false
Performance	num.io.threads	2× CPU cores
	num.network.threads	2-3 per listener
Monitoring	metric.reporters	JMX or Prometheus

---

Related Documentation

• Configuration Overview - Configuration guide
• Topic Configuration - Topic settings
• Tiered Storage - Remote storage configuration
• Performance - Performance tuning
• Security - Security configuration