Getting Started with Kafka

This guide covers Kafka installation, client setup, and basic operations to begin working with event streaming.

---

Prerequisites

Requirement	Minimum	Recommended
Java	JDK 11 (3.x) / JDK 17 (4.0)	JDK 17 or 21
Memory	4 GB	8+ GB
Disk	10 GB	SSD recommended
OS	Linux, macOS, Windows	Linux for production

---

Version Compatibility

JDK Compatibility

Kafka components have different JDK support levels. Kafka 4.0 introduced significant changes to Java version requirements.

Kafka 4.0+ (applies to 4.0, 4.1, and 4.2):

Component	Java 11	Java 17	Java 21
Kafka Clients	✅	✅	✅
Kafka Streams	✅	✅	✅
Kafka Connect	❌	✅	✅
Kafka Server	❌	✅	✅

Legend: ✅ Supported | ❌ Not Supported

Java 8 and 11 Changes in Kafka 4.0

• Java 8 support is completely removed in Kafka 4.0
• Java 11 is no longer supported for Connect and Server components
• Clients and Streams still support Java 11 for backward compatibility

Kafka 3.x:

Component	Java 8	Java 11	Java 17
All Components	✅	✅	✅

Client-Broker Compatibility

Kafka maintains bidirectional compatibility between clients and brokers within certain version ranges.

Client/Broker forward compatibility with Kafka 4.x:

Client Version	Module	4.x Compatibility	Notes
0.x, 1.x, 2.0	All	❌	Pre-0.10.x protocols removed in 4.0 (KIP-896)
2.1 - 2.8	Client	⚠️ Partial	Some consumer/producer changes
2.1 - 2.8	Streams	⚠️ Limited	API changes may affect applications
2.1 - 2.8	Connect	⚠️ Limited	Connector compatibility varies
3.x	All	✅ Full	Fully compatible
4.0 - 4.1	All	✅ Full	Fully compatible with 4.2 brokers

Legend: ✅ Full | ⚠️ Partial | ❌ Not Compatible

Kafka 4.0+ Protocol Changes

• Pre-0.10.x protocol versions are fully removed in Kafka 4.0 (KIP-896)
• The --zookeeper option in AdminClient commands has been removed; use --bootstrap-server instead
• Clients older than 2.1 must be upgraded before connecting to Kafka 4.0+ brokers

General compatibility guarantees:

• Clients can communicate with brokers of the same or adjacent major versions
• Protocol versioning allows automatic feature negotiation
• Older clients may not access features introduced in newer broker versions

KRaft Version Compatibility

KRaft clusters have specific version requirements for controllers and brokers.

Server compatibility with Kafka 4.x:

KRaft Cluster Version	4.x Dynamic Voter	4.x Static Voter
Before 3.2.x	❌	❌
3.3.x - 3.8.x	❌	✅
3.9.x	✅	✅
4.0.x - 4.3.x	✅	✅

Static to Dynamic Voter Migration

Upgrading a cluster from static voter to dynamic voter configuration is not supported. See KAFKA-16538 for details.

Feature Version Requirements

Feature	Minimum Version	Notes
KRaft (production)	3.3	ZooKeeper still required for some features until 3.5
KRaft (full feature parity)	3.5	All features available without ZooKeeper
Tiered Storage	3.6	Early access
Share Groups	4.0	KIP-932
ZooKeeper removal	4.0	KRaft required
Streams Rebalance Protocol	4.1 (early access), 4.2 (GA)	KIP-1071
Share Groups v2	4.2	RENEW ack type (KIP-1222), acquire modes (KIP-1206)
Standardized CLI arguments	4.2	KIP-1147

---

Installation Options

Option	Use Case	Complexity
Local Installation	Development, learning	Low
Docker	Development, CI/CD	Low
Kubernetes	Production, cloud-native	Medium

---

Quick Start

1. Download and Extract

# Download Kafka
curl -O https://downloads.apache.org/kafka/4.3.0/kafka_2.13-4.3.0.tgz

# Extract
tar -xzf kafka_2.13-4.3.0.tgz
cd kafka_2.13-4.3.0

2. Start Kafka (KRaft Mode)

Kafka 3.3+ supports KRaft mode, eliminating the ZooKeeper dependency.

# Generate cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# Format storage
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

# Start Kafka
bin/kafka-server-start.sh config/kraft/server.properties

3. Create a Topic

bin/kafka-topics.sh --create \
  --topic quickstart-events \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

4. Produce Messages

bin/kafka-console-producer.sh \
  --topic quickstart-events \
  --bootstrap-server localhost:9092

Type messages and press Enter to send each one. Press Ctrl+C to exit.

5. Consume Messages

bin/kafka-console-consumer.sh \
  --topic quickstart-events \
  --from-beginning \
  --bootstrap-server localhost:9092

---

Client Drivers

Kafka provides official and community clients for multiple languages.

Language	Client	Documentation
Java	Apache Kafka Client	Driver Guide
Python	confluent-kafka-python	Driver Guide
Go	confluent-kafka-go	Driver Guide
Node.js	kafkajs	Driver Guide
.NET	confluent-kafka-dotnet	Driver Guide

→ Client Drivers

---

Verification

After installation, verify the cluster is operational:

# Check broker status
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# List topics
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

# Describe cluster
bin/kafka-metadata.sh --snapshot /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log --command "describe"

---

Configuration

Essential Broker Settings

Property	Description	Default
broker.id	Unique broker identifier	Required
listeners	Network listeners	PLAINTEXT://:9092
log.dirs	Data directory	/tmp/kafka-logs
num.partitions	Default partitions for new topics	1
default.replication.factor	Default replication factor	1

Production Recommendations

Setting	Development	Production
log.dirs	Single directory	Multiple directories on separate disks
num.partitions	1-3	Based on throughput requirements
default.replication.factor	1	3
min.insync.replicas	1	2

→ Configuration Reference

---

Next Steps

Goal	Documentation
Understand Kafka concepts	Event Streaming Concepts
Build a producer application	Producer Guide
Build a consumer application	Consumer Guide
Set up Kafka Connect	Kafka Connect Guide
Configure for production	Operations Guide

---

Related Documentation

• Installation Guide - Detailed installation procedures
• Client Drivers - Language-specific client setup
• Concepts - Kafka fundamentals
• Operations - Production configuration