Apache Kafka Documentation

Apache Kafka® is a distributed event streaming platform designed for high-throughput, fault-tolerant data pipelines. This documentation provides comprehensive coverage of Kafka architecture, operations, and best practices for production deployments.

Documentation Scope

This reference documentation covers Apache Kafka versions 2.8 through 4.3.x, with particular emphasis on KRaft-mode deployments. Kafka 4.0 (March 2025) removed ZooKeeper entirely—KRaft is now the only supported metadata management mode.

Version Range	ZooKeeper Mode	KRaft Mode	Documentation Status
2.8.x	✅	⚠️ Early Access	Legacy reference
3.0.x - 3.2.x	✅	⚠️ Preview	Legacy reference
3.3.x - 3.5.x	✅	✅	Supported
3.6.x - 3.9.x	⚠️ Deprecated	✅	Fully Documented
4.0.x	❌ Removed	✅	Fully Documented
4.1.x	❌	✅	Fully Documented
4.2.x	❌	✅	Fully Documented
4.3.x	❌	✅	Current (4.3.1)

Legend: ✅ Production Ready | ⚠️ Limited/Deprecated | ❌ Not Supported

---

What's New

The current release on each supported version stream:

Stream	Latest	Released
4.3.x	4.3.1	—
4.2.x	4.2.0	February 2026
4.1.x	4.1.1	November 2025
4.0.x	4.0.1	October 2025

Kafka 4.3.1 is a patch release for the 4.3 line, addressing RocksDB memory leaks (KAFKA-20616, KAFKA-20688), group-coordinator errors after partition leadership changes, and stale changelog offsets on restart. The 4.3.0 release (May 2026) introduced broker and log-directory cordoning, OAuth2 private_key_jwt client assertions, and header-aware Kafka Streams state stores.

For per-release highlights across the 4.3.x, 4.2.x, 4.1.x, 4.0.x, and 3.x streams, plus major-version feature summaries, see the dedicated Release Notes page.

---

Getting Started

New to Apache Kafka? Begin with installation and initial configuration.

• Installation

  ---

  Deploy Kafka on Linux, containerized environments, or managed cloud services.

  Installation Guide

• Client Drivers

  ---

  Configure client libraries for Java, Python, Go, and other languages.

  Driver Documentation

---

Core Concepts

Understand the fundamental concepts underpinning Kafka's distributed architecture.

• Architecture Patterns

  ---

  Event sourcing, CQRS, and streaming architectures built on Kafka.

  Architecture Patterns

• Delivery Semantics

  ---

  At-least-once, at-most-once, and exactly-once delivery guarantees.

  Delivery Semantics

• Data Integration

  ---

  Integrating Kafka with external systems using Connect and Streams.

  Data Integration

• Multi-Datacenter

  ---

  Cross-datacenter replication strategies and disaster recovery.

  Multi-Datacenter

---

Architecture

Deep dive into Kafka's internal architecture and distributed systems design.

• Topics & Partitions

  ---

  Topics, partitions, partition leaders, ISR, and leader election.

  Topics & Partitions

• Brokers

  ---

  Broker architecture, request handling, and controller responsibilities.

  Broker Architecture

• Topology

  ---

  Cluster topology, rack awareness, and network configuration.

  Cluster Topology

• Storage Engine

  ---

  Log segments, compaction, and retention policies.

  Storage Engine

• Replication

  ---

  Partition replication, ISR management, and leader election.

  Replication

• Fault Tolerance

  ---

  Failure detection, recovery mechanisms, and high availability.

  Fault Tolerance

• Transaction Coordinator

  ---

  Exactly-once semantics, producer IDs, and two-phase commit.

  Transactions

---

Producers and Consumers

Configuration and best practices for Kafka clients.

• Producers

  ---

  Producer configuration, batching, compression, and delivery guarantees.

  Producer Guide

• Consumers

  ---

  Consumer groups, offset management, and rebalancing strategies.

  Consumer Guide

---

Stream Processing

Build real-time stream processing applications.

• Kafka Connect

  ---

  Source and sink connectors for data integration pipelines.

  Kafka Connect

• Kafka Streams

  ---

  Stream processing DSL for stateful transformations.

  Kafka Streams

• Schema Registry

  ---

  Schema management for Avro, Protobuf, and JSON Schema.

  Schema Registry

---

Operations

Production deployment, monitoring, and maintenance procedures.

• CLI Tools

  ---

  Command-line tools for cluster administration and troubleshooting.

  CLI Reference

• Configuration

  ---

  Broker, producer, and consumer configuration reference.

  Configuration Guide

• Monitoring

  ---

  JMX metrics, health checks, and alerting strategies.

  Monitoring Guide

• Backup & Restore

  ---

  Data backup strategies and disaster recovery procedures.

  Backup & Restore

• Maintenance

  ---

  Rolling upgrades, partition reassignment, and cluster expansion.

  Maintenance

• Performance Tuning

  ---

  Throughput optimization, latency tuning, and capacity planning.

  Performance

---

Security

Authentication, authorization, and encryption for Kafka deployments.

• Authentication

  ---

  SASL/SCRAM, SASL/GSSAPI (Kerberos), and mTLS authentication.

  Authentication

• Authorization

  ---

  ACL-based authorization and role-based access control.

  Authorization

• Encryption

  ---

  TLS/SSL encryption for data in transit.

  Encryption

---

Cloud Deployments

Deploy Kafka on cloud platforms and container orchestration systems.

• AWS

  ---

  Amazon MSK and self-managed Kafka on EC2.

  AWS Deployment

• Azure

  ---

  Azure Event Hubs for Kafka and AKS deployments.

  Azure Deployment

• Google Cloud

  ---

  Self-managed Kafka on GKE and Compute Engine.

  GCP Deployment

• Kubernetes

  ---

  Strimzi operator and StatefulSet deployments.

  Kubernetes

---

Troubleshooting

Diagnostic procedures and solutions for common issues.

• Common Errors

  ---

  Error codes, root causes, and resolution procedures.

  Error Reference

• Diagnosis

  ---

  Diagnostic procedures for cluster issues.

  Diagnosis Guide

• Log Analysis

  ---

  Log patterns, analysis techniques, and monitoring.

  Log Analysis

---

Quick Reference

• Reference

  ---

  Configuration reference, CLI commands, and metrics catalog.

  Quick Reference

• Migration

  ---

  ZooKeeper to KRaft migration and version upgrade guides.

  Migration Guide

---

Version Compatibility

This documentation follows Apache Kafka's semantic versioning. Behavioral differences between versions are explicitly noted throughout.

KRaft Migration Timeline

Milestone	Version	Date	Status
KRaft Early Access	2.8.0	April 2021	Development only
KRaft Preview	3.0.0 - 3.2.x	Sept 2021 - May 2022	Testing environments
KRaft Production Ready	3.3.0	October 2022	General availability
ZooKeeper Deprecated	3.6.0	October 2023	Migration recommended
Dynamic KRaft Quorum	3.9.0	November 2024	Add/remove controllers
ZooKeeper Removed	4.0.0	March 2025	KRaft required
Streams Rebalance Protocol GA	4.2.0	February 2026	Server-side rebalancing
Previous Release	4.3.0	May 2026	Broker cordoning, OAuth private_key_jwt
Current Release	4.3.1	—	Latest stable (patch)

Kafka 4.0+ Migration Path

Kafka 4.0 does not support ZooKeeper mode. To upgrade:

1. If on Kafka < 3.3: Upgrade to 3.9.x first
2. Migrate from ZooKeeper to KRaft mode
3. Then upgrade to Kafka 4.0+

Direct upgrade from ZooKeeper mode to 4.0 is not possible.

Documentation Conventions

This documentation uses RFC 2119 terminology (must, should, may) to indicate requirement levels. Version-specific behaviors are explicitly noted with the applicable Kafka version range.

---

Related Resources

• Apache Kafka Official Documentation
• Kafka Improvement Proposals (KIPs)
• AxonOps Kafka Monitoring - Monitor Kafka clusters with AxonOps