Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

KRaft: Kafka Raft Consensus

KRaft (Kafka Raft) is Kafka's built-in consensus protocol that replaced Apache ZooKeeper for metadata management. Introduced in Kafka 2.8 and production-ready since Kafka 3.3, KRaft simplifies Kafka's architecture by eliminating the external ZooKeeper dependency.

Why KRaft Replaced ZooKeeper

Section titled “Why KRaft Replaced ZooKeeper”

Architectural Simplification

Section titled “Architectural Simplification”
Before and After: ZooKeeper vs KRaftBefore and After: ZooKeeper vs KRaftZooKeeper Mode (Legacy)Kafka ClusterZooKeeper EnsembleKRaft ModeKafka ClusterBroker 1Broker 2Broker 3ZK 1ZK 2ZK 3Controller 1Controller 2Controller 3Broker 1Broker 2Two separate systems to operateDifferent failure modesSplit-brain risks between Kafka and ZKSingle systemUnified failure handlingNo external dependencies

Benefits of KRaft

Section titled “Benefits of KRaft”
AspectZooKeeper ModeKRaft Mode
Operational complexityTwo systems to manageSingle system
ScalingHigher metadata overhead; lower practical limitsHigher practical partition counts
Recovery timeMinutes (controller failover)Seconds
Metadata propagationPull-based, eventually consistentFetch-based, ordered
SecuritySeparate auth for ZKUnified Kafka security
MonitoringTwo metric systemsSingle metric system

Raft Consensus Protocol

Section titled “Raft Consensus Protocol”

KRaft implements the Raft consensus algorithm for leader election and log replication among controllers.

Raft Fundamentals

Section titled “Raft Fundamentals”
Raft Node StatesRaft Node StatesFollowerReceives log entries from leaderResponds to leader heartbeatsVotes in electionsCandidateRequests votes from peersVotes for itselfWaits for election resultLeaderSends heartbeatsReplicates log entriesCommits entries when quorum reachedStartElection timeout(no heartbeat received)Discovers leaderor higher termReceives majority votesElection timeout(split vote)Discovers higher term

Leader Election

Section titled “Leader Election”

When the controller leader fails, remaining controllers elect a new leader:

KRaft Leader ElectionKRaft Leader ElectionController 1Controller 2Controller 3Controller 1(Leader)Controller 1(Leader)Controller 2(Follower)Controller 2(Follower)Controller 3(Follower)Controller 3(Follower)Normal OperationHeartbeat (term=5)Heartbeat (term=5)AckAckLeader FailureCRASHNo more heartbeatsElection timeoutIncrement term to 6Become candidateElectionRequestVote(term=6, lastLogIndex, lastLogTerm)Grant vote(candidate log up-to-date)VoteGrantedReceived majority (2/3)Become leaderNew LeaderHeartbeat (term=6)Controller 2 is now leader

Election rules:

  1. Term — Logical clock incremented on each election
  2. Vote — Each controller votes once per term
  3. Log completeness — Only vote for candidates with up-to-date logs
  4. Majority — Candidate needs (n/2) + 1 votes to become leader

Log Replication

Section titled “Log Replication”

The leader replicates metadata log entries to followers:

Log Replication and CommitLog Replication and CommitController 1Controller 2Controller 3Controller 1(Leader)Controller 1(Leader)Controller 2(Follower)Controller 2(Follower)Controller 3(Follower)Controller 3(Follower)Client RequestCreateTopic requestAppend to local log(index=100, uncommitted)ReplicationAppendEntries(entries=[100])AppendEntries(entries=[100])Append to logSuccess(matchIndex=100)Append to logSuccess(matchIndex=100)CommitQuorum reached (3/3)Commit index=100Heartbeat(commitIndex=100)Heartbeat(commitIndex=100)Entry 100 now committedApplied to state machineTopic creation complete

Commit rules:

  • Entry is committed when replicated to a majority of controllers
  • Only entries from the current term can be committed directly
  • Committing an entry commits all prior entries

Quorum and Fault Tolerance

Section titled “Quorum and Fault Tolerance”
ControllersQuorumTolerated Failures
110 (no fault tolerance)
321
532
743

Recommendation: Use 3 controllers for most deployments. Use 5 for large clusters requiring higher availability.

Metadata Log

Section titled “Metadata Log”

All cluster metadata is stored in a replicated log, not in an external system.

Log Structure

Section titled “Log Structure”
Metadata Log StructureMetadata Log StructureMetadata LogOffset 0: FeatureLevelRecordOffset 1: RegisterBrokerRecord(broker=1)Offset 2: RegisterBrokerRecord(broker=2)Offset 3: TopicRecord(name=orders)Offset 4: PartitionRecord(topic=orders, p=0)Offset 5: PartitionChangeRecord(leader=1)...Offset N: ConfigRecord(retention.ms=...)Append-only logEach entry is a metadata changeReplicated across all controllers

Record Types

Section titled “Record Types”
CategoryRecord Types (non-exhaustive)
ClusterFeatureLevelRecord, ZkMigrationStateRecord
BrokersRegisterBrokerRecord, UnregisterBrokerRecord, BrokerRegistrationChangeRecord, FenceBrokerRecord, UnfenceBrokerRecord
TopicsTopicRecord, RemoveTopicRecord
PartitionsPartitionRecord, PartitionChangeRecord
ConfigurationConfigRecord, RemoveConfigRecord
SecurityClientQuotaRecord, UserScramCredentialRecord, AccessControlEntryRecord
ProducersProducerIdsRecord

Snapshots

Section titled “Snapshots”

To prevent unbounded log growth, controllers periodically create snapshots:

Snapshot and Log CompactionSnapshot and Log CompactionBefore SnapshotAfter SnapshotLog Segment 1(offsets 0-999)Log Segment 2(offsets 1000-1999)Log Segment 3(offsets 2000-2500)Snapshot @ offset 2000(full state)Log Segment 3(offsets 2000-2500)Snapshot contains:- All broker registrations- All topics and partitions- All configurations- Current leader/ISR state

Snapshot configuration:

# Minimum records between snapshots
metadata.log.max.record.bytes.between.snapshots=20971520


# Maximum time between snapshots
metadata.log.max.snapshot.interval.ms=3600000

Storage Location

Section titled “Storage Location”
Terminal window
# Metadata log directory structure
/var/kafka-logs/__cluster_metadata-0/
├── 00000000000000000000.log      # Log segment
├── 00000000000000000000.index    # Offset index
├── 00000000000000000000.timeindex
├── 00000000000000001000.log      # Next segment
├── 00000000000000001000-checkpoint.snapshot  # Snapshot
└── leader-epoch-checkpoint

Controller Communication

Section titled “Controller Communication”

Controller-to-Controller (Raft)

Section titled “Controller-to-Controller (Raft)”

Controllers communicate using the Raft protocol over a dedicated listener:

# Controller listener configuration
controller.listener.names=CONTROLLER
listeners=CONTROLLER://0.0.0.0:9093


# Inter-controller security
listener.security.protocol.map=CONTROLLER:SSL

Controller-to-Broker (Metadata Fetch)

Section titled “Controller-to-Broker (Metadata Fetch)”

In KRaft, brokers fetch metadata updates from the controller log (unlike ZooKeeper watch-based updates):

Metadata Fetch by BrokersMetadata Fetch by BrokersControllerBroker 1Broker 2Controller(Leader)Controller(Leader)Broker 1Broker 1Broker 2Broker 2Broker RegistrationBrokerRegistrationRequestBrokerRegistrationResponseBrokerHeartbeatRequestBrokerHeartbeatResponseMetadata FetchFetchRequest(__cluster_metadata)Broker fetches fromits last known offsetFetchResponse(new metadata records)Apply metadata updatesContinuous Syncloop[Every fetch interval]FetchRequest(offset=N)FetchResponse(records N+1...)

Key difference from ZooKeeper:

AspectZooKeeper ModeKRaft Mode
PropagationWatch-based, asyncFetch-based, controlled
ConsistencyEventualOffset-based, ordered
LatencyVariablePredictable

Quorum Configuration

Section titled “Quorum Configuration”

Controller Quorum Voters

Section titled “Controller Quorum Voters”
# Define the controller quorum
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093


# Format: node.id@host:port
# All controllers must have the same voter list

Deployment Modes

Section titled “Deployment Modes”

Combined Mode (Development/Small Clusters)

Section titled “Combined Mode (Development/Small Clusters)”

Controllers and brokers run in the same JVM:

# server.properties for combined mode
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@node1:9093,2@node2:9093,3@node3:9093
Combined ModeCombined ModeNode 1Node 2Node 3ControllerBrokerControllerBrokerControllerBrokerSimpler deploymentShared resourcesSuitable for < 10 brokersRaftRaftRaft

Isolated Mode (Production/Large Clusters)

Section titled “Isolated Mode (Production/Large Clusters)”

Dedicated controller nodes separate from brokers:

# controller.properties (controller-only nodes)
process.roles=controller
node.id=1
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093


# broker.properties (broker-only nodes)
process.roles=broker
node.id=101
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
Isolated ModeIsolated ModeController TierBroker TierController 1Controller 2Controller 3Broker 101Broker 102Broker 103Broker 104Broker 105Dedicated resourcesIndependent scalingRecommended for > 10 brokersRaftRaftRaft

Controller Sizing

Section titled “Controller Sizing”

Kafka controllers keep cluster metadata in memory and on disk.

Cluster SizeController CountController Resources
Development1 (no HA)1 CPU, 1GB RAM
Small (< 10 brokers)3 (combined mode)2 CPU, 4GB RAM
Medium (10-50 brokers)3 (isolated)4 CPU, 8GB RAM
Large (50+ brokers)5 (isolated)8 CPU, 16GB RAM

Sourcing

The table above reflects repository guidance. Kafka's KRaft ops docs note that typical clusters can use ~5 GB memory and ~5 GB disk for the metadata log directory.

Failover Behavior

Section titled “Failover Behavior”

Controller Leader Failure

Section titled “Controller Leader Failure”
Controller Failover TimelineController Failover TimelineController 1LeaderController 2FollowerCandidateLeaderController 3FollowerFollowerCluster StateNormalElectionNormalLeader servingFailover (~100-200ms)0100150200

Failover characteristics:

MetricTypical Value
Detection time≤ controller.quorum.fetch.timeout.ms (default 2000ms)
Election time≤ controller.quorum.election.timeout.ms (default 1000ms)
Total failoverDetection + election + metadata catch-up

Broker Behavior During Failover

Section titled “Broker Behavior During Failover”
Broker Behavior During Controller FailoverBroker Behavior During Controller FailoverBrokerController 1Controller 2BrokerBrokerController 1(Old Leader)Controller 1(Old Leader)Controller 2(New Leader)Controller 2(New Leader)Normal OperationBrokerHeartbeatRequestBrokerHeartbeatResponseController FailureBrokerHeartbeatRequestNo response (timeout)Retry with backoffNew Leader ElectedBrokerHeartbeatRequestBrokerHeartbeatResponseFetchRequest(__cluster_metadata)FetchResponseBroker continues serving clientsthroughout controller failover.Metadata updates delayed butread/write operations continue.

Key point: Broker data operations (produce/consume) continue during controller failover. Only metadata operations (topic creation, leader election) are temporarily blocked.

Split-Brain Prevention

Section titled “Split-Brain Prevention”

Raft's quorum requirement prevents split-brain:

Split-Brain PreventionSplit-Brain PreventionNetwork PartitionPartition APartition BController 1Controller 2Quorum (2/3)Can elect leaderController 3No quorum (1/3)Cannot elect leaderOnly the partition with quorumcan elect a leader and make progress.Minority partition is read-only.Raft OKNetwork partitionNetwork partition

Migration from ZooKeeper

Section titled “Migration from ZooKeeper”

Migration Modes

Section titled “Migration Modes”
ZooKeeper to KRaft MigrationZooKeeper to KRaft MigrationZooKeeperBrokers use ZKController in ZK modeKRaft (Migration)Controllers runningZK still authoritativeDual-write modeKRaftZK removedControllers authoritativeFull KRaft modeInitial stateStart migrationRollback (if needed)Complete migration

Migration Steps

Section titled “Migration Steps”
  1. Deploy controller quorum:
Terminal window
# Format controller storage
kafka-storage.sh format -t $(kafka-storage.sh random-uuid) \
  -c controller.properties
  1. Enable migration mode:
# In broker server.properties
zookeeper.metadata.migration.enable=true
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
  1. Start controllers and migrate:
Terminal window
# Controllers will sync metadata from ZooKeeper
# Monitor migration progress
kafka-metadata-shell.sh --snapshot /var/kafka-logs/__cluster_metadata-0/*.log \
  describe
  1. Restart brokers in KRaft mode:
# Remove ZK config, enable KRaft
process.roles=broker
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093
# Remove: zookeeper.connect=...

Rollback Capability

Section titled “Rollback Capability”

During migration, rollback is possible until finalization:

PhaseRollback PossibleData Safe
Controllers deployedYesYes
Dual-write modeYesYes
Brokers migratedYes (restart with ZK)Yes
Migration finalizedNoN/A

Troubleshooting

Section titled “Troubleshooting”

Common Issues

Section titled “Common Issues”
IssueSymptomResolution
No leader electedLEADER_NOT_AVAILABLE errorsCheck controller connectivity, verify quorum voters
Metadata out of syncBrokers have stale topic infoCheck broker fetch lag from controllers
Controller OOMController crashesIncrease heap, check for partition explosion
Slow electionsLong failover timeCheck network latency between controllers

Diagnostic Commands

Section titled “Diagnostic Commands”
Terminal window
# Check controller quorum status
kafka-metadata-quorum.sh --bootstrap-controller ctrl1:9093 \
  describe --status


# View current controller leader
kafka-metadata-quorum.sh --bootstrap-controller ctrl1:9093 \
  describe --status | rg -i leader


# Check metadata log lag
kafka-metadata-shell.sh --snapshot /var/kafka-logs/__cluster_metadata-0/*.log \
  log | tail -20


# Verify broker registration
kafka-broker-api-versions.sh --bootstrap-server broker1:9092

Key Metrics

Section titled “Key Metrics”
MetricDescriptionAlert Condition
kafka.controller:type=KafkaController,name=ActiveControllerCountActive controllers≠ 1
kafka.controller:type=ControllerEventManager,name=EventQueueSizePending controller events> 1000
kafka.server:type=MetadataLoader,name=CurrentMetadataOffsetBroker metadata offsetLag > 1000
kafka.raft:type=RaftMetrics,name=CommitLatencyAvgRaft commit latency> 100ms

Summary

Section titled “Summary”
AspectDetail
WhatBuilt-in consensus replacing ZooKeeper
ProtocolRaft (leader election, log replication)
Storage__cluster_metadata topic
Quorum3 or 5 controllers recommended
Failover2-5 seconds typical
DeploymentCombined (small) or isolated (large) mode
Section titled “Related Documentation”