Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

No Active Kafka Controller

Use this playbook when the cluster-wide sum of ActiveControllerCount is not one, metadata operations time out, or brokers cannot register.

Existing data traffic may continue briefly, but leader failover, broker registration, topic changes, ACL changes, and other metadata operations are at risk. Treat loss of a quorum majority as a critical incident.

Terminal window
kafka-metadata-quorum.sh \
--bootstrap-controller controller1:9093 \
describe --status
kafka-metadata-quorum.sh \
--bootstrap-controller controller1:9093 \
describe --replication
kafka-cluster.sh list-endpoints \
--bootstrap-controller controller1:9093

Collect controller logs, host metrics, disk statistics, TLS/SASL errors, and process start times from every voter.

  1. Confirm a majority of configured voters is running and mutually reachable on the controller listener.
  2. Compare cluster ID, node.id, directory ID, and quorum membership with the intended topology.
  3. Check metadata-log capacity, inode use, mount state, I/O latency, and kernel errors.
  4. Check clock synchronization, certificate validity, SASL credentials, and listener mapping.
  5. Compare voter log-end offsets, last fetch timestamps, and the high watermark.

No leader with fewer than a majority available is expected. A majority that is reachable but cannot elect usually points to security mismatch, conflicting voter configuration, severe pauses, or metadata-storage failure.

  • Restore the existing voters and network path before changing membership.
  • Restart only one controller at a time when a majority remains healthy.
  • For a permanently failed voter, follow the documented controller replacement/removal procedure and retain its node and directory identity evidence.
  • If dynamic quorum membership is enabled, add or remove controllers through kafka-metadata-quorum.sh; do not edit membership ad hoc.

Danger

Do not run kafka-storage.sh format, delete the metadata log, or generate a new cluster ID. Those actions can create a different cluster or destroy the best recovery copy.

  • describe --status reports the expected cluster ID and one leader.
  • A majority is caught up and the high watermark advances.
  • The cluster-wide sum of ActiveControllerCount is one.
  • Brokers register and a harmless metadata read succeeds.
  • Election, append, and metadata-loader error rates return to baseline.

Use three or five dedicated controllers across failure domains. Monitor voter lag, election latency, metadata errors, disk health, certificate expiry, and time synchronization. Test controller maintenance without taking down a majority.

See KRaft operations.