Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Adaptive Balancer

The AxonOps Kafka Adaptive Balancer moves partition replicas and partition leadership between the brokers of an Apache Kafka cluster so that disk, network, CPU, replica, and leader load stay within capacity limits and close to the cluster average. It builds a model of the cluster from the topology and metrics that AxonOps already collects, evaluates that model against a set of goals, and produces a list of proposed moves. An operator reviews the proposals and then executes them, as a dry run or live, from the Adaptive Balancing section of the AxonOps dashboard.

The balancer works with the existing components of an AxonOps installation: axon-server, the AxonOps server that stores metrics and serves the AxonOps API; axon-dash, the AxonOps web dashboard; and axon-agent, the AxonOps agent on each Kafka broker that collects Kafka and host metrics. Getting Started describes these components and how they are installed.

The main concepts are:

  • Cluster model: the balancer’s in-memory picture of the brokers, racks, partitions, replica placement, load, and capacity of one cluster, rebuilt on a fixed interval.
  • Goals: rules the model is evaluated against. A hard goal, such as keeping each broker’s disk usage below a capacity threshold, must be satisfied. A soft goal, such as keeping leader counts close to the average, is improved where possible.
  • Analysis: one run of the enabled goals against the cluster model. An analysis produces proposals: replica moves, leader changes, and disk moves.
  • Execution: applying the proposals of an analysis through AxonOps, as a dry run that submits nothing or as a live run that reassigns partitions.
  • Maintenance window: a marker on a broker that excludes it as a destination for moves and, with drain set, moves its replicas to other brokers.
  • The Kafka cluster must be monitored by AxonOps with the Kafka agent and host metrics on every broker. See Kafka agent setup. The balancer reads broker, topic, and partition metrics from the Kafka agent and CPU and disk metrics from the host collectors.
  • At least one broker’s agent must hold an active connection to Kafka. axon-server relays every reassignment through such an agent and rejects the request with available kafka nodes not found when none has reported in the last 10 seconds.
  • The balancer needs an AxonOps credential with the Admin role on the organisation. Reassignments and history writes are rejected for any lower role. On AxonOps Cloud, AxonOps provisions this credential; on a self-hosted installation, the operator creates an API token as described in Kafka Adaptive Balancer Setup.
  • Brokers must support the AlterPartitionReassignments Admin API (Kafka Improvement Proposal KIP-455, Apache Kafka 2.4 and later). axon-server relays reassignments through brokers running in either ZooKeeper mode or KRaft mode. ZooKeeper mode exists up to Apache Kafka 3.9; Apache Kafka 4.0 and later run only in KRaft mode.
  • For leader changes to take effect, the brokers must run with auto.leader.rebalance.enable=true, which is the default in Apache Kafka (see the broker configuration reference). The balancer changes a partition’s preferred leader, which is the first replica in the partition’s replica list; Kafka moves leadership to the preferred leader only when it runs a preferred leader election. See Leader imbalance.
  • Rack-aware goals use broker.rack. Where it is not set, each host is treated as a rack.

Each reassignment the balancer submits is recorded in the AxonOps audit log with the action “Alter Kafka topic partition assignments”.

The balancer is a standalone service, axon-kafka-adaptive-balancer. It never connects to Kafka directly. It reads cluster topology and metrics from axon-server, and it submits partition reassignments to axon-server, which relays each one to axon-agent on a Kafka broker. The agent applies the reassignment through Kafka’s AlterPartitionReassignments Admin API (KIP-455).

The browser does not contact the balancer either. The Adaptive Balancing pages call axon-dash, and axon-dash forwards each request to the balancer from the server side, using the Base URL stored for the selected Kafka cluster. The bearer token for the balancer is held in the browser’s local storage; axon-dash forwards it to the balancer unchanged. The balancer must therefore be reachable from the axon-dash host, not from operators’ browsers.

The balancer is outside the Kafka data path. When it stops, analysis, execution tracking, and anomaly detection stop. Kafka producers, consumers, replication, and the existing partition placement are unaffected, and reassignments already submitted to Kafka continue until Kafka completes them.

The following diagram shows the component path.

Kafka Adaptive Balancer component path axon-server stores the Base URL for each Kafka cluster, the balancer’s history (analyses, executions, and anomalies), maintenance windows, and each cluster’s goal and model settings. It serves the topology, metrics, and reassignment APIs the balancer uses. It also raises AxonOps alerts from the anomalies the balancer reports, and stores each cluster’s anomaly alerts setting. It does not run, proxy, or monitor the balancer.

For each registered cluster, the balancer rebuilds the cluster model every model refresh interval, 5 minutes by default. The model records each broker and its rack, every partition with its replica placement and leader, partition sizes, and per-broker CPU, disk, and network load. Rate metrics are averaged over the metrics window, 1 hour by default; sizes are taken from the latest sample. Both intervals are set per cluster under Settings → Kafka. Brokers without broker.rack are treated as one rack per host.

The Overview page (Adaptive Balancing → Overview) shows the model’s state. The Balancer Status card gives the last and next model refresh, the refresh interval, and the metrics window, and the Cluster Status card charts partitions, leaders, log size, CPU, and bytes in and out per broker or per rack, from the same metrics the balancer reads.

Adaptive Balancing Overview page showing the Balancer Status card with Status ACTIVE, refresh times, a 5m refresh interval, and a 24h metrics window, above the Cluster Status card with pie charts of partitions, leaders, log size, CPU, bytes in, and bytes out per broker

Kafka Adaptive Balancer Goals and Cluster Model describes how load and capacity are derived and what happens when metrics are missing.

Goals run in a fixed built-in order, hard goals before soft goals, and each goal can veto a move that would break a goal that ran before it. Thirteen goals are enabled by default on a newly registered cluster, and the enabled set is changed per cluster on the Goals page. Kafka Adaptive Balancer Goals and Cluster Model lists every goal, its threshold, and its default state.

An analysis runs the enabled goals against a copy of the cluster model and records the difference between the original and the optimised placement as proposals. Each proposal has one of three types:

Proposal typeExecution page labelEffect
INTER_BROKER_MOVEReplica moveMoves a partition replica from one broker to another
LEADERSHIP_MOVELeader changeMakes another replica of a partition its preferred leader
INTRA_BROKER_MOVEDisk moveMoves a replica between log directories on the same broker

An analysis places no limit on the number of proposals or the volume of data they move.

An execution applies the proposals of the most recent analysis. A dry run builds and validates the execution plan without submitting anything to Kafka. A live execution submits reassignments through AxonOps in phases, replica moves first and leader changes last, and paces them with limits on concurrent movements, movements per broker, and data in flight. The balancer does not set a Kafka replication throttle.

Rebalancing a Kafka Cluster with the Adaptive Balancer describes the workflow from analysis to execution, and the configuration reference lists the pacing defaults.

The balancer checks every registered cluster for three kinds of anomaly. Detection runs by default. Its thresholds and intervals are built in and cannot be changed from the dashboard or from the configuration file, where detection can only be turned off as a whole. Anomaly detection in the configuration reference describes that setting and a configuration block that disables every detector without an error.

DetectorChecked everyReported whenSeverity
Goal violation1 minuteA broker uses more than 80% of its disk capacity, 70% of its CPU capacity, or 80% of its inbound or outbound network capacityMEDIUM above the threshold, HIGH above 1.2 times the threshold, CRITICAL above 1.5 times the threshold
Broker failure30 secondsA broker that hosts replicas has been absent for more than 30 seconds, including a broker that was already down when the balancer started. The first report follows about 30 to 60 seconds after the broker is first seen down, and the report repeats every 5 minutes while the broker stays down. A broker that joins or leaves the cluster is reported once.CRITICAL for an absent broker, HIGH for a broker that leaves, INFO for a broker that joins
Load imbalance5 minutesThe coefficient of variation (standard deviation divided by mean) of disk, CPU, network, replica count, or leader count across brokers exceeds its threshold. Requires at least 3 live brokers. For disk, CPU, and network, only brokers with a metric sample are counted, and the check is skipped when fewer than 3 have one.MEDIUM, or HIGH when the variation exceeds twice the threshold

While a condition persists, the balancer records it again on every detection cycle, each time as a new anomaly with its own identifier. Each anomaly carries a subject that identifies what it is about: the goal name for a goal violation; broker <id>, broker <id> added, or broker <id> removed for the broker failure detector; and the resource (DISK, LEADERSHIP, REPLICA, CPU, NETWORK_IN, or NETWORK_OUT) for a load imbalance.

Descriptions name each broker and, where AxonOps reports one, its host, for example “Broker 3 (host-a) is OFFLINE”, “New broker 3 (host-a) added to cluster”, and “Broker 3 (host-a) removed from cluster”. A load imbalance description gives the measured and threshold skew and lists the brokers more than one standard deviation from the mean, for example “DISK imbalance detected: 43.50% skew (threshold: 20.00%); overloaded: broker 1 (host-a); underloaded: broker 4 (host-d)”.

Detected anomalies appear in the Recent Anomalies table on the Overview page, with the columns Last Detected, First Detected, Count, Type, Severity, and Description. Each row groups the detections that share a type and subject, the same grouping that anomaly alerts use; a gap of more than 15 minutes between two detections starts a new row. Type, Severity, and Description come from the latest detection in the row. Severities are CRITICAL (red), HIGH (orange), MEDIUM, LOW, and INFO. Timestamps are shown in the timezone selected in AxonOps. The table shows at most 20 grouped rows, and shows “No anomalies detected.” when there are none. The same message appears when axon-dash cannot reach the balancer, because the Overview page does not show connection errors.

Recent Anomalies table on the Overview page, with two grouped LOAD_IMBALANCE rows showing the last and first detection times, the detection count, MEDIUM severity, and the disk skew description

The balancer does not execute a rebalance automatically in response to an anomaly. It may start an analysis in response to one. The Overview does not show whether an analysis was started this way; such an analysis is listed on the Analyses page like any other and becomes the most recent analysis for the cluster, so it is the only one that can be executed until a newer one runs.

axon-server raises AxonOps alerts from the anomalies the balancer reports, and sends them to the cluster’s integrations.

Each anomaly severity maps to an alert severity as follows:

Anomaly severityAxonOps alertRouting slot
CRITICALCriticalError
HIGHErrorError
MEDIUMWarningWarning
LOW, INFONone; shown only in Recent AnomaliesNone

The alert is named Adaptive Balancer - <type> - <subject>, for example Adaptive Balancer - GOAL_VIOLATION - DiskCapacityGoal, or Adaptive Balancer - <type> when the anomaly has no subject. Its message is the anomaly description. Balancer alerts appear among the cluster’s active alerts and in the alert and event log.

The alert lifecycle is as follows:

  • One alert exists per cluster, anomaly type, and subject. Repeated detections of the same condition update that alert rather than raising new ones.
  • An alert fires on the first report, with no pending period.
  • A repeated report at the same alert level does not notify again. HIGH and CRITICAL are handled as one critical level for state changes: a change between them does not notify again, and the alert keeps the severity it was raised with. A change between MEDIUM (warning) and HIGH or CRITICAL updates the alert and notifies again.
  • An alert that the balancer has not re-reported for 15 minutes is resolved, with the message “The Adaptive Balancer has not reported this for 15m0s; treating it as resolved”, and a resolved notification is sent.
  • After axon-server restarts, its open balancer alerts are resolved 15 minutes later unless the balancer reports them again.

Routing is set per cluster on Settings → Alert Routing, under the Adaptive Balancer routing type; the integrations themselves are defined on Settings → Integrations (see Routing). Routing has three severity slots, Info, Warning, and Error, and no Critical slot, so CRITICAL and HIGH anomalies are routed through the Error slot and MEDIUM anomalies through the Warning slot. The Adaptive Balancer routing type is empty by default, and each slot it does not override falls back to the same slot of the Global routing. A resolved notification is sent at info severity, through the Info slot of the same routing type, and also to every integration that was notified when the alert fired.

The failure semantics of balancer alerts are as follows:

  • An alert exists only while the balancer keeps re-reporting the condition. When the balancer stops, its host is lost, or it cannot reach axon-server, the open balancer alerts are resolved after 15 minutes, and resolved notifications are sent, even if the condition persists.
  • Turning off Raise alerts from anomalies also sends resolved notifications for conditions that may still be true. Integrations such as PagerDuty close the corresponding incidents on a resolved notification.
  • A planned broker restart produces a broker-removed anomaly (HIGH) and a broker-offline anomaly (CRITICAL), and therefore alerts. Maintenance windows do not suppress them.
  • Balancer alerts should not be the only monitoring of broker loss or disk capacity. AxonOps’ own broker and host alerts should remain in place.

Alerting is controlled per cluster by the Raise alerts from anomalies switch on the Adaptive Balancer card under Settings → Kafka, on AxonOps Cloud and self-hosted alike. The switch is on by default and is saved as soon as it is changed, independently of the card’s Save button. Changing it requires the Admin or SuperUser role.

Turning the switch off resolves every open balancer alert on the cluster and sends the resolved notifications; the balancer still records anomalies, and they still appear in Recent Anomalies. Turning it on again raises alerts only from the next report.

The execution items below are described with their failure semantics in Rebalancing a Kafka Cluster with the Adaptive Balancer, and the cluster model items in Kafka Adaptive Balancer Goals and Cluster Model.

Execution:

  • Execute (live) starts moving data immediately, with no confirmation. Only the most recent analysis can be executed. Its proposals are executed as computed; apart from confirming that the target brokers are alive, nothing checks them against changes in the cluster since the analysis ran, so a fresh analysis should be run immediately before executing.
  • Cancelling an execution stops new movements only. Reassignments already submitted continue in Kafka to completion, and nothing is rolled back.
  • An execution whose movements partly failed, or that reached the 2-hour execution limit, finishes in state COMPLETED. The Failed proposals count shows the failures.
  • A balancer restart during an execution marks it FAILED. Reassignments already submitted keep running, and the execution cannot be resumed.
  • Nothing prevents two executions on the same cluster at the same time. A new execution should not be started while another is RUNNING.
  • The data rate limit caps the bytes in flight, not bandwidth. No Kafka replication throttle is set.
  • A leader change completes in the balancer when the target broker is first in the replica list and in sync. Leadership itself moves only when Kafka runs a preferred leader election.
  • A partition with a replication factor of 1 is moved only off a drained or dead broker unless Allow single-replica moves is on. Under KIP-455, Kafka adds the new replica and lets it catch up before it removes the old one. When the source broker is alive, removing the old replica makes Kafka elect the new one as leader, and the partition is briefly unavailable during that leader change. When the source broker is dead, the balancer still proposes the move, because the partition is already unavailable, but the movement generally cannot complete: the new replica has no leader to copy from and, with the default unclean.leader.election.enable=false, cannot join the in-sync replicas (ISR). The movement fails after movement_stall_timeout (5 minutes) unless the dead broker returns or an operator enables unclean leader election for the topic, which discards any data that was only on the dead replica.
  • Disk moves are recorded as completed without moving any data, because AxonOps has no API for moves between log directories.

Cluster model:

  • Missing metrics are treated as zero load, and missing capacities fall back to defaults. The balancer logs a warning naming the brokers without a sample, but the analysis page does not report it. CPU load is host CPU, so other processes on a broker host count as broker load.
  • Load rates lag an execution by up to one metrics window. An analysis run within one metrics window of the last execution may propose moving load that has already moved.
  • JBOD (just a bunch of disks: several log directories on separate disks of one broker) is not modelled per disk.

Deployment:

  • Balancer state is held in memory, and history is persisted to axon-server and reloaded on start. The balancer has no mechanism that detects or prevents a second instance, so exactly one instance must run per organisation. Two instances would each hold their own state and submit reassignments independently.
  • Stored analyses, executions, and anomalies are deleted by axon-server after its events retention period, 1 week by default, and AxonOps provides no other way to delete them. History retention gives the full rules.
  • The balancer’s listeners use plain HTTP.
  • Clusters are registered with the balancer when it starts. A cluster added later requires a balancer restart.