Kafka Config Drift Detection

Config Drift Detection reads the configuration state of a Kafka cluster on a fixed interval, compares each reading against the previously recorded state, and writes every difference to a change history with a full before-and-after diff. Each recorded change is attributed either to the AxonOps user who made it or to a change made outside AxonOps, and a change made outside AxonOps raises a critical alert.

Five resource types are covered, each tracked independently:

Topics: topic metadata and the full topic configuration
ACLs: every access control list (ACL) binding on the cluster
Connectors: Kafka Connect connectors and their configuration, per Connect cluster
Schemas: Schema Registry subjects, versions, and schema content
Cluster: broker configuration for every broker in the cluster

Availability

Requirement	Value
Cluster type	Apache Kafka clusters only
Role	Admin or SuperUser
Location	Cluster → Settings → Config Drift Detection

The cluster must already be connected to AxonOps and visible in the UI. See Kafka agent setup for connecting a cluster.

Three resource types have their own prerequisites, and a resource type whose prerequisite is unmet records nothing even when its toggle is on:

Schemas requires the Kafka Schema Registry feature to be enabled for the organisation. Schema Registry is a separate service (Confluent Schema Registry, or an API-compatible implementation such as Karapace or Apicurio Registry); it is not part of Apache Kafka itself. See Schema Registry.
Connectors requires at least one Kafka Connect cluster to be configured in AxonOps. See Kafka Connect.
ACLs requires an authorizer to be configured on the Kafka cluster, through authorizer.class.name on the brokers. Brokers running without an authorizer reject the request to list ACLs, and the check ends without recording anything.

Enabling drift detection

Drift detection is enabled in two places, and both are required before anything is recorded.

Storage is enabled once for the organisation. On the Configuration tab of the Config Drift Detection page, turn on Enabled in the Storage section and select Save. This switch is shared by every cluster in the organisation, so enabling it for one Kafka cluster enables it for all of them. It does not on its own cause anything to be recorded, because each cluster still selects which resource types to track.

Resource tracking is enabled per cluster. In the Config Drift Detection section of the same tab, turn on each resource type to track: Topics, ACLs, Connectors, Schemas, Cluster. Each toggle saves immediately and applies only to the cluster currently selected. A resource type left off is not read, and produces no history entries or alerts.

To confirm the feature is working, wait one check interval (300 seconds) after enabling a resource type, then open the Changes tab. An entry with the change source Snapshot should appear for that resource type. Where no entry appears after several intervals, check that both the organisation Enabled switch and the resource type toggle are on, that the resource type’s prerequisites are met, and that the axon-server log does not report a read failure.

Change history is stored in the AxonOps database.

Turning tracking off

Disabling the organisation Enabled switch stops the checker. It does not delete the recorded change history or the stored state, both of which remain available.

Re-enabling resumes from the stored state rather than starting a new baseline, and the same is true after an axon-server restart. Any configuration change made while tracking was off is therefore detected on the first cycle after it is turned back on. Because the attribution record expires within one check interval, such a change is attributed to a source outside AxonOps and raises a critical alert, even where an AxonOps user made it.

Tracked resource types

Resource type	Captured state	Recorded as
`topics`	Topic summary plus every topic configuration entry	One entry per topic
`acls`	Each ACL binding, identified by resource type, pattern type, and resource name	One entry per ACL resource
`connectors`	Connector definition and configuration	One entry per Connect cluster
`schemas`	Subject, version, schema content and references, and subject configuration	One entry per subject
`cluster`	Broker configuration for every broker in the cluster	A single entry for the cluster

State is recorded per item, so a diff names the topic, ACL, or subject that changed rather than reporting the whole resource type as different. Connectors are the exception: they are recorded one entry per Connect cluster, so a connector change produces a diff against that Connect cluster’s entry.

Topic and broker configuration entries

The topic configuration captured is the full effective configuration, not only the entries set explicitly on the topic. Each entry records where its value came from, so an inherited value can be told apart from one set on the topic.

Changing a broker-level default therefore alters the effective configuration of every topic that inherits it, and each of those topics is recorded as changed. The change is real, but its origin is the broker default rather than the topics themselves.

Credentials in captured configuration

Credentials are not captured. Brokers return configuration entries marked sensitive with an empty value, so broker and topic secrets never leave the cluster. For Kafka Connect, axon-agent replaces credential values with ******** before sending the configuration to axon-server, so the real value is absent from the change history, the diff view, and the state export.

Connect masking works on the name of the configuration key, matching names that contain password, secret, credential, token, or an API, access, or private key. A credential stored under a key name that does not follow one of those conventions is captured as written.

Because the mask is a constant, drift detection cannot see a change behind it. Rotating a connector password leaves ******** in place on both readings, so the rotation produces no diff, no history entry, and no alert.

How detection works

AxonOps reads the configuration of every enabled resource type every 300 seconds and compares each reading against the last one it recorded. Where nothing has changed, the cycle ends and nothing is written. Where something has changed, AxonOps identifies the individual topics, ACLs, subjects, or connectors that differ, records each one with a before-and-after diff, and raises a critical alert for any change it cannot attribute to an AxonOps user.

The following diagram shows one drift detection cycle for a single resource type.

First observation

The first time a resource type is observed for a cluster, its state is recorded as a snapshot with the change source Snapshot, and no alert is raised. There is no earlier state to compare against, so the reading establishes the baseline rather than reporting drift.

One exception applies. Where a resource type comes into existence through an AxonOps action, such as the first connector created on a Connect cluster, the reading is recorded as a change rather than a snapshot so that the user who made it is still credited.

An axon-server restart does not produce a fresh snapshot or a spurious alert. The baseline is restored from the stored state instead of being recorded again.

Change attribution

Every change made through AxonOps is recorded against the cluster, the resource type, the item, and the user who made it. That record is held for the length of the check interval plus 60 seconds, which is long enough for the next check to find the change and match it to its author.

Change source	Meaning	Alert raised
Snapshot	Baseline reading for a resource type not previously observed	❌
AxonOps	A matching AxonOps operation was recorded within the attribution window	❌
Outside AxonOps	No matching AxonOps operation was found	✅

Legend: ✅ alert raised | ❌ no alert.

Where more than one AxonOps operation on the same item falls inside the window, the most recent one is used.

Creations and deletions are matched separately. An external deletion of an item that AxonOps created moments earlier is attributed to a source outside AxonOps, not to the user who created it.

A single check cycle may contain both kinds of change. Each item is attributed on its own, so one external edit does not reclassify the legitimate AxonOps changes made in the same cycle.

The Change Source column in the Changes table shows AxonOps for any change attributed to a user. The diff view shows the AxonOps username itself.

Alerting

A change attributed to a source outside AxonOps raises an alert with the following properties.

Property	Value
Title	`Configuration drift: <resource type>`
Severity	Critical
Summary	A per-item description of what changed, followed by the change reference: the identifier of the stored history entry, shown as the Commit column on the Changes tab
Routing	The cluster’s configured notification integrations
Retention	The cluster’s active alert retention for cluster settings

The alert is delivered through the same notification integrations as every other AxonOps alert. See Notification Integrations for configuring the destinations.

Reviewing changes

The Changes tab lists the recorded changes for the selected cluster over a time range.

From and To set the range. Both default to a window ending at the current time.
Limit caps the number of returned entries at 50, 100, 200, or 500.
The refresh control re-runs the query.

The table has the following columns.

Column	Content
Date	When the change was recorded, in the browser’s local time
Change Source	`Snapshot`, `AxonOps`, or `Outside AxonOps`
Resource Type	The resource type the change belongs to
Message	The commit message for the change
Commit	The first eight characters of the change reference

Selecting a row opens the diff for that change. The diff shows one block per changed item, with the previous state on the left and the current state on the right, and highlights the lines that differ. The header of the diff view shows the full change reference, the message, the author, and the timestamp.

Because the Limit control caps the result set, a range containing more changes than the limit returns only the most recent entries within it. Narrow the range rather than raising the limit when a complete record of a period is needed.

Downloading the current state

The Cluster State section of the Configuration tab exports the recorded state as JSON. Select a resource type, or All resources, and select Download. The file is saved as config.json, or as config-<resource-type>.json where a single resource type is selected. The name does not identify the cluster, so exports from several clusters must be renamed to be told apart.

The export contains the state as it is currently recorded, which is the state as of the last completed check cycle, not a fresh read of the cluster.

Failure semantics

Drift detection reads live cluster state, and each cycle can partially fail. The contract is as follows.

A failed read is never recorded as a deletion. The affected check ends without writing, and the next successful cycle picks up from the last recorded state.
Where no Kafka broker is reachable, the cluster, topics, and ACL checks are skipped for that cycle. The schemas and connectors checks still run, because they read from the Schema Registry and Connect rather than from a broker.
A genuine deletion is recorded, including the deletion of the last topic, ACL, subject, or connector. Losing contact with every broker is not treated as the removal of every broker.
Where the topic list is read but the topic configurations cannot be, the topics are stored without their configuration. The next successful cycle restores the configuration and records that as a change.
Where a change cannot be written to the history, the alert is still raised and the failure is logged, but that change does not appear on the Changes tab and does not reappear on a later cycle. A drift alert with no matching history entry indicates this condition, and the axon-server log records the underlying failure.

Kafka Topics: creating and configuring the topics whose changes are tracked
Topic Access Control List (ACL): the ACL bindings tracked by the acls resource type
Kafka Connect: the Connect clusters whose connectors are tracked
Schema Registry: the subjects tracked by the schemas resource type
Kafka Advanced Checks: the other per-cluster checks configured on the Kafka settings page
Notification Integrations: where drift alerts are delivered