Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Cassandra Time Synchronization Monitoring

Time synchronization is an operational dependency and should be monitored as one. What the signals below protect is the timestamp ordering described in How Cassandra uses time: a host whose clock has stopped being disciplined writes data that is wrong on read, without producing an error. Silence from the time daemon is not evidence that the clock is right; chronyd continues running with Leap status: Not synchronised when it has lost all its sources.

The diagram below shows what is being measured and by which component on a single host. chronyd reads chrony.conf, talks to the configured Network Time Protocol (NTP) servers, and disciplines the system clock. The AxonOps agent reads the same system clock and independently queries the NTP servers, which is how it produces an offset figure rather than merely reporting what chronyd believes.

Time management and measurement on a Cassandra host cluster_host Linux host (Cassandra node or client application host) conf chrony.conf chronyd chronyd conf->chronyd configuration clock System clock chronyd->clock disciplines ntp NTP servers (internal tier or cloud time service) chronyd->ntp queries agent axon-agent agent->clock reads agent->ntp queries

What to collect

Section titled “What to collect”

What to collect from every Cassandra node and every client application host:

  • Offset against the synchronization source, from chronyc tracking (System time and Last offset).
  • Pairwise offset across the cluster, computed by comparing the per-node offsets centrally. This is the measurement that corresponds to what Cassandra depends on, and it is the one a per-node check cannot produce.
  • Jitter, as RMS offset and the per-source standard deviation from chronyc sourcestats. Rising jitter precedes rising offset.
  • Stratum, and the identity of the reference source. A node that has silently fallen back to a different source, or to a much higher stratum, has lost its intended reference.
  • Source reachability, from the Reach column of chronyc sources, and the count of sources currently selectable. Persistent values below 377 indicate loss.
  • Synchronization state, from Leap status. Anything other than Normal is an alert in its own right.
  • Step events, from the chrony log. A step on a running Cassandra node is an incident, not a routine correction.

The commands that produce these readings, and what healthy output looks like, are covered in Configuring chrony.

The degradation from three reachable tier servers to one is the case worth designing the alerting around, because the obvious signal does not fire for it. While at least one source is still selected and the minsources threshold is met, Leap status stays Normal and the reported offset stays small, so a check that watches only those two fields sees a healthy node. Two signals do catch it: the source reachability count, which drops as servers are lost, and the pairwise node-to-node offset comparison, which widens once different nodes fall back to different remaining sources. Both should be alerted on independently of Leap status, since by the time Leap status changes the node has already stopped being disciplined.

Suggested thresholds

Section titled “Suggested thresholds”

Suggested starting thresholds, to be tuned to the workload rather than treated as limits:

SignalWarningInvestigate immediately
Offset against sourceabove 10 msabove 100 ms
Pairwise skew between any two nodesabove 10 msabove 100 ms
Leap statusnot Normal for more than one poll intervalnot Normal sustained
Source reachabilityReach below 377all sources unreachable

A workload that issues conflicting writes to the same cells at intervals shorter than 10 ms requires correspondingly tighter thresholds, derived as described in How much skew is tolerable.

The AxonOps agent

Section titled “The AxonOps agent”

The AxonOps agent (axon-agent) measures the NTP offset on each monitored Cassandra host and reports it as the host_ntp_offset_seconds metric, alongside the other host metrics, which makes the per-node offsets available for the cluster-wide comparison described above. Where a dedicated internal NTP tier is in use, or where the agent’s automatic detection of the configured time source fails, which is the common case for Cassandra deployed on Kubernetes, the NTP: section of /etc/axonops/axon-agent.yml lists the servers the agent measures against, with port 123 assumed if no port is given. See AxonOps agent configuration for the agent configuration file.

Section titled “Related pages”