Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

NTP Fundamentals for Cassandra

System clocks drift on their own and require active correction. This page covers how the Network Time Protocol (NTP) measures and corrects that drift, how time servers are organized into a stratum hierarchy, and which of absolute and relative accuracy Cassandra depends on.

Clock drift

Section titled “Clock drift”

A computer’s system clock is maintained by counting oscillations of a quartz crystal. The oscillator’s frequency is never exactly nominal, and it varies with temperature and load, so an undisciplined clock drifts away from real time at a rate that changes over the day. The magnitude is small per second and accumulates without bound: a clock left unsynchronized for weeks ends up wrong by an amount that matters to Cassandra long before it becomes visible to a human.

Virtual machines are more exposed than physical hardware. Guest timekeeping depends on the hypervisor scheduling the guest, so a loaded host produces larger and less predictable error than an idle one, and operations such as live migration, suspend and resume, or restoring from a snapshot can introduce a large discrete jump rather than gradual drift.

What NTP does

Section titled “What NTP does”

NTP corrects this. An NTP client periodically exchanges packets with one or more NTP servers over User Datagram Protocol (UDP) port 123. Each exchange records four timestamps (when the request left the client, when it arrived at the server, when the reply left the server, and when it arrived back at the client), from which the client computes both the round-trip delay and its own offset from the server’s clock. The client then disciplines the local clock in one of two ways:

  • Slewing: adjusting the rate at which the clock advances so the offset is removed gradually. Time remains monotonic, and no jump is observable to applications.
  • Stepping: setting the clock directly to the corrected value. This is fast, but it makes time jump, and a backward step means the same wall-clock instant occurs twice.

Which of the two a host performs, and when, is a configuration decision with direct consequences for a running Cassandra process. Configuring chrony covers how to constrain stepping to the safe window.

The stratum hierarchy

Section titled “The stratum hierarchy”

NTP servers are organized by stratum, a number counting hops away from a reference clock. A stratum 0 device is the reference clock itself, an atomic clock or a receiver for a global navigation satellite system (GNSS) that recovers time from the satellite signal. Stratum 0 devices are not network hosts and are not queried over NTP; they are attached to a server. That server is stratum 1, a server synchronizing to a stratum 1 server is stratum 2, and so on. A client normally selects the source with the lowest stratum and the most credible measurement.

The diagram below shows the hierarchy from a reference clock down to the Cassandra nodes and client application hosts that consume time from it.

NTP stratum hierarchy cluster_s0 Stratum 0: reference clocks, not network hosts cluster_s1 Stratum 1: servers attached directly to a reference clock cluster_s2 Stratum 2: servers synchronizing to stratum 1, where an internal tier sits cluster_s3 Stratum 3: hosts synchronizing to the internal tier gnss GNSS receiver s1a NTP server gnss->s1a direct attachment atomic Atomic clock s1c NTP server atomic->s1c direct attachment s1b NTP server s1a->s1b peer s2a NTP server s1a->s2a s1b->s1c peer s2b NTP server s1b->s2b s2c NTP server s1c->s2c s2a->s2b peer s3a Cassandra node s2a->s3a s2b->s2c peer s3b Cassandra node s2b->s3b s3c Client application host s2c->s3c hops Each hop adds network delay and dispersion, so accuracy degrades with distance from the reference clock.

Servers within one stratum peer with each other, shown by the bidirectional arrows. Peering does not move a server up or down a stratum; it gives the members of a tier a measurement of one another so the tier can identify and discard a member whose clock has gone wrong.

An internal NTP tier built for a Cassandra deployment typically sits at stratum 2, synchronizing to stratum 1 servers upstream, which puts the Cassandra nodes and client application hosts that synchronize to it at stratum 3.

Each hop adds network delay and dispersion to the estimate, so accuracy degrades with distance from the reference clock. That is why the recommended design keeps the number of hops between a Cassandra node and a trustworthy reference small, and keeps the last hop short and local.

Absolute compared with relative accuracy

Section titled “Absolute compared with relative accuracy”

Two different properties follow from this, and Cassandra cares about only one of them:

  • Absolute accuracy is how close a node’s clock is to real (UTC) time.
  • Relative accuracy is how closely two nodes agree with each other.

Cassandra’s ordering depends on relative accuracy. A cluster whose nodes are all one second behind UTC but agree with each other to within 100 microseconds behaves correctly. A cluster whose nodes are each within 10 ms of UTC but which disagree with each other by 20 ms does not.

Section titled “Related pages”