Configuring chrony for Cassandra

chrony is the default Network Time Protocol (NTP) implementation on current RHEL-family distributions and on recent Debian and Ubuntu releases, and should be preferred over the legacy ntpd for new deployments. It converges faster after start, handles intermittent network connectivity without losing its frequency estimate, and copes better with the discrete clock jumps that virtual machines experience (chrony comparison of NTP implementations).

The directives shown on this page are those of current chrony releases. Where a directive was introduced in a specific version, that version is stated; nts requires chrony 4.0 or later. The topology these files implement is described in Deployment architecture, and the reason a Cassandra deployment depends on this configuration at all is described in How Cassandra uses time.

Prerequisite: chrony must be installed

Most general-purpose distribution images install chrony, but minimal, container-base, and hardened images frequently omit it, and a host with no time daemon reports no error while its clock free-runs. Confirm the package is present before configuring it:

# RHEL, CentOS Stream, Rocky, Alma, Amazon Linux
sudo dnf install -y chrony

# Debian, Ubuntu
sudo apt-get install -y chrony

chronyd --version

Installing the package does not by itself make the host correct: on a machine that has been running unsynchronized, the first start of chronyd may find a large offset, which is addressed in the applying section below.

Configuration on a Cassandra node or client application host

# /etc/chrony.conf (Debian and Ubuntu use /etc/chrony/chrony.conf)

# The internal NTP tier. Every Cassandra node and every client application
# host lists exactly these servers and no external ones, and marks the same
# server as preferred so the whole cluster tracks one common source.
server ntp1.internal.example.com iburst prefer
server ntp2.internal.example.com iburst
server ntp3.internal.example.com iburst

# Persist the measured oscillator rate error so it is applied immediately at
# the next start instead of being relearned.
driftfile /var/lib/chrony/drift

# Allow a step, rather than a slew, only for the first three updates after
# chronyd starts, and only if the offset exceeds one second.
makestep 1.0 3

# Copy system time to the real-time clock periodically (Linux).
rtcsync

# Require at least two agreeing sources before the clock is disciplined.
# Trade-off: see the note below on what happens when only one source is left.
minsources 2

logdir /var/log/chrony
log tracking measurements statistics

The directives that matter for a Cassandra host:

server <host> iburst names an upstream server. iburst sends a rapid burst of packets at start so the first usable measurement is available in seconds rather than minutes.
prefer biases source selection toward one server, as described below.
driftfile records the local oscillator's rate error. Without it, chronyd relearns the rate after every restart and the clock is less accurate in the interim.
makestep <threshold> <limit> permits chronyd to step the clock when the offset exceeds <threshold> seconds, but only for the first <limit> clock updates after chronyd starts. makestep 1.0 3 therefore allows a large correction at boot, when nothing is running yet, and forces every later correction to be slewed. A negative <limit> would allow stepping at any time and must not be used on a Cassandra host.
rtcsync keeps the hardware real-time clock aligned so the machine starts with an approximately correct clock after a reboot.
minsources 2 sets the number of sources that must agree before chronyd will discipline the clock. The chrony default is 1.

minsources is a trade-off rather than pure hardening, and the boundary is worth stating exactly. With three tier servers configured and minsources 2, the loss of one server is absorbed: two remain, they agree, and the clock continues to be disciplined normally. The loss of a second leaves one reachable source, which is below the threshold, so chronyd stops actively disciplining the clock. It does not step, jump, or start following the single remaining source blindly; it continues to advance the clock using the rate correction recorded in the drift file, and the accumulated error grows from that point at the oscillator's residual rate error. The clock is therefore free-running, and chronyc tracking reports Leap status: Not synchronised.

The alternative, leaving the default of 1, keeps a node disciplined against a single surviving server, at the cost of following that server without any second opinion, which is the exact case where a server that has itself gone wrong drags the node with it. minsources 2 buys protection against that in exchange for requiring two live sources. Either setting makes the number of reachable sources an operational signal that has to be monitored, which Monitoring covers.

An NTP client corrects an offset in one of two ways, and the difference governs what is safe on a running node: stepping sets the clock directly, so time jumps discontinuously, while slewing changes the rate at which the clock advances so the offset is removed gradually and time stays monotonic. What NTP does describes both.

Do not step the clock while Cassandra is running

Stepping the clock backwards on a running node makes every subsequent write from that node carry a timestamp older than data the same node has already written. Those writes are stored and acknowledged, and then permanently ignored on read. Stepping forwards has the mirror effect: writes and tombstones carry future timestamps that shadow legitimate writes until the cluster's wall clock catches up. Cassandra reports no error in either case.

Corrections must be applied by slewing while Cassandra is running. A node found to be badly wrong should be drained and the Cassandra service stopped before chronyc makestep, chronyd -q, ntpdate, or any equivalent one-shot correction is run.

A node whose clock has been stepped backwards while serving traffic has written data of unknown correctness for the duration of the offset. Recovery is an anti-entropy repair over the tables that node coordinated during that window, run once the clock is correct and stable with nodetool, Cassandra's administrative command-line tool: nodetool repair -full <keyspace> <table>. Repair reconciles replicas to the highest-timestamp version, so it removes divergence between replicas but does not undo a wrong timestamp; a write that was stamped ahead of the cluster still wins until it is overwritten with a correct, higher timestamp. Where a specific value is known to be wrong, it must be rewritten explicitly. See Repair Architecture for the repair options and their coverage.

Preferring one common source on every node

The prefer option on a server line tells chronyd to bias source selection toward that server. Among the sources that pass chronyd's selection tests, the preferred one is chosen as the synchronization reference even when another source presents a slightly better measurement. If the preferred server becomes unreachable, or is rejected as an outlier by the other configured sources, chronyd falls back to the remaining servers automatically, and returns to the preferred server once it recovers. No operator action is involved in either transition.

The pattern this supports is to give every Cassandra node and every client application host the same set of internal tier servers, and to mark the same server prefer on all of them. Every node then disciplines its clock against one common source rather than against whichever tier member each node independently rates highest. Node-to-node skew is consequently bounded by each node's own offset to that single server, rather than by the spread across three tier members plus the differing paths to them. This tightens relative accuracy, which is the property Cassandra's timestamp ordering depends on. In this arrangement the other two tier servers are fallbacks and cross-checks rather than co-equal sources.

Two caveats apply. First, the improvement holds only while the preferred server is being followed: during a failover, nodes may briefly settle on different fallback sources, so the tier members must still be tightly peered with one another to bound how far apart those fallbacks can be. Second, at least three sources should remain configured on every node. prefer biases selection among the sources that pass the falseticker tests; it does not override those tests, and it is the presence of two other sources that lets chronyd identify a preferred server whose time has gone wrong and reject it. Marking a server prefer and configuring it as the only source removes that protection.

The AWS configuration, server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4, uses the same option for the same reason.

Applying the configuration to a Cassandra node

Writing the configuration file is not the hazardous part; starting or restarting chronyd is, because the first three clock updates after chronyd starts are permitted to step under makestep 1.0 3. On a node whose clock has drifted past one second, enabling chronyd while Cassandra is serving traffic produces exactly the step the warning above prohibits.

Measure the offset before starting the daemon. chronyd's -Q mode performs the measurement and prints the result without adjusting the clock at all:

# Measure only. -Q never sets the clock.
sudo chronyd -Q 'server ntp1.internal.example.com iburst'

# On a host where chronyd already runs, the equivalent reading is:
chronyc tracking

The output line to read is of the form System clock wrong by 0.000431 seconds (ignored). Decide from its magnitude:

Offset comfortably below the makestep threshold of one second: chronyd may be enabled and started with Cassandra running. The correction is slewed.
Offset at or above the threshold, or unknown: drain the node and stop Cassandra first, then start chronyd, let it settle, and start Cassandra again once chronyc tracking reports Leap status: Normal.

# Only after the offset check above.
sudo systemctl enable --now chronyd
sudo systemctl restart chronyd     # after any later edit to chrony.conf
chronyc tracking
chronyc sources -v

The same procedure applies to a client application host, minus the drain: the application should be stopped rather than left issuing writes across a clock step.

Re-applying an unchanged configuration and restarting chronyd is idempotent and safe on a node that is already synchronized, since the offset at restart is small and every correction is slewed. It is not safe on a node that has accumulated a large offset, for example one whose chronyd has been stopped or unable to reach its sources for a long period. Configuration management that restarts chronyd must therefore run the offset check above rather than assuming a restart is inert.

Configuration on the internal NTP tier

# /etc/chrony.conf on each internal NTP server (ntp0, ntp1, ntp2)

# Upstream reference. Use the public pool as shown, an appliance disciplined
# by a global navigation satellite system (GNSS), a national time service,
# or an authenticated external service.
# Only these three servers reach outside the network.
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst

# Symmetric peering with the other members of the tier so the tier measures
# and reconciles its own members. Each server lists the other two; the lines
# below are the ones for ntp0.
peer ntp2.internal.example.com iburst
peer ntp3.internal.example.com iburst

# Serve time only to the Cassandra and application networks.
allow 10.20.0.0/16
allow 10.30.0.0/16

# If every upstream source is lost, continue serving at a high stratum so
# clients keep a common reference. The orphan option makes the tier select a
# single member deterministically instead of the members following each other
# in a loop.
local stratum 10 orphan

driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

Notes on the tier configuration:

allow <network> is what makes chronyd serve time to clients. chronyd serves nothing by default, so the allow directives must be present on the tier servers and must not be present on Cassandra nodes.
local stratum 10 orphan is a degraded-mode fallback. The high stratum means any real upstream source is preferred whenever one is available. The orphan option is what prevents the peers from circularly following one another once the external reference is gone.
The peers are at the same stratum as each other and at a higher stratum than the upstream source, so peering refines agreement between the tier members rather than overriding the upstream reference.

Apply the tier configuration with sudo systemctl restart chronyd on one server at a time, confirming with chronyc tracking and chronyc sources -v that each server has resynchronized before moving to the next. The stepping precaution that governs Cassandra nodes does not apply to a tier server, which runs no Cassandra process, but restarting the whole tier at once removes every source the cluster has and should be avoided. If tier members peer across datacenters, the firewall must permit UDP 123 between them in both directions, which Security covers.

Verifying synchronization

chronyc tracking reports the state of the local clock:

Reference ID    : 0A140001 (ntp1.internal.example.com)
Stratum         : 3
Ref time (UTC)  : Mon Aug 24 09:14:22 2026
System time     : 0.000012431 seconds slow of NTP time
Last offset     : -0.000004512 seconds
RMS offset      : 0.000021837 seconds
Frequency       : 12.401 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.043 ppm
Root delay      : 0.000842 seconds
Root dispersion : 0.000391 seconds
Update interval : 64.2 seconds
Leap status     : Normal

A healthy Cassandra host shows Leap status: Normal (any other value, in particular Not synchronised, means the clock is not being disciplined), a System time offset well under a millisecond on a local network, an RMS offset of the same order, a stable Stratum, and a Reference ID naming one of the intended internal servers.

chronyc sources -v lists the sources and how each is being used:

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp1.internal.example.com              2   6   377     41   -4512ns[-4512ns] +/-  431us
^+ ntp2.internal.example.com              2   6   377     38   +6120ns[+6120ns] +/-  455us
^+ ntp3.internal.example.com              2   6   377     44   -8901ns[-8901ns] +/-  502us

The mode and state characters are the diagnostic. ^* marks the source currently selected as the synchronization reference, ^+ marks other sources being combined with it, ^- marks a source excluded by the combining algorithm, ^? marks a source that is unreachable, and ^x marks a falseticker whose time disagrees with the majority. Reach is an octal register of the last eight polls, so 377 means all eight succeeded and anything persistently lower indicates packet loss or an unavailable server.

chronyc sourcestats reports the regression chrony has fitted per source, including the number of samples retained, the estimated frequency offset, and the standard deviation of the residuals. A source with a large standard deviation relative to the others is contributing noise and should be investigated.

For a cluster-wide view, the offsets reported on individual nodes are not sufficient on their own, because two nodes can each report a small offset against different sources while disagreeing with each other. Collecting the offset from every node into one place and comparing them pairwise is what verifies relative synchronization.

If synchronization does not establish

Three symptoms account for most cases where a host will not synchronize.

chronyd is not running. systemctl status chronyd reports the failure, and journalctl -u chronyd carries the reason, most often a syntax error in chrony.conf or a directory named in driftfile or logdir that does not exist. chronyd exits on a configuration error rather than starting with a partial configuration.

Sources show ^?. The host is not receiving replies. Confirm the tier hostnames resolve, then confirm UDP 123 is open in both directions between the host and every tier server, since NTP replies arrive as UDP and a firewall permitting only the outbound direction blocks them. The firewall scoping this requires is described in Security. chronyc sources -v continues to show ^? for a source until a reply arrives; chronyc ntpdata <server> reports what has and has not been received for one source.

Leap status: Not synchronised with sources reachable. Either too few sources meet the minsources threshold, discussed above, or the sources disagree enough that chronyd cannot select among them, which appears as ^x falseticker markings. Two tier servers with materially different time, or one smeared and one stepping source in the same list, both produce this. Resolve the disagreement at the tier rather than by lowering the threshold on the client.

If a large offset has accumulated during any of the above, the node's clock must be corrected under the stepping precaution described earlier: drain the node and stop Cassandra before applying a step.

Other time daemons

ntpd equivalents

Deployments that already run ntpd express the same architecture with server and peer lines, restrict directives in place of chrony's allow, and tos orphan <stratum> in place of local stratum <n> orphan. State is inspected with ntpq -p, which produces a source list comparable to chronyc sources, and ntpq -c rv, which reports the equivalent of chronyc tracking. The architectural guidance in this section applies unchanged; only the syntax differs. New deployments should use chrony.

systemd-timesyncd

systemd-timesyncd is an SNTP client. It queries one server at a time, does not run the source selection and combining algorithms that let a client identify a misbehaving server among several, and cannot serve time to other hosts. It is acceptable for hosts with no timestamp-ordering requirement. Cassandra nodes and client application hosts that generate write timestamps should run chrony instead.

Only one daemon may discipline the clock

Two time services correcting the same clock fight each other and produce worse results than either alone. Before enabling chrony, find which other time daemons are installed and stop them:

# Which time daemons exist on this host.
systemctl list-unit-files 'systemd-timesyncd*' 'ntp*' 'ntpd*' 'chrony*'

# Disable the ones that are present. A unit that is not installed reports
# "Unit file ... does not exist", which is safe to ignore; any other failure
# is not, so the output is read rather than discarded.
sudo systemctl disable --now systemd-timesyncd
sudo systemctl disable --now ntpd

Starting chronyd is a separate step, and on a host running Cassandra it is subject to the offset check in the applying section above, because the first clock updates after chronyd starts are permitted to step.

Virtualized environments add a second source of correction. On VMware, the periodic time synchronization performed by VMware Tools must not be combined with chrony; one or the other should discipline the guest clock, and for a Cassandra guest that should be chrony. On Hyper-V and Azure, the recommended arrangement is chrony taking time from the host-provided PTP device rather than a separate guest time-sync service also adjusting the clock. Container workloads do not have their own clock: a container shares the host kernel's clock, so the time daemon runs on the host, not in the container.

The kernel clocksource

chrony disciplines the system clock, but the kernel does not measure time by itself: it reads a hardware counter, the clocksource, and derives the system clock from it. An unstable or low-resolution clocksource limits how well the system clock can be maintained regardless of how good the NTP configuration is, so the clocksource is worth checking on any host whose timekeeping matters.

On modern x86_64 hardware the preferred clocksource is the TSC (Time Stamp Counter), a per-core counter that is cheap to read and high resolution. The kernel uses it only when the CPU reports the counter as constant and invariant, meaning it advances at a fixed rate independent of the core's current frequency and power state. On KVM guests the paravirtual kvm-clock is typical, and the hypervisor supplies the reading.

The active and available clocksources are readable from sysfs:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

A host with a reliable TSC that is nevertheless running on hpet or acpi_pm has fallen back, usually because the kernel's clocksource watchdog observed the TSC misbehaving or because firmware reported it as unusable. Both fallbacks are slower to read and coarser than the TSC, and the fallback itself is a signal that the platform's timekeeping should be investigated rather than simply overridden.

The kernel boot parameters clocksource=tsc and tsc=reliable exist to force the choice, but tsc=reliable disables the watchdog that would otherwise detect a TSC drifting or jumping between cores. It should be set only where the hardware's TSC is known to be invariant and the fallback has been traced to a firmware reporting defect rather than a real instability, and it should not be applied as a general tuning step.

On older hardware without an invariant TSC, CPU frequency scaling changes the rate at which the counter advances, which is the reason such systems fall back to a slower but rate-stable clocksource. Cassandra hosts on that class of hardware should be verified individually rather than assumed correct.

Further material on Linux-level time configuration for Cassandra hosts is available in the AxonOps article Linux time management for Cassandra.

Deployment architecture - the topology these files implement
Monitoring - collecting these readings continuously rather than by hand
Security - firewall rules and authenticated sources