Cassandra Time Synchronization Architecture

The design goal is to make every machine that generates a Cassandra timestamp, meaning cluster nodes and client application hosts, agree with each other as closely as possible. The means is a source set that is identical for every one of those machines and near enough to be measured accurately. Which machines generate those timestamps, and why their clocks decide the outcome of conflicting writes, is covered in How Cassandra uses time.

The diagram below shows the recommended topology for a single datacenter. Three internal Network Time Protocol (NTP) servers, one per availability zone, are the only hosts that reach the upstream sources, and they peer with one another. Every Cassandra node and client application host lists all three and marks the same one as preferred. A cluster spanning several datacenters repeats this pattern once per datacenter rather than extending one tier across the network between them.

The tier members synchronize to the upstream sources and peer with each other, and every Cassandra node and client application host in that datacenter synchronizes to the tier and to nothing else. The upstream reference can be the public pool as shown, an on-site appliance disciplined by a global navigation satellite system (GNSS), or an external service authenticated with Network Time Security, which Security covers.

Do Not Do This

Problem: Configuring each Cassandra node to synchronize independently to pool.ntp.org or another public NTP service. The pool resolves to a different, rotating set of servers per client, so each node ends up disciplined against different upstream clocks over different network paths. The arrangement optimizes each node's absolute accuracy and never measures the node-to-node agreement that Cassandra's timestamp ordering actually depends on. It also makes every node dependent on internet reachability, and exposes every node to unauthenticated time from servers outside the organization's control.

Symptoms: Each node reports a healthy offset against its own sources while pairwise skew across the cluster is materially larger; updates that appear not to take effect; deleted rows that reappear; repair that does not converge.

Instead: Point every Cassandra node and every client application host at the same small internal NTP tier, and let that tier be the only component that talks to external time sources.

# BAD: each node picks its own upstream servers, no relative guarantee
pool 2.pool.ntp.org iburst

# GOOD: every node disciplines against the same three internal servers,
# preferring the same one of them
server ntp1.internal.example.com iburst prefer
server ntp2.internal.example.com iburst
server ntp3.internal.example.com iburst

Establish what already exists before designing anything

Enterprise and self-hosted environments usually have time synchronization infrastructure and policy already in place: a corporate NTP service, GPS-disciplined appliances in the datacenter, an Active Directory domain whose controllers serve time to domain members, or a security or compliance policy mandating specific time sources. The first step is therefore to find out what exists and who owns it, not to design a tier.

Where a managed service already exists, the work is to verify it against the requirements set out in this section rather than to build something new. The questions are whether it delivers tight relative accuracy across every Cassandra node and client application host, whether one consistent discipline applies across every replicating datacenter (all smeared or all non-smeared), whether it is reachable from the Cassandra and application networks, and whether its state is monitored. Standing up a parallel, unsanctioned NTP tier alongside a sanctioned corporate service can breach policy on its own, and it produces exactly the mixed-source condition this section warns against. The internal-tier design that follows applies where no suitable service exists, and otherwise is the specification against which an existing service should be evaluated.

The internal NTP tier

For on-premises and self-managed deployments, the recommended arrangement is:

Three or more dedicated internal NTP servers, placed in different racks or availability zones so that the loss of one fault domain does not remove time service.
Each internal server synchronizes to trusted upstream sources, and peers symmetrically with the other internal servers so that the tier measures and reconciles its own members' clocks.
Every Cassandra node and every client application host lists all internal servers and none of the upstream ones.

This produces tight relative synchronization across the cluster (all nodes discipline against the same small set of servers, over short low-latency internal paths), acceptable absolute accuracy inherited from the upstream sources, tolerance of losing an internal server, and a single controlled point of egress for NTP traffic.

Three servers is the practical minimum. With two servers, a client that sees them disagree has no basis to decide which is wrong. With three, the client's source selection algorithm can identify and discard a single misbehaving server.

Three also remains adequate as the fleet grows. An NTP exchange is a single small request and response, so the load a client places on a server is negligible and does not scale with how much work the client is doing. Tier members are added for fault-domain coverage, such as a fourth or fifth availability zone, rather than to absorb query load. A large fleet is a reason to check that every fault domain has a nearby tier member, not a reason to size the tier by node count.

Preferring one common source

The prefer option shown against ntp1.internal.example.com in the diagram is what makes every node track one common server rather than whichever tier member each node independently rates highest, which bounds node-to-node skew by each node's offset to that single server. The option's exact selection and failover behaviour, and the two caveats that apply to it, are covered in Configuring chrony.

The tier in a multi-datacenter cluster

Each datacenter should run its own internal NTP tier, synchronized to upstream sources near it. Pointing one datacenter's Cassandra nodes at another datacenter's NTP servers across a wide area network (WAN) link degrades accuracy, because NTP estimates offset from the round-trip delay on the assumption that the outbound and return paths take similar time. WAN paths are frequently asymmetric, and the error that asymmetry introduces is roughly half the difference between the two directions, which on a long or congested link is far larger than the millisecond-scale budget the cluster needs.

Consistency between the tiers comes from the upstream sources rather than from cross-WAN client synchronization. Every tier, in every datacenter, synchronizes to sources of the same kind and the same discipline, so all of them converge on the same time even though no node crosses the WAN to get it. A peer relationship between the tiers across datacenters may be added on top of that as a cross-check, which gives each tier a measurement of the other and makes a divergence visible, but it is not the primary mechanism and should not replace local upstream sources.

The alternative, one shared tier serving every datacenter, is simpler to build and operate: one set of servers, one set of upstream relationships, one configuration. Its costs are that the WAN sits inside the synchronization path for every remote node, with the asymmetry error described above, and that the loss of the site hosting the tier stops active clock discipline in every datacenter at once rather than in one. Per-datacenter tiers cost more infrastructure and more configuration, and in exchange remove both the WAN from the sync path and the cross-datacenter blast radius. For a production cluster whose datacenters replicate with each other, the per-datacenter design should be preferred; a shared tier is defensible only where the datacenters are close, the link between them is low-latency and symmetric, and the loss of one site taking time service with it is an accepted risk.

The constraint that does not relax with distance is the discipline itself. Every datacenter that replicates with any other belongs to one correctness domain and must use the same handling of leap seconds, either all smeared or all stepping, as described below.

Alternatives and their trade-offs

Arrangement	Relative accuracy	Absolute accuracy	Availability	Notes
Dedicated internal tier of three or more peered servers, one tier per datacenter	Good	Good	Good	Recommended for on-premises and self-managed deployments
Cloud provider time service	Good	Good	Good	Recommended only where the whole cluster runs in that one provider. A cluster spanning providers, or spanning cloud and on-premises, must not let each site use its local provider's default: see the multi-cloud rule below
Full mesh peering of all Cassandra nodes with orphan mode	Good	Degrades over time without an external reference	Good	Configuration must be regenerated whenever the cluster is scaled
A single internal NTP server	Good while it is up	Good	Single point of failure	Its failure leaves the whole cluster free-running
Each node synchronizing independently to public pools	Poor	Good per node	Depends on internet reachability	Anti-pattern for Cassandra

Full mesh peering (every node peering with every other node, with orphan mode selecting a fallback leader when no external reference is available) gives excellent relative synchronization, which is the property Cassandra needs. Its drawbacks are that the whole cluster's absolute time drifts together when the external reference is lost, and that the configuration on every node changes every time a node is added or removed. For clusters that change size, the dedicated tier is easier to operate.

Mixing external servers and internal peers naively does not work as intended. Source selection prefers a lower stratum, so a server that lists both an external stratum 2 source and same-stratum internal peers will follow the external source and treat the peers as nearly irrelevant. Peering supplies mutual measurement between servers of comparable stratum; it does not override a better-rated source.

Cloud time services

Each major cloud provider operates a time service reachable from inside its own network, either link-local or supplied by the hypervisor. These sources have short and stable path delay, are consistent across the instances in that environment, and remove the need to build and operate an internal tier. In a single-provider deployment, the provider's service is the recommended choice.

Provider	Source	Directive
AWS	Amazon Time Sync Service, link-local at `169.254.169.123`, or `fd00:ec2::123` on IPv6 (Nitro instances)	`server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4`
Google Cloud	The metadata server, `metadata.google.internal`	`server metadata.google.internal iburst`
Azure	Host clock exposed as a Precision Time Protocol (PTP) device at the stable symlink `/dev/ptp_hyperv`	`refclock PHC /dev/ptp_hyperv poll 3 dpoll -2 offset 0 stratum 2`

AWS

The Amazon Time Sync Service is reachable from every EC2 instance at the link-local address 169.254.169.123, and over IPv6 at fd00:ec2::123 on Nitro-based instances. Amazon Linux 2023 and recent Amazon Linux 2 AMIs are configured to use it by default. The chrony directive for it is server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4 (EC2 time synchronization documentation). It uses the same prefer option, for the same reason, as the internal-tier configuration above: the link-local endpoint is the source every instance in that environment should track, and any additional source is a fallback.

The link-local NTP endpoint presents a leap-smeared view of UTC.

Instances launched into a placement group with the precision-time strategy receive a higher-accuracy source, and Linux instances there can additionally take time from the Nitro PTP Hardware Clock exposed by the ENA driver, configured in chrony as a refclock PHC /dev/ptp_ena source, which reaches microsecond-level synchronization. The PTP Hardware Clock does not smear leap seconds, so it and the smeared NTP endpoint are not interchangeable references and must not be mixed across a cluster without accounting for that difference. For Cassandra, the ordinary link-local NTP endpoint on every node is sufficient and is the simplest consistent choice.

The link-local services share a limit of 1024 packets per second per instance, covering NTP together with the instance metadata service and the Amazon-provided DNS. NTP polling at the recommended interval is far below this, but the limit is shared, so an instance already saturating it with metadata or DNS traffic can lose NTP packets.

Google Cloud

Compute Engine instances are preconfigured to synchronize to the metadata server at metadata.google.internal, and that is the source they should use (Compute Engine time synchronization documentation). External NTP sources must not be configured alongside the metadata server. For hosts outside Google Cloud, the public service is time.google.com.

Both the metadata server and time.google.com smear leap seconds over a 24-hour window (Google Public NTP FAQ), which is why neither may be mixed with a non-smearing source.

Azure

The recommended time source for Linux virtual machines is the Azure host's own clock, exposed to the guest as a PTP device. Recent images ship a udev rule that creates the stable symlink /dev/ptp_hyperv; the underlying ptp0 or ptp1 index varies with device enumeration order at boot, so the configuration must reference the symlink rather than an index. The chrony directive for it is a refclock PHC /dev/ptp_hyperv source (Azure time synchronization documentation).

Host time on Azure derives from Microsoft-operated stratum 1 servers backed by GPS. Because the guest reads it from the hypervisor rather than over the network, it is not subject to the variable network delay that limits the accuracy of an external NTP source.

One time discipline per cluster

A Cassandra cluster, meaning every datacenter that replicates to any other, is a single correctness domain and must use one consistent time discipline throughout:

All-AWS cluster: the Amazon Time Sync endpoint on every node and every client application host.
All-GCP cluster: the metadata server on every node and every client application host.
All-Azure cluster: the host PTP device on every node and every client application host.

Hybrid and multi-cloud clusters are the difficult case, because the provider services differ in leap-second handling: the AWS link-local NTP endpoint and Google's time services smear, while a self-operated GPS or NTS tier normally does not. A cluster spanning providers, or spanning cloud and on-premises, should synchronize every node against one consistently disciplined source set rather than letting each site use its local provider's default. Where that is not possible, the divergence during a leap event, up to approximately one second, must be treated as a real skew risk and planned for.

Kubernetes

The clock is a property of the Kubernetes node, not of the pod. Containers share the host kernel and therefore the host's clock; there is no separate per-container clock to configure. Pods also run without CAP_SYS_TIME by default and so cannot adjust the clock even if a time daemon were present.

Two consequences follow for Cassandra on Kubernetes:

An NTP daemon must not be run inside the Cassandra container. It would either fail for lack of the capability or, if the capability were granted, discipline the host clock from inside a pod, which is not a supportable arrangement.
Time synchronization is configured on the Kubernetes node's operating system, exactly as for any other host. The correctness of Cassandra timestamps on Kubernetes depends entirely on the node configuration.

On managed Kubernetes, the node images generally get this right, and the operational task is verification rather than configuration:

EKS node AMIs run chrony against the Amazon Time Sync Service at 169.254.169.123 by default.
AKS Linux nodes run chrony synchronized to the Azure host clock through the PTP device.
GKE node images synchronize to Google's internal time service.

Verification is performed on the node itself, through a node shell or a privileged debug pod, using the chronyc tracking and chronyc sources checks described in Configuring chrony. Custom and self-built node images must be confirmed to carry the same configuration, since a hardened or minimal image can omit the time daemon entirely.

On self-managed Kubernetes, the cluster operator owns node time synchronization. The chrony configuration and internal NTP tier guidance in this section apply unchanged, and must be applied identically across every node pool that can schedule Cassandra pods, and across every datacenter that replicates to another. Heterogeneous node pools, or a Cassandra cluster spanning Kubernetes nodes and non-Kubernetes hosts, are subject to the same rule: one source discipline for the whole cluster.

Monitoring likewise runs at node level. Reading the time inside a Cassandra pod returns the node clock, which is adequate for a spot check, but continuous measurement should observe the node through a host-level agent or a DaemonSet rather than through the Cassandra pod.

Leap seconds and smeared time

A leap second is an extra second inserted into UTC to keep it aligned with the Earth's rotation. Time services handle it in one of two ways:

Stepping (or inserting) the leap second at the moment it occurs, which is the classic NTP behaviour.
Smearing it, by slightly slowing or speeding the served clock over a window of hours so that no discontinuity ever occurs. Google smears over 24 hours, and the Amazon Time Sync Service NTP endpoint also smears.

The two behaviours diverge by up to one second while a smear is in progress.

Mixing smeared and non-smeared sources within one cluster must not be done. A cluster whose nodes take time from both kinds of source will develop up to a second of relative skew during the smear window, which is orders of magnitude beyond the tolerance described in Clock skew failure modes. A hybrid deployment with on-premises nodes on a stepping internal tier and cloud nodes on a smearing provider service is exactly this defect. The AWS Nitro PTP Hardware Clock is a further instance of it: it does not smear, while the link-local NTP endpoint in the same environment does.

A client may also mark a smearing source as a falseticker, meaning a source whose reported time disagrees with the majority of the others and which is therefore excluded from selection. Losing a source that way reduces the number of usable sources.

No leap second has been scheduled since the one inserted at the end of 2016, so no cluster has been exposed to a smear since then. The mechanism remains in place and a future leap second can still be announced, so the discipline must be consistent across the cluster before one is scheduled rather than after.

All nodes in all datacenters of one cluster, and all client application hosts that generate timestamps for it, must use the same time discipline. Multi-datacenter and hybrid deployments have to plan this before the second datacenter is built, not after.

Configuring chrony - the configuration files that implement this topology
Security - firewall scoping and authenticated time sources for the tier
Multi-Datacenter - deployments spanning datacenters