Cassandra Clock Skew Failure Modes

Every failure below is silent. Cassandra 3.11, 4.x, and 5.0 do not compare node clocks, do not reject a write whose timestamp precedes existing data, and emit no warning when a newer write is shadowed by an older one. The write is acknowledged at the requested consistency level, and the defect surfaces later as a read that returns the wrong value.

The mechanisms these failures act on are described in How Cassandra uses time.

---

Updates that disappear

Consider two nodes whose clocks differ by 200 ms, and an application that updates the same cell twice in quick succession.

1. The first update is coordinated by the node whose clock is 200 ms fast, and is stored with timestamp T + 200ms.
2. The second update, issued 50 ms later and coordinated by the node whose clock is correct, is stored with timestamp T + 50ms.
3. On the next read, the first update wins, because its timestamp is higher.

The second update is not lost in transit and not rejected. It is stored, replicated, and then permanently ignored on every read and every compaction. To the application, the update simply did not take effect.

---

Deletes that do not delete, and data that returns

A DELETE is a write. It stores a tombstone, a marker recording that the data was deleted, and that marker carries a timestamp of its own. A tombstone shadows only the data whose timestamp is lower than its own. Skew therefore breaks deletion in two directions:

• A tombstone written from a node with a slow clock carries a timestamp lower than data that was written earlier in real time from a node with a fast clock. The tombstone does not shadow that data, and the row remains readable after a successful DELETE.
• A tombstone written from a node with a fast clock carries a timestamp in the future relative to the rest of the cluster. It shadows not only the intended data but also every legitimate write to those cells until the cluster's wall clock passes the tombstone's timestamp. Writes issued during that window are accepted and stored, but not visible on read.

The second case is often described as data resurrection when it interacts with gc_grace_seconds, the per-table period a tombstone must survive before compaction is allowed to discard it. If a tombstone is purged while data it was supposed to shadow still exists on another replica, a later repair propagates the surviving data back and the deleted row reappears.

---

TTL expiry and writetime

In Cassandra 3.11, 4.x, and 5.0, a cell written with a time to live (TTL) records both the write timestamp and a local deletion time computed from the writing node's clock. A node whose clock is ahead computes an expiry point that is early relative to the rest of the cluster, so the data expires sooner than the TTL specified. A node whose clock is behind expires it later.

The writetime() function returns the stored timestamp, not a measurement of when the write actually happened. Under skew, writetime() values from different coordinators are not comparable and must not be used to reconstruct the order of events.

---

Symptoms in a running cluster

Clock skew rarely presents as "the clocks are wrong". It presents as:

• A value that appears or disappears depending on which coordinator serves the query, because the replica set returns different versions and the resolution depends on which replicas responded.
• Read repair and anti-entropy repair that appear not to converge: repair propagates the highest-timestamp version, which is the wrong one, so every repair reinstates the defect.
• Ordering anomalies in read-modify-write workloads, where a read returns a value that a subsequent write was supposed to have replaced.
• Rows that reappear after a successful delete, particularly following a repair.
• Counters and TTL-bearing data behaving inconsistently between datacenters.

---

Cross-node request timeout accounting

cross_node_timeout propagates the coordinator's request deadline to replicas so that a replica can abandon work for a request that has already timed out. The setting has existed at least since Cassandra 2.2, where it defaulted to false, meaning replicas assumed the coordinator had forwarded the request instantly. Cassandra 4.0 changed the default to true (CASSANDRA-15216). Cassandra 4.1 renamed the setting internode_timeout as part of the configuration and JVM parameter standardization (CASSANDRA-15234), and that is also the name and default in 5.0. Under either name the mechanism compares a timestamp created on one node against the clock of another, and it therefore assumes node clocks are modestly in sync. Under significant skew, replicas either discard work for requests that have not in fact timed out, or continue processing requests that have. See the cassandra.yaml reference for the full parameter descriptions.

---

Lightweight transactions are not a substitute for synchronized clocks

Lightweight transactions (LWT) use Paxos, a consensus protocol, to serialize competing operations on a partition, which does provide linearizable semantics for the operations that go through it. That serialization does not remove the dependency on clocks:

• A Paxos ballot, the proposal number the protocol orders competing operations by, carries a timestamp taken from the proposing coordinator's clock, and ballots are ordered primarily by that timestamp. In Cassandra 4.1 and later, when paxos_variant is set to v2 (CEP-14, CASSANDRA-17164; v1 remains the default through 5.0), the ballot combines that timestamp with a random 64-bit value that supplies uniqueness rather than identifying the node. In Cassandra 3.11, and in 4.1 and 5.0 under the default paxos_variant: v1, the ballot is a TimeUUID whose non-timestamp portion is a per-JVM random clock_seq_and_node field. Under either variant the ordering is decided by the coordinator's clock.
• A coordinator whose clock is ahead of the cluster always generates higher ballots, so it wins contention consistently, and coordinators with slower clocks can be repeatedly pre-empted.
• The value committed by a successful LWT is written with a timestamp derived from the winning ballot, so it is subject to the same last-write-wins comparison against any non-LWT write to the same cells.

That last property is what makes the combination dangerous rather than merely inefficient.

LWT is not a substitute for synchronized clocks

Mixing LWT and non-LWT writes against the same cells is already an anti-pattern; under clock skew it is a data-integrity defect. LWT should not be introduced as a way to tolerate unsynchronized clocks.

---

What guardrails do and do not detect

A guardrail is a configurable limit that makes the coordinator warn on or reject a request breaching it, introduced by the Guardrails Framework (CEP-3). Two sets of guardrails touch write timestamps.

Cassandra 4.1 added user_timestamps_enabled, which allows an operator to reject statements that specify USING TIMESTAMP, removing the application's ability to supply a timestamp at all.

Cassandra 5.0 added minimum_timestamp_warn_threshold, minimum_timestamp_fail_threshold, maximum_timestamp_warn_threshold, and maximum_timestamp_fail_threshold (CASSANDRA-18352), which warn on or reject a write whose timestamp falls outside a configured window relative to the coordinator's clock. These apply to the resolved write timestamp whatever its origin, covering USING TIMESTAMP, the native protocol default timestamp field, and a timestamp the coordinator assigned itself.

Both sets compare a request's timestamp against the coordinator's own clock. Neither compares that clock to any other node's, so neither detects disagreement between the clocks of Cassandra nodes, and neither is a substitute for time synchronization.

---

How much skew is tolerable

There is no single threshold, because the tolerance is a property of the workload rather than of Cassandra:

• Append-only and read-mostly workloads tolerate the most. If two writes never target the same cell, there is no conflict for a timestamp to resolve incorrectly.
• Workloads that perform concurrent read-modify-write against the same cells, or that write and then delete the same rows within a short window, need relative synchronization in the low-millisecond range.

The useful way to express the requirement: cluster-wide relative skew, meaning how far two nodes' clocks are from each other rather than from UTC, should stay below the shortest interval at which the application issues conflicting writes to the same cell. If an application can update the same cell twice within 5 ms, then relative skew above 5 ms can invert the order of those two updates. Because that interval is usually short and rarely known precisely, keeping relative skew in the low single-digit milliseconds or better is the practical target for a production cluster.

Two properties of unsynchronized or poorly synchronized deployments make this harder than it appears:

• Drift is continuous and load-dependent. An unsynchronized host does not sit at a fixed error; it accumulates error, and the rate is worse on virtual machines under load than on idle physical hardware. Any deployment without an active time daemon will exceed a millisecond-scale budget.
• Independent synchronization optimizes the wrong property. When each node synchronizes on its own to a public Network Time Protocol (NTP) pool, each node selects a different set of upstream servers, over different network paths, with different asymmetries and different residual errors. Each node's reported offset against its own sources can look healthy while node-to-node agreement is materially worse, because nothing in that arrangement measures or corrects the difference between the nodes.

What public NTP pools actually deliver

A measurement of pool.ntp.org servers published by the SANS Internet Storm Center on 21 October 2025 found most pool servers reporting an offset under 10 ms, with a small tail of servers beyond 100 ms (SANS ISC diary 32390). That distribution is adequate for general-purpose timekeeping, which is what the pool exists to provide.

It is not what Cassandra depends on. Two consequences follow:

• Each node that resolves pool.ntp.org is handed a different server, so two nodes in the same cluster can sit at opposite ends of that distribution. Node-to-node offsets of tens of milliseconds are plausible even when every node individually reports a healthy offset against its own upstream.
• The NTP Pool project's monitoring system removes a server from rotation only once its offset is large, on the order of 100 ms or more. A server can therefore be tens of milliseconds off and remain in the pool in good standing. The pool's scoring tolerates offsets far larger than a Cassandra cluster should.

The pool is a reasonable upstream reference for a small internal tier, which measures its members against each other and against several pool servers at once. It is a poor per-node configuration for a cluster whose correctness depends on relative agreement.

---

Related pages

• Deployment architecture - the topology that prevents these failures
• Monitoring - the signals that detect skew before it corrupts data
• Gossip failures playbook - gossip symptoms clock skew can produce
• Schema disagreement playbook - schema propagation symptoms clock skew can produce