Cassandra Assess Node Failure

One or more nodes stop serving requests. Choosing the recovery procedure is the harder part, and the choice depends on what the failure actually cost. This playbook works out that cost and whether a restore is warranted.

The behaviour described applies to Cassandra 5.0, 4.1 and 4.0. It is the same on all three unless stated otherwise.

Two Different Questions

Node loss raises two questions with different answers. Answering one of them with a fact about the other leads to the wrong recovery action.

Availability: Can the Application Read and Write?

LOCAL_QUORUM is the level normally used in a multi-datacenter deployment. It needs floor(RF/2) + 1 replicas of that partition to respond: two of three at RF 3, three of five at RF 5. The arithmetic uses the local datacenter's replication factor and counts only the replicas in that datacenter. A partition is affected only when the unavailable nodes leave fewer than that many local replicas able to respond. The worked examples below use RF 3 in one datacenter, where the threshold is two.

Two nodes down in a six node cluster at RF 3 is therefore a partial outage. The partitions whose replica set contains both failed nodes lose quorum; every other partition is served normally. Quorum is a per-partition property, never a cluster-wide one.

The common belief that RF 3 means a cluster can only ever lose one node is true per partition and false per cluster. With more nodes than the replication factor, losing two produces a partial outage rather than a total one. nodetool getendpoints settles the question for any specific partition.

Data Safety: Was Anything Acknowledged Lost?

For any given range, at most two of its three replicas were on the two failed nodes, so a third copy survives on a node that stayed up. The coordinator sends every write to all replicas of the partition regardless of consistency level, so a write acknowledged at LOCAL_QUORUM with RF 3 in the local datacenter was still sent to all three of that datacenter's replicas. Losing it outright requires every replica that received it to lose its host inside the same unsynced commit log window. See What Is Lost When a Node Stops for that window and its bounds.

Hinted handoff and repair rebuild the missing copies once the nodes return.

A Restore Addresses Only the Second Question

A partial outage ends when the nodes come back, which is a restart. Where a restore procedure finishes by restarting nodes, the restart is what ended the outage, and the restore only rolled data back to the backup point on top of it.

Error Semantics During a Partial Outage

What the application observes, and what may safely be retried:

Error	Outcome	Client action
`UnavailableException`	The coordinator did not attempt the write	Safe to retry
`WriteTimeoutException`	Undefined; the write may have landed on some replicas	Retry is usually appropriate, subject to idempotency
`ReadTimeoutException`	No data returned	Safe to retry

UnavailableException is raised before any replica is contacted, so no state changed. A timeout leaves the outcome undefined. Writes are last-write-wins, so retrying the same mutation converges to the same result. That reasoning does not apply to counter updates or to conditional (LWT) statements. The per-statement tables give the full contract: INSERT failure semantics and DELETE failure semantics. Idempotency and retry behaviour are covered under driver retry policies.

Symptoms

One or more nodes report DN in nodetool status when run from a node that stayed up
Application errors confined to a subset of partitions rather than all traffic
UnavailableException or timeouts at LOCAL_QUORUM or QUORUM, while LOCAL_ONE and ONE still succeed
A node reporting itself UN while every other node reports it DN
Client connections refused on port 9042 by a node whose process is still present

Prerequisites

Shell access to the affected nodes and to at least one node that stayed up, with sudo or root privileges; every command in the Resolution and Recovery sections either changes service state or moves files owned by the Cassandra user
nodetool access to a node that stayed up, for an accurate view of cluster state
Read access to /var/log/cassandra/system.log and to the systemd journal
The replication factor of each affected keyspace, read with DESCRIBE KEYSPACE <keyspace> in cqlsh, and the consistency levels the application uses, read from its driver configuration

Diagnosis

Step 1: Establish Cluster State From a Node That Stayed Up

nodetool status

Run this from a node that is serving normally. A node always lists itself as up regardless of its own state, so its local view must not be used to judge whether it is serving.

A node reports itself UN even when it has stopped serving

Under disk_failure_policy: stop a node stops its native transport and gossip while the JVM keeps running. On 5.0, 4.1 and 4.0 it still reports itself UN while every peer reports it DN, because a node always reports its own state as alive. Judge node state from a peer, and confirm on the node itself with nodetool info, which reports the gossip and native transport flags.

Step 2: Determine Which Partitions Lost Quorum

nodetool getendpoints my_keyspace my_table 'partition_key_value'

The command lists the replicas for that partition. At LOCAL_QUORUM, count only the listed endpoints in the local datacenter: the partition lost quorum if the down nodes leave fewer than floor(RF/2) + 1 of them able to respond, which is two of three at RF 3. At QUORUM, count the listed endpoints in every datacenter.

Step 3: Check Whether the Process Is Still Running

systemctl status cassandra
nodetool info | grep -E "Gossip active|Native Transport active"

A running process with Gossip active: false or Native Transport active: false is serving nothing. The usual cause is disk_failure_policy: stop firing on a filesystem error. See Disk Failure Policy.

Step 4: Determine How the Process Stopped

# Clean stop through systemd or SIGTERM
journalctl -u cassandra --since "2 hours ago" | tail -50

# OOM killer
dmesg -T | grep -i "killed process"
journalctl -k | grep -i "out of memory"

# JVM crash report, written to the working directory of the process unless
# -XX:ErrorFile relocates it; 5.0, 4.1 and 4.0 do not set -XX:ErrorFile by default
sudo find / -path /proc -prune -o -path /sys -prune -o \
     -name 'hs_err_pid*.log' -mtime -1 -print 2>/dev/null

# Host reboot
uptime -s
last reboot | head -3

The distinction that matters is between loss of the process and loss of the host. A kill -9, an OOM kill or a JVM crash leaves acknowledged data in the OS page cache, so the cost is bounded at roughly the last 100 milliseconds of acknowledged writes on that node. An OS crash, hypervisor reset or power failure discards the page cache and costs the full unsynced window, up to 15 seconds at the default commitlog_sync_period of 10 seconds. Both bounds assume the default uncompressed, unencrypted commit log; see What Is Lost When a Node Stops for the exceptions.

A crash report is written only when the JVM itself crashed. An OOM kill by the kernel, a kill -9 and a host power loss produce none, and -XX:ErrorFile can direct the report outside the paths searched above. The absence of an hs_err_pid file therefore does not establish that the JVM did not crash; it must be read alongside the journal and dmesg output rather than on its own.

Step 5: Check How Long the Node Has Been Down

Hints are held for max_hint_window, which defaults to 3 hours on 5.0, 4.1 and 4.0 (max_hint_window_in_ms on 4.0). Writes issued while a replica is down are retained by the coordinators for that window and delivered when the replica returns.

# From a node that stayed up, on all versions
nodetool statushandoff

# Cassandra 5.0 and 4.1 only; the command does not exist in 4.0
nodetool listpendinghints

# Cassandra 4.0 equivalent: the hint files held by this coordinator
sudo ls -l /var/lib/cassandra/hints/

nodetool statushandoff reports only whether hinted handoff is enabled, on 5.0, 4.1 and 4.0. nodetool listpendinghints was added in Cassandra 4.1 and is absent from 4.0. It reports the hints held by the node it runs on rather than a cluster-wide total, so it must be run on each node that coordinated writes during the outage. With no hints held it prints This node does not have any pending hints. Otherwise it prints a table with one row per target host ID. On 4.0 the equivalent check is the contents of hints_directory, which is /var/lib/cassandra/hints in the packaged configuration. A directory with no .hints files means the node holds no hints.

Stopped transports count as down to the coordinators, so hints accumulate normally in the disk_failure_policy: stop case as well.

Step 6: Determine Whether Any Range Has Lost Every Replica

This step is the only test in the diagnosis that identifies destroyed data. Where the failure is node loss rather than a logical deletion, it is the only positive trigger for a restore. It compares the replica set of each affected range against the nodes that are unrecoverable, which is a stronger condition than down.

A node is unrecoverable when its data directories cannot be read back: the storage was destroyed or wiped, an instance backed by ephemeral storage was terminated, or the filesystem is beyond repair. A node that is merely down, whose data directories are intact, is not unrecoverable. Neither is a node that fails to start on a corrupt commit log segment or a corrupt SSTable, which Case 4 addresses.

Recoverability comes from infrastructure evidence rather than from Cassandra: the cloud console state of the instance and its volumes, the storage controller or RAID status, or the result of a filesystem check. No nodetool command reports it. A node whose data directories have not been confirmed readable must be treated as unrecoverable for this test until they are. Treating an undiagnosed node as recoverable can produce a conclusion of no data loss that later proves wrong.

If more than one node in the same incident is being quarantined or scrubbed under Case 4, repeat this step before concluding: taking a second replica of the same range out of service can change the answer.

For each affected keyspace and table, list the replicas:

# Replica set for one specific partition
nodetool getendpoints my_keyspace my_table 'partition_key_value'

# Every token range in the keyspace with the endpoints holding it
nodetool describering my_keyspace

nodetool describering is present in 5.0, 4.1 and 4.0 and prints one line per token range, each naming the endpoints that hold it. With virtual nodes the output is long, so it should be filtered against the unrecoverable addresses rather than read in full:

# Ranges whose listed endpoints include an unrecoverable node
nodetool describering my_keyspace | grep -E '10\.0\.1\.7|10\.0\.1\.8'

Compare each replica set against the unrecoverable nodes:

If every range retains at least one replica on a node whose data directories survive, no data has been destroyed. Repair restores the missing copies on nodes that return to service, and node replacement restores them on nodes that do not. A restore is not warranted.
If any range has its entire replica set on unrecoverable nodes, the data in that range exists nowhere in the cluster. That range is data loss, and it is the case a restore addresses. See When a Restore Is Warranted.

Record the replica sets before removing any node from the ring. nodetool describering and nodetool getendpoints report the current ring only. Once nodetool removenode or nodetool assassinate removes an unrecoverable node, its address disappears from that output, and the replica sets shown afterwards no longer answer this question.

Resolution

Cases 1 to 4 apply to one node at a time. Work through them in order for each affected node, and act on the first case that matches that node. Case 5 applies once for the incident, after every affected node is back in service.

The restart commands are safe to re-run. systemctl start on a unit that is already running is a no-op, and systemctl restart performs the same clean stop and start on each invocation. The repair in Case 5 is also safe to re-run; see that case for the detail.

Case 1: The Process Is Running but Not Serving

Restart the node.

sudo systemctl restart cassandra

systemctl restart sends SIGTERM, which runs the drain shutdown hook, so memtables are flushed and the commit log is synced before the JVM exits. Nothing acknowledged is lost. Fix the underlying filesystem error before the restart, or the policy fires again.

Case 2: The Process Stopped Cleanly

A stop through systemctl stop, nodetool drain or any other SIGTERM path runs the same drain shutdown hook. Nothing acknowledged is lost. Restart the node.

sudo systemctl start cassandra

Case 3: The Process Died Hard, or the Host Was Lost

Restart the node and let the commit log replay. Replay reapplies mutations that were not yet flushed to SSTables, bounded as described in Step 4.

sudo systemctl start cassandra
grep -iE "commitlog|replay" /var/log/cassandra/system.log | tail -20

An OOM kill and a filled disk both recur once the node is back under the same load. Start the node to restore availability, then address the cause before the node is left running unattended. See Recover from OOM and Handle Full Disk.

Case 4: The Node Will Not Start

A corrupt commit log segment or a corrupt SSTable can stop a node starting. If Step 6 found that every range this node holds keeps at least one replica on a surviving node, fix or replace this node rather than restore the cluster. If Step 6 found a range whose entire replica set is unrecoverable, that does not hold for that range, and When a Restore Is Warranted applies to it.

The procedures here run on a stopped node. Files are moved to a quarantine location rather than deleted, so the evidence is still there if the first diagnosis is wrong.

Corrupt commit log segment

Commit log replay failures are logged before the JVM exits, and the message names the segment file:

sudo grep -E "Exiting due to error while processing commit log|Failed commit log replay|checksum failure|Could not read commit log descriptor" \
    /var/log/cassandra/system.log | tail -20

On 5.0, 4.1 and 4.0 the message takes forms such as Mutation checksum failure at 42315 in CommitLog-7-1699999999999.log or Could not read commit log descriptor in file /var/lib/cassandra/commitlog/CommitLog-7-1699999999999.log. The exit itself is logged as Exiting due to error while processing commit log during initialization. The file name identifies the segment. The default commitlog_directory is /var/lib/cassandra/commitlog; confirm the configured value in cassandra.yaml before acting.

Move the named segment to a directory outside commitlog_directory and start the node:

sudo mkdir -p /var/lib/cassandra/commitlog-quarantine
sudo chown cassandra:cassandra /var/lib/cassandra/commitlog-quarantine
sudo mv /var/lib/cassandra/commitlog/CommitLog-7-1699999999999.log \
        /var/lib/cassandra/commitlog-quarantine/
sudo systemctl start cassandra

The quarantine directory must not sit inside commitlog_directory, or the segment is picked up again on the next start.

A successful start logs replay of the remaining segments with no further commit log error, reports both transports active, and is seen as UN by its peers:

sudo grep -iE "commitlog|replay" /var/log/cassandra/system.log | tail -20
nodetool info | grep -E "Gossip active|Native Transport active"

# From a node that stayed up
nodetool status

If startup fails again naming a different segment, repeat the move for that segment. The mutations in each quarantined segment are not replayed on this node; the other replicas still hold them, and the repair in Case 5 restores them.

Recover from OOM gives moving the whole commit log directory aside as a last resort. When the log names a single segment, quarantining that segment is the smaller action and should be preferred: it gives up the unflushed mutations in one segment rather than in all of them. Move the whole directory only when it cannot be read at all, or when quarantining segments one at a time has not produced a start.

Corrupt SSTable

The exception names the data file path:

sudo grep -E "CorruptSSTableException|Corrupted: " /var/log/cassandra/system.log | tail -20

The message is Corrupted: followed by the file path on 5.0, 4.1 and 4.0. The path names the -Data.db component under the table directory, in the form /var/lib/cassandra/data/<keyspace>/<table>-<table-id>/<generation>-Data.db. The generation prefix is everything in the file name before -Data.db, and every file sharing that prefix is a component of the same SSTable.

With the node stopped, scrub the table offline. sstablescrub <keyspace> <table> ships with 5.0, 4.1 and 4.0 and requires the node to be down; running it as the Cassandra user keeps file ownership intact:

sudo -u cassandra sstablescrub my_keyspace my_table
sudo systemctl start cassandra

If scrub cannot recover the file, move every component of that SSTable out of the data directory. All components must be moved together; leaving one behind produces a further startup failure. Substitute the table directory and the generation prefix taken from the Corrupted: path:

sudo mkdir -p /var/lib/cassandra/sstable-quarantine
sudo chown cassandra:cassandra /var/lib/cassandra/sstable-quarantine

# List the components first, then move the same set
sudo ls /var/lib/cassandra/data/my_keyspace/my_table-1f3c.../nb-12-big-*
sudo mv /var/lib/cassandra/data/my_keyspace/my_table-1f3c.../nb-12-big-* \
        /var/lib/cassandra/sstable-quarantine/
sudo systemctl start cassandra

The same success check applies: the node starts, nodetool info reports both transports active, and a peer reports it UN. Removing an SSTable discards the rows it held on this node, so the repair in Case 5 is required rather than optional after this path.

A node that cannot be recovered at all

Replace it. See Replace Dead Node.

Case 5: Every Node Is Back

Once nodetool status reports UN for every node, repair the keyspaces the failed nodes held.

# On the recovered node, once per keyspace it holds
nodetool repair --full my_keyspace

On 5.0, 4.1 and 4.0, -pr restricts repair to each node's primary range, the range it owns rather than the ranges it replicates. A single -pr run therefore appears to complete while leaving most of the recovered node's replicated ranges unrepaired. Either of these covers the affected set: nodetool repair --full my_keyspace on the recovered node, which covers every range that node replicates, or nodetool repair -pr my_keyspace run on every node in turn. A managed repair service that tracks coverage across the cluster is the third option where one is in use.

Repair is safe to re-run. A repeated run compares the same ranges again and streams only what still differs, and a run that fails partway leaves the ranges it already completed repaired, so it can be started again.

A node that was down for less than max_hint_window has its writes covered by hints, so the repair confirms consistency rather than restoring it. A node that was down for longer than the window has writes that were never hinted, so the repair is required rather than optional. Repair must complete within gc_grace_seconds of the outage to avoid resurrecting deleted data. See Repair.

When a Restore Is Warranted

Step 6 is the test. A restore is warranted when that step finds a range whose entire replica set is unrecoverable, and when data was destroyed logically, because every replica then holds the destroyed state. In every other case the data still exists on a node whose data directories survive, and repair or node replacement recovers it.

Do Not Restore to Recover a Node That Stopped

Problem: A restore rolls data back to the backup point. It recovers nothing that was written after the backup, and it discards everything written between the backup and the restore on the nodes it is applied to. A stopped node has lost at most the bound in Step 4, and its data still exists on the other replicas, so a restore removes more data than the failure did.

Symptoms: Rows written since the last backup are missing after the restore completes. The cluster becomes available at the same moment for an unrelated reason: the restore procedure ends by restarting the nodes, and that restart is what ended the outage.

Instead: Restore only when data has been destroyed logically, or when every replica of a range is gone. An accidental DROP, TRUNCATE or DELETE, or a release writing incorrect data, is a restore. Nodes stopping, crashing or losing a disk are handled by replication, hints and repair, and by replacing a node that will not come back.

Recovery

Verify Availability Is Restored

# Every node UN, from a node that stayed up
nodetool status

# Transports active on the recovered node
nodetool info | grep -E "Gossip active|Native Transport active"

Verify Hints Have Drained

# Cassandra 5.0 and 4.1
nodetool listpendinghints

# Cassandra 4.0
sudo ls -l /var/lib/cassandra/hints/

On 5.0 and 4.1, This node does not have any pending hints means the node the command ran on holds none; the command reports only that node, so it must be run on every node that coordinated writes during the outage. On 4.0 the command does not exist, and a hints_directory with no .hints files is the same result.

Verify Consistency

# On the recovered node, once per keyspace it holds
nodetool repair --full my_keyspace

Use one of the routes given in Case 5, so every range the recovered node replicates is covered. Confirm the repair completes without error.

Command	Purpose
`nodetool status`	Cluster-wide node state, run from a node that stayed up
`nodetool info`	Gossip and native transport flags on the local node
`nodetool getendpoints`	Replicas holding a specific partition
`nodetool describering`	Every token range in a keyspace with the endpoints holding it
`nodetool listpendinghints`	Hints held by the local node; Cassandra 5.0 and 4.1 only
`nodetool statushandoff`	Whether hinted handoff is enabled
`nodetool drain`	Flush memtables and stop accepting writes before a stop
`nodetool repair --full`	Rebuild missing replicas after the outage
`sstablescrub`	Offline scrub of a corrupt SSTable on a stopped node

Commit Log - What a stopped node costs, by failure mode
Hinted Handoff - The hint window and its limits
Consistency Levels - Quorum arithmetic per partition
Backup and Restore - When a restore is and is not the answer
Replace Dead Node - Replacing a node that will not return
Recover from OOM - Diagnosing and preventing OOM kills
Handle Full Disk - Disk exhaustion and the disk failure policy
Driver Retry Policies - Idempotency and retry behaviour