Cassandra SSTable Corruption

A single SSTable on a node has become unreadable, so reads that touch it fail with CorruptSSTableException. Corruption arises from hardware faults (bad sectors, bit rot), a process or host that stopped mid-write, or a filesystem fault. The data in a corrupt SSTable is recoverable from other replicas when the replication factor is greater than 1.

This is distinct from SSTable Count Explosion: that playbook addresses too many SSTables, this one addresses an SSTable whose contents cannot be read.

Prerequisites

Filesystem access to the data directory holding the affected SSTable.
The ability to stop and start the node, since the offline tools require a stopped node and the nodetool steps require a running one.
The replication factor of the affected keyspace, which determines whether the data can be recovered from another replica. Confirm it with DESCRIBE KEYSPACE <keyspace> in cqlsh (the replication clause) or nodetool describering <keyspace> (the number of distinct endpoints per range).

Several steps below change the node's run state. Each step states whether the node must be running or stopped; the states are not interchangeable, and the procedure assumes they are followed in order.

Symptoms

Read errors returned to clients for data on the affected node
CorruptSSTableException in system.log
A specific compaction or read repeatedly failing against the same file

Diagnosis

Step 1: Identify the Corrupt SSTable (node running or stopped)

grep -i corrupt /var/log/cassandra/system.log

The exception message names the SSTable file whose read failed. Note its keyspace, table, and generation so the correct components can be acted on later. This step reads only the log and does not depend on the node's run state.

Step 2: Verify the Table (node running)

nodetool verify <keyspace> <table>

nodetool verify is a JMX call and requires the node to be running. It checks SSTable integrity against stored checksums and reports any file that fails.

Step 3: Inspect an SSTable Offline (node stopped)

The offline tools read SSTable files directly and require the node to be stopped first. sstableverify validates checksums and sstablemetadata reads an SSTable's metadata. Running them against a live node's data directory can produce invalid results, and the tool does not check the node's run state for the operator. Stop the node before running either:

sudo systemctl stop cassandra
sstableverify <keyspace> <table>
sstablemetadata <sstable_path>

Step 4: Return the Node to Service Before Any nodetool Fix (node running)

The resolution steps below that call nodetool (verify, repair, scrub) are JMX calls and require a running node. If the node was stopped for Step 3, start it and confirm it reaches UN before proceeding:

sudo systemctl start cassandra
nodetool status

Each node in the nodetool status output must show UN for its own address before any nodetool-based resolution is attempted.

Resolution

The correct path depends on whether another replica holds the data. Confirm the replication factor of the affected keyspace (see Prerequisites) before choosing a case.

Case 1: Replication Factor Greater Than 1

When other replicas hold the same data, remove the corrupt SSTable's components and restore them from a replica by repair. Removing one SSTable's components loses only the data it held on this node; repair streams that data back from the replicas that still hold it.

The SSTable's data is unavailable on this node until repair completes

Removing an SSTable's components makes the rows it held unreadable on this node until nodetool repair streams them back. This path is safe only at replication factor greater than 1, where another replica holds the same rows. Confirm RF > 1 for the keyspace first. Move the components aside rather than deleting them, so they can be restored if repair does not recover the data; delete them only after repair is confirmed complete.

The node must be stopped to move SSTable files, because a running node holds them open. Confirm which components the glob matches before moving anything:

# Dry run: list the components the glob matches, and confirm they belong
# only to the corrupt SSTable identified in Step 1
ls -la <corrupt_sstable>*

With the match confirmed, stop the node, move the components to a holding directory, then return the node to service:

# Stop the node before touching SSTable files
sudo systemctl stop cassandra

# Move the components aside (preserves a rollback window)
mkdir -p /tmp/corrupt_sstable
mv <corrupt_sstable>* /tmp/corrupt_sstable/

# Start the node and confirm it reaches UN
sudo systemctl start cassandra
nodetool status

Once the node shows UN, restore the removed data from replicas and confirm the result:

# Restore the removed data from replicas
nodetool repair <keyspace> <table>

# Confirm the previously failing reads now succeed and verify reports no corruption
nodetool verify <keyspace> <table>

Delete the held-aside components only after repair is confirmed complete and the data is restored:

rm -r /tmp/corrupt_sstable

Case 2: No Other Replica

When no other replica holds the data (replication factor 1, or all replicas affected), scrub is the only option. Confirm no replica holds the data before scrubbing: DESCRIBE KEYSPACE <keyspace> in cqlsh shows the replication factor in its replication clause, and nodetool describering <keyspace> shows the endpoints per token range. Where a replica exists, Case 1 recovers the data in full instead.

Scrub discards unreadable data

nodetool scrub rewrites the SSTable and drops any row it cannot read. At replication factor 1 the dropped rows cannot be recovered from another replica, so scrub causes permanent data loss for the unreadable portion. Confirm no replica holds the data before scrubbing; where a replica exists, Case 1 recovers the data in full instead.

Scrub runs on a running node and rewrites the SSTable into new files before removing the old ones, so it needs free disk headroom roughly the size of the table being scrubbed. A volume without that headroom can fill during the rewrite and turn the recovery into a disk-full incident; see Handle Full Disk. Confirm available space first:

df -h /var/lib/cassandra
nodetool scrub <keyspace> <table>

Recovery

Confirm reads that previously failed now succeed.
Re-run nodetool verify <keyspace> <table> and confirm it reports no corrupt SSTables.
If the corrupt file was removed (Case 1), confirm nodetool repair completed so the missing data was streamed back from replicas.

An interrupted scrub or repair is safe to re-run. Scrub rewrites into new files and removes the originals only on success, so an interrupted run leaves the SSTable in its prior state and can be started again. Repair is idempotent in effect: a re-run re-compares the ranges the interrupted run did not finish. In Case 1, do not delete the held-aside components until a repair has completed in full, so a re-run always has the original data to fall back on.

Command	Purpose
`nodetool verify`	Check SSTable integrity against checksums on a running node
`nodetool scrub`	Rebuild an SSTable, dropping unreadable rows
`nodetool repair`	Restore removed data from replicas
`sstableverify`	Validate SSTable checksums offline
`sstablemetadata`	Read an SSTable's metadata offline

SSTable Count Explosion - Runaway SSTable counts
Handle Full Disk - Disk space exhaustion
Repair Failures - Repair problems
Node Fails to Start - Startup failures, including commit log corruption