sstableverify

Validates SSTable integrity by checking data structure and checksums without modifying files.

Synopsis

sstableverify [options] <keyspace> <table>

Description

sstableverify performs comprehensive integrity checks on SSTable files to detect corruption before it causes operational problems. Unlike sstablescrub, this tool is read-only and does not modify any data, making it safe for diagnostic purposes.

The tool validates:

Data file checksums - Verifies CRC32 checksums match stored values
Index consistency - Ensures index entries point to valid data locations
Row structure - Validates row format and column data
Bloom filter integrity - Checks filter file consistency

Cassandra Must Be Stopped

Cassandra must be completely stopped before running sstableverify. Running this tool while Cassandra is active may produce inconsistent results or cause issues with active SSTables.

How It Works

Verification Levels

Level	Components Checked	Use Case
Basic	Checksums only	Quick health check
Extended (`-e`)	Checksums + row reads	Deep validation
Token Check (`-t`)	Token range validation	Ring consistency

Arguments

Argument	Description
`keyspace`	Name of the keyspace containing the table
`table`	Name of the table to verify

Options

Option	Description
`-f, --force`	Required. Force verification to proceed (safety flag)
`-e, --extended-verify`	Extended verification - read and validate every row
`-t, --token_range <left,right>`	Only verify SSTables containing tokens in range (comma-separated, can be specified multiple times)
`-q, --quick`	Quick verification (less thorough)
`-c, --check_version`	Check SSTable version compatibility
`-r, --mutate_repair_status`	Mutate repair status metadata
`-v, --verbose`	Verbose output showing progress
`-h, --help`	Display help information
`--debug`	Enable debug logging

Force Flag Required

The -f or --force flag is required to run verification (CASSANDRA-17017). This is a safety measure to prevent accidental execution.

Extended Verification (`-e`)

Extended mode performs a complete read of every row in the SSTable:

Examples

Basic Verification

# Stop Cassandra first
sudo systemctl stop cassandra

# Verify a specific table (--force is required)
sstableverify -f my_keyspace my_table

# Start Cassandra
sudo systemctl start cassandra

Extended Verification

# Full row-by-row verification
sstableverify -f -e my_keyspace my_table

Verbose Output

# See progress during verification
sstableverify -f -v my_keyspace my_table

Verify Token Range

# Only verify SSTables containing specific tokens (comma-separated range)
sstableverify -f -t -9223372036854775808,0 my_keyspace my_table

# Multiple token ranges can be specified
sstableverify -f -t -9223372036854775808,0 -t 0,9223372036854775807 my_keyspace my_table

Verify All Tables in Keyspace

#!/bin/bash
KEYSPACE="$1"
DATA_DIR="/var/lib/cassandra/data"

# Get all tables in keyspace
for table_dir in ${DATA_DIR}/${KEYSPACE}/*/; do
    table_name=$(basename "$table_dir" | cut -d'-' -f1)
    echo "Verifying ${KEYSPACE}.${table_name}..."
    sstableverify "$KEYSPACE" "$table_name"
    if [ $? -ne 0 ]; then
        echo "ERROR: Corruption found in ${KEYSPACE}.${table_name}"
    fi
done

Understanding Output

Successful Verification

Verifying BigTableReader(path='/var/lib/cassandra/data/my_keyspace/my_table-abc123/nb-1-big-Data.db')
Deserializing sstable metadata for /var/lib/cassandra/data/my_keyspace/my_table-abc123/nb-1-big-Data.db
Checking computed hash of BigTableReader...
Verifying BigTableReader(path='/var/lib/cassandra/data/my_keyspace/my_table-abc123/nb-2-big-Data.db')
...

No errors printed indicates all SSTables passed verification.

Failed Verification

Verifying BigTableReader(path='/var/lib/cassandra/data/my_keyspace/my_table-abc123/nb-1-big-Data.db')
Error verifying /var/lib/cassandra/data/my_keyspace/my_table-abc123/nb-1-big-Data.db:
Corrupted: Cannot read sstable at position 12345

Exit Codes

Code	Meaning
0	All SSTables verified successfully
1	Corruption detected in one or more SSTables
2	Tool error (permissions, missing files, etc.)

When to Use sstableverify

Scenario 1: Routine Health Checks

#!/bin/bash
# weekly_verify.sh - Run weekly via cron

KEYSPACES="system system_auth my_keyspace"
LOG_FILE="/var/log/cassandra/verify_$(date +%Y%m%d).log"

echo "Starting SSTable verification: $(date)" >> $LOG_FILE

for ks in $KEYSPACES; do
    for table_dir in /var/lib/cassandra/data/$ks/*/; do
        table=$(basename "$table_dir" | cut -d'-' -f1)
        echo "Verifying $ks.$table" >> $LOG_FILE
        sstableverify -f "$ks" "$table" >> $LOG_FILE 2>&1
    done
done

echo "Verification complete: $(date)" >> $LOG_FILE

Scenario 2: After Hardware Issues

# After disk errors, power loss, or memory issues
# Verify all tables before starting Cassandra

sudo systemctl stop cassandra

# Verify critical tables first (--force required)
sstableverify -f -e system_auth roles
sstableverify -f -e my_keyspace critical_table

# If issues found, scrub before starting
# sstablescrub my_keyspace corrupted_table

sudo systemctl start cassandra

Scenario 3: Before Major Operations

# Verify before upgrades or migrations
sstableverify -f my_keyspace my_table

# If clean, proceed with operation
if [ $? -eq 0 ]; then
    echo "SSTables healthy, proceeding..."
else
    echo "Corruption detected, run sstablescrub first"
fi

Scenario 4: Investigating Read Errors

# When seeing read errors in logs
# Error: "CorruptSSTableException" or "Cannot read SSTable"

# 1. Stop Cassandra
sudo systemctl stop cassandra

# 2. Verify the suspect table (--force required)
sstableverify -f -e -v my_keyspace problematic_table 2>&1 | tee verify.log

# 3. Identify corrupted SSTables from output
grep -i "error\|corrupt" verify.log

# 4. Scrub to fix
sstablescrub my_keyspace problematic_table

Tool	Purpose	Modifies Data	When to Use
`sstableverify`	Check integrity	No	Diagnostics, health checks
`sstablescrub`	Fix corruption	Yes	After verify finds issues
`nodetool verify`	Online verification	No	While Cassandra running
`nodetool scrub`	Online scrub	Yes	Minor corruption, online

Decision Flow

Performance Considerations

Verification Time Estimates

Data Size	Basic Mode	Extended Mode
1 GB	~10 seconds	~1 minute
10 GB	~1 minute	~10 minutes
100 GB	~10 minutes	~1-2 hours
1 TB	~1-2 hours	~10-20 hours

Resource Usage

CPU: Moderate (checksum calculations)
Memory: Low (streams data)
Disk I/O: High (reads all SSTable data)

Optimizing Large Verifications

# Verify tables in parallel (if I/O allows)
sstableverify -f keyspace table1 &
sstableverify -f keyspace table2 &
sstableverify -f keyspace table3 &
wait

# Or limit to specific token ranges (comma-separated)
sstableverify -f -t -9223372036854775808,0 keyspace table &
sstableverify -f -t 0,9223372036854775807 keyspace table &
wait

Troubleshooting

Permission Denied

# Run as cassandra user
sudo -u cassandra sstableverify my_keyspace my_table

# Or fix ownership
sudo chown -R cassandra:cassandra /var/lib/cassandra/data/

Out of Memory

# Increase heap for verification
export JVM_OPTS="-Xmx4G"
sstableverify my_keyspace my_table

Tool Cannot Find SSTables

# List SSTables first
sstableutil my_keyspace my_table

# Verify directory structure
ls -la /var/lib/cassandra/data/my_keyspace/my_table-*/

# Check for in-progress compactions
ls /var/lib/cassandra/data/my_keyspace/my_table-*/*.log

False Positives

Verification may report issues that are not actual corruption:

In-progress compaction logs - Temporary files during compaction
Transaction logs - Pending operations

# Clean up transaction logs before verification
# (Only if Cassandra is stopped and was cleanly drained)
find /var/lib/cassandra/data/ -name "*.log" -type f

Best Practices

sstableverify Guidelines

Run regularly - Schedule weekly or monthly verification
Use extended mode - For thorough checks after incidents
Verify before upgrades - Ensure clean state before major changes
Keep Cassandra stopped - Required for accurate results
Check all critical tables - System tables and important application tables
Review logs first - Check for corruption indicators before verification
Plan for time - Extended verification can take hours on large datasets

Limitations

Does not fix corruption (use sstablescrub for that)
Requires Cassandra to be stopped
Extended mode is slow on large datasets
Cannot verify SSTables being actively written

Command	Relationship
sstablescrub	Fix corruption found by verify
nodetool verify	Online verification (less thorough)
nodetool scrub	Online scrub
sstablemetadata	View SSTable properties
sstableutil	List SSTable files