Cassandra SSTable Count Explosion

A table accumulates far more SSTables than its compaction strategy would normally hold — hundreds to thousands of small files — while generic compaction throughput still looks healthy. Each read then consults more SSTables, so read latency climbs and heap pressure grows from the per-SSTable bloom filters and index summaries held in memory.

This is distinct from general compaction backlog: here the count runs away, usually because SSTables are being produced faster than compaction consumes them (a flush or streaming storm), or because compaction is blocked or disabled for a specific table.

---

Symptoms

• SSTable count per table in the hundreds or thousands (nodetool tablestats)
• Read latency degrading as more SSTables are touched per read
• High SSTables per read in nodetool tablehistograms
• Large number of small *-Data.db files in the table data directory
• Heap pressure from bloom filters and index summaries
• Pending compactions high, or zero yet the count does not fall
• A sharp climb during or shortly after a repair, particularly a subrange or full repair

---

Diagnosis

Step 1: Identify the Offending Table

# Rank tables by SSTable count
nodetool tablestats my_keyspace | grep -E "Table:|SSTable count"

# Or count files directly for one table
ls /var/lib/cassandra/data/my_keyspace/my_table-*/*-Data.db | wc -l

Target: Generally < 20 SSTables per table for size-tiered, higher during transient bursts.

Step 2: Check Whether Compaction Is Enabled

nodetool statusautocompaction my_keyspace my_table

Auto-compaction stays disabled until re-enabled or the node restarts, so a prior nodetool disableautocompaction is a common cause.

Step 3: Check the Compaction Backlog

nodetool compactionstats -H

Problem indicators:

• High pending count that keeps growing
• No active compactions despite a large pending count

Step 4: Check Throughput and Concurrency

nodetool getcompactionthroughput      # default 64 MB/s
nodetool getconcurrentcompactors

Step 5: Check SSTables per Read

nodetool tablehistograms my_keyspace my_table

The SSTables column shows how many SSTables a read consults. Values consistently above single digits confirm read amplification from the count.

Step 6: Check Flush Frequency

grep -iE "flushing|writing memtable" /var/log/cassandra/system.log | tail -30

Frequent flushes of small memtables produce many small SSTables. Contributing factors include a low memtable_flush_period_in_ms, commit log pressure forcing flushes, heap shared across a very large number of tables, and repeated manual nodetool flush or node restarts.

Step 7: Check for Compaction Errors

grep -iE "CompactionExecutor|corrupt|marking.*unfinished" /var/log/cassandra/system.log | tail -30

A compaction that repeatedly throws on a corrupt or oversized SSTable leaves the rest of the backlog unprocessed.

Step 8: Check the Compaction Strategy

cqlsh -e "SELECT compaction FROM system_schema.tables WHERE keyspace_name = 'my_keyspace' AND table_name = 'my_table';"

---

Resolution

Case 1: Auto-Compaction Disabled

nodetool enableautocompaction my_keyspace my_table
nodetool statusautocompaction my_keyspace my_table   # confirm

Case 2: Compaction Falling Behind Write Rate

When SSTables are produced faster than compaction consumes them, raise throughput and concurrency to hardware limits:

nodetool setcompactionthroughput 128     # MB/s; 0 removes the limit
nodetool setconcurrentcompactors 4

These changes are runtime-only and reset on restart. Persist them in cassandra.yaml (compaction_throughput, concurrent_compactors) once a value is confirmed safe for the hardware.

Case 3: Flush Storm

Frequent flushing of small memtables is the most common source of a runaway count.

• Avoid unnecessary nodetool flush and frequent restarts.
• Review memtable_flush_period_in_ms; a low non-zero value forces periodic flushes regardless of memtable fullness (0 disables periodic flushing).
• Confirm the commit log is not undersized, which forces flushes of the dirtiest tables to release commit log segments.
• On nodes hosting a very large number of tables, memtable heap is divided across all of them, so each flushes while small; reducing table count or increasing memtable space raises flush size.

Case 4: Repair or Streaming Storm

Streaming during bootstrap, nodetool rebuild, and repair land many small SSTables that compaction consolidates once the operation completes. Raise throughput (Case 2) to let compaction catch up rather than forcing a major compaction mid-repair.

One combination produces a far larger explosion and must be avoided: full or subrange repair run against a table that has already been incrementally repaired.

Do Not Subrange-Repair Incrementally Repaired Tables

Problem: Subrange repair is always a full, non-incremental repair — it cannot mark data as repaired because it performs no anti-compaction. Incremental repair splits each table into repaired and unrepaired SSTable sets tracked by repairedAt metadata. Replicas compact those sets independently, so an SSTable can be compacted away on one node before another, leaving the same partitions marked repaired on some replicas and unrepaired on others. A later full or subrange repair sees those partitions as inconsistent and overstreams large volumes of data, each stream landing as new SSTables.

Symptoms: SSTable count climbs sharply on the repaired table during or immediately after the repair. Strategies that maintain many small range-partitioned SSTables amplify the effect: under LeveledCompactionStrategy the overstreamed data floods L0 and can create tens of thousands of small SSTables, and under the Unified Compaction Strategy (UCS, Cassandra 5.0, CEP-26) the same overstreaming lands across its density-level shards, multiplying the file count. Either can overwhelm a node.

Instead: Keep one repair model per table. Repair incrementally repaired tables incrementally, or reset the tables to a fully unrepaired state before switching to full or subrange repair — stop the node and clear the repaired status with sstablerepairedset --really-set --is-unrepaired. Cassandra 4.0 redesigned incremental repair (CASSANDRA-9143) to reduce this inconsistency, but mixing repair models on one table remains an anti-pattern.

Case 5: Idle Table Below min_threshold (STCS)

Size-tiered compaction only compacts once a size bucket holds min_threshold SSTables (default 4). A table receiving a slow trickle of writes can hold a stable handful of small SSTables that never reach the threshold. This is usually benign; if read amplification is measurable, lower the threshold or run a one-time major compaction:

ALTER TABLE my_table WITH compaction = {
    'class': 'SizeTieredCompactionStrategy',
    'min_threshold': 2
};

Case 6: Disk Full Blocking Compaction

Compaction needs free space to write its output. When the disk fills, compaction stalls and the count climbs. See Handle Full Disk.

nodetool clearsnapshot --all
df -h /var/lib/cassandra

Case 7: Corrupt or Stuck SSTable

If a single SSTable repeatedly fails compaction, identify it from the log, then scrub it. Prefer the offline sstablescrub (node stopped) over nodetool scrub when the node can be taken out of rotation.

# Online, node running
nodetool scrub my_keyspace my_table

# Offline, node stopped
sstablescrub my_keyspace my_table

Scrub discards unreadable data

Scrub rebuilds SSTables and drops rows it cannot deserialize. Run a full repair afterwards to restore any dropped data from other replicas.

Case 8: Last Resort — Major Compaction

A major compaction collapses the table's SSTables in one pass. It is resource-intensive and, for size-tiered, produces a single very large SSTable; use --split-output to emit several smaller ones instead.

nodetool compact --split-output my_keyspace my_table

Major compaction is a symptom fix

A major compaction clears the count once but does not address why SSTables accumulated. Resolve the root cause (Cases 1–4) or the count returns.

---

Recovery

Verify the Count Falls and Stabilizes

watch -n 30 'nodetool tablestats my_keyspace.my_table | grep "SSTable count"'

Verify Read Amplification Improves

# SSTables-per-read should drop
nodetool tablehistograms my_keyspace my_table

---

Monitoring

Metric	Warning	Critical
SSTable count per table	> 20	> 50
SSTables per read	> 4	> 10
Pending compactions	> 20	> 100
Disk usage	> 70%	> 85%

Per-table SSTable count, SSTables-per-read, and pending compactions are exposed as Cassandra table metrics; track them per table so a runaway count is caught before read latency degrades. See Monitoring.

---

Prevention

1. Match compaction to write rate - keep throughput and concurrency ahead of ingest.
2. Keep auto-compaction enabled - re-enable promptly after any maintenance that disabled it.
3. Avoid unnecessary flushes - do not flush or restart nodes more than needed.
4. Size the commit log adequately - prevent commit-log-driven flush storms.
5. Maintain free disk - keep utilization below 70% so compaction always has working space.
6. Limit table count - fewer tables per node means larger, less frequent flushes.

---

Related Commands

Command	Purpose
nodetool tablestats	SSTable count per table
nodetool tablehistograms	SSTables consulted per read
nodetool statusautocompaction	Whether auto-compaction is enabled
nodetool enableautocompaction	Re-enable auto-compaction
nodetool setcompactionthroughput	Adjust compaction throughput
nodetool setconcurrentcompactors	Adjust concurrent compactors
nodetool compact	Force compaction

Related Documentation

• Compaction Issues - General compaction backlog and stuck compactions
• Compaction Management - Compaction strategies and tuning
• Repair Failures - Repair problems, including repair-model conflicts
• Handle Full Disk - Disk space exhaustion
• Tombstone Accumulation - Tombstone-driven read amplification