Cassandra Large Partition Issues
Large partitions are a common cause of performance problems in Cassandra. They cause slow reads, OOM during compaction, and uneven data distribution.
Symptoms
Section titled “Symptoms”- Slow reads on specific partition keys
- OOM errors during compaction
- “Compacting large partition” warnings in logs
- Uneven disk usage across nodes
- Read timeouts on specific queries
- High GC pressure when accessing certain data
Diagnosis
Section titled “Diagnosis”Step 1: Check for Large Partition Warnings
Section titled “Step 1: Check for Large Partition Warnings”grep -i "large partition\|compacting large\|Writing large partition" /var/log/cassandra/system.log | tail -20Example warning:
WARN Writing large partition my_keyspace/my_table:key123 (150.2 MiB) to sstableStep 2: Check Table Statistics
Section titled “Step 2: Check Table Statistics”nodetool tablestats my_keyspace.my_tableKey metrics:
Compacted partition maximum bytes: Largest partitionCompacted partition mean bytes: Average partition size
Target: Keep partitions under 100 MB.
Step 3: Identify Specific Large Partitions
Section titled “Step 3: Identify Specific Large Partitions”# Use sstablepartitions tool (Cassandra 4.0+)sstablepartitions /var/lib/cassandra/data/my_keyspace/my_table-*/nb-*-big-Data.db --min-size 50MBOr query with tracing:
TRACING ON;SELECT * FROM my_table WHERE partition_key = 'suspect_key' LIMIT 1;TRACING OFF;Step 4: Check Partition Size Configuration
Section titled “Step 4: Check Partition Size Configuration”grep -i "partition_size\|compaction_large" /etc/cassandra/cassandra.yamlStep 5: Monitor During Access
Section titled “Step 5: Monitor During Access”# Watch for GC during large partition accessnodetool gcstats
# Watch heap usagenodetool info | grep HeapResolution
Section titled “Resolution”Immediate: Handle OOM During Compaction
Section titled “Immediate: Handle OOM During Compaction”Temporarily skip large partition:
# Not recommended for production, but for emergency# Reduce compaction throughputnodetool setcompactionthroughput 16Raise the heap (temporary):
This is a workaround, not a fix. A partition that does not fit is still too large after the heap grows, and the next one larger than it fails the same way.
The heap is not written in an options file on 5.0, 4.1 or 4.0. cassandra-env.sh computes it from host memory at every start, so raising it means overriding that computation on the node, and the override applies to every start until it is removed. Heap Sizing gives the computed value per release and host size, and Changing the Heap gives the two supported mechanisms and the pairing rule that stops the node starting when only one of -Xms and -Xmx is set. Every heap change takes effect at the next start, so it costs a restart of the node in each direction.
Short-term: Adjust the Reporting Threshold
Section titled “Short-term: Adjust the Reporting Threshold”Raising the threshold at which a large partition is reported stops the warnings. The partition itself is unchanged, so this is worth doing only to keep the log readable while the data model is fixed:
# cassandra.yaml, on 5.0, where the guardrail defaults to null (disabled)# 4.1: compaction_large_partition_warning_threshold: 100MiB# 4.0: compaction_large_partition_warning_threshold_mb: 100partition_size_warn_threshold: 100MiBLong-term: Fix Data Model
Section titled “Long-term: Fix Data Model”Problem Pattern 1: Unbounded partition growth
-- Bad: All events for a user in one partitionCREATE TABLE user_events ( user_id uuid, event_time timestamp, data text, PRIMARY KEY (user_id, event_time));-- Partition grows foreverSolution: Add time bucketing
-- Good: Events bucketed by dayCREATE TABLE user_events ( user_id uuid, day date, event_time timestamp, data text, PRIMARY KEY ((user_id, day), event_time));-- Partition limited to one day of eventsProblem Pattern 2: Wide rows with many columns
-- Bad: Thousands of columns per rowCREATE TABLE sensor_data ( sensor_id uuid, metric_name text, value double, PRIMARY KEY (sensor_id, metric_name));-- Can have millions of metrics per sensorSolution: Add bucketing or limit scope
-- Good: Bucket by time periodCREATE TABLE sensor_data ( sensor_id uuid, hour timestamp, metric_name text, value double, PRIMARY KEY ((sensor_id, hour), metric_name));Problem Pattern 3: Collection columns
-- Bad: Large collectionsCREATE TABLE users ( user_id uuid PRIMARY KEY, followers set<uuid> -- Can grow to millions);Solution: Use separate table
-- Good: Separate relationship tableCREATE TABLE user_followers ( user_id uuid, follower_id uuid, followed_at timestamp, PRIMARY KEY (user_id, follower_id));Data Migration Strategy
Section titled “Data Migration Strategy”To fix existing large partitions:
-- 1. Create new table with better modelCREATE TABLE user_events_v2 (...);
-- 2. Migrate data with bucketing-- Use Spark, application code, or COPY command
-- 3. Update application to use new table
-- 4. Drop old table after verificationDROP TABLE user_events;Recovery
Section titled “Recovery”Verify Partition Sizes
Section titled “Verify Partition Sizes”# After data model fix, check new partition sizesnodetool tablestats my_keyspace.my_table_v2 | grep "partition"Monitor for Recurrence
Section titled “Monitor for Recurrence”Set up alerting:
- Alert on “Writing large partition” log entries
- Alert on max partition size > 50MB
- Monitor specific problematic partition keys
Partition Size Guidelines
Section titled “Partition Size Guidelines”| Size | Status | Action |
|---|---|---|
| < 10 MB | Good | No action needed |
| 10-50 MB | Warning | Monitor, plan for growth |
| 50-100 MB | Problem | Redesign data model |
| > 100 MB | Critical | Immediate remediation required |
Estimating Partition Size
Section titled “Estimating Partition Size”Partition size ≈ (number of rows) × (average row size)Row size ≈ sum of column sizes + clustering key size + 23 bytes overheadDesign Targets
Section titled “Design Targets”| Metric | Target |
|---|---|
| Partition size | < 100 MB |
| Rows per partition | < 100,000 |
| Cells per partition | < 100,000 |
Prevention
Section titled “Prevention”- Design for bounded partitions - Always include time or other limiting factor
- Avoid unbounded collections - Use separate tables instead
- Monitor partition sizes - Alert before they become critical
- Test with realistic data - Load test with expected data volumes
- Review data model changes - Check for partition size impact
Related Commands
Section titled “Related Commands”| Command | Purpose |
|---|---|
nodetool tablestats | Check partition size metrics |
sstablepartitions | Find large partitions in SSTables |
nodetool gcstats | Monitor GC during access |
Related Documentation
Section titled “Related Documentation”- Data Modeling - Partition design best practices
- GC Pause Issues - GC problems from large partitions
- Compaction Management - Compaction and large partitions
- JVM Tuning - The computed heap per release, and how to change it