Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Cassandra Large Partition Issues

Large partitions are a common cause of performance problems in Cassandra. They cause slow reads, OOM during compaction, and uneven data distribution.


  • Slow reads on specific partition keys
  • OOM errors during compaction
  • “Compacting large partition” warnings in logs
  • Uneven disk usage across nodes
  • Read timeouts on specific queries
  • High GC pressure when accessing certain data

Step 1: Check for Large Partition Warnings

Section titled “Step 1: Check for Large Partition Warnings”
Terminal window
grep -i "large partition\|compacting large\|Writing large partition" /var/log/cassandra/system.log | tail -20

Example warning:

WARN Writing large partition my_keyspace/my_table:key123 (150.2 MiB) to sstable
Terminal window
nodetool tablestats my_keyspace.my_table

Key metrics:

  • Compacted partition maximum bytes: Largest partition
  • Compacted partition mean bytes: Average partition size

Target: Keep partitions under 100 MB.

Step 3: Identify Specific Large Partitions

Section titled “Step 3: Identify Specific Large Partitions”
Terminal window
# Use sstablepartitions tool (Cassandra 4.0+)
sstablepartitions /var/lib/cassandra/data/my_keyspace/my_table-*/nb-*-big-Data.db --min-size 50MB

Or query with tracing:

TRACING ON;
SELECT * FROM my_table WHERE partition_key = 'suspect_key' LIMIT 1;
TRACING OFF;

Step 4: Check Partition Size Configuration

Section titled “Step 4: Check Partition Size Configuration”
Terminal window
grep -i "partition_size\|compaction_large" /etc/cassandra/cassandra.yaml
Terminal window
# Watch for GC during large partition access
nodetool gcstats
# Watch heap usage
nodetool info | grep Heap

Temporarily skip large partition:

Terminal window
# Not recommended for production, but for emergency
# Reduce compaction throughput
nodetool setcompactionthroughput 16

Raise the heap (temporary):

This is a workaround, not a fix. A partition that does not fit is still too large after the heap grows, and the next one larger than it fails the same way.

The heap is not written in an options file on 5.0, 4.1 or 4.0. cassandra-env.sh computes it from host memory at every start, so raising it means overriding that computation on the node, and the override applies to every start until it is removed. Heap Sizing gives the computed value per release and host size, and Changing the Heap gives the two supported mechanisms and the pairing rule that stops the node starting when only one of -Xms and -Xmx is set. Every heap change takes effect at the next start, so it costs a restart of the node in each direction.

Short-term: Adjust the Reporting Threshold

Section titled “Short-term: Adjust the Reporting Threshold”

Raising the threshold at which a large partition is reported stops the warnings. The partition itself is unchanged, so this is worth doing only to keep the log readable while the data model is fixed:

# cassandra.yaml, on 5.0, where the guardrail defaults to null (disabled)
# 4.1: compaction_large_partition_warning_threshold: 100MiB
# 4.0: compaction_large_partition_warning_threshold_mb: 100
partition_size_warn_threshold: 100MiB

Problem Pattern 1: Unbounded partition growth

-- Bad: All events for a user in one partition
CREATE TABLE user_events (
user_id uuid,
event_time timestamp,
data text,
PRIMARY KEY (user_id, event_time)
);
-- Partition grows forever

Solution: Add time bucketing

-- Good: Events bucketed by day
CREATE TABLE user_events (
user_id uuid,
day date,
event_time timestamp,
data text,
PRIMARY KEY ((user_id, day), event_time)
);
-- Partition limited to one day of events

Problem Pattern 2: Wide rows with many columns

-- Bad: Thousands of columns per row
CREATE TABLE sensor_data (
sensor_id uuid,
metric_name text,
value double,
PRIMARY KEY (sensor_id, metric_name)
);
-- Can have millions of metrics per sensor

Solution: Add bucketing or limit scope

-- Good: Bucket by time period
CREATE TABLE sensor_data (
sensor_id uuid,
hour timestamp,
metric_name text,
value double,
PRIMARY KEY ((sensor_id, hour), metric_name)
);

Problem Pattern 3: Collection columns

-- Bad: Large collections
CREATE TABLE users (
user_id uuid PRIMARY KEY,
followers set<uuid> -- Can grow to millions
);

Solution: Use separate table

-- Good: Separate relationship table
CREATE TABLE user_followers (
user_id uuid,
follower_id uuid,
followed_at timestamp,
PRIMARY KEY (user_id, follower_id)
);

To fix existing large partitions:

-- 1. Create new table with better model
CREATE TABLE user_events_v2 (...);
-- 2. Migrate data with bucketing
-- Use Spark, application code, or COPY command
-- 3. Update application to use new table
-- 4. Drop old table after verification
DROP TABLE user_events;

Terminal window
# After data model fix, check new partition sizes
nodetool tablestats my_keyspace.my_table_v2 | grep "partition"

Set up alerting:

  • Alert on “Writing large partition” log entries
  • Alert on max partition size > 50MB
  • Monitor specific problematic partition keys

SizeStatusAction
< 10 MBGoodNo action needed
10-50 MBWarningMonitor, plan for growth
50-100 MBProblemRedesign data model
> 100 MBCriticalImmediate remediation required
Partition size ≈ (number of rows) × (average row size)
Row size ≈ sum of column sizes + clustering key size + 23 bytes overhead
MetricTarget
Partition size< 100 MB
Rows per partition< 100,000
Cells per partition< 100,000

  1. Design for bounded partitions - Always include time or other limiting factor
  2. Avoid unbounded collections - Use separate tables instead
  3. Monitor partition sizes - Alert before they become critical
  4. Test with realistic data - Load test with expected data volumes
  5. Review data model changes - Check for partition size impact

CommandPurpose
nodetool tablestatsCheck partition size metrics
sstablepartitionsFind large partitions in SSTables
nodetool gcstatsMonitor GC during access