Cassandra Recover from OOM

A node stops after running out of memory. Two different failures carry that name and lead to different fixes:

• The JVM throws java.lang.OutOfMemoryError when a Java allocation cannot be satisfied. The shipped configuration writes a heap dump and then kills the process.
• The Linux OOM killer terminates the process when memory runs out on the host, or in the cgroup, the kernel control group that caps the memory of a container. No heap dump is written, and the trigger is the total memory of the process rather than the heap alone.

The restart is the same either way. What has to change afterwards is not, so the two must be told apart before anything is tuned.

The behaviour described applies to Cassandra 5.0, 4.1 and 4.0. Heap, collector and heap dump defaults differ between those releases and within 5.0, so every claim below names the versions it holds for. What the Shipped Configuration Already Does collects those defaults in one place.

---

Symptoms

• The Cassandra process is absent, and peers report the node DN in nodetool status
• java.lang.OutOfMemoryError in /var/log/cassandra/system.log
• java invoked oom-killer and Out of memory: Killed process <pid> (java) in dmesg or the kernel journal
• An .hprof heap dump written at the time of the stop
• The node starts and then stops again within minutes, which indicates the trigger is still present
• Lengthening GC pauses in the period before the stop; see GC Pause Issues

---

Prerequisites

• A shell on the affected node with sudo or root privileges; the steps below change service state and read files owned by the Cassandra user
• nodetool access to a node that stayed up, for an accurate view of cluster state
• Read access to /var/log/cassandra/system.log and to the kernel journal
• Free space on the target filesystem of at least the max heap, since a heap dump is written on OutOfMemoryError and approaches the size of the live heap
• The location of the configuration the node actually runs, which is /etc/cassandra in the packaged layout and $CASSANDRA_HOME/conf in a tarball install

---

Return the Node to Service

Availability is restored first, subject to one condition before the start and one after it.

Copy the heap dump off the node before starting it. The dump is a file on disk, and a second OutOfMemoryError writes a second dump into the same space. Step 3 gives the paths to check.

sudo systemctl start cassandra
sudo grep -iE "commitlog|replay" /var/log/cassandra/system.log | tail -20

The packages install an LSB-tagged SysV init script rather than a systemd unit file, on 5.0, 4.1 and 4.0. systemd generates a unit from that script, so systemctl start cassandra works on a systemd host.

The process was killed rather than drained, so the memtables, the in-memory write buffers described in Write Path, were not flushed. Commit log replay reapplies the unflushed mutations on start. What a hard stop costs is bounded as described in What Is Lost When a Node Stops.

Starting the node is safe to repeat, but it does not address the cause. Where the trigger is a client query, a running repair or a compaction that resumes on start, the node reaches the same state again.

Do not add an automatic restart before the cause is known

A node that stops a second time should be left stopped until the diagnosis below has identified the trigger. Nothing in the shipped configuration restarts the process, and a Restart= directive added to a unit file turns a repeating OOM into a restart loop that reads as a running node. Each cycle also consumes coordinator resources and accumulates hints on the peers.

---

Diagnosis

Step 1: Determine Whether the JVM Threw or the Kernel Killed

# JVM OutOfMemoryError
sudo grep -n "java.lang.OutOfMemoryError" /var/log/cassandra/system.log | tail -20

# Kernel OOM killer
dmesg -T | grep -iE "oom-killer|killed process"
journalctl -k --since "2 hours ago" | grep -i "out of memory"

An OutOfMemoryError in system.log means the JVM could not satisfy an allocation and then killed itself through the shipped -XX:OnOutOfMemoryError command. A kernel message naming the Java process means the host or cgroup ran out of memory and the process was terminated from outside; there is no OutOfMemoryError line and no heap dump in that case.

The distinction decides the fix. A JVM error points at the heap or at a specific off-heap pool. A kernel kill points at the total resident memory of the process against the memory available to it, which is heap plus direct buffers plus native allocations plus thread stacks, and it can be triggered by a heap that is correctly sized for the host but too large for the cgroup.

If neither is present, the process stopped for another reason. Assess Node Failure covers the other stop paths and what each one costs.

Step 2: Read the OutOfMemoryError Qualifier

The text after OutOfMemoryError names the exhausted pool and is the strongest single piece of evidence available.

Message	Exhausted resource	Where to look
Java heap space	On-heap	Workload at the time, then the heap size (Step 4 and Step 5)
GC overhead limit exceeded	On-heap, with the collector unable to reclaim	Same as Java heap space
Direct buffer memory	Off-heap direct buffers, which hold network and compression buffers	-XX:MaxDirectMemorySize, which is half the max heap on 5.0.4 and later, and unset on 5.0.0 to 5.0.3, 4.1 and 4.0, where the JVM defaults it to the max heap size
Map failed	Address space or mmap count, not heap	vm.max_map_count and the SSTable count on the node; see SSTable Count Explosion
unable to create new native thread	Process or thread limits, not heap	The nofile and nproc limits for the Cassandra user

The last two are not solved by changing the heap, and enlarging the heap makes Map failed more likely rather than less, because it leaves less address space and less memory for mappings.

On 5.0.0 to 5.0.3, 4.1 and 4.0, -XX:MaxDirectMemorySize is not set, and the JVM then allows the direct buffer pool to reach the max heap size. A Direct buffer memory error on those versions therefore means the pool grew to roughly the size of the heap before failing. Total memory of the process is close to twice the heap at that point, which is the condition that provokes a kernel kill in a container.

Step 3: Locate and Preserve the Heap Dump

# Path a running node was given; a stopped node has no process to read
ps -o args= -p "$(pgrep -fn CassandraDaemon)" | tr ' ' '\n' | grep HeapDumpPath

# Common locations
sudo ls -lh /var/log/cassandra/*.hprof /var/lib/cassandra/*.hprof 2>/dev/null

On 5.0.4 and later, cassandra-env.sh appends -XX:HeapDumpPath on every start, defaulting to the log directory, which is /var/log/cassandra in the Debian packaging. Because that flag is appended after the options files are read, it overrides any -XX:HeapDumpPath set in jvm-server.options. CASSANDRA_HEAPDUMP_DIR is the setting that changes the destination, and it takes effect at the next start. On 5.0.0 to 5.0.3, 4.1 and 4.0 the flag is set only when CASSANDRA_HEAPDUMP_DIR is set, and with it unset the JVM writes java_pid<pid>.hprof into the working directory of the process.

A heap dump can fill the filesystem holding the logs

The dump approaches the size of the live heap, so a node running the 5.0.4 default heap on a large host writes tens of gibibytes into the log directory, which is commonly a small filesystem. A full data or commit log filesystem then triggers disk_failure_policy on the next write. Set CASSANDRA_HEAPDUMP_DIR to a filesystem with at least the max heap free, and remove dumps once they have been analysed. See Handle Full Disk.

The dump is written when the error is thrown, before the configured OnOutOfMemoryError command runs, so the kill does not truncate it.

Where the dump is too large to keep, -Dcassandra.printHeapHistogramOnOutOfMemoryError=true ships commented out in cassandra-env.sh on 5.0, 4.1 and 4.0. It writes a class histogram to the log instead, which identifies the dominant object type without the space cost.

Step 4: Establish the Effective Heap, Collector and Direct Memory

The effective values come from jvm-server.options, the per-JDK options file, and the computation in cassandra-env.sh together. Read them from the process or from nodetool, not from a single file.

# Flags the node actually started with
ps -o args= -p "$(pgrep -fn CassandraDaemon)" | tr ' ' '\n' \
  | grep -E '^-Xm|MaxDirectMemorySize|UseG1GC|UseConcMarkSweepGC|HeapDump'

# Used and max heap as the JVM reports them, on 5.0, 4.1 and 4.0
nodetool info | grep "Heap Memory"

UseG1GC selects G1, the collector that divides the heap into regions and reclaims them incrementally against a pause goal. UseConcMarkSweepGC selects CMS, the older concurrent mark-sweep collector, which has no pause goal. 5.0 ships G1 and 4.1 and 4.0 ship CMS, so a node showing the other one has been configured by hand.

nodetool info prints Heap Memory (MB) as used and max, separated by a slash, on all three versions. The max value is the one to compare against the formula in What the Shipped Configuration Already Does.

Step 5: Identify What the Node Was Doing

# Activity in the minutes before the error
sudo grep -B 200 "java.lang.OutOfMemoryError" /var/log/cassandra/system.log | tail -200

# Specific triggers
sudo grep -iE "large partition|Writing large partition" /var/log/cassandra/system.log | tail -20
sudo grep -iE "Compacting|Compacted" /var/log/cassandra/system.log | tail -20
sudo grep -iE "Validating|Merkle|repair" /var/log/cassandra/system.log | tail -20
sudo grep -iE "Batch for .* is of size|batch_size_(warn|fail)" /var/log/cassandra/system.log | tail -20

The last activity before the error identifies which case in the next section applies. Where several are present, the one that allocates per request scales with client traffic and is the more likely trigger.

---

Resolution

Cases 1 to 6 presuppose a java.lang.OutOfMemoryError in system.log and address what exhausted the heap. Case 7 covers a kernel OOM kill, where no such error was logged. Cases 8 and 9 apply to either.

Work through the cases in order and act on the first that matches. Each one stops the failure recurring; the node itself is already back in service from the section above.

Case 1: A Large Partition Read

Reads of a very large partition allocate in proportion to the data read, and several such reads in flight multiply that. concurrent_reads bounds how many read requests execute at once; it defaults to 32 on 5.0, 4.1 and 4.0, and changing it takes effect at the next start.

The durable fix is in the data model rather than the heap, because no heap size makes an unbounded partition safe to read. See Large Partition Issues.

Neither the settings that report large partitions nor the guardrails that replace them limit memory use. Guardrails are the named limits in cassandra.yaml that warn or reject when a threshold is crossed; see Guardrails. On 5.0, partition_size_warn_threshold and partition_size_fail_threshold both default to null and are therefore disabled. Both only log when enabled, since a partition is measured as it is written to an SSTable rather than as the write arrives. 5.0 deprecates the older settings in favour of those guardrails (CASSANDRA-18500). 4.1 has only compaction_large_partition_warning_threshold and 4.0 only compaction_large_partition_warning_threshold_mb, both always on, both defaulting to 100MiB, and both logging a warning without limiting anything. That 100MiB default is a logging threshold, not a supported maximum.

Case 2: Compaction

Compaction memory scales with the number of compactions running at once and with the size of the partitions being written. Reduce the concurrency first, since it takes effect without a restart:

# Immediate, not persisted across a restart
nodetool setconcurrentcompactors 1

One compactor bounds the memory at the cost of compaction throughput. The backlog grows while writes continue at the same rate, so SSTable counts rise and reads touch more files. Watch pending compactions while the setting is in place, and return the concurrency to its previous value once the memory cause is fixed.

concurrent_compactors in cassandra.yaml persists the change and applies at the next start. It defaults to the smaller of the number of disks and the number of cores, with a minimum of 2 and a maximum of 8, on 5.0, 4.1 and 4.0.

Compaction that is falling behind for other reasons is covered in Compaction Issues and SSTable Count Explosion.

Case 3: Repair Validation

Building Merkle trees for validation allocates on heap. Stop the validation that is running now, then bound the next one.

# Halts in-flight validation on this node, on 5.0, 4.1 and 4.0
nodetool stop VALIDATION

nodetool repair_admin list reports the incremental repair sessions the node is participating in, and nodetool repair_admin cancel --session <id> fails one, on all three versions. Neither command needs a restart, and a repair stopped this way has to be run again once the memory is bounded.

Two settings in cassandra.yaml bound the next repair. Both take effect at the next start.

repair_session_space limits the memory used for Merkle trees for one table and common token range. It defaults to one sixteenth of the heap and cannot be set below one mebibyte. Lowering it produces lower-resolution trees, which can over-stream data during repair.

concurrent_merkle_tree_requests limits how many validation requests one repair command runs at once. It defaults to 0, which is no limit, and a repair covering several tables or several virtual node ranges can therefore exceed repair_session_space several times over. Setting it to 1 keeps one repair command inside that bound. Validation is serialised at that setting, so the repair takes longer to finish. A repair cycle must still complete within gc_grace_seconds, which defaults to 10 days per table on 5.0, 4.1 and 4.0.

# cassandra.yaml on 5.0 and 4.1; the 4.0 name is given in the comment

# One validation request at a time keeps a repair command inside
# repair_session_space even when it covers several tables or vnode ranges
concurrent_merkle_tree_requests: 1

# Only where trees are still too large at that concurrency. The default is
# one sixteenth of the heap; any lower value trades tree resolution, and
# therefore over-streaming, for memory, and must not go below 1MiB
# repair_session_space: 256MiB    # repair_session_space_in_mb: 256 on 4.0

repair_session_space is the 4.1 rename of the 4.0 repair_session_space_in_mb (CASSANDRA-15234, which moved the configuration to size literals). concurrent_merkle_tree_requests is not a rename and carries the same name and default on every version that has it: it was added in 5.0.0, 4.1.5 and 4.0.13 (CASSANDRA-19336) and is absent from earlier patch releases.

Repairing one table at a time, or using subrange repair, also reduces the number of simultaneous validations. See Repair Failures.

Case 4: Large Batches

A batch is held in memory by the coordinator, and a logged batch is also written to the batch log. Reduce the batch size in the application. Batches spanning many partitions should be replaced by individual writes, which the driver can execute concurrently; a batch is not a performance optimisation in Cassandra. batch_size_warn_threshold and batch_size_fail_threshold on 5.0 and 4.1, named batch_size_warn_threshold_in_kb and batch_size_fail_threshold_in_kb on 4.0, report and then reject batches large enough to threaten the coordinator. All three versions default to 5KiB and 50KiB, and a batch approaching the fail threshold is already too large to be a single request.

Case 5: The Heap Is Pinned Below the Shipped Default

Where -Xmx and -Xms are set in jvm-server.options, the calculation in cassandra-env.sh is skipped entirely and the pinned value applies regardless of how much memory the host has. Compare the value from Step 4 against the formula for the running version in What the Shipped Configuration Already Does. On 5.0.4 and later the shipped default is half of host memory up to 31744M under G1, so a heap pinned at a smaller value on a host with more memory is below what the release would have chosen.

Removing the pinned pair returns the node to the shipped formula. Both flags must be removed together, or the start fails.

A heap should be departed from the shipped default only for a measured reason, and the measurement is post-collection occupancy over time rather than a single peak. Raising the heap without changing what allocates moves the failure later rather than removing it, and it lengthens each collection.

Case 6: The Heap Was Raised Above the Shipped Cap

The 31744M cap on 5.0.4 and later, described in What the Shipped Configuration Already Does, sits just under the boundary at which the JVM disables compressed object pointers. Above that boundary every reference widens, so a larger heap holds proportionally less data and each collection has more live data to trace. Raising -Xmx past the cap can therefore reduce usable capacity and lengthen pauses at the same time.

Where pauses rather than allocation failures are the problem, the collector and its settings are the subject rather than the heap size. See GC Pause Issues.

Case 7: A Container Memory Limit Below the Computed Heap

A kernel OOM kill inside a container, with no OutOfMemoryError in the log, usually means the heap was sized from host memory while the process runs under a smaller cgroup limit. Compare the max heap from Step 4 against the limit:

# cgroup v2
cat /sys/fs/cgroup/memory.max

# cgroup v1
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Set MAX_HEAP_SIZE in the environment, or -Xmx and -Xms as a pair in jvm-server.options, to a value that leaves room inside the limit for direct buffers, thread stacks, native allocations and the mapped SSTable pages the process touches. The heap alone must not be sized to the limit. Direct buffers are the largest of those on 5.0.0 to 5.0.3, 4.1 and 4.0, where -XX:MaxDirectMemorySize is unset and the JVM allows the pool to reach the max heap size. Setting that flag explicitly puts a known bound on the pool.

MAX_HEAP_SIZE may be set on its own under G1, which is the default on 5.0. On 4.1 and 4.0, which default to CMS, MAX_HEAP_SIZE and HEAP_NEWSIZE must be set together, and setting one alone stops the node starting with a message asking for the pair.

Case 8: The Node Will Not Start After the OOM

A start that fails during commit log replay is a corrupt segment rather than a memory problem, and it has its own procedure. Assess Node Failure gives the full sequence: read the segment name from the replay error, move that one segment outside commitlog_directory, and start the node again, repeating for any further segment named.

Do Not Delete the Commit Log Directory to Get a Node Started

Problem: Removing every file in commitlog_directory discards all unflushed mutations on the node, not only the ones in the segment that failed to replay. The mutations were acknowledged to clients and are lost from this replica, with no way back if the diagnosis was wrong.

Symptoms: The node starts, and rows written shortly before the stop are missing from it until a repair runs. Where another replica of the same range was also cleared, they are missing everywhere.

Instead: Quarantine the single segment named in the replay error, as described in Assess Node Failure. Where the directory cannot be read at all, or where quarantining segments one at a time has not produced a start, move the whole directory to a quarantine location outside commitlog_directory rather than deleting it, so the segments can be put back if the diagnosis was wrong, and repair the node afterwards.

Case 9: More Than One Node in the Same Incident

Simultaneous OOMs on several nodes point at a cluster-wide trigger rather than at one host: a query pattern that reads a large partition, a repair running across the cluster, or a schema change that enlarged a partition.

Confirm the shared trigger rather than assuming it. Run the Step 5 log check on each affected node and compare the last activity before the error, and compare the Step 2 qualifiers. The same qualifier and the same preceding activity across nodes identify one trigger; different ones mean the nodes failed for different reasons and each needs its own case above.

Start the nodes one at a time and let each reach UN before starting the next, so the returning nodes are not all taking hint delivery and read traffic at once. Identify the trigger before restoring full client traffic, because restarting into the same workload reproduces the failure.

---

Recovery

Verify the Node Is Serving

# From a node that stayed up
nodetool status

# On the recovered node
nodetool info | grep -E "Gossip active|Native Transport active|Heap Memory"

Verify the Heap Settles

Heap usage climbs and falls as collections run, so a single reading proves nothing. The node is stable when occupancy after collection returns to a similar level over successive cycles rather than trending upward under steady load.

nodetool gcstats

Verify Consistency

# On the recovered node, once per keyspace it holds
nodetool repair --full my_keyspace

On 5.0, 4.1 and 4.0, -pr restricts repair to the primary range of the node it runs on, so a single -pr run leaves most of the recovered node's replicated ranges unrepaired. Either nodetool repair --full my_keyspace on the recovered node, or nodetool repair -pr my_keyspace run on every node in turn, covers the affected ranges; a managed repair service that tracks coverage is the third option. Repair is safe to re-run. See Repair.

A node down for less than max_hint_window, which defaults to 3 hours on 5.0, 4.1 and 4.0, has its missed writes covered by hints, so the repair confirms consistency rather than restoring it.

---

What the Shipped Configuration Already Does

Most memory-related JVM settings are already set by the packaged configuration. Adding them again changes nothing, and the settings that matter operationally are the ones that are not set.

On 5.0, 4.1 and 4.0 alike, the shipped configuration writes a heap dump on OutOfMemoryError (-XX:+HeapDumpOnOutOfMemoryError) and then kills the process (-XX:OnOutOfMemoryError=kill -9 %p). The collector follows the release: 5.0 ships G1 as the default (CASSANDRA-18027), and 4.1 and 4.0 ship CMS with G1 present but commented out. The collector flags live in the per-JDK options file rather than in jvm-server.options: 5.0 reads jvm11-server.options under JDK 11 and jvm17-server.options under JDK 17, and 4.1 and 4.0 read jvm11-server.options under JDK 11.

The settings that differ between releases are these:

Setting	5.0.4 and later	5.0.0 to 5.0.3	4.1 and 4.0
Max heap, when -Xmx is unset in jvm-server.options	min(1/2 RAM, 31744M) under G1, which 5.0 ships; the min(1/2 RAM, 15872M) branch applies only where CMS has been configured by hand	max(min(1/2 RAM, 1024M), min(1/4 RAM, 8192M))	max(min(1/2 RAM, 1024M), min(1/4 RAM, 8192M))
-XX:MaxDirectMemorySize	half the max heap	not set, so the JVM defaults it to the max heap size	not set, so the JVM defaults it to the max heap size
-XX:HeapDumpPath	always set, to CASSANDRA_HEAPDUMP_DIR and defaulting to the log directory	set only when CASSANDRA_HEAPDUMP_DIR is set	set only when CASSANDRA_HEAPDUMP_DIR is set

The heap formula, the direct memory default and the unconditional heap dump path all changed in 5.0.4 (CASSANDRA-20296). Where -XX:MaxDirectMemorySize is not set, the JVM defaults it to the max heap size, so the network and compression buffers that fill that pool can add close to a second heap of resident memory.

The Heap Is Not in Any Options File

conf/cassandra-env.sh computes the heap size at start and appends -Xms and -Xmx to the command line. The computation is skipped only when both -Xmx and -Xms are set in jvm-server.options; setting one without the other makes the start fail with a message asking for the flags to be set in pairs. Reading jvm-server.options alone therefore does not establish the heap of a running node.

The JVM Is Already Killed on OutOfMemoryError

cassandra-env.sh sets JVM_ON_OUT_OF_MEMORY_ERROR_OPT="-XX:OnOutOfMemoryError=kill -9 %p" on 5.0, 4.1 and 4.0, and bin/cassandra puts it on the command line in both the foreground and the background start paths. -XX:+ExitOnOutOfMemoryError and -XX:+CrashOnOutOfMemoryError ship commented out in the same file as alternatives to swap in, the second of which also writes a core dump.

Nothing in the Shipped Configuration Restarts the Process

The Debian and RPM packages install an LSB-tagged SysV init script and no systemd unit file, on 5.0, 4.1 and 4.0. systemd generates a unit from the ### BEGIN INIT INFO header of that script, which is why systemctl start cassandra works, but a generated unit carries no Restart= directive. An automatic restart requires a unit file supplied by the operator, and one must not be added before the cause is understood.

The Heap Is Sized From Host Memory

On Linux, cassandra-env.sh reads free -m, which reports the memory of the host. In a container with a cgroup memory limit, free normally still reports the host, so the computed heap can exceed the limit the process actually runs under. Because -Xmx is then set explicitly, the container-aware heap sizing in the JVM does not correct it, and the kernel kills the process before the JVM ever throws.

---

Monitoring

Heap occupancy and GC behaviour are continuous metrics: they need history, so a rise over hours is visible, and alerting, so the rise is acted on before the allocation fails. Both are exposed as JVM metrics through JMX and belong in whatever monitoring the cluster already has. See Monitoring.

The thresholds worth alerting on follow from the shipped collector configuration, and from the memory limit the process runs under, rather than from a round number.

Signal	Warning	Critical	Basis
Heap occupancy after a collection, 5.0	Sustained above 70%	Does not fall below 70% after a full collection	5.0 ships -XX:InitiatingHeapOccupancyPercent=70, so G1 starts a concurrent cycle at that point
Heap occupancy after a collection, 4.1 and 4.0	Sustained above 75%	Does not fall below 75% after a full collection	4.1 and 4.0 ship -XX:CMSInitiatingOccupancyFraction=75 with -XX:+UseCMSInitiatingOccupancyOnly
Stop-the-world pause duration, 5.0	Repeatedly above 300ms	Approaching the client request timeout	5.0 ships -XX:MaxGCPauseMillis=300, so pauses persistently above it mean G1 is missing its goal
Stop-the-world pause duration, 4.1 and 4.0	Repeatedly above the established baseline for the node	Approaching read_request_timeout or write_request_timeout	CMS is shipped with no pause target, so the bounds that matter are those two cassandra.yaml settings, named read_request_timeout_in_ms and write_request_timeout_in_ms on 4.0
Process resident memory against the cgroup limit, where the node runs in a container	Sustained above 80% of the limit	Approaching the limit	The kernel kills the process at the limit, and the limit is read from memory.max on cgroup v2 and memory.limit_in_bytes on v1

The heap and pause signals are blind to a kernel OOM kill, because the JVM never sees the shortage and reports a healthy heap up to the moment the process is terminated. A containerised node therefore needs the cgroup signal as well.

Alert on occupancy after a collection rather than on instantaneous heap used. Instantaneous usage reaches high values normally between collections, so an alert on it fires on healthy nodes and is then ignored.

In AxonOps, heap occupancy is reported by the jvm_Memory_ metric: its scope attribute distinguishes HeapMemoryUsage from NonHeapMemoryUsage, and its function attribute gives used, max and committed. Collection counts and times are reported per collector by jvm_GarbageCollector_*. System Dashboard Metrics Mapping lists the JVM metrics and their attributes, and Set up alert rules covers building the thresholds above into alerts.

For looking at one node during an incident, a polling loop is adequate and immediate:

watch -n 10 'nodetool info | grep "Heap Memory"'
watch -n 30 'nodetool gcstats'

Neither retains history, neither alerts, and neither survives the session, so they supplement monitoring during an incident and do not replace it.

---

Prevention

• Leave the heap to the formula the running version ships, and depart from it only against a measurement of post-collection occupancy over time. On 5.0.4 and later that default is larger than on 4.1 and 4.0 for the same host.
• Set CASSANDRA_HEAPDUMP_DIR to a filesystem with at least the max heap free; it takes effect at the next start. The dump is enabled by default on 5.0, 4.1 and 4.0, and the default destination on 5.0.4 and later is the log directory.
• Size the heap against the cgroup limit rather than host memory wherever Cassandra runs in a container, and set -XX:MaxDirectMemorySize explicitly on 5.0.0 to 5.0.3, 4.1 and 4.0 so the direct buffer pool has a known bound.
• Bound repair memory with concurrent_merkle_tree_requests on 5.0, on 4.1.5 and later, and on 4.0.13 and later, and with repair_session_space where the trees are still too large at that concurrency. Both take effect at the next start.
• Keep partitions bounded in the data model. A partition must fit comfortably within the read-path headroom of the node, and the logging thresholds and guardrails in cassandra.yaml report partitions that exceed a size rather than preventing them.
• Treat the memtable pools as a fixed share of the heap rather than as a lever. memtable_heap_space and memtable_offheap_space both default to one quarter of the heap when omitted on 5.0, 4.1 and 4.0 (memtable_heap_space_in_mb and memtable_offheap_space_in_mb on 4.0). Lowering memtable_heap_space returns heap to the read path at the cost of flushing more often. memtable_offheap_space does nothing unless memtable_allocation_type is set to offheap_objects, since all three versions ship heap_buffers. All three settings take effect at the next start.

---

Related Commands

Command	Purpose
nodetool info	Used and max heap as the JVM reports them
nodetool gcstats	Collection counts and times since the last call
nodetool status	Cluster-wide node state, run from a node that stayed up
nodetool setconcurrentcompactors	Reduce compaction parallelism without a restart
nodetool stop VALIDATION	Halt in-flight repair validation without a restart
nodetool repair_admin	List and fail incremental repair sessions
nodetool flush	Flush memtables and release the memory they hold
nodetool repair --full	Rebuild the ranges the node missed while it was down

Related Documentation

• Assess Node Failure - What a stopped node costs, and the corrupt commit log segment procedure
• GC Pause Issues - Long pauses without an allocation failure
• Large Partition Issues - The dominant read-path source of heap pressure
• High Memory Usage - Memory growth short of an OOM
• Handle Full Disk - Disk exhaustion, including from heap dumps
• SSTable Count Explosion - Per-SSTable structures and mmap count
• Guardrails - The cassandra.yaml limits that warn or reject on a threshold
• Repair - Repair scope, coverage and scheduling
• Write Path - Memtables and the memtable pools
• Commit Log - What a hard stop costs