Cassandra Memory Management

Cassandra runs as a Java process, so the memory a node uses falls into three pools, each managed by a different component:

The JVM heap, the region of memory the Java Virtual Machine allocates objects in. The garbage collector reclaims space in the heap once the objects holding it are no longer referenced, and it stops application threads to do part of that work.
Off-heap memory, also called native memory, which the process requests from the operating system directly. The garbage collector neither traces nor frees it; Cassandra allocates and releases it itself.
The operating system page cache, the kernel's copy of recently read file data, held in RAM that nothing else has claimed. Cassandra does not size it. The kernel fills it as files are read and evicts from it when the memory is needed elsewhere.

Every read and write a node serves touches more than one of these. A write lands in a memtable, the in-memory buffer holding recent writes for a table, which sits on the heap or off it depending on configuration. A read first consults a bloom filter, the off-heap structure that reports whether an SSTable may hold a partition. The node then reads the SSTable itself, an immutable data file on disk that the page cache may already hold.

Where the Data Lives

Pool	Bounded by	Reclaimed by
JVM heap	`-Xmx`, the maximum heap size the JVM is given, computed by `cassandra-env.sh` at start	The garbage collector, which pauses application threads for part of its work
Off-heap	Per-structure configuration, and for direct buffers `-XX:MaxDirectMemorySize`	Cassandra, when the structure it belongs to is discarded
Page cache	Whatever RAM is free	The kernel, on demand, without involving the process

Why Memory Is Divided This Way

The division follows from what garbage collection costs. Long-lived data on the heap is live data the collector accounts for on every cycle. The pauses that result raise the tail of the latency distribution on the node, because every request in flight waits for the pause to end. A pause long enough to stop the node answering gossip causes its peers to mark it down. Gossip is the protocol that carries node state between peers.

Cassandra's largest structures are also its longest-lived. Bloom filters, compression metadata and the partition index exist for the lifetime of an SSTable and can reach gigabytes on a node holding many partitions. Placing them on the heap would make the collector trace that data repeatedly for no benefit, since none of it becomes garbage until the SSTable is compacted away. With the data off-heap, the collector sees a reference to a native region rather than the gigabytes held in it. The cost of that placement is that Cassandra must manage the memory itself, and that the data is invisible to heap dumps and standard Java profilers.

The page cache exists for the opposite reason: the kernel manages it, so it costs the node nothing to maintain. The kernel fills it with SSTable data as reads occur and reclaims it as soon as something else needs the RAM, so memory left to the page cache is never wasted. The heap and the page cache draw on the same host memory, which is why a larger heap is not free even when it fits.

What Is Tuned, and What Is Not

The heap is not sized by hand on a default install. cassandra-env.sh computes it from host memory at every start, and the formula changed at 5.0.4, so the same host receives a substantially different heap on 5.0.4 and later than on 5.0.0 to 5.0.3, 4.1 and 4.0. A memory plan must therefore name the release it applies to. Departing from the computed value should rest on a measurement rather than a preference.

Beyond the heap, four things determine how the rest of the memory on a node behaves:

Where memtables are allocated. On-heap memtables add write-path churn to the collector's work; moving them off-heap removes it, at a small read cost.
How much of the heap is given to caches. The key cache and the row cache are on-heap by design, so cached data is live data. The row cache is disabled by default on 5.0, 4.1 and 4.0.
Total resident memory. Heap, off-heap pools, thread stacks and the JVM's own structures are separate budgets that all draw on host memory.
Host settings that undo the rest. Swap allows the heap to be paged to disk, and transparent huge pages introduce allocation stalls unrelated to garbage collection.

Off-heap memory counts against host memory

The resident memory of a Cassandra process is the heap plus everything it allocates outside the heap. Sizing a host or container from -Xmx alone leaves the off-heap pools unaccounted for, and the kernel can terminate the process while the JVM still reports a healthy heap.

The Pages in This Section

Question	Page
Why the heap is capped where it is, which collector a release runs, and where the heap sits in process memory	JVM Architecture
Which Cassandra structures sit on the heap, which sit off it, how they scale, and how to size and monitor them	Cassandra Memory Structures
Swap, transparent huge pages, memory map limits and NUMA on the host	Linux Memory Configuration

The settings themselves, the heap value per release and host size, and the procedure for changing them are in JVM Tuning.

Storage Engine Overview - Where the read and write paths allocate
Write Path - Memtable flush process
Read Path - Cache behaviour during reads
Performance - How memory fits into node sizing
Recover from OOM - Diagnosis after a node is lost to memory exhaustion