Cassandra Memory Structures

Cassandra places each of its in-memory structures on the JVM heap, in off-heap native memory, or leaves the data to the operating system page cache. This page states where each structure sits, how its size scales, and how it is configured and observed.

Memory Management introduces the three pools and the reasoning behind the division. JVM Architecture covers the heap itself and the garbage collector.

---

Heap Components

The JVM heap holds Cassandra's managed data structures. These are subject to garbage collection.

Component	Description	Configuration
Memtables	In-memory write buffer per table	memtable_heap_space (memtable_heap_space_in_mb on 4.0)
Key Cache	Maps partition keys to SSTable offsets	key_cache_size (key_cache_size_in_mb on 4.0)
Row Cache	Caches entire rows (use sparingly)	row_cache_size (row_cache_size_in_mb on 4.0)

The setting names above are the ones used on 5.0 and 4.1. Cassandra 4.1 moved the configuration to size literals, so each of these settings lost its _in_mb suffix and now takes a value carrying its unit (CASSANDRA-15234). The 4.0 name is given in parentheses throughout this page.

Memtables

Memtables buffer writes before flushing to SSTables. Each table has one active memtable. Memtable covers what a memtable holds, where it sits in the write path and what triggers a flush; this section covers only the memory it occupies.

cassandra.yaml

# Total heap space for memtables across all tables, on 5.0 and 4.1
# 4.0: memtable_heap_space_in_mb: 4096
memtable_heap_space: 4096MiB

# Flush threshold (fraction of memtable space)
# Note: memtable_cleanup_threshold is deprecated on 5.0, 4.1 and 4.0 and is
# computed by the node. Set it only to override that computed value.
# memtable_cleanup_threshold: 0.11

Larger memtables flush less often and hold more heap while they fill. On write-heavy workloads, off-heap memtables keep that growth out of the heap.

Key Cache

The key cache stores partition key to SSTable offset mappings, so a read of a frequently accessed partition skips the partition index lookup. Key Cache covers what it holds and where it sits in the read path, including why the trie-indexed bti format on 5.0 does not use it the same way.

cassandra.yaml

# Key cache size on 5.0 and 4.1, defaulting to the smaller of 5% of heap or 100MiB
# 4.0: key_cache_size_in_mb: 100
key_cache_size: 100MiB

# Save interval (0 disables saving)
key_cache_save_period: 14400

# Keys to save (empty = all)
key_cache_keys_to_save:

Row Cache

The row cache stores whole rows on heap. Row Cache covers what it holds and the workloads it helps or hurts; the heap it occupies is the concern here.

cassandra.yaml

# Disabled by default, on 5.0 and 4.1
# 4.0: row_cache_size_in_mb: 0
row_cache_size: 0MiB

# Save interval
row_cache_save_period: 0

Row Cache Caution

Row cache is disabled by default for good reason. Enable only for specific tables with read-heavy, rarely-updated data. Consider the key cache or OS page cache first.

---

Off-Heap Memory

Off-heap memory is native memory the process allocates outside the JVM heap. The garbage collector neither traces nor frees it, so data held there does not add to the live set a collection cycle walks. Why Memory Is Divided This Way gives the reasoning behind the split, and Garbage Collection and Cassandra covers what a pause costs under each collector.

How Cassandra Uses Off-Heap Memory

The structures Cassandra places off the heap are the large, long-lived ones:

Component	Why Off-Heap?
Bloom filters	Bloom filters are large (10 bits per partition key × number of SSTables), long-lived (exist for SSTable lifetime), and accessed on every read. On-heap, they would consume gigabytes and be scanned by every GC cycle.
Partition index	In the default big format the off-heap index summary (a sampling of the on-disk partition index) grows with partition count; the trie-indexed bti format (5.0+, opt-in) holds a trie partition index in its place. A table with 100 million partitions could have a multi-gigabyte index. Off-heap placement prevents this from bloating GC pause times.
Compression metadata	Stores byte offsets for each compressed chunk in an SSTable. Grows proportionally to data size and SSTable count. Rarely changes once written.
Off-heap memtables	Memtables have high object churn: data is constantly written and then flushed. This churn creates GC pressure. Off-heap memtables keep write-path allocations out of the heap.
Chunk cache	Caches decompressed SSTable blocks. Can grow to multiple gigabytes for read-heavy workloads. Off-heap prevents cache growth from impacting GC.
Networking buffers	Direct ByteBuffers enable zero-copy I/O between the network stack and Cassandra. Data can be sent/received without copying through the heap.

What Stays on the Heap

The rest is the other side of that list. Request and query state is discarded within the request that created it, which is the allocation pattern generational collection handles cheapest. Structures carrying many internal references do not flatten into a native region without being serialised on every access.

Cassandra's Memory Allocators

Cassandra implements custom memory allocators to manage off-heap memory efficiently:

• BufferPool: Manages pools of direct ByteBuffers for networking and chunk cache, avoiding allocation overhead
• NativeAllocator: Uses Unsafe for bloom filters and index structures with explicit lifecycle management
• Slab allocation: Reduces fragmentation by allocating fixed-size chunks rather than variable-sized blocks

Off-Heap Trade-offs

Advantage	Disadvantage
Not traced or collected by the garbage collector	Manual memory management required
Can exceed heap size limits	Memory leaks if not freed properly
Better cache locality for large structures	Slightly slower allocation than heap
Reduced heap pressure	Harder to debug (not visible in heap dumps)
Enables larger working sets	Must account for in capacity planning

Monitoring Off-Heap Usage

# Total off-heap memory used by Cassandra
nodetool info | grep "Off Heap"

# Native memory tracking (JVM flag required)
# Add to jvm-server.options: -XX:NativeMemoryTracking=summary
jcmd <pid> VM.native_memory summary

Off-Heap Components in Detail

Component	Description	Memory Scaling
Bloom filters	Probabilistic existence checks	~10 bits per partition key per SSTable
Compression metadata	Chunk offset mappings	Proportional to data size
Partition index	Big-format index summary; trie index in bti (5.0+)	Proportional to partition count
Memtables	Write buffer (if configured)	memtable_offheap_space (memtable_offheap_space_in_mb on 4.0)
Chunk cache	Decompressed SSTable chunks	file_cache_size (file_cache_size_in_mb on 4.0)

Off-Heap Memtables

Moving memtables off-heap reduces GC pressure significantly for write-heavy workloads.

cassandra.yaml

# Off-heap memtable allocation (choose based on workload)
# memtable_allocation_type: offheap_objects  # Write-heavy: lowest GC
# memtable_allocation_type: offheap_buffers  # Read-heavy: minimal read impact
memtable_allocation_type: offheap_objects

Type	Description	Best For
heap_buffers	All memtable data on heap (default)	Low-memory environments, simple deployments
offheap_buffers	Cell names/values in DirectBuffers, metadata on heap	Read-heavy workloads, large cell values (blobs, long strings)
offheap_objects	Entire cells off-heap, only pointers on heap	Write-heavy workloads, small cell values (ints, UUIDs), lowest GC pressure

offheap_objects gives the lowest GC pressure and suits write-heavy workloads. It adds a small read cost, because data is copied back to the heap when it is read, and it needs the JNA library. offheap_buffers suits read-heavy workloads with large values: the read cost is smaller and so is the reduction in GC pressure. heap_buffers is the default on 5.0, 4.1 and 4.0, and is what applies where JNA is unavailable.

Chunk Cache

The chunk cache stores decompressed SSTable chunks, reducing CPU overhead for repeated reads.

cassandra.yaml

# Auto-sized by default to the smaller of 1/4 of heap or 512MiB (allocated off-heap)
# Manually set if needed, on 5.0 and 4.1
# 4.0: file_cache_size_in_mb: 2048
file_cache_size: 2048MiB

---

OS Page Cache

The operating system automatically caches recently accessed file data in unused RAM. This is Cassandra's primary read cache for SSTable data.

How It Works

• SSTable data cached after first read
• No Cassandra configuration required
• Automatically sized to available RAM
• Shared across all processes
• Evicted under memory pressure (LRU)

Sizing

Page Cache = Total RAM - JVM Heap - Off-Heap - OS Overhead

Example (64GB server, Cassandra 5.0.4 and later):
- JVM Heap: 31GB, the computed value at this host size
- Off-heap: 4-6GB
- OS overhead: 4GB
- Page cache: 23-25GB available

The heap in that example is the 5.0.4 and later value; on 5.0.0 to 5.0.3, 4.1 and 4.0 the computed heap on the same host is far smaller, so the page cache share is correspondingly larger. Heap sizing per version gives the figure for each host size and release.

Maximizing Page Cache Effectiveness

• Leave the heap at the value the release computes unless a measurement supports departing from it
• Leave sufficient free RAM
• Avoid memory-hungry co-located processes
• Use SSDs for faster cache misses

---

Memory Sizing Example

64GB Server Configuration on 5.0.4 and Later

The division above holds on 5.0.4 and later; on 5.0.0 to 5.0.3, 4.1 and 4.0 the computed heap on this host is several times smaller, which moves the same memory into the page cache row. A memory plan must therefore name the release it applies to.

Configuration

cassandra.yaml

# Settings shown with the 5.0 and 4.1 names; the 4.0 name is in the comment

# Memtables
memtable_heap_space: 4096MiB      # 4.0: memtable_heap_space_in_mb: 4096
memtable_allocation_type: offheap_buffers

# Key cache
key_cache_size: 100MiB            # 4.0: key_cache_size_in_mb: 100

# Row cache (disabled)
row_cache_size: 0MiB              # 4.0: row_cache_size_in_mb: 0

The heap needs no entry in jvm-server.options here. cassandra-env.sh computes it at every start, and pinning -Xmx and -Xms overrides that computation on every host the configuration is deployed to. JVM Tuning covers the two supported ways of overriding it and the measurement a departure should rest on.

---

Monitoring Memory

Heap and Off-Heap Usage

# Overall memory status
nodetool info

# Heap memory
nodetool info | grep "Heap Memory"

# Off-heap memory
nodetool info | grep "Off Heap Memory"

# GC statistics
nodetool gcstats

Per-Table Memory

# Table statistics including bloom filter size
nodetool tablestats keyspace.table

# Bloom filter memory
nodetool tablestats | grep -i bloom

JMX Metrics

# Heap
java.lang:type=Memory/HeapMemoryUsage

# Memtables
org.apache.cassandra.metrics:type=Table,name=MemtableOnHeapSize
org.apache.cassandra.metrics:type=Table,name=MemtableOffHeapSize

# Caches
org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Size
org.apache.cassandra.metrics:type=Cache,scope=RowCache,name=Size

# Bloom filters
org.apache.cassandra.metrics:type=Table,name=BloomFilterOffHeapMemoryUsed

---

Troubleshooting

High Heap Usage

Symptoms:

• Long GC pauses
• Heap usage consistently >70%
• OutOfMemoryError

Solutions:

1. Move memtables off-heap (memtable_allocation_type: offheap_buffers)
2. Reduce key cache size
3. Disable row cache if enabled
4. Reduce number of tables (see below)

Memory Pressure from Many Tables

Each table requires memory for:

• One memtable
• Bloom filters per SSTable
• Index structures per SSTable

A node should hold no more than about 200 tables. Each table consumes memory whether or not it holds data.

Bloom Filter Memory

Bloom filter memory scales with partition count and SSTable count:

Memory ≈ partitions × SSTables × bits_per_key

Example:
- 100 million partitions
- 20 SSTables average
- 10 bits per key
- ≈ 2.5GB bloom filter memory

Raising bloom_filter_fp_chance reduces the memory each filter needs, at the cost of more false positives and so more SSTable reads that return no row. Lowering the SSTable count through compaction removes whole filters. Modelling fewer, larger partitions reduces the number of keys each filter holds, within the partition size the data model can carry.

-- Increase false positive rate to reduce memory
ALTER TABLE my_table WITH bloom_filter_fp_chance = 0.1;

---

Configuration Reference

Workload-Specific Settings

Write-Heavy:

# 4.0: memtable_heap_space_in_mb: 4096
memtable_heap_space: 4096MiB
memtable_allocation_type: offheap_objects  # Lowest GC pressure for high write rates
memtable_flush_writers: 4

Read-Heavy:

memtable_allocation_type: offheap_buffers  # Minimal read overhead
# 4.0: key_cache_size_in_mb: 200
key_cache_size: 200MiB
# Ensure sufficient page cache for working set

Mixed:

# 4.0: memtable_heap_space_in_mb: 2048 and key_cache_size_in_mb: 100
memtable_heap_space: 2048MiB
memtable_allocation_type: offheap_objects
key_cache_size: 100MiB

---

Related Documentation

• Memory Management - The three memory pools and why a node divides memory that way
• JVM Architecture - The heap, the collector, and process memory outside the heap
• Linux Memory Configuration - Kernel settings, swap, THP, and NUMA
• Storage Engine Overview - Architecture overview
• Write Path - Memtable flush process
• Read Path - Cache behaviour during reads