nodetool gcstats

Displays garbage collection statistics for the Cassandra JVM.

---

Synopsis

nodetool [connection_options] gcstats [options]

See connection options for connection options.

Options

Option	Description
-F, --format <format>	Output format: json, yaml, or table (default)
-H, --human-readable	Display sizes in human-readable format

Description

nodetool gcstats shows cumulative garbage collection metrics including:

• GC invocation counts
• Total GC time
• Time since last GC

Collection that takes an increasing share of node time raises request latency and, at the extreme, stops the node responding to peers. The values come from the JVM garbage collection MXBeans over JMX (java.lang:type=GarbageCollector) rather than from Cassandra's own metrics, so they report what the JVM observed rather than what Cassandra measured.

Elapsed time is not stop-the-world pause time

The elapsed time gcstats reports is the total time the collector was active, which is not the time application threads were stopped. G1, Shenandoah and ZGC do much of their work alongside application threads, so their elapsed time covers concurrent phases as well as pauses; a fully stop-the-world collector is the only case in which the two are equal. Use the GC log for pause durations, and read gcstats as a measure of collection pressure over an interval.

---

Output Example

       Interval (ms) Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed (ms)   GC Reclaimed (bytes)         GC Count   Allocated Direct Memory Bytes  Max Direct Memory Bytes  Reserved Direct Memory Bytes
                 986                 123                45678                   45              129536000             1234                        67108864                134217728                     134217728

---

Output Fields

Field	Description
Interval (ms)	Time since gcstats was last called
Max GC Elapsed (ms)	Longest single GC pause
Total GC Elapsed (ms)	Cumulative GC time
Stdev GC Elapsed (ms)	Standard deviation of GC times
GC Reclaimed (bytes)	Memory reclaimed by GC (in bytes; use -H for human-readable units)
GC Count	Number of GC invocations
Allocated Direct Memory Bytes	Currently allocated off-heap (direct) memory
Max Direct Memory Bytes	Maximum off-heap memory configured
Reserved Direct Memory Bytes	Reserved off-heap memory

---

Interpreting Results

Healthy GC

Max GC Elapsed (ms)Total GC Elapsed (ms)   Collections
                50                 5000          1000

Expect Max GC Elapsed to sit below the pause goal set for the running collector. 5.0 sets -XX:MaxGCPauseMillis=300 for G1, and 4.1 and 4.0 under G1 set 300 and 500 respectively; CMS, which 4.1 and 4.0 start with, has no goal, so the comparison there is against the baseline established for the node under representative load. Monitoring thresholds per version gives the figure for each release and the setting it derives from.

GC Pressure Signs

Max GC Elapsed (ms)Total GC Elapsed (ms)   Collections
               2000               500000         50000

Pressure shows as Max GC Elapsed repeatedly above the goal for the running collector, as a collection count rising while the workload is unchanged, and as average elapsed time per collection trending upward across intervals. The last of those needs successive samples, since one call reports a single interval.

Long GC Pauses

Max GC Elapsed (ms): 2345

A pause approaching read_request_timeout or write_request_timeout in cassandra.yaml (read_request_timeout_in_ms and write_request_timeout_in_ms on 4.0) fails the requests in flight on the node, whatever the collector. A pause long enough to stop the node answering gossip causes peers to mark it down, after which hints accumulate and coordinators route around it until it returns. Both consequences follow from the pause duration against those timeouts rather than from a fixed number of milliseconds.

---

When to Use

Routine Monitoring

# Check GC health periodically
nodetool gcstats

Investigating Latency Issues

When experiencing slow queries:

nodetool gcstats
nodetool tpstats
nodetool proxyhistograms

After Heap Tuning

Verify GC behavior after JVM changes:

# Before change
nodetool gcstats > gc_before.txt

# Make JVM changes, restart

# After change
nodetool gcstats > gc_after.txt
diff gc_before.txt gc_after.txt

During Load Testing

Monitor GC under stress:

watch -n 5 'nodetool gcstats'

---

GC Analysis Over Time

Comparing Intervals

# Take samples
nodetool gcstats
sleep 60
nodetool gcstats

The difference shows GC activity in that interval:

Metric	Sample 1	Sample 2	Delta	Interpretation
Collections	1000	1050	50	~1 GC/sec
Total GC	5000	7000	2000ms	2 sec GC in 60 sec
GC %	-	-	3.3%	Acceptable

Calculating GC Overhead

GC overhead = (Total GC Elapsed / Node Uptime) × 100

Overhead	Assessment
< 5%	Normal
5-10%	Monitor closely
> 10%	Tuning needed
> 20%	Critical

---

Common GC Issues

Frequent Full GC

High collection count with large Max GC indicates full GCs:

Causes:

• The heap is too small for the live set the workload retains
• An object is held past its use, so the live set grows across cycles
• Individual allocations are large enough to bypass the young generation, which under G1 means humongous allocations occupying whole regions

Solutions:

• Check what is retained on the heap before raising it: a larger heap without a change in allocation moves the failure later and lengthens each collection
• Review the workload for the usual sources of retention, large partitions and unbounded reads
• Where the heap is undersized, compare it against the value the running release computes for the host (heap sizing per version), since a heap below that figure has been pinned by hand

Long GC Pauses

Causes:

• A stop-the-world collection traces the whole heap in one pause, which is what CMS falls back to on 4.1 and 4.0 when concurrent collection cannot keep up
• The old generation has no room for objects surviving a young collection, so promotion fails and a full collection follows
• The live set itself is large, so every cycle traces and copies more of it

Solutions:

• Reduce the allocation rate first: large partitions, unbounded reads and oversized batches produce most of it
• Confirm the collector matches the release, G1 on 5.0 and CMS on 4.1 and 4.0, since a hand-configured collector explains behaviour the shipped settings do not
• Change one garbage collector setting at a time, against a pause profile measured over representative load

Memory Not Reclaimed

Low GC Reclaimed despite high collections:

Causes:

• Most of what the heap holds is long-lived, such as on-heap caches and memtables, so a collection finds little to reclaim
• An object is held past its use, so the live set grows across cycles and does not fall back
• The node is short of memory off the heap rather than on it, which no heap collection can relieve

Solutions:

• Compare the key cache and row cache sizes against the heap, since on-heap cached data is live data by design (Cassandra memory structures)
• Take a heap dump and identify what holds the largest retained sets before changing any collector setting
• Read off-heap usage with nodetool info | grep "Off Heap Memory", since a node under off-heap pressure shows the same low reclaim figure and responds to none of the heap changes above

---

JVM GC Settings

The collector and its settings ship active in the per-JDK options file and do not need adding. The heap is computed by cassandra-env.sh at start rather than written in an options file, and GC logging is already enabled, writing to /var/log/cassandra/gc.log in 10 rotating files of 10MiB.

Heap sizing and garbage collector settings per version gives the full set of shipped values, what changing each one costs, and how to measure the result.

Analyzing GC Logs

Use tools like:

• GCViewer
• GCEasy
• VisualVM

---

Correlation with Other Metrics

GC and Thread Pools

nodetool gcstats
nodetool tpstats

High GC often correlates with:

• Blocked threads
• Pending tasks
• Dropped messages

GC and Latency

nodetool gcstats
nodetool proxyhistograms

GC pauses directly impact p99 latencies.

GC and Heap

nodetool gcstats
nodetool info | grep "Heap Memory"

Near-full heap triggers more frequent GC.

---

Examples

Basic Check

nodetool gcstats

Monitor Continuously

watch -n 10 'nodetool gcstats'

Compare Across Nodes

for node in node1 node2 node3; do
    echo "=== $node ==="
    ssh "$node" "nodetool gcstats"
done

Alert on High GC

#!/bin/bash
# Threshold is the pause goal the running collector was given: 300 under G1 on
# 5.0. Under CMS on 4.1 and 4.0 there is no goal, so use the baseline measured
# for the node instead.
threshold=300
max_gc=$(nodetool gcstats | tail -1 | awk '{print $2}')
if [ "$max_gc" -gt "$threshold" ]; then
    echo "WARNING: Max GC elapsed ${max_gc}ms exceeds ${threshold}ms"
fi

---

Best Practices

nodetool gcstats reports an interval, so a single call establishes nothing. Take a baseline over a period of representative load, keep the reading in continuous monitoring with history and alerting rather than calling the command by hand, and correlate pause durations against read and write latency before concluding that collection is the cause of a latency spike. GC logging is already enabled on 5.0, 4.1 and 4.0 and gives the per-collection detail this command summarises.

Heap size and collector are not changed in response to this output alone. JVM Tuning gives the shipped values per version, which settings are worth changing, and the measurement each change should be made against.

---

Related Commands

Command	Relationship
info	Heap memory usage
tpstats	Thread pool status
proxyhistograms	Latency histograms
tablestats	Table metrics