Cassandra Troubleshooting Guide

Most Cassandra problems resolve to a small number of root causes:

• High read latency is most often caused by tombstones or large partitions rather than CPU.
• Out-of-memory errors usually come from a partition that grew too large for the heap during compaction rather than insufficient RAM.
• Timeouts often indicate compaction falling behind.

The diagnosis process is consistent: check the logs for warnings, review the metrics (p99 latency, pending compactions, and thread pool stats), and use nodetool to inspect specific tables.

This guide provides procedures for diagnosing and resolving common issues. The step-by-step runbooks are in the troubleshooting playbooks; the sibling sections cover systematic diagnosis, log analysis, and the common errors reference.

Troubleshooting Framework

The SDRR framework structures problem resolution:

1. Symptoms: What observable behaviors indicate the problem?
2. Diagnosis: How to identify the root cause
3. Resolution: Step-by-step fix procedures
4. Recovery: Verification and prevention

---

Quick Reference: Common Issues

Performance Issues

Symptom	Likely Cause	Quick Action
High read latency	Data model, tombstones	Check nodetool tablestats
High write latency	Disk I/O, compaction	Check nodetool compactionstats
Request timeouts	Overload, disk issues	Check nodetool tpstats
Slow startup	Large commitlog	Check commitlog size

Playbooks: GC Pause Issues, Large Partition Issues, Tombstone Accumulation, Compaction Issues.

Availability Issues

Symptom	Likely Cause	Quick Action
Node down	OOM, disk full, crash	Check logs, systemctl status
Node marked down (DN)	Network, gossip, GC pause	Check nodetool gossipinfo, port connectivity
Nodes not joining	Network, schema	Check gossip, firewalls
Inconsistent data	Repair needed	Run nodetool repair

Playbooks: Node Fails to Start, Recover from OOM, Gossip Failures, Connectivity Issues, Repair Failures.

Resource Issues

Symptom	Likely Cause	Quick Action
High CPU	GC, compaction	Check GC logs, compaction
High memory	Heap settings	Review JVM configuration
Disk full	Large tables, snapshots	Clean snapshots, add capacity

Playbooks: High CPU Usage, High Memory Usage, Recover from OOM, Handle Full Disk.

---

First Response Checklist

When an issue occurs, gather this information first:

1. Check Node Status

# Cluster overview
nodetool status

# Node info
nodetool info

# Thread pool stats
nodetool tpstats

# Compaction status
nodetool compactionstats

2. Check Logs

# Recent errors
tail -100 /var/log/cassandra/system.log | grep -i error

# Warnings
tail -100 /var/log/cassandra/system.log | grep -i warn

# GC issues
grep "GC pause" /var/log/cassandra/gc.log | tail -20

3. Check Resources

# Disk space
df -h /var/lib/cassandra

# Memory
free -h

# CPU
top -b -n 1 | head -20

# I/O
iostat -x 1 5

4. Check Metrics

# Table stats for specific keyspace
nodetool tablestats my_keyspace

# Pending compactions
nodetool compactionstats | grep pending

# Dropped messages
nodetool tpstats | sed -n '/Message type/,$p'

Dropped-message counts are cumulative since the node last started, so a single reading does not show whether messages are being dropped now. Sample twice about 60 seconds apart and compare; only a rising delta between the two samples indicates active drops.

---

Diagnostic Commands Reference

nodetool Commands

Command	Purpose
nodetool status	Cluster and node health
nodetool info	Node information
nodetool tpstats	Thread pool statistics
nodetool tablestats <ks>	Table statistics
nodetool compactionstats	Compaction status
nodetool netstats	Network status
nodetool gossipinfo	Gossip state
nodetool describecluster	Cluster description
nodetool proxyhistograms	Request latencies
nodetool tablehistograms <ks> <table>	Table latencies

Log Locations

Log	Location	Purpose
System log	/var/log/cassandra/system.log	Main application log
Debug log	/var/log/cassandra/debug.log	Detailed debug info
GC log	/var/log/cassandra/gc.log	Garbage collection
Audit log	/var/log/cassandra/audit/audit.log	Security audit

Key Metrics to Check

# JMX metrics via nodetool
nodetool gcstats                    # GC statistics
nodetool getlogginglevels           # Current log levels
nodetool statusbinary               # CQL port status
nodetool statusgossip               # Gossip status

---

Issue Categories

Timeout Issues

Requests failing due to time limits. Defaults vary by version and can be overridden in cassandra.yaml, so confirm the value in use against the running node’s cassandra.yaml rather than relying on the figures below. Cassandra 5.0 and 4.1 use unsuffixed duration keys with unit-bearing values (CASSANDRA-15234), for example read_request_timeout: 5000ms; Cassandra 4.0 uses the _in_ms-suffixed name with a plain-number value, read_request_timeout_in_ms: 5000.

Error	Default	Configuration key (5.0 / 4.1)	Configuration key (4.0)
Read timeout	5000ms	read_request_timeout	read_request_timeout_in_ms
Write timeout	2000ms	write_request_timeout	write_request_timeout_in_ms
Range timeout	10000ms	range_request_timeout	range_request_timeout_in_ms
Counter write timeout	5000ms	counter_write_request_timeout	counter_write_request_timeout_in_ms
Truncate timeout	60000ms	truncate_request_timeout	truncate_request_timeout_in_ms

Common causes: Overload, slow disks, network issues, data model problems.

Consistency Issues

Data inconsistency between replicas:

Symptom	Cause	Action
Different data per query	Repair needed	Run repair
Unavailable errors	Insufficient replicas	Check node status
Timeout with QUORUM	Multiple nodes slow	Check all replicas

Compaction Issues

Compaction falling behind:

Symptom	Cause	Action
High pending compactions	Write-heavy, slow disk	Increase throughput
Large SSTable files	No compaction	Check strategy
High read latency	Many SSTables	Force compaction

Memory Issues

Heap exhaustion or GC pressure:

Symptom	Cause	Action
OOM errors	Heap too small, large partitions	Increase heap, fix data model
Long GC pauses	Heap too large	Reduce heap size
Off-heap OOM	Bloom filters, compression	Adjust off-heap settings

---

Emergency Procedures

Node Unresponsive

# 1. Check if process is running
ps aux | grep cassandra

# 2. Check for OOM kills
dmesg | grep -i "killed process"

# 3. Check system log
tail -100 /var/log/cassandra/system.log

# 4. If hung, get thread dump
jstack $(pgrep -f CassandraDaemon) > /tmp/threaddump.txt

# 5. If necessary, restart
sudo systemctl restart cassandra

Disk Full

# 1. Check disk usage
df -h /var/lib/cassandra

# 2. Find large files
du -sh /var/lib/cassandra/*

# 3. Clear snapshots
nodetool clearsnapshot --all

# 4. If still full, consider temporary cleanup
ls -la /var/lib/cassandra/data/<keyspace>/<table>*/

Cluster Partition

# 1. Check gossip state on both sides
nodetool gossipinfo

# 2. Verify network connectivity
nc -zv <other-node-ip> 7000
nc -zv <other-node-ip> 9042

# 3. Check firewall rules
sudo iptables -L -n

# 4. If network is fine, check for zombie nodes
nodetool status | grep -E "DN|UJ|UL"

---

Getting Help

Information to Collect

When seeking help, gather:

1. Cassandra version: nodetool version
2. Cluster configuration: nodetool describecluster
3. Node status: nodetool status
4. Recent logs: Last 100 lines of system.log
5. Error messages: Exact exception text
6. Recent changes: What changed before the issue

Resources

• Apache Cassandra Slack - Community chat
• AxonOps Community - Professional support
• Stack Overflow - Q&A

---

Next Steps

• Monitoring Guide - Proactive monitoring
• Performance Tuning - Performance optimization