Cassandra Log Analysis

Guide to analyzing Cassandra logs for troubleshooting.

Log Locations

/var/log/cassandra/
├── system.log      # Main application log
├── debug.log       # Debug-level logging
├── gc.log          # Garbage collection
└── audit/          # Audit logs (if enabled)

Key Patterns to Search

Errors and Exceptions

# All errors
grep -i "error\|exception" /var/log/cassandra/system.log | tail -50

# Stack traces
grep -A 20 "Exception" /var/log/cassandra/system.log

# Specific errors
grep -i "ReadTimeoutException" /var/log/cassandra/system.log
grep -i "WriteTimeoutException" /var/log/cassandra/system.log
grep -i "UnavailableException" /var/log/cassandra/system.log

Performance Issues

# Slow queries
grep -i "slow" /var/log/cassandra/system.log

# Tombstone warnings
grep -i "tombstone" /var/log/cassandra/system.log

# Large partitions
grep -i "large partition\|large row" /var/log/cassandra/system.log

# Compaction
grep -i "compaction\|compacted" /var/log/cassandra/system.log

Cluster Events

# Node status changes
grep -i "is now\|state jump\|marking" /var/log/cassandra/system.log

# Gossip events
grep -i "gossip" /var/log/cassandra/system.log

# Streaming
grep -i "streaming\|stream" /var/log/cassandra/system.log

GC Log Analysis

# Long pauses
grep -E "pause.*[0-9]{3,}ms" /var/log/cassandra/gc.log

# Full GC events
grep -i "full gc\|to-space" /var/log/cassandra/gc.log

# Heap after GC
grep -E "Heap:.* -> " /var/log/cassandra/gc.log

Common Log Messages

Normal Operations

INFO  - Starting listening for CQL clients
INFO  - Completed loading
INFO  - Starting compaction
INFO  - Compacted to [sstable]

Warning Signs

WARN  - Dropping MUTATION message
WARN  - Detected GC pause of Xms
WARN  - Large partition in sstable
WARN  - Tombstones scanned X for query

Critical Issues

ERROR - Exception in thread
ERROR - Error writing to channel
FATAL - Cannot start Cassandra

Log Aggregation with AxonOps

AxonOps provides centralized log collection and analysis for Cassandra clusters, eliminating the need to SSH into individual nodes or maintain separate logging infrastructure.

Key Features

Feature	Description
Centralized collection	Logs from all nodes aggregated in one interface
Real-time streaming	View logs as they occur across the cluster
Pattern detection	Automatic identification of error patterns
Metric correlation	Link log events to performance metrics
Search and filter	Full-text search with time-based filtering
Retention management	Configurable log retention policies

Getting Started

1. Enable log collection - Configure the axon-agent to collect Cassandra logs
2. View logs - Access centralized logs from the AxonOps dashboard
3. Set up alerts - Configure alerts for specific log patterns (errors, warnings)

See Setup Log Collection for configuration details and Logs & Events for usage.

---

Next Steps

• Diagnosis Guide - Systematic diagnosis
• Common Errors - Error reference
• Monitoring - Proactive monitoring