Cassandra Decommission Node

Decommissioning removes a node from the cluster by streaming its data to remaining nodes. This is the proper way to permanently remove a healthy node.

---

Prerequisites

• Node is UP and operational
• Cluster has sufficient capacity after removal
• No other operations (repair, bootstrap) running
• Recent backup available

When NOT to Use Decommission

• If node is DOWN: Use nodetool removenode instead
• If node is unresponsive: Use nodetool assassinate (last resort)
• If removing multiple nodes: Decommission one at a time

---

Pre-Decommission Checklist

Step 1: Verify Cluster Health

nodetool status

All other nodes should show UN (Up/Normal).

Step 2: Check Capacity After Removal

# Current data per node
nodetool status | awk '/^UN/ {print $1, $3}'

# Verify remaining nodes can absorb data
# Rule: Each remaining node should have < 70% disk after decommission

Step 3: Run Repair First (Optional but Recommended)

# Ensures data is consistent before streaming
nodetool repair -pr

Step 4: Disable Auto-Bootstrap (Precaution)

Ensure auto_bootstrap: false won't be an issue if node accidentally restarts.

Step 5: Notify Stakeholders

Decommission can take hours and impacts cluster performance during streaming.

---

Decommission Procedure

Step 1: Start Decommission

On the node to be removed:

nodetool decommission

This command:

1. Stops accepting new writes
2. Streams data to other replicas
3. Leaves the token ring
4. Shuts down (in some versions)

Step 2: Monitor Progress

In another terminal:

# Watch streaming progress
nodetool netstats

# Watch decommission status
nodetool status | grep -E "UL|UJ|UN"
# UL = Leaving, UJ = Joining, UN = Normal

Step 3: Wait for Completion

Decommission can take hours depending on data size. Do not interrupt.

# Monitor from another node
watch -n 60 'nodetool status'

Step 4: Verify Completion

From another node:

nodetool status

The decommissioned node should no longer appear.

Step 5: Clean Up (After Decommission)

On the decommissioned node (if still running):

# Stop Cassandra if still running
sudo systemctl stop cassandra

# Optionally remove data
sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*
sudo rm -rf /var/lib/cassandra/saved_caches/*

---

Troubleshooting

Decommission Seems Stuck

# Check netstats for streaming activity
nodetool netstats

# Check logs
tail -100 /var/log/cassandra/system.log | grep -i "stream\|decommission"

If truly stuck (no progress for hours):

1. Check disk space on receiving nodes
2. Check network connectivity
3. Check for errors in logs

Decommission Failed Mid-Way

# Check node status
nodetool status

# If node shows UL (Leaving) but decommission failed:
# Option 1: Restart and retry
sudo systemctl restart cassandra
nodetool decommission

# Option 2: Cancel and diagnose
nodetool netstats  # Check for issues

Node Reappears After Decommission

The node's data directories may not have been cleared:

# On the decommissioned node, clear all data
sudo systemctl stop cassandra
sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*

Not Enough Space on Remaining Nodes

# Cancel decommission if possible (Ctrl+C usually works)
# Then:
# 1. Add more nodes first, OR
# 2. Clean up snapshots on remaining nodes
nodetool clearsnapshot --all
# 3. Add disk capacity

---

Streaming Performance

Monitor Streaming

# Detailed streaming info
nodetool netstats

# Streaming throughput
nodetool getstreamthroughput

Speed Up Streaming (If Needed)

# Increase throughput (value in Mb/s by default)
nodetool setstreamthroughput 400

# Or specify MiB/s explicitly with -m flag
nodetool setstreamthroughput -m 50  # 50 MiB/s

# On receiving nodes, adjust concurrent compactors
nodetool setconcurrentcompactors 2

Streaming Throughput Units

nodetool setstreamthroughput uses megabits per second (Mb/s) by default. Use the -m flag to specify MiB/s instead.

---

Recovery If Interrupted

If decommission is interrupted:

Node Still in Cluster (UL Status)

# Restart Cassandra
sudo systemctl restart cassandra

# Retry decommission
nodetool decommission

Node Removed but Data Remains

# Clear data and repurpose hardware
sudo systemctl stop cassandra
sudo rm -rf /var/lib/cassandra/*

# Or re-add as new node (requires different IP or cleared system tables)

---

Best Practices

Practice	Reason
Decommission one node at a time	Prevents overload
Run repair before decommission	Ensures data consistency
Monitor during process	Catch issues early
Have backup ready	Recovery option
Plan maintenance window	Performance impact
Verify capacity first	Prevent disk full

---

Time Estimation

Data per Node	Expected Duration
< 100 GB	1-2 hours
100-500 GB	2-8 hours
500 GB - 1 TB	8-24 hours
> 1 TB	24+ hours

Duration depends on:

• Network bandwidth
• Disk I/O speed
• Stream throughput settings
• Concurrent operations

---

Related Procedures

Scenario	Procedure
Node is DOWN	Replace Dead Node
Adding capacity	Add Node
Node unresponsive	Use nodetool removenode or assassinate

Related Commands

Command	Purpose
nodetool decommission	Remove healthy node
nodetool removenode	Remove dead node
nodetool netstats	Monitor streaming
nodetool setstreamthroughput	Adjust streaming speed
nodetool status	Verify cluster state