nodetool resumehandoff

Resumes the delivery of hints that was previously paused.

---

Synopsis

nodetool [connection_options] resumehandoff

See connection options for connection options.

Description

nodetool resumehandoff resumes the delivery of stored hints to their target nodes after it was paused with pausehandoff. When resumed, the node begins replaying accumulated hints to recovered replicas, restoring data consistency.

This command only affects hint delivery—it does not change whether new hints are being stored (controlled by enablehandoff/disablehandoff).

Resume After Pause

Use this command after pausehandoff to restore normal hint delivery behavior. If hints were disabled with disablehandoff, use enablehandoff instead.

---

Behavior

When hint delivery is resumed:

1. Hint delivery threads become active
2. Accumulated hints begin replaying to target nodes
3. Recovered replicas receive their missing writes
4. Consistency is progressively restored

Delivery Process

Resume Hint Delivery:

1. resumehandoff command received
2. Hint delivery threads activated
3. For each target node:
   - Check if node is UP via gossip
   - Begin streaming hints to node
   - Mark delivered hints for deletion
4. Consistency restored without repair

---

Examples

Basic Usage

nodetool resumehandoff

Resume and Monitor

nodetool resumehandoff

# Watch hint delivery progress
watch -n 2 'nodetool tpstats | grep -i hint'

---

When to Use

After Temporary Pause

Resume normal operation after a maintenance pause:

# Maintenance complete, resume delivery
nodetool resumehandoff

# Verify delivery is active
nodetool tpstats | grep -i hint

After Node Stabilization

When a recovering node is ready to receive hints:

# Node has stabilized after recovery
nodetool resumehandoff

# Monitor hint delivery
nodetool listpendinghints

After Troubleshooting

When hint-related investigation is complete:

# Issue resolved, resume delivery
nodetool resumehandoff

# Check for pending hints
nodetool listpendinghints

During Maintenance Window

Resume hint delivery during off-peak hours:

# Off-peak hours, safe to resume
nodetool resumehandoff

# Monitor progress until hints are delivered
watch 'nodetool listpendinghints'

---

Impact Assessment

Immediate Effects

Aspect	Impact
Hint delivery threads	Activate immediately
Network traffic	Increases (hint streaming)
Target node load	Increases (processing hints)
Coordinator disk	Begins freeing (delivered hints removed)

Short-term Effects

Aspect	Impact
Pending hint count	Decreases progressively
Data consistency	Improves as hints delivered
Hints table size	Shrinks as hints are delivered
Read performance	May improve on target nodes

Gradual Delivery

Hints are delivered gradually based on hinted_handoff_throttle to prevent overwhelming target nodes.

---

Monitoring After Resume

Track Hint Delivery

# Watch hint delivery progress
watch -n 5 'nodetool listpendinghints'

Monitor Thread Activity

# Check hint delivery threads
nodetool tpstats | grep -i hint

Example output when delivering:

Pool Name                         Active   Pending      Completed   Blocked  All time blocked
HintedHandoff                          2         0           1523         0                 0

Track Hints Directory Size

# Monitor hints directory shrinking (hints stored as files)
watch -n 30 'du -sh /var/lib/cassandra/hints/'

---

Workflow: Complete Pause/Resume Cycle

hint_pause_resume_cycle.sh

#!/bin/bash
echo "=== Hint Delivery Pause/Resume Cycle ==="

# 1. Initial status
echo "1. Initial Status:"
echo "Pending hints:"
nodetool listpendinghints
echo ""
echo "Hint threads:"
nodetool tpstats | grep -i hint

# 2. Pause delivery
echo ""
echo "2. Pausing hint delivery..."
nodetool pausehandoff

# 3. Perform maintenance or wait
echo ""
echo "3. Hint delivery paused. Perform maintenance or press Enter to resume."
read -p "Press Enter to continue..."

# 4. Resume delivery
echo ""
echo "4. Resuming hint delivery..."
nodetool resumehandoff

# 5. Monitor delivery progress
echo ""
echo "5. Monitoring delivery (Ctrl+C to exit):"
while true; do
    pending=$(nodetool listpendinghints 2>/dev/null | tail -n +2 | awk '{sum+=$2} END {print sum+0}')
    echo "$(date '+%H:%M:%S') - Pending hints: $pending"

    if [ "$pending" -eq 0 ]; then
        echo "All hints delivered!"
        break
    fi
    sleep 5
done

echo ""
echo "=== Cycle Complete ==="

---

Throttling Hint Delivery

Default Throttling

Hint delivery is throttled by default to prevent overwhelming target nodes. Parameter names vary by version:

Cassandra Version	Throttle Parameter	Example
Pre-4.1	hinted_handoff_throttle_in_kb	1024
4.1+	hinted_handoff_throttle	1024KiB

# cassandra.yaml (4.1+)
hinted_handoff_throttle: 1024KiB
max_hints_delivery_threads: 2

# cassandra.yaml (Pre-4.1)
# hinted_handoff_throttle_in_kb: 1024
# max_hints_delivery_threads: 2

Adjust During Delivery

# If delivery is too slow and nodes can handle more
nodetool sethintedhandoffthrottlekb 2048

# If target nodes are struggling
nodetool sethintedhandoffthrottlekb 512

Monitor Delivery Rate

#!/bin/bash
# Monitor hint delivery rate

prev_completed=0
while true; do
    completed=$(nodetool tpstats 2>/dev/null | grep "HintedHandoff" | awk '{print $4}')
    if [ -n "$prev_completed" ] && [ "$prev_completed" -gt 0 ]; then
        rate=$((completed - prev_completed))
        echo "$(date '+%H:%M:%S') - Hints delivered in last interval: $rate"
    fi
    prev_completed=$completed
    sleep 10
done

---

Cluster-Wide Operations

Resume on All Nodes

resume_handoff_cluster.sh

#!/bin/bash
echo "Resuming hint delivery cluster-wide..."

# Get list of node IPs from local nodetool status
nodes=$(nodetool status | grep "^UN" | awk '{print $2}')

for node in $nodes; do
    echo -n "$node: "
    ssh "$node" 'nodetool resumehandoff 2>/dev/null && echo "resumed" || echo "FAILED"'
done

echo ""
echo "Verification (hint thread activity):"
for node in $nodes; do
    echo "=== $node ==="
    ssh "$node" "nodetool tpstats 2>/dev/null | grep -i hint"
done

Monitor Cluster-Wide Delivery

monitor_cluster_hint_delivery.sh

#!/bin/bash
echo "=== Cluster Hint Delivery Status ==="

# Get list of node IPs from local nodetool status
nodes=$(nodetool status | grep "^UN" | awk '{print $2}')

total_pending=0
for node in $nodes; do
    pending=$(ssh "$node" "nodetool listpendinghints 2>/dev/null | tail -n +2 | awk '{sum+=$2} END {print sum+0}')"
    echo "$node: $pending pending hints"
    total_pending=$((total_pending + pending))
done

echo ""
echo "Total cluster pending hints: $total_pending"

---

Troubleshooting

Hints Not Being Delivered After Resume

# Check if target nodes are UP
nodetool status

# Check hint thread activity
nodetool tpstats | grep -i hint

# Check for errors in logs
grep -i "hint" /var/log/cassandra/system.log | tail -20

Slow Hint Delivery

# Check current throttle setting
nodetool gethintedhandoffthrottlekb

# Increase if nodes can handle more
nodetool sethintedhandoffthrottlekb 2048

# Check hint delivery threads
nodetool tpstats | grep -i hint

Hint Delivery Errors

# Check logs for hint delivery failures
grep -i "hint" /var/log/cassandra/system.log | grep -i "error\|fail" | tail -20

# Check target node connectivity
nodetool ring | head -20

# Verify target node is responsive
ssh <target_node> "nodetool info"

Hints Expired During Pause

If hints expired while delivery was paused:

# Check pending hints (may show 0 if expired)
nodetool listpendinghints

# Run repair to restore consistency
nodetool repair -pr

---

Recovery After Extended Pause

If hints were paused for an extended period:

recovery_after_extended_pause.sh

#!/bin/bash
echo "=== Recovery After Extended Hint Pause ==="

# 1. Resume hint delivery
echo "1. Resuming hint delivery..."
nodetool resumehandoff

# 2. Check for pending hints
echo ""
echo "2. Pending hints:"
nodetool listpendinghints

pending=$(nodetool listpendinghints 2>/dev/null | tail -n +2 | awk '{sum+=$2} END {print sum+0}')

if [ "$pending" -eq 0 ]; then
    echo ""
    echo "WARNING: No pending hints found."
    echo "If hints were paused longer than the hint window, they may have expired."
    echo ""
    echo "3. Running repair to restore consistency..."
    nodetool repair -pr
else
    echo ""
    echo "3. $pending hints pending, monitoring delivery..."
    # Monitor until complete
    while [ "$pending" -gt 0 ]; do
        sleep 10
        pending=$(nodetool listpendinghints 2>/dev/null | tail -n +2 | awk '{sum+=$2} END {print sum+0}')
        echo "Pending: $pending"
    done
    echo "All hints delivered!"
fi

echo ""
echo "=== Recovery Complete ==="

---

Best Practices

Resume Guidelines

1. Resume promptly - Don't leave hints paused beyond the hint window
2. Monitor after resume - Watch hint delivery progress
3. Check throttling - Adjust if delivery is too slow or fast
4. Verify target nodes - Ensure target nodes are UP before resuming
5. Plan for expired hints - Run repair if hints may have expired
6. Resume cluster-wide - If paused cluster-wide, resume on all nodes

After Extended Pause

If hints were paused longer than max_hint_window (default 3 hours), some hints may have expired. Run nodetool repair -pr after resuming to ensure full consistency.

---

Related Commands

Command	Relationship
pausehandoff	Pause hint delivery
enablehandoff	Enable hint storage
disablehandoff	Disable hint storage and delivery
statushandoff	Check handoff status
listpendinghints	List pending hints
truncatehints	Remove all hints
sethintedhandoffthrottlekb	Control delivery throttle