Hinted Handoff

Hinted handoff handles writes to temporarily unavailable replicas by storing the write locally on the coordinator and replaying it when the replica recovers. This allows writes to succeed at the requested consistency level even when a subset of replicas is unreachable, without requiring the replica to catch up via repair.

How It Works

The coordinator stores a hint — the mutation plus target endpoint — in a local hints file (durable via the commit log). When the target node is detected as available again, the coordinator delivers accumulated hints in order, throttled to avoid overwhelming the recovering node.

Configuration

# Enable/disable hinted handoff
hinted_handoff_enabled: true

# How long to store hints (default 3 hours)
# Hints older than this are dropped
max_hint_window: 3h                    # 4.1+ (duration format)
# max_hint_window_in_ms: 10800000      # Pre-4.1

# Throttle hint delivery to avoid overwhelming recovering nodes
hinted_handoff_throttle: 1024KiB       # 4.1+ (data size format)
# hinted_handoff_throttle_in_kb: 1024  # Pre-4.1

# Maximum delivery threads
max_hints_delivery_threads: 2

# Hint storage directory
hints_directory: /var/lib/cassandra/hints

Parameter	Pre-4.1	4.1+
Hint window	`max_hint_window_in_ms` (milliseconds)	`max_hint_window` (duration: `3h`, `30m`)
Delivery throttle	`hinted_handoff_throttle_in_kb` (KB/s)	`hinted_handoff_throttle` (data size: `1024KiB`)

Hint Window

The hint window (max_hint_window in 4.1+, max_hint_window_in_ms in earlier versions) defines how long the coordinator accumulates hints for an unavailable replica. Once the window expires, new writes to that key no longer generate hints.

Node down for 2 hours (window = 3 hours):
- Hints accumulated for 2 hours
- Node recovers
- All hints replayed ✓

Node down for 5 hours (window = 3 hours):
- Hints accumulated for first 3 hours
- After 3 hours, new writes stop generating hints
- Node recovers
- Only 3 hours of hints replayed
- 2 hours of writes are missing ✗
→ Run repair to recover missing data

When a node has been down longer than the hint window, hinted handoff alone is insufficient. A full or incremental repair must be run after the node rejoins. See Repair Architecture.

Hints and Consistency Level ANY

Consistency Level ANY

ANY counts a stored hint as an acknowledgment, meaning a write can succeed even when all target replicas are unreachable.

The hint is written only to the coordinator's commit log — it is not replicated.
If the coordinator is permanently lost before hint delivery, the data is lost.
ANY should not be used for data where durability is required.

Monitoring

# Check hint accumulation
nodetool tpstats | grep Hint

# View hint files
ls -la /var/lib/cassandra/hints/

# Hinted handoff control
nodetool disablehandoff   # Disable hinted handoff
nodetool enablehandoff    # Enable hinted handoff
nodetool pausehandoff     # Pause hint delivery
nodetool resumehandoff    # Resume hint delivery

JMX metrics:

org.apache.cassandra.metrics:type=Storage,name=TotalHints
org.apache.cassandra.metrics:type=HintsService,name=HintsSucceeded
org.apache.cassandra.metrics:type=HintsService,name=HintsFailed
org.apache.cassandra.metrics:type=HintsService,name=HintsTimedOut

Troubleshooting: Hints Accumulating

# Check hint backlog size
ls -la /var/lib/cassandra/hints/

# Verify target node is reachable
nodetool status

# Check hint delivery threads are not saturated
nodetool tpstats | grep Hint

If hints are not draining after a node recovers, verify that hinted_handoff_enabled is true and that the delivery thread count (max_hints_delivery_threads) is sufficient for the write rate. Large backlogs that persist after node recovery may indicate the target is still unhealthy or the throttle is set too low.

Limitations

Limitation	Implication
Hints expire	Long outages require repair
Hints use disk	Large hint backlog consumes storage
Single copy	Coordinator failure loses hints
Not for schema	DDL changes do not use hints

Best Practices

Practice	Rationale
Keep enabled	Handles transient failures automatically
Set `max_hint_window` to match expected outage duration	Prevents silent data gaps on recovery
Monitor hint accumulation	Large backlogs indicate sustained availability problems
Do not rely solely on hints	Scheduled repair is still required for complete convergence

Replica Synchronization - Overview of all convergence mechanisms
Read Repair - Divergence detection during read operations
Repair Architecture - Merkle trees, repair modes, gc_grace_seconds, and scheduling
Repair Operations - Operational procedures for repair