Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Cassandra Node Replacement

This section examines the architectural considerations for handling failed nodes in a Cassandra cluster. Understanding failure classification, recovery strategies, and the mechanics of node replacement is essential for designing resilient cluster topologies.

For operational procedures and step-by-step instructions, see Operations: Cluster Management.

Failure Classification

Section titled “Failure Classification”

Types of Node Failures

Section titled “Types of Node Failures”

Node failures fall into distinct categories, each requiring different recovery approaches:

Node Failure ClassificationNode Failure ClassificationNode FailureTransient Failure• Temporary outage• Restart recovers• Data intactPermanent Failure• Hardware failure• Data loss• Node unrecoverableRecovery: RestartHints replay automaticallyRecovery: ReplaceNew node takes tokens(new Host ID)Recovery: RemoveReduce cluster size
Failure TypeCharacteristicsRecovery Approach
TransientNode temporarily unavailable, data intactWait for restart, hints replay
RecoverableNode can restart, possible data issuesRestart + repair
PermanentHardware failed, node cannot returnReplace or remove

Failure Detection

Section titled “Failure Detection”

Cassandra detects node failures through the gossip protocol’s Phi Accrual Failure Detector. When a node stops responding to gossip messages, other nodes independently calculate a suspicion level (φ). Once φ exceeds the configured threshold (default: 8), the node is marked DOWN locally.

Key characteristics:

  • Local determination: Each node independently decides if a peer is DOWN
  • Not gossiped: DOWN status is not propagated; each node must observe failure directly
  • Adaptive: Phi Accrual adjusts to network latency variations

See Gossip Protocol for failure detection mechanics.

Decision Framework

Section titled “Decision Framework”

Replace vs Remove

Section titled “Replace vs Remove”

When a node permanently fails, the architectural decision between replacement and removal affects cluster capacity, data distribution, and recovery time:

Replace vs Remove DecisionReplace vs Remove DecisionNode Permanently FailedNeed to maintaincluster capacity?YesNo(reducing capacity)Have replacementhardware available?YesNo(hardware pending)REPLACENew node assumesfailed node's tokensREMOVE nowADD new node laterREMOVERedistribute datato remaining nodes

Comparison of Approaches

Section titled “Comparison of Approaches”
FactorReplaceRemove
Cluster capacityMaintainedReduced
Data movementStream to new node onlyRedistribute across all remaining nodes
Token ownershipNew node assumes dead node’s tokensTokens redistributed to existing nodes
Disk usage impactIsolated to new nodeIncreased on all remaining nodes
Time to completeLonger (full data stream to one node)Shorter (parallel redistribution)
Network impactConcentrated streamingDistributed streaming

Capacity Constraints

Section titled “Capacity Constraints”

Before choosing an approach, verify capacity constraints:

ConstraintReplaceRemove
Minimum nodesN ≥ RF after operationN - 1 ≥ RF after operation
Disk headroomNew node needs capacity for its shareRemaining nodes need capacity for redistributed data
Network bandwidthStreaming to single nodeParallel streaming between remaining nodes

Replacement Architecture

Section titled “Replacement Architecture”

Token Assumption Model

Section titled “Token Assumption Model”

Node replacement operates on the principle of token assumption—the new node takes ownership of the failed node’s token ranges without redistributing tokens across the cluster:

Token Assumption During ReplacementToken Assumption During ReplacementBefore FailureNode B FailedAfter ReplacementToken Ring Node A: tokens 0-33Node B: tokens 34-66Node C: tokens 67-100Token Ring Node A: tokens 0-33Node B: DOWNNode C: tokens 67-100Token Ring Node A: tokens 0-33Node D: tokens 34-66Node C: tokens 67-100failurereplace

Advantages of token assumption:

  • No token recalculation required
  • Other nodes unaffected
  • Predictable data movement
  • Faster recovery than full rebalance

Replacement Process Flow

Section titled “Replacement Process Flow”
Node Replacement ProcessNode Replacement ProcessIdentify Dead NodeObtain IP and Host IDProvision New NodeSame Cassandra versionConfigure ReplacementSet replace_address_first_bootNew Node StartsContacts seedsToken AssumptionClaims dead node's rangesData StreamingReceives from replicasNORMAL StateServes client trafficRepairEnsures consistencyPreparationExecutionCompletion

Replace Address Mechanism

Section titled “Replace Address Mechanism”

The replace_address_first_boot JVM option instructs a new node to assume a dead node’s identity:

OptionBehaviorRecommendation
replace_address_first_bootCleared after successful first bootPreferred—prevents accidental re-replacement
replace_addressPersists across restartsLegacy—can cause issues on restart

Architectural behavior:

  1. New node contacts seeds with replacement intent
  2. Cluster validates dead node is actually DOWN
  3. New node receives dead node’s token assignments
  4. Streaming begins from surviving replicas
  5. Upon completion, new node announces NORMAL status
  6. Dead node’s gossip state is eventually purged

Same-IP vs Different-IP Replacement

Section titled “Same-IP vs Different-IP Replacement”
ScenarioToken HandlingHost IDConfiguration
Same IPAssumed from dead nodeNew ID generatedreplace_address_first_boot with same IP
Different IPAssumed from dead nodeNew ID generatedreplace_address_first_boot with dead node’s IP

Both scenarios require the replace_address_first_boot option—the IP in the option always refers to the dead node’s address, regardless of the new node’s IP.

Removal Architecture

Section titled “Removal Architecture”

Token Redistribution Model

Section titled “Token Redistribution Model”

Node removal triggers token redistribution—the dead node’s token ranges are reassigned to remaining nodes:

Node Removal - Token RedistributionNode Removal - Token RedistributionDead Node IdentifiedHost ID knownRemoval InitiatedFrom any live nodeOwnership UpdateNode removed from ringData StreamingBetween remaining nodesRemoval CompleteNode purged from ring

Removal vs Decommission

Section titled “Removal vs Decommission”
Aspectremovenodedecommission
PreconditionNode is DOWNNode is UP and operational
Initiated fromAny live nodeThe departing node itself
Data sourceRemaining replicas stream to each otherDeparting node streams its data out
CoordinationDistributed among remaining nodesCentralized on departing node
Use caseUnplanned failurePlanned capacity reduction

Assassinate Operation

Section titled “Assassinate Operation”

The assassinate operation forcibly removes a node from gossip state without data redistribution:

CharacteristicDescription
PurposeEmergency removal of stuck nodes
Data handlingNone—no streaming occurs
RiskPotential data loss if replicas insufficient
Post-actionFull repair required on all nodes

Use only when:

  • Node is unresponsive but not fully DOWN
  • removenode fails or hangs indefinitely
  • Emergency cluster recovery is required

Multiple Failure Scenarios

Section titled “Multiple Failure Scenarios”

Concurrent Failure Analysis

Section titled “Concurrent Failure Analysis”

When multiple nodes fail, data availability depends on the relationship between failures and replication:

Multiple Failure Impact AnalysisMultiple Failure Impact AnalysisAssess Failure ScopeCount failed nodes per token rangeSurviving Replicas ≥ 1for all ranges?YesNoData RecoverableSequential replacement possibleReplace One at a TimeRepair between eachData At RiskBackup restoration may be requiredRestore from BackupThen rebuild cluster

Quorum Impact Matrix

Section titled “Quorum Impact Matrix”
Failed NodesRF=3 AvailabilityRecovery Strategy
1QUORUM available (2 of 3)Standard replacement
2ONE available onlyUrgent replacement, sequential
3 (same range)Data unavailableBackup restoration required

Recovery Ordering

Section titled “Recovery Ordering”

When multiple nodes require replacement:

  1. Assess scope: Identify which token ranges are affected
  2. Prioritize: Replace nodes affecting most-critical ranges first
  3. Sequential execution: Complete one replacement before starting next
  4. Repair between: Run repair after each replacement to ensure consistency
  5. Verify coverage: Confirm all ranges have sufficient replicas before proceeding

Data Consistency Considerations

Section titled “Data Consistency Considerations”

Streaming During Replacement

Section titled “Streaming During Replacement”

During replacement, the new node receives data from surviving replicas:

Source SelectionCriteria
Prefer local DCMinimize cross-datacenter traffic
Replica availabilitySelect from nodes holding required data
Avoid concurrent streamersPrevent resource contention

Post-Replacement Consistency

Section titled “Post-Replacement Consistency”

After replacement completes, data consistency may require repair:

ScenarioRepair Recommendation
Short downtime (< hint window)Hints cover gap; repair optional
Extended downtime (> hint window)Repair required for missed writes
Multiple failuresFull repair recommended
Consistency-critical dataAlways run repair

The hint window (default: 3 hours) determines whether hinted handoff can cover writes during downtime. Beyond this window, hints expire and repair becomes necessary.

Section titled “Related Documentation”