Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Cassandra Multi-Datacenter Operations

This guide covers procedures for adding and removing datacenters in a Cassandra cluster.


Expanding to a new datacenter provides geographic redundancy and reduced latency for regional users.

RequirementVerification
Existing cluster healthyAll nodes UN
Network connectivityNew DC can reach existing DCs
Cross-DC latency acceptable< 100ms recommended
Same Cassandra versionMatch existing cluster
Hardware provisionedNodes ready in new DC

Replication strategy:

The cluster must use NetworkTopologyStrategy for multi-DC deployments:

-- Check current replication
DESCRIBE KEYSPACE my_keyspace;
-- Must be NetworkTopologyStrategy, not SimpleStrategy

Node count:

  • New datacenter should have at least RF nodes
  • Typically match node count of existing DC

Network requirements:

PortPurpose
7000Internode (gossip, streaming)
7001Internode SSL
9042Client connections

Step 1: Prepare keyspace replication (existing datacenter only)

Before adding any nodes, ensure every keyspace uses NetworkTopologyStrategy scoped to the existing datacenter only. Convert from SimpleStrategy if needed. Do not add the new datacenter yet — replication is extended only after the new nodes have joined (Step 4), so the new datacenter never receives traffic before it can serve it.

-- User keyspaces: convert SimpleStrategy to NetworkTopologyStrategy (existing DC only)
ALTER KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
-- System keyspaces
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_distributed WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};

New datacenter added later

The new datacenter is intentionally left out of the replication map until the nodes have joined the ring. It is added in Step 4.

Pin clients to the existing datacenter

Before the new nodes join gossip, pin application drivers to the existing datacenter so they never coordinate through the empty new-DC nodes:

// Driver 4.x
CqlSession.builder()
.withLocalDatacenter("dc1")
.build();
// Legacy driver 3.x
Cluster.builder()
.withLoadBalancingPolicy(
DCAwareRoundRobinPolicy.builder().withLocalDc("dc1").build())
.build();

Pair this with LOCAL_ONE or LOCAL_QUORUM consistency — not ONE or QUORUM, which can span both data centers — so reads and writes stay within the pinned datacenter.

Step 2: Configure nodes in new datacenter

On each new node:

cassandra.yaml
cluster_name: 'ProductionCluster' # Must match
num_tokens: 16 # Match existing cluster value (default is 16 for 4.0+)
# Seeds from BOTH datacenters
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "dc1-node1,dc1-node2,dc2-node1"
# This node's address
listen_address: <node_ip>
rpc_address: <node_ip>
# Snitch for multi-DC
endpoint_snitch: GossipingPropertyFileSnitch
# Join the ring without streaming data; populated later via rebuild
auto_bootstrap: false
cassandra-rackdc.properties
dc=dc2
rack=rack1

Step 3: Start nodes in new datacenter

Add nodes roughly 2 minutes apart so gossip settles between joins, and do NOT wait for bootstrap:

Terminal window
# New DC nodes start with auto_bootstrap: false
# They join the ring but receive no data yet
# Start first node
sudo systemctl start cassandra
# Verify it joins (shows UN but with 0 load)
nodetool status
# Start remaining nodes, ~2 minutes apart

Bootstrap Disabled

For new datacenter nodes, set auto_bootstrap: false. Data will be populated via rebuild, not bootstrap.

Step 4: Extend replication to the new datacenter

Now that all new nodes have joined and report UN, extend NetworkTopologyStrategy so it also covers the new datacenter. New writes begin replicating to it immediately.

-- User keyspaces
ALTER KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3,
'dc2': 3 -- New datacenter
};
-- System keyspaces (critical!)
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3,
'dc2': 3
};
ALTER KEYSPACE system_distributed WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3,
'dc2': 3
};
ALTER KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3,
'dc2': 3
};

Include system_auth

Include system_auth when extending replication, and rebuild it (Step 5) so authentication works locally in the new datacenter.

Step 5: Rebuild data in new datacenter

On each node in the new datacenter, run rebuild to back-fill the data that bootstrap skipped:

Terminal window
# Rebuild from the existing (source) datacenter
nodetool rebuild dc1
# This streams all data for this node's token ranges from dc1

Start with one node, then add parallel rebuilds while client latency stays stable, backing off as soon as it climbs. This paces streaming against live load instead of overwhelming the source datacenter.

Terminal window
# Monitor rebuild progress
nodetool netstats
# Watch for completion
tail -f /var/log/cassandra/system.log | grep -i rebuild

Step 6: Verify completion

Terminal window
# All nodes UN with data
nodetool status
# Example output:
# Datacenter: dc1
# UN 10.0.1.1 245.5 GB 16 ...
# UN 10.0.1.2 238.2 GB 16 ...
# UN 10.0.1.3 251.8 GB 16 ...
#
# Datacenter: dc2
# UN 10.0.2.1 243.1 GB 16 ... <-- Data present
# UN 10.0.2.2 240.7 GB 16 ...
# UN 10.0.2.3 248.3 GB 16 ...

Step 7: Redirect client traffic to the new datacenter

Once the new datacenter is fully rebuilt and consistent, repoint application drivers to prefer it. Switch the local datacenter from the old DC to the new DC and roll the applications:

// Driver 4.x
CqlSession.builder()
.addContactPoint(new InetSocketAddress("dc2-node1", 9042))
.withLocalDatacenter("dc2") // was dc1
.build();

Continue to use LOCAL_ONE or LOCAL_QUORUM consistency — not ONE or QUORUM, which can span both data centers — so reads and writes stay within the new datacenter.

Repoint only after rebuild completes

Do not switch clients to the new datacenter until rebuild (Step 5) has finished and consistency is verified (Step 6). Repointing earlier serves reads from an unbuilt datacenter.

Data to RebuildPer Node
100 GB1-2 hours
500 GB4-8 hours
1 TB8-16 hours

Total time ≈ per-node time × number of new-DC nodes when rebuilding sequentially. Running rebuilds in parallel (Step 5) reduces wall-clock time as live load allows.


Datacenter removal takes two forms. A planned removal consolidates the cluster while the datacenter is still running and can be shut down in a controlled order. Recovery from an unplanned loss applies when the datacenter is already gone and must be removed from the cluster once the loss is confirmed.

Two methods are used for planned removal:

  • Method 1, decommission each node. Each node is removed from the ring one at a time while it is still running. This fits smaller datacenters where nodes leaving the ring gracefully one at a time is acceptable and there is no time pressure.
  • Method 2, stop the datacenter and remove its nodes. Replication is removed first, every node in the datacenter is stopped together, and the dead endpoints are then removed from gossip. This is the common approach for large datacenters, particularly those with high vnode counts (the number of token ranges each node owns; see virtual nodes), because per-node decommission is slow and each departure triggers streaming and topology recalculation.
AspectMethod 1: decommissionMethod 2: stop and remove
Node state during removalRunning; leaves the ring one node at a timeStopped; removed as an already-dead endpoint
StreamingMinimal; the datacenter is removed from the replication map firstMinimal; the datacenter is removed from the replication map first
Ring changePer-node ring departure while the node is liveRemoval of endpoints that are already dead
Elapsed timeGrows with node count and vnode countShorter overall, but endpoint removal must still be paced one node at a time (see Method 2, Step 7)
Typical fitSmaller datacenters, no time pressureLarge production datacenters

In both methods as documented here, the datacenter is removed from the replication map before any node leaves, so streaming is minimal in both. The difference is that decommission performs a per-node ring departure while the node is still live, whereas Method 2 removes endpoints that are already dead. Method 2 is what large production clusters typically use.

These prerequisites apply to both methods. A datacenter that has already failed cannot meet them; see Recovering from an Unplanned Datacenter Loss.

RequirementVerification
All nodes in the DC healthynodetool status shows UN
Data present in the remaining DCsFull repair completed, or recent successful repairs verified
Remaining DCs sized for the dataRF in remaining DCs satisfied by available nodes
No clients using the removed DCTraffic shifted away and verified
No client pinned to the removed DCDriver local-datacenter and contact points updated

Verify consistency before the datacenter is stopped

Problem: The datacenter being removed may hold the only up-to-date copy of some data. Once it is removed from the replication map and stopped, that data is unrecoverable.

Symptoms: Missing or stale rows in the remaining datacenters after removal, with no source left to repair from.

Instead: Run a full repair covering all keyspaces, or verify recent successful repairs, before removing the datacenter from the replication map. The datacenter must still be running and replicated at this point so that it can act as a source.

Every node in the datacenter stays running and is taken out of the ring individually, after replication has been withdrawn. The prerequisites for planned removal apply in full, including verification that a full repair has established the data in the remaining datacenters.

Step 1: Redirect client traffic

Update clients to no longer contact the datacenter being removed:

// Java driver - update contact points
CqlSession session = CqlSession.builder()
.addContactPoint(new InetSocketAddress("dc1-node1", 9042))
.addContactPoint(new InetSocketAddress("dc1-node2", 9042))
// Remove dc2 contact points
.withLocalDatacenter("dc1")
.build();

Verify that traffic has actually stopped reaching the datacenter, using the checks in Method 2, Step 3.

Step 2: Update keyspace replication

Remove the datacenter from all keyspaces:

-- User keyspaces
ALTER KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
-- dc2 removed
};
-- System keyspaces
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_distributed WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};

Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.

Step 3: Decommission all nodes in the datacenter

Remove nodes one at a time:

Terminal window
# On each node in dc2
nodetool decommission
# Wait for completion before starting next

nodetool decommission blocks until the node has left the ring, and progress can be watched with nodetool netstats. If a decommission is interrupted, re-run it on the same node; if the node cannot resume, remove it with nodetool removenode.

Removing the datacenter from the replication map (Step 2) before decommissioning means each departing node no longer owns replicas, so it has little or nothing to stream out. This limits decommission streaming and avoids the CPU spikes that full decommission streaming can cause.

Step 4: Verify removal

Terminal window
# dc2 should not appear
nodetool status
# Only dc1 remains
# Datacenter: dc1
# UN 10.0.1.1 245.5 GB 256 ...
# UN 10.0.1.2 238.2 GB 256 ...
# UN 10.0.1.3 251.8 GB 256 ...

Step 5: Update seed lists

Remove dc2 seeds from all remaining nodes:

# cassandra.yaml on dc1 nodes
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "dc1-node1,dc1-node2" # dc2 seeds removed

Seed list changes take effect on restart, so the rolling restart may be deferred, but the configuration must not continue to reference hosts that are about to disappear.

Method 2: Stop the Datacenter and Remove Its Nodes

Section titled “Method 2: Stop the Datacenter and Remove Its Nodes”

The steps must run in this order: consistency is established while the datacenter still exists, traffic is moved away and verified, replication is withdrawn, and only then are the nodes stopped and their endpoints removed from gossip one at a time.

Step 1: Verify the remaining datacenters are consistent

Run a full repair covering all keyspaces, or verify that recent repairs completed successfully, before any cutover.

Terminal window
# Run on every node, in every datacenter, including the one being removed
nodetool repair --full -pr

-pr repairs only each node's primary ranges, so full ring coverage requires the command to run on every node in every datacenter, including the datacenter being removed while it is still a consistency source.

This step confirms the remaining datacenters are consistent with the datacenter being removed while that datacenter still exists as a source. Removing it from the replication map and stopping it without this step risks losing data that was only up to date there.

Step 2: Redirect client traffic away from the datacenter

Redeploy applications with contact points and local datacenter set to a remaining datacenter, and remove the old datacenter's hosts from driver contact points. Traffic that is already local to other datacenters stays where it is.

// Driver 4.x: dc2 contact points removed, local DC moved to dc1
CqlSession.builder()
.addContactPoint(new InetSocketAddress("dc1-node1", 9042))
.addContactPoint(new InetSocketAddress("dc1-node2", 9042))
.withLocalDatacenter("dc1")
.build();

Step 3: Verify no client traffic reaches the datacenter

On the nodes of the datacenter being removed, the completed counts for ReadStage and WriteStage should stop incrementing, allowing for background operations.

Terminal window
# On each dc2 node, sample twice and compare completed counts
nodetool tpstats | grep -E "ReadStage|WriteStage"
# Established client connections on the native transport port
ss -tn state established '( sport = :9042 )'

The expected result is no established client connections on port 9042 to the nodes in the datacenter being removed.

Step 4: Remove the datacenter from all keyspace replication maps

Alter every user keyspace, plus system_auth, system_distributed, and system_traces, so that NetworkTopologyStrategy lists only the remaining datacenters.

-- User keyspaces
ALTER KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
-- dc2 entry removed entirely
};
-- System keyspaces
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_distributed WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};

The datacenter's entry must be removed from the map rather than set to a replication factor of 0. Removing the entry is the standard path, and once the datacenter no longer exists a zero entry would require a second ALTER anyway.

Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.

After this step the nodes in the old datacenter own no replicas, which is what makes their later removal cheap: there is nothing to stream.

Step 5: Update seed lists

Remove the old datacenter's seed hosts from cassandra.yaml on all remaining nodes.

# cassandra.yaml on dc1 nodes
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "dc1-node1,dc1-node2" # dc2 seeds removed

Seed list changes take effect on restart, so the rolling restart may be deferred, but the configuration must not continue to reference hosts that are about to disappear.

Step 6: Stop Cassandra on all nodes in the datacenter

Terminal window
# On every dc2 node
sudo systemctl stop cassandra

The stopped nodes report DN in nodetool status from the remaining datacenters.

Step 7: Remove the dead nodes, one node at a time

Removal runs from a node in a remaining datacenter, against one dead node at a time. nodetool removenode takes the host ID, which is the UUID column in nodetool status, while nodetool assassinate takes the node's IP address.

Terminal window
# Read the host IDs of the stopped dc2 nodes from a dc1 node
nodetool status
# Datacenter: dc2
# -- Address Load Tokens Owns Host ID Rack
# DN 10.0.2.1 245.3 GB 256 ? c0f8f95d-9d3a-4f0e-9a1e-2f6b1d5c7a41 rack1
# DN 10.0.2.2 240.7 GB 256 ? 7b2e4c18-51aa-4d63-8f77-9c0a3e2b6d55 rack2
# One node at a time, run from a node in a remaining datacenter
nodetool removenode c0f8f95d-9d3a-4f0e-9a1e-2f6b1d5c7a41
# If a removal hangs
nodetool removenode force
# Last resort, for an endpoint removenode cannot clear (takes the IP address)
nodetool assassinate 10.0.2.1

After removenode force, confirm the endpoint is actually gone by checking nodetool status and nodetool gossipinfo from several surviving nodes, and assassinate it if it remains. Because the datacenter was already removed from every replication map in Step 4, there is nothing to stream, so a forced removal does not create a consistency gap here.

Operators frequently run assassinate directly, but removenode is the correct process, with removenode force and then assassinate as escalations. Per-command behavior is documented in Removing Nodes.

Each removal causes every surviving node to update its endpoint and token metadata. With high vnode counts, for example 256 vnodes per node, removing nodes too quickly backs up gossip and token metadata calculations, causing CPU and GC spikes and inconsistent ring views across the cluster. Cassandra 4.0 lowered the default num_tokens from 256 to 16 (CASSANDRA-13701), partly to reduce the cost of topology changes; clusters running high vnode counts are more exposed to this effect. A removal command can also return before the change has propagated through gossip, so a command returning is not confirmation that the change has taken effect.

Between each removal, verify cluster stability before proceeding:

Terminal window
nodetool tpstats # GossipStage pending tasks back to zero or normal baseline
nodetool describecluster # exactly one schema version
nodetool status # removed node no longer listed
nodetool gossipinfo # removed endpoint absent

CPU and GC on the surviving nodes should be back to baseline, and no nodes should be flapping. These checks must be run from several nodes, not one: gossip and topology state are local views, and one healthy-looking node does not mean the change has propagated cluster-wide.

Verify the target and pace the removals

Before every decommission, removenode, or assassinate, confirm that the target is the intended node and that it is the node meant to leave, by checking the IP address and host ID against nodetool status. decommission runs on the node being removed itself, so confirm which node the shell is connected to.

Remove one node at a time. Move to small batches, for example one node per rack or availability zone, only if the cluster stays stable between passes, and stay serial where there is no time pressure. Topology recalculation has a fixed CPU cost; pacing spreads it over time. The procedure must not be rushed.

nodetool assassinate is available from Cassandra 2.2 (CASSANDRA-7935); earlier releases expose the equivalent unsafeAssassinateEndpoint operation on the Gossiper JMX MBean.

Step 8: Final verification

Terminal window
# Only the remaining datacenters appear
nodetool status
# One schema version cluster-wide
nodetool describecluster
# No trace of the removed endpoints
nodetool gossipinfo

Client health in the remaining datacenters should also be confirmed before the change is considered complete.

Recovering from an Unplanned Datacenter Loss

Section titled “Recovering from an Unplanned Datacenter Loss”

An entire datacenter may be lost permanently, through site destruction or region failure. All of its nodes show DN and will not return. The prerequisites for planned removal cannot be met, so the datacenter is removed from the cluster once the loss is confirmed.

This procedure applies only once the loss is confirmed to be permanent. A transient network partition or a regional outage that will heal is not a lost datacenter. Any node removed from the ring by this procedure must never rejoin the cluster carrying its previous state; if hardware from the lost site is ever reused, its data directories must be wiped before the node is started.

Immediate impact

The remaining datacenters continue to serve LOCAL_ONE and LOCAL_QUORUM traffic. Requests at QUORUM, EACH_QUORUM, and ALL that required replicas in the lost datacenter fail or degrade until it is removed from replication. Clients pinned to the lost datacenter must be redeployed against a remaining datacenter.

Data exposure

The coordinator sends every mutation to all replicas in all datacenters regardless of consistency level; the consistency level determines only which acknowledgements the coordinator waits for. The remaining datacenters therefore normally already hold the data. Exposure is limited to mutations that had not yet reached any replica in a remaining datacenter at the instant of failure, together with hints stored only on the lost nodes. Neither can be recovered by any tool. This behavior applies to all supported Cassandra versions.

Step 1: Redirect clients to the remaining datacenters

Redeploy applications with contact points and local datacenter set to a surviving datacenter.

Step 2: Remove the dead datacenter from replication

Alter every user keyspace, plus system_auth, system_distributed, and system_traces, so that only the surviving datacenters remain in the replication map.

-- User keyspaces
ALTER KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
-- lost datacenter removed
};
-- System keyspaces
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_distributed WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};
ALTER KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3
};

Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.

Alter replication before removing any node

Problem: nodetool removenode restores replica counts for the ranges the dead node owned. If the lost datacenter is still in the replication map, each removal streams data between the surviving nodes to satisfy a replication factor for a datacenter that no longer exists.

Symptoms: Long-running removals, heavy streaming and CPU load on the survivors, and removals that stall or have to be forced.

Instead: Run the ALTER KEYSPACE statements for all keyspaces first, so the dead nodes own no replicas and node removal has nothing to stream.

Step 3: Remove each dead node

Take the host IDs from nodetool status, which are the UUID column, and remove one node at a time from a live node.

Terminal window
# Host IDs of the dead nodes
nodetool status
# One node at a time
nodetool removenode <host-id>
# If a removal hangs
nodetool removenode force
# Last resort, for an endpoint removenode cannot clear (takes the IP address)
nodetool assassinate <ip>

The full progression, the pacing rules, and the stability checks between removals are described in Method 2, Step 7 and apply unchanged here. Per-command behavior is documented in Removing Nodes.

Step 4: Update seed lists

Remove the lost datacenter's seed hosts from cassandra.yaml on the remaining nodes.

Step 5: Resume the repair cadence

Once the topology has settled, resume the normal repair schedule as ordinary anti-entropy between the surviving replicas. Repair does not recover data that was lost with the datacenter.

Step 6: Replace the datacenter, if required

Building a replacement datacenter follows the Adding a Datacenter procedure on this page.


Renaming a datacenter requires migrating to a new DC configuration.

  1. Add nodes with new DC name (as if adding new datacenter)
  2. Rebuild data to new DC
  3. Update replication to include new DC name
  4. Redirect clients to new DC name
  5. Remove old DC

Complex Operation

DC renaming is effectively adding a new DC and removing the old one. Plan for significant downtime or accept temporary doubled hardware.


Consistency LevelMulti-DC Behavior
ONESatisfied by any DC
LOCAL_ONEMust be satisfied in coordinator's DC
QUORUMMajority across ALL DCs
LOCAL_QUORUMMajority in coordinator's DC only
EACH_QUORUMMajority in EACH DC
ALLAll replicas in ALL DCs
Use CaseWrite CLRead CL
Strong local consistencyLOCAL_QUORUMLOCAL_QUORUM
Global strong consistencyQUORUMQUORUM
Availability priorityLOCAL_ONELOCAL_ONE
Cross-DC reads during DC failureQUORUMQUORUM

LOCAL_QUORUM Recommendation

For most multi-DC deployments, LOCAL_QUORUM provides the best balance of consistency and availability. It ensures strong consistency within each DC while tolerating complete DC failure.


Symptoms: nodetool netstats shows no progress

Terminal window
# Check source DC health
nodetool status
# Check network connectivity
nc -zv dc1-node1 7000
# Check logs
grep -i "stream\|rebuild" /var/log/cassandra/system.log | tail -50

Solutions:

  1. Verify cross-DC network connectivity
  2. Check source DC capacity (may be overwhelmed)
  3. Increase streaming timeouts

Symptoms: Nodes can't authenticate after joining

Cause: system_auth not replicated to new DC before nodes joined

Solution:

-- Update system_auth replication
ALTER KEYSPACE system_auth WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc1': 3,
'dc2': 3
};
-- Run repair on system_auth
nodetool repair system_auth

Symptoms: Client requests slow when coordinator in different DC

Solutions:

  1. Use LOCAL_QUORUM instead of QUORUM
  2. Configure client with correct local DC:
.withLocalDatacenter("dc1")
  1. Review cross-DC network performance