Cassandra Multi-Datacenter Operations
This guide covers procedures for adding and removing datacenters in a Cassandra cluster.
Adding a Datacenter
Section titled “Adding a Datacenter”Expanding to a new datacenter provides geographic redundancy and reduced latency for regional users.
Prerequisites
Section titled “Prerequisites”| Requirement | Verification |
|---|---|
| Existing cluster healthy | All nodes UN |
| Network connectivity | New DC can reach existing DCs |
| Cross-DC latency acceptable | < 100ms recommended |
| Same Cassandra version | Match existing cluster |
| Hardware provisioned | Nodes ready in new DC |
Planning Considerations
Section titled “Planning Considerations”Replication strategy:
The cluster must use NetworkTopologyStrategy for multi-DC deployments:
-- Check current replicationDESCRIBE KEYSPACE my_keyspace;
-- Must be NetworkTopologyStrategy, not SimpleStrategyNode count:
- New datacenter should have at least RF nodes
- Typically match node count of existing DC
Network requirements:
| Port | Purpose |
|---|---|
| 7000 | Internode (gossip, streaming) |
| 7001 | Internode SSL |
| 9042 | Client connections |
Procedure
Section titled “Procedure”Step 1: Prepare keyspace replication (existing datacenter only)
Before adding any nodes, ensure every keyspace uses NetworkTopologyStrategy scoped to the existing datacenter only. Convert from SimpleStrategy if needed. Do not add the new datacenter yet — replication is extended only after the new nodes have joined (Step 4), so the new datacenter never receives traffic before it can serve it.
-- User keyspaces: convert SimpleStrategy to NetworkTopologyStrategy (existing DC only)ALTER KEYSPACE my_keyspace WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
-- System keyspacesALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_distributed WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_traces WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};New datacenter added later
The new datacenter is intentionally left out of the replication map until the nodes have joined the ring. It is added in Step 4.
Pin clients to the existing datacenter
Before the new nodes join gossip, pin application drivers to the existing datacenter so they never coordinate through the empty new-DC nodes:
// Driver 4.xCqlSession.builder() .withLocalDatacenter("dc1") .build();
// Legacy driver 3.xCluster.builder() .withLoadBalancingPolicy( DCAwareRoundRobinPolicy.builder().withLocalDc("dc1").build()) .build();Pair this with LOCAL_ONE or LOCAL_QUORUM consistency — not ONE or QUORUM, which can span both data centers — so reads and writes stay within the pinned datacenter.
Step 2: Configure nodes in new datacenter
On each new node:
cluster_name: 'ProductionCluster' # Must matchnum_tokens: 16 # Match existing cluster value (default is 16 for 4.0+)
# Seeds from BOTH datacentersseed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "dc1-node1,dc1-node2,dc2-node1"
# This node's addresslisten_address: <node_ip>rpc_address: <node_ip>
# Snitch for multi-DCendpoint_snitch: GossipingPropertyFileSnitch
# Join the ring without streaming data; populated later via rebuildauto_bootstrap: falsedc=dc2rack=rack1Step 3: Start nodes in new datacenter
Add nodes roughly 2 minutes apart so gossip settles between joins, and do NOT wait for bootstrap:
# New DC nodes start with auto_bootstrap: false# They join the ring but receive no data yet
# Start first nodesudo systemctl start cassandra
# Verify it joins (shows UN but with 0 load)nodetool status
# Start remaining nodes, ~2 minutes apartBootstrap Disabled
For new datacenter nodes, set auto_bootstrap: false. Data will be populated via rebuild, not bootstrap.
Step 4: Extend replication to the new datacenter
Now that all new nodes have joined and report UN, extend NetworkTopologyStrategy so it also covers the new datacenter. New writes begin replicating to it immediately.
-- User keyspacesALTER KEYSPACE my_keyspace WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3 -- New datacenter};
-- System keyspaces (critical!)ALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
ALTER KEYSPACE system_distributed WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
ALTER KEYSPACE system_traces WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};Include system_auth
Include system_auth when extending replication, and rebuild it (Step 5) so authentication works locally in the new datacenter.
Step 5: Rebuild data in new datacenter
On each node in the new datacenter, run rebuild to back-fill the data that bootstrap skipped:
# Rebuild from the existing (source) datacenternodetool rebuild dc1
# This streams all data for this node's token ranges from dc1Start with one node, then add parallel rebuilds while client latency stays stable, backing off as soon as it climbs. This paces streaming against live load instead of overwhelming the source datacenter.
# Monitor rebuild progressnodetool netstats
# Watch for completiontail -f /var/log/cassandra/system.log | grep -i rebuildStep 6: Verify completion
# All nodes UN with datanodetool status
# Example output:# Datacenter: dc1# UN 10.0.1.1 245.5 GB 16 ...# UN 10.0.1.2 238.2 GB 16 ...# UN 10.0.1.3 251.8 GB 16 ...## Datacenter: dc2# UN 10.0.2.1 243.1 GB 16 ... <-- Data present# UN 10.0.2.2 240.7 GB 16 ...# UN 10.0.2.3 248.3 GB 16 ...Step 7: Redirect client traffic to the new datacenter
Once the new datacenter is fully rebuilt and consistent, repoint application drivers to prefer it. Switch the local datacenter from the old DC to the new DC and roll the applications:
// Driver 4.xCqlSession.builder() .addContactPoint(new InetSocketAddress("dc2-node1", 9042)) .withLocalDatacenter("dc2") // was dc1 .build();Continue to use LOCAL_ONE or LOCAL_QUORUM consistency — not ONE or QUORUM, which can span both data centers — so reads and writes stay within the new datacenter.
Repoint only after rebuild completes
Do not switch clients to the new datacenter until rebuild (Step 5) has finished and consistency is verified (Step 6). Repointing earlier serves reads from an unbuilt datacenter.
Duration Estimates
Section titled “Duration Estimates”| Data to Rebuild | Per Node |
|---|---|
| 100 GB | 1-2 hours |
| 500 GB | 4-8 hours |
| 1 TB | 8-16 hours |
Total time ≈ per-node time × number of new-DC nodes when rebuilding sequentially. Running rebuilds in parallel (Step 5) reduces wall-clock time as live load allows.
Removing a Datacenter
Section titled “Removing a Datacenter”Datacenter removal takes two forms. A planned removal consolidates the cluster while the datacenter is still running and can be shut down in a controlled order. Recovery from an unplanned loss applies when the datacenter is already gone and must be removed from the cluster once the loss is confirmed.
Two methods are used for planned removal:
- Method 1, decommission each node. Each node is removed from the ring one at a time while it is still running. This fits smaller datacenters where nodes leaving the ring gracefully one at a time is acceptable and there is no time pressure.
- Method 2, stop the datacenter and remove its nodes. Replication is removed first, every node in the datacenter is stopped together, and the dead endpoints are then removed from gossip. This is the common approach for large datacenters, particularly those with high vnode counts (the number of token ranges each node owns; see virtual nodes), because per-node decommission is slow and each departure triggers streaming and topology recalculation.
Choosing a Method
Section titled “Choosing a Method”| Aspect | Method 1: decommission | Method 2: stop and remove |
|---|---|---|
| Node state during removal | Running; leaves the ring one node at a time | Stopped; removed as an already-dead endpoint |
| Streaming | Minimal; the datacenter is removed from the replication map first | Minimal; the datacenter is removed from the replication map first |
| Ring change | Per-node ring departure while the node is live | Removal of endpoints that are already dead |
| Elapsed time | Grows with node count and vnode count | Shorter overall, but endpoint removal must still be paced one node at a time (see Method 2, Step 7) |
| Typical fit | Smaller datacenters, no time pressure | Large production datacenters |
In both methods as documented here, the datacenter is removed from the replication map before any node leaves, so streaming is minimal in both. The difference is that decommission performs a per-node ring departure while the node is still live, whereas Method 2 removes endpoints that are already dead. Method 2 is what large production clusters typically use.
Prerequisites for Planned Removal
Section titled “Prerequisites for Planned Removal”These prerequisites apply to both methods. A datacenter that has already failed cannot meet them; see Recovering from an Unplanned Datacenter Loss.
| Requirement | Verification |
|---|---|
| All nodes in the DC healthy | nodetool status shows UN |
| Data present in the remaining DCs | Full repair completed, or recent successful repairs verified |
| Remaining DCs sized for the data | RF in remaining DCs satisfied by available nodes |
| No clients using the removed DC | Traffic shifted away and verified |
| No client pinned to the removed DC | Driver local-datacenter and contact points updated |
Verify consistency before the datacenter is stopped
Problem: The datacenter being removed may hold the only up-to-date copy of some data. Once it is removed from the replication map and stopped, that data is unrecoverable.
Symptoms: Missing or stale rows in the remaining datacenters after removal, with no source left to repair from.
Instead: Run a full repair covering all keyspaces, or verify recent successful repairs, before removing the datacenter from the replication map. The datacenter must still be running and replicated at this point so that it can act as a source.
Method 1: Decommission Each Node
Section titled “Method 1: Decommission Each Node”Every node in the datacenter stays running and is taken out of the ring individually, after replication has been withdrawn. The prerequisites for planned removal apply in full, including verification that a full repair has established the data in the remaining datacenters.
Step 1: Redirect client traffic
Update clients to no longer contact the datacenter being removed:
// Java driver - update contact pointsCqlSession session = CqlSession.builder() .addContactPoint(new InetSocketAddress("dc1-node1", 9042)) .addContactPoint(new InetSocketAddress("dc1-node2", 9042)) // Remove dc2 contact points .withLocalDatacenter("dc1") .build();Verify that traffic has actually stopped reaching the datacenter, using the checks in Method 2, Step 3.
Step 2: Update keyspace replication
Remove the datacenter from all keyspaces:
-- User keyspacesALTER KEYSPACE my_keyspace WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3 -- dc2 removed};
-- System keyspacesALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_distributed WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_traces WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.
Step 3: Decommission all nodes in the datacenter
Remove nodes one at a time:
# On each node in dc2nodetool decommission
# Wait for completion before starting nextnodetool decommission blocks until the node has left the ring, and progress can be watched with nodetool netstats. If a decommission is interrupted, re-run it on the same node; if the node cannot resume, remove it with nodetool removenode.
Removing the datacenter from the replication map (Step 2) before decommissioning means each departing node no longer owns replicas, so it has little or nothing to stream out. This limits decommission streaming and avoids the CPU spikes that full decommission streaming can cause.
Step 4: Verify removal
# dc2 should not appearnodetool status
# Only dc1 remains# Datacenter: dc1# UN 10.0.1.1 245.5 GB 256 ...# UN 10.0.1.2 238.2 GB 256 ...# UN 10.0.1.3 251.8 GB 256 ...Step 5: Update seed lists
Remove dc2 seeds from all remaining nodes:
# cassandra.yaml on dc1 nodesseed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "dc1-node1,dc1-node2" # dc2 seeds removedSeed list changes take effect on restart, so the rolling restart may be deferred, but the configuration must not continue to reference hosts that are about to disappear.
Method 2: Stop the Datacenter and Remove Its Nodes
Section titled “Method 2: Stop the Datacenter and Remove Its Nodes”The steps must run in this order: consistency is established while the datacenter still exists, traffic is moved away and verified, replication is withdrawn, and only then are the nodes stopped and their endpoints removed from gossip one at a time.
Step 1: Verify the remaining datacenters are consistent
Run a full repair covering all keyspaces, or verify that recent repairs completed successfully, before any cutover.
# Run on every node, in every datacenter, including the one being removednodetool repair --full -pr-pr repairs only each node's primary ranges, so full ring coverage requires the command to run on every node in every datacenter, including the datacenter being removed while it is still a consistency source.
This step confirms the remaining datacenters are consistent with the datacenter being removed while that datacenter still exists as a source. Removing it from the replication map and stopping it without this step risks losing data that was only up to date there.
Step 2: Redirect client traffic away from the datacenter
Redeploy applications with contact points and local datacenter set to a remaining datacenter, and remove the old datacenter's hosts from driver contact points. Traffic that is already local to other datacenters stays where it is.
// Driver 4.x: dc2 contact points removed, local DC moved to dc1CqlSession.builder() .addContactPoint(new InetSocketAddress("dc1-node1", 9042)) .addContactPoint(new InetSocketAddress("dc1-node2", 9042)) .withLocalDatacenter("dc1") .build();Step 3: Verify no client traffic reaches the datacenter
On the nodes of the datacenter being removed, the completed counts for ReadStage and WriteStage should stop incrementing, allowing for background operations.
# On each dc2 node, sample twice and compare completed countsnodetool tpstats | grep -E "ReadStage|WriteStage"
# Established client connections on the native transport portss -tn state established '( sport = :9042 )'The expected result is no established client connections on port 9042 to the nodes in the datacenter being removed.
Step 4: Remove the datacenter from all keyspace replication maps
Alter every user keyspace, plus system_auth, system_distributed, and system_traces, so that NetworkTopologyStrategy lists only the remaining datacenters.
-- User keyspacesALTER KEYSPACE my_keyspace WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3 -- dc2 entry removed entirely};
-- System keyspacesALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_distributed WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_traces WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};The datacenter's entry must be removed from the map rather than set to a replication factor of 0. Removing the entry is the standard path, and once the datacenter no longer exists a zero entry would require a second ALTER anyway.
Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.
After this step the nodes in the old datacenter own no replicas, which is what makes their later removal cheap: there is nothing to stream.
Step 5: Update seed lists
Remove the old datacenter's seed hosts from cassandra.yaml on all remaining nodes.
# cassandra.yaml on dc1 nodesseed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "dc1-node1,dc1-node2" # dc2 seeds removedSeed list changes take effect on restart, so the rolling restart may be deferred, but the configuration must not continue to reference hosts that are about to disappear.
Step 6: Stop Cassandra on all nodes in the datacenter
# On every dc2 nodesudo systemctl stop cassandraThe stopped nodes report DN in nodetool status from the remaining datacenters.
Step 7: Remove the dead nodes, one node at a time
Removal runs from a node in a remaining datacenter, against one dead node at a time. nodetool removenode takes the host ID, which is the UUID column in nodetool status, while nodetool assassinate takes the node's IP address.
# Read the host IDs of the stopped dc2 nodes from a dc1 nodenodetool status
# Datacenter: dc2# -- Address Load Tokens Owns Host ID Rack# DN 10.0.2.1 245.3 GB 256 ? c0f8f95d-9d3a-4f0e-9a1e-2f6b1d5c7a41 rack1# DN 10.0.2.2 240.7 GB 256 ? 7b2e4c18-51aa-4d63-8f77-9c0a3e2b6d55 rack2
# One node at a time, run from a node in a remaining datacenternodetool removenode c0f8f95d-9d3a-4f0e-9a1e-2f6b1d5c7a41
# If a removal hangsnodetool removenode force
# Last resort, for an endpoint removenode cannot clear (takes the IP address)nodetool assassinate 10.0.2.1After removenode force, confirm the endpoint is actually gone by checking nodetool status and nodetool gossipinfo from several surviving nodes, and assassinate it if it remains. Because the datacenter was already removed from every replication map in Step 4, there is nothing to stream, so a forced removal does not create a consistency gap here.
Operators frequently run assassinate directly, but removenode is the correct process, with removenode force and then assassinate as escalations. Per-command behavior is documented in Removing Nodes.
Each removal causes every surviving node to update its endpoint and token metadata. With high vnode counts, for example 256 vnodes per node, removing nodes too quickly backs up gossip and token metadata calculations, causing CPU and GC spikes and inconsistent ring views across the cluster. Cassandra 4.0 lowered the default num_tokens from 256 to 16 (CASSANDRA-13701), partly to reduce the cost of topology changes; clusters running high vnode counts are more exposed to this effect. A removal command can also return before the change has propagated through gossip, so a command returning is not confirmation that the change has taken effect.
Between each removal, verify cluster stability before proceeding:
nodetool tpstats # GossipStage pending tasks back to zero or normal baselinenodetool describecluster # exactly one schema versionnodetool status # removed node no longer listednodetool gossipinfo # removed endpoint absentCPU and GC on the surviving nodes should be back to baseline, and no nodes should be flapping. These checks must be run from several nodes, not one: gossip and topology state are local views, and one healthy-looking node does not mean the change has propagated cluster-wide.
Verify the target and pace the removals
Before every decommission, removenode, or assassinate, confirm that the target is the intended node and that it is the node meant to leave, by checking the IP address and host ID against nodetool status. decommission runs on the node being removed itself, so confirm which node the shell is connected to.
Remove one node at a time. Move to small batches, for example one node per rack or availability zone, only if the cluster stays stable between passes, and stay serial where there is no time pressure. Topology recalculation has a fixed CPU cost; pacing spreads it over time. The procedure must not be rushed.
nodetool assassinate is available from Cassandra 2.2 (CASSANDRA-7935); earlier releases expose the equivalent unsafeAssassinateEndpoint operation on the Gossiper JMX MBean.
Step 8: Final verification
# Only the remaining datacenters appearnodetool status
# One schema version cluster-widenodetool describecluster
# No trace of the removed endpointsnodetool gossipinfoClient health in the remaining datacenters should also be confirmed before the change is considered complete.
Recovering from an Unplanned Datacenter Loss
Section titled “Recovering from an Unplanned Datacenter Loss”An entire datacenter may be lost permanently, through site destruction or region failure. All of its nodes show DN and will not return. The prerequisites for planned removal cannot be met, so the datacenter is removed from the cluster once the loss is confirmed.
This procedure applies only once the loss is confirmed to be permanent. A transient network partition or a regional outage that will heal is not a lost datacenter. Any node removed from the ring by this procedure must never rejoin the cluster carrying its previous state; if hardware from the lost site is ever reused, its data directories must be wiped before the node is started.
Immediate impact
The remaining datacenters continue to serve LOCAL_ONE and LOCAL_QUORUM traffic. Requests at QUORUM, EACH_QUORUM, and ALL that required replicas in the lost datacenter fail or degrade until it is removed from replication. Clients pinned to the lost datacenter must be redeployed against a remaining datacenter.
Data exposure
The coordinator sends every mutation to all replicas in all datacenters regardless of consistency level; the consistency level determines only which acknowledgements the coordinator waits for. The remaining datacenters therefore normally already hold the data. Exposure is limited to mutations that had not yet reached any replica in a remaining datacenter at the instant of failure, together with hints stored only on the lost nodes. Neither can be recovered by any tool. This behavior applies to all supported Cassandra versions.
Step 1: Redirect clients to the remaining datacenters
Redeploy applications with contact points and local datacenter set to a surviving datacenter.
Step 2: Remove the dead datacenter from replication
Alter every user keyspace, plus system_auth, system_distributed, and system_traces, so that only the surviving datacenters remain in the replication map.
-- User keyspacesALTER KEYSPACE my_keyspace WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3 -- lost datacenter removed};
-- System keyspacesALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_distributed WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};
ALTER KEYSPACE system_traces WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3};Confirm schema agreement with nodetool describecluster, which must report a single schema version, before proceeding.
Alter replication before removing any node
Problem: nodetool removenode restores replica counts for the ranges the dead node owned. If the lost datacenter is still in the replication map, each removal streams data between the surviving nodes to satisfy a replication factor for a datacenter that no longer exists.
Symptoms: Long-running removals, heavy streaming and CPU load on the survivors, and removals that stall or have to be forced.
Instead: Run the ALTER KEYSPACE statements for all keyspaces first, so the dead nodes own no replicas and node removal has nothing to stream.
Step 3: Remove each dead node
Take the host IDs from nodetool status, which are the UUID column, and remove one node at a time from a live node.
# Host IDs of the dead nodesnodetool status
# One node at a timenodetool removenode <host-id>
# If a removal hangsnodetool removenode force
# Last resort, for an endpoint removenode cannot clear (takes the IP address)nodetool assassinate <ip>The full progression, the pacing rules, and the stability checks between removals are described in Method 2, Step 7 and apply unchanged here. Per-command behavior is documented in Removing Nodes.
Step 4: Update seed lists
Remove the lost datacenter's seed hosts from cassandra.yaml on the remaining nodes.
Step 5: Resume the repair cadence
Once the topology has settled, resume the normal repair schedule as ordinary anti-entropy between the surviving replicas. Repair does not recover data that was lost with the datacenter.
Step 6: Replace the datacenter, if required
Building a replacement datacenter follows the Adding a Datacenter procedure on this page.
Changing Datacenter Names
Section titled “Changing Datacenter Names”Renaming a datacenter requires migrating to a new DC configuration.
Procedure
Section titled “Procedure”- Add nodes with new DC name (as if adding new datacenter)
- Rebuild data to new DC
- Update replication to include new DC name
- Redirect clients to new DC name
- Remove old DC
Complex Operation
DC renaming is effectively adding a new DC and removing the old one. Plan for significant downtime or accept temporary doubled hardware.
Cross-DC Consistency Considerations
Section titled “Cross-DC Consistency Considerations”Consistency Level Behavior
Section titled “Consistency Level Behavior”| Consistency Level | Multi-DC Behavior |
|---|---|
ONE | Satisfied by any DC |
LOCAL_ONE | Must be satisfied in coordinator's DC |
QUORUM | Majority across ALL DCs |
LOCAL_QUORUM | Majority in coordinator's DC only |
EACH_QUORUM | Majority in EACH DC |
ALL | All replicas in ALL DCs |
Recommended Settings
Section titled “Recommended Settings”| Use Case | Write CL | Read CL |
|---|---|---|
| Strong local consistency | LOCAL_QUORUM | LOCAL_QUORUM |
| Global strong consistency | QUORUM | QUORUM |
| Availability priority | LOCAL_ONE | LOCAL_ONE |
| Cross-DC reads during DC failure | QUORUM | QUORUM |
LOCAL_QUORUM Recommendation
For most multi-DC deployments, LOCAL_QUORUM provides the best balance of consistency and availability. It ensures strong consistency within each DC while tolerating complete DC failure.
Troubleshooting
Section titled “Troubleshooting”Rebuild Stalled
Section titled “Rebuild Stalled”Symptoms: nodetool netstats shows no progress
# Check source DC healthnodetool status
# Check network connectivitync -zv dc1-node1 7000
# Check logsgrep -i "stream\|rebuild" /var/log/cassandra/system.log | tail -50Solutions:
- Verify cross-DC network connectivity
- Check source DC capacity (may be overwhelmed)
- Increase streaming timeouts
Authentication Failures in New DC
Section titled “Authentication Failures in New DC”Symptoms: Nodes can't authenticate after joining
Cause: system_auth not replicated to new DC before nodes joined
Solution:
-- Update system_auth replicationALTER KEYSPACE system_auth WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
-- Run repair on system_authnodetool repair system_authCross-DC Latency Issues
Section titled “Cross-DC Latency Issues”Symptoms: Client requests slow when coordinator in different DC
Solutions:
- Use
LOCAL_QUORUMinstead ofQUORUM - Configure client with correct local DC:
.withLocalDatacenter("dc1")- Review cross-DC network performance
Related Documentation
Section titled “Related Documentation”- Cluster Management Overview - Operation selection
- Adding Nodes - Bootstrap procedures
- Removing Nodes - Decommission, removenode, and assassinate behavior
- Repair - Anti-entropy repair before and after topology changes
- Multi-Datacenter Architecture - The architecture behind these procedures
- Consistency Levels - Multi-DC consistency