Second Datacenter Migration (DSE 5.1 and Earlier)

This page describes an alternative migration path available for DSE 5.1 and earlier releases. Instead of standing up a separate target cluster and proxying traffic to it, a native Apache Cassandra datacenter is added to the existing ring, data is replicated and rebuilt onto it under controlled load, validated, and the DSE datacenter is then decommissioned, leaving a pure Apache Cassandra cluster.

For DSE 6.0+, see DSE 6.0+: expert-only below. The CDM + ZDM Proxy path is the safer, recommended choice. The same applies when the preference is to avoid adding a datacenter to the existing ring.

Why this path suits DSE 5.1 and earlier

Each DSE release is a fork of a specific Apache Cassandra version and, up to and including DSE 5.1, remains gossip-, streaming-, and SSTable-compatible with that underlying open-source version:

DSE release	Underlying Apache Cassandra version to add as the new DC
DSE 5.1	Cassandra 3.11
DSE 5.0	Cassandra 3.0
DSE 4.7 and 4.8	Cassandra 2.1
DSE 4.0 to 4.6	Cassandra 2.0

That compatibility is what allows an open-source datacenter running the matching Cassandra version to join the existing DSE ring. DSE 5.1, for example, reports itself as Apache Cassandra 3.11 (it is DataStax's fork of Apache 3.11), so an open-source Cassandra 3.11 datacenter joins the ring. DSE 6.0+ does not maintain that compatibility (BTI SSTables and EverywhereStrategy system keyspaces); see DSE 6.0+: expert-only.

The compatibility is real but conditional: several proprietary DSE classes will prevent an open-source node from joining until they are remediated (see Prerequisites).

The new datacenter joins at the underlying Cassandra version (3.11 for DSE 5.1, older for earlier releases), because the ring is compatible only at that version. Reaching the 5.0 end-state requires a subsequent, separate rolling upgrade; see Upgrade after migration to 5.0. The older the DSE release, the more upgrade hops that involves.

DSE 6.0+: expert-only

DSE 6.0+ introduced the proprietary BTI SSTable format and assigns the EverywhereStrategy replication class to its internal system keyspaces. Neither exists in open-source Cassandra, which is why open-source nodes do not join a DSE 6.0+ ring out of the box.

This does not make a second-datacenter migration categorically impossible on DSE 6.0+, but it does make it highly nuanced and expert-only. Any attempt must be proven with a proof of concept and thorough testing against the actual configuration before production, and AxonOps engagement is recommended for guidance.

For the overwhelming majority of DSE 6.0+ migrations, the safest and recommended path is the CDM + ZDM Proxy data migration to a separate target cluster. Use that unless there is a specific, tested reason not to.

When to choose this path

This path (second datacenter)	CDM + ZDM Proxy
Uses only Cassandra's native multi-DC machinery; no third-party component enters the application path	Introduces the proxy into the application path
The DSE datacenter stays intact and authoritative until validation passes, so rollback before decommission is to remove the new DC	Rollback is reconfiguring the proxy/CDM
Data movement rate is fully controllable via streaming throttles	Migration rate governed by CDM throughput
Allows correction of settings (num_tokens, topology, hardware) on the new DC from day one	Target cluster built fresh regardless
Reaches the DSE release's underlying Cassandra version first; a subsequent upgrade to 5.0 is required	Can target Cassandra 5.0 directly

Because a hybrid ring requires the open-source side to run the DSE release's underlying Cassandra version, reaching the 5.0 end-state is a two-stage process: migrate at the matching version, then perform a rolling upgrade. See Upgrade after migration to 5.0.

Prerequisites: remediating proprietary classes

Before the first open-source node joins the ring, the following DSE-specific behaviours must be remediated, or open-source nodes will fail to start or behave incorrectly. These remediations are applied to the DSE side first, so that both datacenters share open-source-compatible infrastructure before the ring becomes hybrid. This in-ring path is available because DSE 5.x remains SSTable- and ring-compatible with open-source Cassandra; see SSTable compatibility for why DSE 6.0+ cannot use it.

Proprietary element	Problem	Remediation
EverywhereStrategy on DSE system keyspaces	The class does not exist in open-source Cassandra; it is loaded by fully-qualified name, so it cannot be stubbed or aliased. Open-source nodes crash-loop with ClassNotFoundException.	Rewrite the affected keyspaces to NetworkTopologyStrategy with zero replicas in the open-source DC before any open-source node joins.
DseAuthenticator / DseAuthorizer / DseRoleManager	Exist only on DSE's classpath; an open-source node configured with them fails at startup. They can also accept connections while stamping sessions anonymous if incompletely configured.	Move the DSE side to the open-source stack (PasswordAuthenticator / CassandraAuthorizer / CassandraRoleManager) so both DCs share identical authentication.
DseQueryHandler (DSE-only protocol versions, continuous paging)	Drivers that negotiated DSE_V1/DSE_V2 or continuous paging fail against open-source nodes.	Review and update driver configuration (see Component Migration).
Dual-file configuration (cassandra.yaml + dse.yaml)	Open-source Cassandra reads only cassandra.yaml; anything governed by dse.yaml is invisible to it.	Classify every operative dse.yaml stanza: carry it over in open-source-equivalent form, discard it, or compensate for it.
Transparent Data Encryption (TDE)	Encrypted on-disk bytes depend on keys in a DSE-only keyspace, readable only inside DSE.	Confirm TDE status during assessment; if enabled, resolve it before migrating, because encrypted data cannot be read by an open-source node.
DSE-only system keyspaces (e.g. dse_system, dse_insights, dse_security, and others) and DSE UDTs	Present on every DSE 5.1 cluster regardless of features used.	Remove them, in dependency order, before decommissioning the DSE nodes, and never restore a backup that contains them.

Never restore a DSE-era backup into the open-source cluster

Problem: A DSE-era backup preserves DSE's original system_schema rows, which include EverywhereStrategy, a replication strategy class that does not exist in open-source Cassandra.

Symptoms: Every open-source node crash-loops on startup when it loads the restored schema and cannot resolve EverywhereStrategy.

Instead: Migrate user-table data with CDM over the joined ring; never restore a full DSE backup. Backup-and-restore is not a viable migration approach for this path.

Migration procedure

1. Apply the prerequisite remediations above to the DSE datacenter while it is still the only datacenter.
2. Add the new datacenter, empty. Provision open-source Apache Cassandra nodes running the DSE release's underlying version (3.11 for DSE 5.1; see the version table) and have them join with auto_bootstrap: false. They take ownership of token ranges and participate in gossip but hold no data and receive no streams. This separates the topology change from the bulk data transfer, so data-movement rate can be controlled via streaming throttles during the subsequent rebuild step rather than coupling both operations in a single, hard-to-pause join. Build the new DC with corrected settings, notably a sensible num_tokens (for example, 6 vnodes instead of the legacy 256), since the migration is the right moment to make that change.
3. Verify the topology in isolation. Confirm a consistent ring view from both sides, schema agreement, and the intended num_tokens, while the DSE datacenter continues to serve traffic unaffected.

Configure datacenter-aware routing before adding the new DC to replication

Once the new datacenter is in the keyspace replication map but before nodetool rebuild has populated it, application drivers without datacenter affinity can route reads to the empty new datacenter and receive missing or inconsistent rows. Application drivers must use datacenter-aware load balancing (for example, DCAwareRoundRobinPolicy in the Java driver, or local_dc in the Python driver) pinned to the DSE datacenter as the local DC, and that pinning must remain until the new datacenter passes full validation. Audit driver configuration before this step.

4. Alter user keyspaces to replicate into the new datacenter (add the new DC to the NetworkTopologyStrategy replication map).
5. Rebuild under control. Run nodetool rebuild node by node on the new DC, streaming from the DSE datacenter with throughput throttled and observed. This is the controlled bulk transfer, decoupled from the topology change. See Adding a Datacenter for the full node-by-node procedure and throttle settings.

   # On each new-DC node, stream data from the DSE datacenter
   nodetool rebuild -- <dse_datacenter_name>

6. Repair the new datacenter to establish full consistency once rebuild completes.

   nodetool repair -pr --full

7. Validate the new datacenter against the application and against the baselines; see Validation.

Removing the DSE datacenter is irreversible

The DSE datacenter must be decommissioned only after the new datacenter is validated and authoritative. The replication change, node decommission, and removal of DSE-only system keyspaces must be performed in order; this step cannot be reversed.

8. Decommission the DSE datacenter once validation passes: remove the DSE DC from keyspace replication, decommission the DSE nodes, and remove the DSE-only system keyspaces in dependency order. The end state is a pure Apache Cassandra cluster.

Rollback

Until the DSE datacenter is decommissioned, rollback is low-risk: stop replicating to the new datacenter, remove it from the keyspace replication maps, and decommission the new (open-source) nodes. The DSE datacenter remains authoritative throughout, so no data is at risk before the decommission step.

Upgrade after migration to 5.0

This path leaves the cluster on the DSE release's underlying Cassandra version: 3.11 for DSE 5.1, older for earlier releases. Cassandra does not permit an entire major generation to be skipped (a 3.11 cluster cannot upgrade directly to 5.0), so reaching the 5.0 end-state requires multiple hops: 3.11 → 4.0 → 4.1 → 5.0 from DSE 5.1, and correspondingly more hops from older DSE releases.

Per hop, per node, rolling:

verify cluster health
  → nodetool drain
  → stop Cassandra
  → upgrade packages
  → apply configuration changes for the new series
  → start Cassandra
  → verify the node rejoins healthy
  → proceed to the next node

Once every node runs the new version, rewrite SSTables across the cluster:

nodetool upgradesstables

No streaming operations during a mixed-version window

Problem: While a ring contains more than one major Cassandra version, streaming uses wire formats and gossip assumptions that differ across versions.

Symptoms: Repairs, bootstraps, decommissions, rebuilds, and schema changes can fail mid-operation or move data incorrectly.

Instead: Complete the rolling upgrade to a single version on every node before running any streaming operation, and keep the mixed-version window short.

What each hop delivers:

• 3.11 → 4.0 → 4.1: the 4.x series adds a hardened storage engine and zero-copy streaming, full-query and audit logging, virtual tables, dependable incremental repair, and Java 11 support over the 3.x baseline. The 4.0 hop provides an intermediate validation checkpoint before proceeding to 4.1. The 3.x series is end-of-life; 4.1 is the minimum supportable open-source baseline.
• 4.1 → 5.0: Storage-Attached Indexes (SAI), the Unified Compaction Strategy, trie-based memtables and SSTables, vector search, and Java 17 support: the strategic end-state.

Throughout rebuild, repair, decommission, and the subsequent upgrades, AxonOps provides per-node visibility including streaming progress, repair coverage, version state, and upgradesstables progress, which OpsCenter cannot provide for an open-source cluster.

---

Next: Phase 4: Validate the Cassandra migration.