DSE Migration Troubleshooting: ZDM & CDM Caveats
The CDM + ZDM Proxy migration mechanics have behaviours that are not obvious from a first read of the tooling and that have tripped up real migrations. This page collects the most important ones as a troubleshooting reference. Read it before running the data migration.
Each item follows a Symptoms → Diagnostics → Resolution → Root Cause structure.
Before migration
Section titled “Before migration”Dedicated migration infrastructure is required
Section titled “Dedicated migration infrastructure is required”The ZDM Proxy and CDM run on their own instances, not on the source or target clusters. Plan for at least 3 proxy instances, a jumphost/monitoring instance, and a CDM host. See ZDM Proxy and CDM Setup.
System tables require manual setup
Section titled “System tables require manual setup”Symptoms: writes to the target fail immediately after the proxy is deployed; clients see authentication failures or "keyspace/table does not exist" errors.
Diagnostics: check whether the target cluster has the keyspaces, tables, roles,
permissions, and user passwords that the application expects. Confirm every target
node is UN.
Resolution: recreate all keyspaces, tables, roles, permissions, and user passwords on the target before deploying the proxy.
Root cause: CDM migrates user table data only; it does not migrate system tables, so authentication credentials and schema are not copied automatically. The proxy begins dual-writing the moment it is deployed, and a write to a target that is missing the schema or roles fails and can propagate the error back to the client.
Review CDM collection, UDT, and TTL caveats
Section titled “Review CDM collection, UDT, and TTL caveats”Symptoms: after a CDM run, some rows differ between source and target, typically rows involving collections, user-defined types, or per-cell TTLs.
Diagnostics: run CDM in validation mode and inspect the mismatch/missing reports.
The detail appears in standard output, and in the cdm_run_details table when run
tracking is enabled (trackRun). Without trackRun, standard output is the only source.
Resolution: understand how CDM handles collections, UDTs, and TTLs before the run, set the job parameters to match the data model, and use the appropriate autocorrect pass to reconcile differences (see Validation).
Root cause: the way collections, UDTs, and per-cell TTLs are read and rewritten during a bulk copy can differ from the source's exact internal representation; knowing this up front enables correct CDM configuration rather than discovering discrepancies during validation.
For assistance running CDM, contact AxonOps.
During migration
Section titled “During migration”Dual-writes start immediately
Section titled “Dual-writes start immediately”The instant the ZDM Proxy is deployed, every write is sent to both clusters. There is no "deploy now, enable later" window. This is why the target schema, roles, and credentials must exist first.
Monitor the proxy layer alongside the clusters
Section titled “Monitor the proxy layer alongside the clusters”Symptoms: the source and target clusters look healthy in cluster monitoring, yet clients report intermittent errors or latency.
Diagnostics: inspect the ZDM Proxy metrics (primary-cluster reads, dual reads, target-cluster reads) in addition to cluster metrics.
Resolution: keep proxy-layer monitoring running alongside cluster monitoring for the whole migration. The AxonOps ZDM Proxy fork exposes additional metrics for deeper visibility.
Root cause: problems in the proxy layer (for example, a degraded write path to one cluster) are invisible to cluster-level monitoring, which only sees each cluster in isolation.
Some CDM errors are unavoidable
Section titled “Some CDM errors are unavoidable”Symptoms: CDM logs retain a small set of errors even after multiple passes, even at low throughput.
Diagnostics: confirm the residual errors are within an acceptable, understood set rather than a sign of a systemic problem.
Resolution: a residual error set is acceptable when it is bounded in size, explained by known source-data conditions, and reconciled against validation counts; an unexplained or growing residual set indicates a systemic problem. When residual errors cannot be eliminated, minimise the time between the final CDM autocorrect validation pass and switching reads to the target, and monitor the CDM verification output closely during that window.
Root cause: certain data shapes and edge cases cannot be perfectly reconciled by a bulk copy; the practical mitigation is to keep the unreconciled window as short as possible.
Counter tables need explicit configuration
Section titled “Counter tables need explicit configuration”Symptoms: counter values are missing on the target after a CDM run.
Diagnostics: check whether counter tables were included in the CDM job.
Resolution: enable autocorrect.missing.counter if the application does not
delete from counter tables.
Root cause: by default CDM does not copy counter tables when missing, because counters cannot be safely re-applied in all cases; they must be opted in explicitly.
Counters do not fit the dual-write flow
Counter values missing or wrong on the target is usually a symptom of a deeper issue: counter updates are not idempotent, so the dual-write and bulk-copy pattern can double-count or drop increments that occur during the migration. Counter tables must be migrated as a separate, dedicated step rather than through the standard flow. See Counter tables need special handling in the data migration phase for the recommended approach.
Finalising migration
Section titled “Finalising migration”System reads stay on the source cluster
Section titled “System reads stay on the source cluster”Symptoms: during cutover, system/system_auth reads still appear to hit the
source (DSE) cluster even after switching application reads to the target.
Diagnostics: distinguish system table reads from user data reads in the proxy metrics.
Resolution: this is expected. Confirm all non-system reads are hitting the target, then point clients directly at the new cluster and remove the proxy. System reads move to the target once the proxy leaves the path.
Root cause: the ZDM Proxy always routes system-table reads to the source cluster for as long as it remains in the query path. See Cutover.
Run repair after CDM validation
Section titled “Run repair after CDM validation”Symptoms: immediately after a CDM run, replica consistency on the new cluster is lower than expected.
Diagnostics: check repair status/coverage on the new cluster.
Resolution: run a rolling full repair over partitioner ranges
(nodetool repair -pr --full) across all new nodes after a successful CDM
validation pass. See Validation.
Root cause: a bulk copy populates data without exercising Cassandra's normal replication/anti-entropy paths, so replicas need a repair to reach full consistency.
How AxonOps helps
Rather than driving this repair by hand across every node, AxonOps automates this. As well as workload-aware Adaptive Repair, AxonOps supports standard Scheduled Repairs, including a full repair over partitioner ranges, with progress tracking and alerting on failures. This is well suited to the post-CDM consistency pass and to ongoing repair once the migration is complete.
How AxonOps helps: monitoring across both clusters
Several of these caveats (proxy-layer issues, repair coverage, and post-copy consistency) are easiest to catch with monitoring that spans both clusters and the proxy. See AxonOps for Cassandra.