Phase 4: Validate a Cassandra Migration
With the data migrated and dual reads enabled (CDM + ZDM path) or the new datacenter rebuilt (second-datacenter path), validate everything before switching clients to the target. The goal is to confirm the target is correct, consistent, and at least as performant as the source, while a fallback is still possible.
1. Cluster health
Section titled “1. Cluster health”Check via the monitoring dashboard:
- All nodes report
UN(Up/Normal) innodetool status. - No active alerts.
- No pending compactions beyond normal levels.
2. Performance comparison against baseline
Section titled “2. Performance comparison against baseline”Compare the target against the baselines captured in Phase 1. Target performance should be within ~10% of baseline; investigate any metric that regresses further.
| Metric | Pre-migration baseline | Post-migration | Status |
|---|---|---|---|
| Read latency (p50) | Pass/Fail | ||
| Read latency (p99) | Pass/Fail | ||
| Write latency (p50) | Pass/Fail | ||
| Write latency (p99) | Pass/Fail | ||
| Throughput (ops/sec) | Pass/Fail | ||
| Disk usage per node | Pass/Fail | ||
| CPU usage per node | Pass/Fail | ||
| Heap usage per node | Pass/Fail | ||
| GC pause (p99) | Pass/Fail | ||
| Pending compactions | Pass/Fail | ||
| Dropped mutations | Pass/Fail | ||
| Read/write timeouts | Pass/Fail |
If the target misses the baseline, check these first
A target that is more than ~10% off baseline usually points to one of a small set of causes, in rough order of likelihood:
- Compaction still catching up. A backlog of pending compactions inflates read latency. Let it drain (or tune the compaction strategy) before judging performance.
- Repair or streaming in progress. Anti-entropy and streaming compete with live traffic. Measure once the cluster is quiescent.
- Undersized or mismatched hardware. Fewer or smaller nodes, slower disks, or less RAM than the source will show up as latency and throughput regressions.
- JVM and heap settings carried over incorrectly, causing GC pauses.
- Cold caches. A freshly built cluster has empty key and chunk caches; latency improves as they warm under representative load.
- Schema or data-model differences, such as a changed compaction strategy or
num_tokens, behaving differently from the source.
3. Data consistency validation (CDM path)
Section titled “3. Data consistency validation (CDM path)”Use CDM's validation mode to investigate any failed migrations. Failed rows appear
in standard output; when run tracking is enabled (trackRun), CDM also records
them in the cdm_run_info and cdm_run_details tables created in each targeted
keyspace. Without run tracking, standard output (captured via log redirection) is
the only source of failed-row detail.
After investigating discrepancies, re-run validation with the appropriate autocorrect option to repair the target:
autocorrect.missing: write rows missing from the target.autocorrect.mismatch: overwrite mismatched rows.autocorrect.missing.counter: by default, counter tables are not copied when missing; enable this if the application does not delete from counter tables.
Tune CDM's job parameters to the data model and data load to optimise the validation pass.
4. Increase replica consistency
Section titled “4. Increase replica consistency”After a successful CDM run, replica consistency on the new cluster can be low. Establish full consistency by running a rolling full repair across all new nodes:
nodetool repair -pr --fullRun this node by node. A workload-aware repair scheduler can perform the same full, partitioner-range repair while throttling during peak traffic.
5. Application-level validation
Section titled “5. Application-level validation”- Run the application's integration test suite against the new cluster.
- Verify that all query patterns return the expected results.
- Test failover: stop one node and confirm reads and writes continue.
6. Verify proxy metrics (CDM path)
Section titled “6. Verify proxy metrics (CDM path)”When deployed, the ZDM Proxy ships migration metrics (for example, via a Prometheus
- Grafana dashboard). Review them in addition to cluster-level monitoring: the proxy layer tracks primary-cluster reads, dual reads, and target-cluster reads, and can surface degradations or failures that do not appear in cluster metrics at all.
How AxonOps helps
The AxonOps ZDM Proxy fork exposes additional metrics beyond the upstream proxy, giving deeper visibility into subtle issues that would otherwise only surface after cutover. Run proxy metrics and cluster monitoring together throughout validation. See AxonOps for Cassandra.
Go / No-Go
Section titled “Go / No-Go”Go: all validation criteria pass; latency and throughput on the target are within ~10% of baseline; no data loss is detected; stakeholders sign off.
Rollback: if validation fails on specific metrics, investigate the root cause before deciding whether to roll back or remediate. Because reads have not yet been switched to the target, rolling back at this stage is low-risk.