Phase 3: Validate the Astra Migration Before Cutover

With the proxy active and dual reads enabled, the migration has reached its safest checkpoint. Writes are already flowing to both Astra DB and the target open-source Apache Cassandra cluster, data has been bulk-copied across, and reads are still served by Astra. This phase verifies that the target is correct, consistent, and at least as performant as the Astra baseline, while rollback remains available without user impact.

This phase assumes completion of Assessment and Data migration; the baseline figures captured during assessment are central to the comparisons below. Work through each section before switching reads to the target in Cutover and drivers. If any check fails, the migration is not yet committed: reads still come from Astra, so remediation carries no user impact.

1. Cluster health

Before comparing anything, confirm the target cluster itself is healthy.

• Every target node must report UN (Up/Normal) in nodetool status. A node showing DN, UJ, or UL means the cluster is not in a steady state and any comparison will be misleading.
• There must be no active alerts on the target cluster (node down, disk pressure, dropped messages, high heap usage).
• Pending compactions should be within normal levels for the hardware in use. A persistent backlog inflates read latency and skews the performance comparison.

nodetool status
nodetool compactionstats
nodetool tpstats

Resolve any health issue here first. A cluster that is still catching up on compactions or streaming will not provide a fair performance reading.

2. Performance comparison against baseline

Compare the target against the baseline captured from the Astra console during Assessment. The goal is parity: the target should land within roughly 10% of the Astra baseline. Investigate larger regressions before proceeding, because they tend to point at data model, compaction, or sizing issues that get harder to address after cutover.

Complete the table with the Astra baseline values alongside equivalent measurements taken from the target under a load profile that resembles production traffic. Tail latencies (p99) are far more sensitive to compaction state and GC behaviour than averages.

Metric	Astra baseline	Target	Status
Read latency (p50)
Read latency (p99)
Write latency (p50)
Write latency (p99)
Throughput (ops/sec)
Disk usage per node
CPU per node
GC pause (p99)
Pending compactions

A target more than roughly 10% off the Astra baseline usually points to one of a small set of causes, in rough order of likelihood: a compaction backlog still draining; repair or streaming running concurrently with the measurement; undersized or mismatched hardware versus the workload; JVM or heap settings causing GC pauses; cold caches on a freshly built cluster that warm under load; or a schema and data-model difference such as a changed compaction strategy or num_tokens. Resolve the cause before judging the result rather than accepting a regression.

3. Data consistency validation

Bulk copying moves the data, but it does not by itself prove the two sides agree. Use the validation mode of the bulk-copy tool (Cassandra Data Migrator, CDM) to identify rows that failed to copy or that differ between Astra and the target.

When validation reports discrepancies:

1. Inspect the tool's log output to understand what failed and why. The logs distinguish missing rows from mismatched rows and record the partition and clustering keys involved.
2. Re-run with the appropriate autocorrect option for the problem found:
   • autocorrect.missing writes rows that exist on Astra but are absent from the target.
   • autocorrect.mismatch overwrites rows whose values differ so the target matches Astra.
   • autocorrect.missing.counter handles counter tables, which are not copied by default. Enable it only if the application does not delete from counter tables, since reconstructing counters is only safe when the source values are authoritative.
3. Tune the job parameters (split sizes, fetch sizes, concurrency, rate limits) to the data model. Wide partitions, large rows, and high cardinality all benefit from different settings than narrow, uniform tables.

Counters are special

Counter columns are not idempotent. Re-running a counter copy can double-count if the application has issued counter updates in the interim, or if rows are deleted and re-added. Only use autocorrect.missing.counter when the source is confirmed as the single source of truth for those counters.

If data was copied with DSBulk or Spark rather than CDM, CDM's built-in row-level validation is not available, so check consistency independently:

• Compare row counts per table on both sides. With DSBulk, run dsbulk count against Astra (using the Secure Connect Bundle) and against the target, and confirm the totals match for each table.
• Spot-check specific partitions. Pick a sample of known partition keys and read them from both Astra and the target, confirming the values match at the cell level. Row counts catch missing data; spot checks catch silently mismatched values.

4. Increase replica consistency

After a successful copy and validation pass, run a rolling full repair across the target cluster. The bulk copy writes data at a chosen consistency level, but a full repair over the primary partitioner ranges guarantees every replica holds a consistent copy before the target serves production reads.

Run the command on each node in sequence. Each node's repair must complete before the next begins:

nodetool repair -pr --full

The -pr flag restricts each node to its primary ranges, so iterating over all nodes covers the full token space exactly once without redundant work. The --full flag runs a complete repair rather than an incremental one.

A workload-aware repair scheduler can perform the same task while throttling activity during peak traffic, which keeps the repair from competing with live load. AxonOps Adaptive Repair and Scheduled Repairs are workload-aware and back off during busy periods; an AxonOps ZDM Proxy fork is also available that exposes additional proxy-layer metrics beyond the standard build.

5. Application-level validation

Infrastructure checks confirm the cluster is sound; application checks confirm it behaves correctly for the workload.

• Run the application's integration test suite against the target by routing it through the proxy with dual reads enabled. This exercises the real driver, real queries, and real consistency expectations.
• Verify that the query patterns return the expected results, paying attention to anything order-dependent, paginated, or relying on secondary indexes or SAI.
• Check any behaviour that depended on Astra guardrails (the limits recorded during Assessment, such as maximum mutation size, page size, or tombstone thresholds). Confirm each one behaves acceptably on the new cluster, since the defaults may differ.
• Test failover: stop one target node and confirm that reads and writes continue at the chosen consistency level. A cluster that cannot tolerate a single node loss is not ready to take production reads.

6. Verify proxy metrics

Cluster-only monitoring cannot see what happens inside the proxy. Review the ZDM Proxy migration metrics alongside normal Cassandra metrics:

• Origin reads: reads served from Astra (the source of truth during this phase).
• Dual reads: reads sent to both Astra and the target for comparison, where enabled.
• Target reads: reads served from the target cluster.

Watch for elevated error rates, latency divergence between origin and target, or read mismatches surfaced by the proxy. These are signals that something is wrong at the proxy layer, and they are invisible when monitoring only the clusters.

Go / No-Go

Make an explicit decision before moving on.

Go when all of the following hold:

• Every check in sections 1 to 6 passes.
• Latency and throughput on the target are within roughly 10% of the Astra baseline.
• Data consistency validation reports no data loss.
• Stakeholders have signed off on the comparison and test results.

No-Go / rollback when validation fails:

• Investigate the root cause first. Do not switch reads on top of an unresolved discrepancy.
• Decide whether to remediate in place (re-copy, re-validate, repair, retune) or to roll back.
• Rolling back at this stage is low-risk: reads have not been switched, so Astra is still serving production traffic exactly as before. The only cost is the time invested.

Once a confident Go is reached, continue to Cutover and drivers.