Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Phase 5: Astra Post-Migration Operations

With data migrated and the application cut over, the operations that Astra previously handled automatically are now owned by the operator. On a managed service these tasks ran behind the scenes: repair was not scheduled, backups were not configured, alerting was not owned, and restarts were not performed manually. On a self-run open-source Apache Cassandra cluster, all of that becomes the operator's responsibility.

Node failure is a foreseeable operational event rather than an emergency: data is replicated across nodes, so the cluster continues to serve reads and writes at the configured consistency level while a single node is down, and replacing it is a documented procedure conducted at a planned time. AxonOps alerts when a node is unhealthy and provides tooling to drive the replacement and follow-up repair.

The operational areas below must be established and tested before the cluster is relied on in production. Each section describes what Astra handled and how to configure the equivalent on an open-source cluster.

For the wider migration context, see the migration overview.

Open-source Cassandra relies on regular anti-entropy repair to keep replicas consistent and to prevent the resurrection of deleted data once tombstones pass gc_grace_seconds. Astra ran this automatically; repair must now be scheduled explicitly. The full-repair cadence must complete every token range within gc_grace_seconds (default 10 days) to prevent tombstone resurrection. At a minimum, schedule a recurring full repair across each node's primary partitioner ranges so that every range is covered without redundant work:

Terminal window
nodetool repair -pr --full

Running -pr on every node together covers the whole token space exactly once. As the cluster grows, a workload-aware scheduler is preferable to a fixed cron job: it can adjust repair intensity to current load, split work into manageable subrange segments, and track coverage per table to confirm that every table is repaired within gc_grace_seconds. AxonOps Adaptive Repair and Scheduled Repairs automate continuous, load-aware repair with per-table coverage reporting.

Astra provided managed backups. On a self-hosted cluster, backups must be configured with a scheduled cadence and a retention policy that matches the recovery objectives. To recover to an arbitrary moment rather than only the last snapshot, enable commitlog archiving so that point-in-time restore (PITR) can replay mutations up to a chosen timestamp. Backup destinations commonly include the local filesystem, SFTP, Amazon S3, Google Cloud Storage, and Azure Blob Storage; an off-host destination is strongly recommended so a node loss does not take its backups with it.

Test restores before production reliance

A backup that has never been restored is not a verified recovery plan. A full restore (and a PITR replay) must be performed into a separate environment and the data verified before these backups are depended on in production. Validate restores on a recurring schedule.

AxonOps Backup provides scheduled, retention-managed backups with commitlog archiving for point-in-time restore across local, SFTP, and major cloud object stores.

On Astra, the service owner watched the platform and acted on problems. On a self-hosted cluster, alerting must be configured explicitly. Configure alert rules for the signals that predict trouble in Cassandra: read and write latency (including high percentiles), pending compactions, long GC pauses, disk usage and free space headroom, dropped mutations, and hint accumulation. Alerts are only useful if they reach the right people, so route notifications to the channels the team already uses, such as Slack, PagerDuty, Microsoft Teams, ServiceNow, or email, and tune thresholds to avoid fatigue. AxonOps Monitoring ships curated Cassandra alert rules and supports custom rules, with routing via AxonOps Integrations.

Configuration changes, version upgrades, and JVM tuning all require restarting nodes, and on a distributed database a rolling restart must be done one node at a time to preserve availability. Astra handled this kind of maintenance as part of the managed service. A safe, automated rolling restart checks each node's health before touching it, waits for the node to rejoin and the cluster to stabilise before moving on, tracks progress so a restart can be resumed, and allows control over node order and parallelism to respect rack and datacentre topology. AxonOps Rolling Restart automates ordered, health-gated node restarts with progress tracking.

Infrastructure metrics indicate that the cluster is up, but not that the application can actually read and write its data with acceptable performance. Schedule application-level service checks, for example custom CQL queries that exercise representative tables, assert expected results, and measure response time, then alert when a check fails or a query runs too slowly. These checks catch end-to-end problems (a degraded keyspace, an overloaded node, a schema drift) that node-level metrics alone can miss. AxonOps Service Checks schedule custom CQL and shell checks with expected results and alerting.

Use a clear decision gate before the migration is considered complete.

  • Go: every operational area above is configured and tested. Repair is scheduled and proven to cover all tables within gc_grace_seconds; backups run on schedule and a restore (including PITR) has been validated; alert rules are active and routing to the right channels; rolling restart is automated and exercised; service health checks are running. The team is trained on running these operations and on the runbooks for responding to alerts.
  • Rollback: rollback to Astra is not applicable once the cluster is operating solely on the new cluster and Astra has been decommissioned. The safety net is no longer a managed source to fall back to; it is the tested backups and restore procedure. Confirm that backups are in place and have been restored successfully before that point is reached.

AxonOps covers this full day-2 surface as a single operations layer: repair, backup and PITR, alerting, rolling restarts, and service checks. See the AxonOps for Cassandra overview.