Orchestrated cluster restore with axon-restore-runner

axon-restore-runner orchestrates a whole-cluster restore of an AxonOps backup onto a target cluster from a bastion host. It discovers the backup and source topology through the AxonOps API, maps each source node to a target node, and submits a durable restore job to every target node over SSH. Each job runs under systemd on its target, so a restore continues if the bastion, the orchestrator, or an SSH connection is interrupted.

Install the tools first: axon-restore-runner on the bastion, and axon-agent (and, for a renamed cluster or a datacenter/rack change, sstable-tools) on every target node. See the setup guide.

To restore a single node, restore one table into a differently named table, or perform a commit-log or point-in-time restore, use the per-node tool directly instead — see Restore individual nodes.

How it works

The diagram shows the runtime topology: the orchestrator on the bastion drives a durable restore job on every target node, and each node pulls its data from remote storage.

The operator drives that flow through the following commands:

Generate the mapping. The orchestrator discovers the source cluster, datacenter, backup, and source agent IDs from the AxonOps API and writes an editable JSON mapping.
Edit the mapping. Each source node is mapped to the correct target node.
Validate the mapping. A local check with no API or node access.
Dry run. A read-only preflight against every target node.
Submit the restore. After the confirmation prompt, a durable systemd job is submitted per node.
Monitor. The orchestrator polls over SSH, or exits and reconnects later with the status command.

An accepted restore belongs to systemd on the target. Closing the orchestrator, losing a polling connection, or rebooting the bastion does not stop the target restore.

Prerequisites

The tools installed as described in the setup guide.
An AxonOps API key exported on the bastion as AXONOPS_API_KEY.
SSH access from the bastion to every target node as root, or through passwordless sudo.
Cassandra stopped on every target node, with empty data_file_directories. The orchestrator verifies each node's data directories are empty; it does not wipe data directories and does not start Cassandra.

Emptying the target data directories is destructive

A restore replaces the contents of the target nodes. Removing the existing Cassandra data on the target is irreversible; confirm the correct cluster is being targeted before clearing any data directory. The orchestrator will not overwrite a non-empty data directory — it fails preflight instead.

Step 1: Generate the source-to-target mapping

Interactive — the command prompts for the source cluster, datacenter, backup, and destination cluster name:

export AXONOPS_API_KEY='replace-with-your-api-key'

axon-restore-runner generate-mapping \
  --axonops-url https://dash.axonops.cloud \
  --org my-organization \
  --output restore-mapping.json

Non-interactive:

axon-restore-runner generate-mapping \
  --axonops-url https://axonops.example.com \
  --org my-organization \
  --cluster production \
  --datacenter dc1 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --destination-cluster restored-production \
  --output restore-mapping.json \
  --non-interactive

The --destination-cluster value determines whether the restore keeps the source cluster name or renames it, which changes the node-side behaviour described in Same cluster name and Different cluster name below. The datacenter and rack a node restores into are set separately, per node, in the mapping — see Rewriting system.local for a rename or topology change.

Step 2: Edit and validate the mapping

The generated JSON records the source backup, the destination cluster, and one entry per node under nodes. Each node maps one source_agent_id to one destination_ip and carries the node's source and destination topology. The ... marks one further entry per source node:

{
  "format_version": 2,
  "created_at": "2026-08-04T09:15:32.000000Z",
  "source": {
    "organization": "my-organization",
    "cluster": "production",
    "datacenter": "dc1",
    "backup_id": "2c1d9aca-5312-11ee-b686-bed50b9335ec",
    "backup_tag": "scheduled-2026-08-03",
    "backup_created_at": "2026-08-03T02:00:00.000000Z"
  },
  "destination": {
    "cluster": "restored-production"
  },
  "nodes": [
    {
      "source_agent_id": "e92e95ce-0f5d-484f-bb7a-c58614b22fd2",
      "source_ip": "10.16.0.64",
      "source_name": "cass-1",
      "source_datacenter": "dc1",
      "source_rack": "rack-a",
      "destination_ip": "10.20.5.130",
      "destination_datacenter": "dc1",
      "destination_rack": "rack-a"
    }
    ...
  ]
}

destination_ip is the address the orchestrator connects to over SSH, and accepts a hostname as well as an IP address. It starts out copied from the source address, which is generally not the address of the corresponding target node, so review every destination_ip and map each source node to the correct target node. Every destination must be unique and SSH-reachable.

destination_datacenter and destination_rack start out copied from the source node's datacenter and rack. Leave them unchanged to restore the topology as it was, or set them to the target node's datacenter and rack to move the restore into a different topology. See Rewriting system.local for a rename or topology change.

Validate the edited file locally. This step does not contact the AxonOps API or the target nodes:

axon-restore-runner validate-mapping restore-mapping.json

Step 3: Dry run

Cassandra must already be stopped on every target. Run the restore with --dry-run:

axon-restore-runner restore restore-mapping.json \
  --storage-config-file /secure/path/storage-config.json \
  --ssh-user admin \
  --ssh-args='-i /secure/path/restore-key' \
  --cassandra-yaml /etc/cassandra/cassandra.yaml \
  --cassandra-lib-dir /usr/share/cassandra/lib \
  --dry-run

The dry run connects to every destination, reads that node's own cassandra.yaml, checks the destination cluster name, data directories, and topology, verifies Cassandra is stopped, and prints the planned mutations. It does not stage files, change the AxonOps identity, or start a restore.

Cassandra configuration is read per node. --cassandra-yaml names a remote path that the orchestrator reads separately from each destination, so nodes whose YAML contents or data-directory paths differ are handled by the same command. For each node the orchestrator reads cassandra.yaml, verifies cluster_name matches the mapping, parses data_file_directories, checks each directory exists and is empty, and verifies the node's snitch publishes the destination_datacenter and destination_rack in the mapping where that snitch's topology can be read. Every data_file_directories entry becomes its own --local-sstable-dir, so nodes with multiple data directories are restored across all of them.

--cassandra-lib-dir names the remote directory holding each target's Cassandra JARs. It is used only when the destination cluster is renamed or moved to a different datacenter or rack, because that is when sstable-tools rewrites system.local. It is optional: omit it and each node's installation is discovered. The examples here pass it explicitly so the whole command is visible.

Step 4: Submit the restore and monitor it

Remove --dry-run and run the same command:

axon-restore-runner restore restore-mapping.json \
  --storage-config-file /secure/path/storage-config.json \
  --ssh-user admin \
  --ssh-args='-i /secure/path/restore-key' \
  --cassandra-yaml /etc/cassandra/cassandra.yaml \
  --cassandra-lib-dir /usr/share/cassandra/lib \
  --poll-interval 5

After the final plan is displayed, type RESTORE exactly to submit. --yes skips that prompt for a controlled non-interactive workflow. For every eligible node, the orchestrator stages the job and invokes systemd-run; the launch SSH connection closes as soon as systemd accepts the job.

Without --detach, the restore command stays attached and polls progress automatically until the jobs reach a terminal state. Each polling cycle opens a fresh, short-lived SSH connection to each target, reads its state, attempt number, exit code, and latest log line, then closes the connection. --poll-interval sets the interval in seconds and defaults to 5.

The command writes a local run record before submitting jobs — by default the mapping filename with a .run.json extension, for example restore-mapping.run.json — and updates it as jobs are accepted. This file is the reconnect handle if the orchestrator is interrupted. It has mode 0600 and does not contain the storage configuration or storage credentials. Keep it.

To submit and exit as soon as every required job has been accepted by systemd, add --detach. With --node-concurrency lower than the number of targets, detached submission waits for job slots so the requested concurrency is not exceeded.

Step 5: Reconnect, monitor, and retry

Show one progress snapshot:

axon-restore-runner status restore-mapping.run.json

Keep polling every 10 seconds and show the last 10 log lines from every node:

axon-restore-runner status restore-mapping.run.json \
  --watch \
  --poll-interval 10 \
  --log-lines 10

status is read-only: it does not submit, retry, or stop jobs, and a failed polling connection does not affect the systemd-owned restore. To retry nodes that reached the failed state, correct the underlying problem and rerun the original restore command with --retry-failed. A deterministic job identifier prevents an already-running or successful job from being submitted twice; a later attempt automatically adds --skip-existing-files.

Step 6: Start Cassandra and verify

The orchestrator does not start Cassandra. Once a node's job reaches succeeded, start Cassandra on that node and confirm it joins the cluster:

service cassandra start
nodetool status

Same cluster name

When the destination cluster name matches the source cluster name, system.local already carries the correct cluster_name, so the cluster name is not rewritten. A datacenter or rack change is independent of the cluster name: if the destination datacenter or rack differs from the source, system.local is still rewritten and sstable-tools is still required on the affected nodes — see Rewriting system.local for a rename or topology change. When the cluster name, datacenter, and rack are all unchanged, sstable-tools is not used.

The AxonOps agent identity on each target is handled as follows:

If a hostId already exists on the target, the hostId and local.db are preserved and the agent is not stopped.
If the AxonOps directory is empty, the mapped source UUID is written as the hostId, owned by axonops:axonops, and the agent is started.

A target whose AxonOps directory is non-empty but has no hostId is treated as inconsistent and fails preflight rather than having unknown contents deleted. The standard identity file is /var/lib/axonops/hostId.

Because the restored cluster and the source cluster share a cluster_name, isolating them on the network is strongly recommended: firewall the source nodes away from the target nodes. The system.peers and system.peers_v2 tables are not restored by default, which avoids token-ownership conflicts, but network isolation should still be applied so the two clusters cannot gossip with each other.

Different cluster name

When the destination cluster name differs from the source cluster name, the AxonOps agent identity is handled differently: the existing hostId and local.db are removed and the agent is started so it creates a new identity for the renamed cluster.

A renamed cluster also rewrites the cluster_name cell of system.local, described together with the datacenter and rack rewrite in Rewriting system.local for a rename or topology change.

Setting the AxonOps host ID explicitly

The identity actions in Same cluster name and Different cluster name are inferred per node. --populate-host-id overrides that inference: it writes each node's mapped source_agent_id to the host ID file (--axonops-host-id-file, default /var/lib/axonops/hostId), owned by --axonops-user/--axonops-group, whatever is already on the node and whatever the cluster is called.

axon-restore-runner restore restore-mapping.json \
  --storage-config-file /secure/path/storage-config.json \
  --ssh-user admin \
  --cassandra-yaml /etc/cassandra/cassandra.yaml \
  --populate-host-id

The agent is stopped for the write and started again. local.db is left untouched, so a node that still holds one from another cluster keeps it; remove it separately if that is not intended. The chosen action is recorded in each node's immutable job manifest, so a job that is already staged keeps the action it was created with even if the option changes on a later run.

Rewriting `system.local` for a rename or topology change

A restored system.local carries the source node's identity. Cassandra refuses to start when any of three columns differs from what the target node itself declares, so each differing column is rewritten with sstable-tools after the restore finishes and before Cassandra is started. All changed columns go into one UPDATE, so system.local never holds a half-applied identity.

`system.local` column	Target node declares it from	New value from
`cluster_name`	`cluster_name` in `cassandra.yaml`	that node's own `cassandra.yaml`
`data_center`	the configured snitch	`destination_datacenter` in the mapping
`rack`	the configured snitch	`destination_rack` in the mapping

The rewrite runs when the destination cluster name differs from the source, when the destination datacenter or rack differs from the source, or both. A topology-only change — same cluster name, different datacenter or rack — still requires sstable-tools on the affected nodes. This path requires sstable-tools on every affected target node (see the setup guide); preflight verifies sstable-tools can run before any download starts, and a node where it cannot run fails preflight with no restore submitted. Nothing is rewritten when the backup restored no system.local; Cassandra then creates system.local from cassandra.yaml itself.

The rewrite is part of the durable node job: it is logged, is covered by a completion marker so a resumed job does not repeat it, and fails the job rather than the node start if it cannot complete. Because sstable-tools runs as root, the rewritten table directory is returned to the Cassandra account afterwards and read back, so a directory Cassandra could not reopen fails the job.

Destination topology in the mapping

The mapping file carries the source and destination topology of every node. source_datacenter and source_rack describe the source node and are what the restored system.local holds before any rewrite. destination_datacenter and destination_rack describe what the target node's snitch publishes, and are copied from the source values initially. Leave them unchanged to restore the topology as it was, or set them to the target node's datacenter and rack to move the restore into a different topology. All mapped nodes must share one destination_datacenter, because one source datacenter owns one set of tokens.

A mapping written by an earlier release (format_version 1) named the source topology datacenter and rack, called the target address destination_host, and carried no destination topology. It is still accepted and read under the current names as a restore that keeps the source datacenter and rack.

Confirming the destination topology against the snitch

The destination datacenter and rack are the operator's values, so preflight confirms them against the target node before writing. Where the snitch keeps its topology decides whether that is possible:

`endpoint_snitch`	Topology read from
`GossipingPropertyFileSnitch`	`cassandra-rackdc.properties`, beside `cassandra.yaml` unless `--cassandra-rackdc-file` overrides it
`SimpleSnitch`	the fixed `datacenter1`/`rack1` it publishes
`PropertyFileSnitch`, the cloud snitches, anything else, or no `--cassandra-yaml`	not read

Where the topology can be read, a node whose mapping contradicts its snitch fails preflight and names both values: rewriting system.local to a rack the snitch does not publish would leave Cassandra refusing to start, which is the failure the rewrite exists to prevent. Where it cannot be read, the mapping is taken on trust and the node's plan is marked (unverified).

Cassandra library directory and SSTable output format

sstable-tools needs the directory holding the Cassandra core JAR to select the matching release adapter. It is discovered per node — so a cluster with mixed installation layouts needs no extra flags — or set explicitly with --cassandra-lib-dir. Where discovery cannot succeed, preflight lists what it tried and asks for --cassandra-lib-dir.

The republished system.local siblings are written in Big (big) or BTI (bti) format, resolved per node in this order: --sstable-output-format if given; otherwise sstable.selected_format from that node's cassandra.yaml; otherwise the format of the restored SSTables in the table directory (a directory containing *-bti-Data.db gets bti, anything else gets big). BTI output requires Cassandra 5.0; selecting bti for an earlier release fails preflight.

For a tarball installation, --cassandra-yaml and --cassandra-lib-dir come from the same installation root, for example --cassandra-yaml /opt/apache-cassandra-5.0.8/conf/cassandra.yaml and --cassandra-lib-dir /opt/apache-cassandra-5.0.8/lib. Both flags name remote paths read from each target node, so all targets must share the layout when --cassandra-lib-dir is given explicitly; omit it to let each node be discovered separately.

Options that control the rewrite

Option	Purpose
`--sstable-tools-path PATH`	Path to the `sstable-tools` launcher. Default `/usr/bin/sstable-tools`.
`--cassandra-lib-dir PATH`	Directory holding this node's Cassandra core JAR. Discovered per node when unset.
`--sstable-tools-java-home PATH`	Java installation used by `sstable-tools`.
`--sstable-output-format big\|bti`	Format for the republished SSTables. Overrides `sstable.selected_format`.
`--cassandra-rackdc-file PATH`	`cassandra-rackdc.properties` on nodes using `GossipingPropertyFileSnitch`. Defaults to the `cassandra.yaml` directory.
`--skip-system-local-update`	Never rewrite `system.local`, and do not check the destination topology.
`--skip-system-local-cluster-name`	Compatibility alias for `--skip-system-local-update`.

Example: restoring into a renamed datacenter

Set each node's destination topology to what the target's snitch publishes. Here dc1/rack-a becomes dc-west/rack-1, and the cluster keeps its name:

{
  "source_agent_id": "e92e95ce-0f5d-484f-bb7a-c58614b22fd2",
  "source_datacenter": "dc1",
  "source_rack": "rack-a",
  "destination_ip": "10.20.5.130",
  "destination_datacenter": "dc-west",
  "destination_rack": "rack-1"
}

--cassandra-rackdc-file is only needed when the target keeps that file away from its cassandra.yaml:

axon-restore-runner restore restore-mapping.json \
  --storage-config-file /secure/path/storage-config.json \
  --ssh-user admin \
  --cassandra-yaml /etc/cassandra/cassandra.yaml \
  --cassandra-rackdc-file /etc/cassandra/conf/cassandra-rackdc.properties \
  --dry-run

Each affected node then issues one statement, for example:

UPDATE system.local USING TIMESTAMP 1785312000000000
   SET data_center = 'dc-west', rack = 'rack-1' WHERE key = 'local';

Renaming the cluster as well adds cluster_name to the same statement.

Restore filesystem ownership

The node-side restore runs as root because it creates and assigns Cassandra files. The orchestrator passes --set-owner cassandra:cassandra --table-uuids --no-host-dir to the restore tool, so Cassandra data ownership is set during download; there is no separate recursive chown. Override the account with --cassandra-user and --cassandra-group when necessary. The storage configuration is transferred over SSH to a root-only file and is not placed in command arguments, the node manifest, or the local run file; by default it is removed when the node-side job finishes.

Command and option reference

The orchestrator has four commands. The options below mirror the tool's built-in help; run axon-restore-runner <command> --help on the bastion for the same detail.

`generate-mapping`

Discovers a backup through the AxonOps v1 API and writes an editable mapping file.

Option	Purpose
`--axonops-url URL`	AxonOps URL; `https://dash.axonops.cloud` for Cloud. Required.
`--org ORGANIZATION`	AxonOps organization name or ID. Required.
`--axonops-api-key KEY`	API key. Prefer the `AXONOPS_API_KEY` environment variable.
`--cluster NAME`	Select a cluster without prompting.
`--datacenter NAME`	Select a datacenter without prompting.
`--backup-id UUID`	Select a backup without prompting.
`--destination-cluster NAME`	Target cluster name. Defaults to the source when non-interactive.
`--non-interactive`	Never prompt; requires `--cluster`, `--datacenter`, and `--backup-id`.
`--output FILE`	Mapping file to create. Default `restore-mapping.json`.
`--force`	Replace an existing output file.
`--axonops-timeout SEC`	API timeout in seconds. Default `30`.
`--axonops-insecure`	Disable TLS verification. Unsafe; use only when required.

The API key is read from AXONOPS_API_KEY, then AXONOPS_TOKEN, then AXONOPS_API_TOKEN.

`validate-mapping`

Checks a mapping file's structure, UUIDs, datacenter consistency, destination address syntax, and duplicate source IDs or destinations. Makes no API or SSH connections.

axon-restore-runner validate-mapping restore-mapping.json

`restore`

Runs preflight, submits durable systemd jobs, and reports progress. The mapping file argument and --storage-config-file FILE are required.

SSH:

Option	Purpose
`--ssh-user USER`	Remote user. Uses `sudo -n` unless USER is root.
`--ssh-args ARGS`	Extra OpenSSH arguments as one quoted string.
`--ssh-connect-timeout SEC`	Connection timeout. Default `10`.

Cassandra discovery:

Option	Purpose
`--cassandra-yaml PATH`	Remote `cassandra.yaml`. Discovers and checks `cluster_name`, `data_file_directories`, and `endpoint_snitch` per node.
`--cassandra-rackdc-file PATH`	Remote `cassandra-rackdc.properties`, read on `GossipingPropertyFileSnitch` nodes to confirm the destination topology. Defaults to the `cassandra.yaml` directory. Requires `--cassandra-yaml`.
`--cassandra-data-dir PATH`	Manual data directory; repeat for multiple. Default `/var/lib/cassandra/data`. Cannot be combined with `--cassandra-yaml`.
`--cassandra-service NAME`	systemd service name. Default `cassandra`.
`--cassandra-user USER`	Restored file owner. Default `cassandra`.
`--cassandra-group GROUP`	Restored file group. Default `cassandra`.
`--cassandra-commitlog-dir PATH`	Commitlog destination. Default `/var/lib/cassandra/commitlog`.

Restore control:

Option	Purpose
`--node-concurrency N`	Maximum simultaneous target restore jobs. Default: all mapped nodes.
`--node-parallelism N`	Compatibility alias for `--node-concurrency`.
`--restore-threads N`	Download threads on each node. Default `4`.
`--restore-commitlogs`	Also restore archived commitlogs.
`--skip-existing-files`	Permit non-empty data on a new job. A restarted durable job adds this automatically.
`--poll-interval SEC`	Progress polling interval. Default `5`.
`--retry-failed`	Submit one new attempt for failed jobs.
`--dry-run`	Preflight and print changes; mutate nothing.
`--yes`	Skip the typed `RESTORE` confirmation.
`--detach`	Exit once every required job is accepted by systemd.
`--run-file FILE`	Choose a different local run file.

Remote paths and AxonOps identity:

Option	Purpose
`--restore-tool-path PATH`	Default `/usr/share/axonops/axon-cassandra-restore`.
`--remote-jobs-dir PATH`	Persistent state and logs. Default `/var/lib/axonops/restore/jobs`.
`--keep-remote-storage-config`	Keep the root-only storage config inside each job directory.
`--axon-agent-service NAME`	Default `axon-agent`.
`--axonops-host-id-file PATH`	Default `/var/lib/axonops/hostId`.
`--axonops-local-db-file PATH`	Default `/var/lib/axonops/local.db`.
`--axonops-user USER`	Default `axonops`.
`--axonops-group GROUP`	Default `axonops`.
`--populate-host-id`	Write each node's mapped source agent ID to the host ID file, overriding the inferred identity behavior.

The system.local rewrite options are listed in Options that control the rewrite.

`status`

Reads a run file and inspects node-side state read-only.

Option	Purpose
`--watch`	Poll until no remote job is active.
`--poll-interval SEC`	Watch interval. Default `5`.
`--log-lines N`	Show N recent log lines per job. Default `0`.
`--ssh-user USER`	Override the user stored in the run file.
`--ssh-args ARGS`	Override stored OpenSSH arguments.
`--ssh-connect-timeout SEC`	Override the stored connection timeout.