Installing the Cassandra restore tools

Restoring a whole Cassandra cluster from AxonOps backups uses tools that run in two places: an orchestrator on a bastion host, and a per-node restore tool on every target Cassandra node.

axon-restore-runner — the orchestrator. It runs on a bastion host, connects to the AxonOps API, generates a source-to-target node mapping, and submits a durable restore job to every target node over SSH.
axon-cassandra-restore — the per-node restore tool, included in the axon-agent package. The orchestrator runs it on each target node to download SSTables from remote storage.
sstable-tools — rewrites the cluster_name value in each node’s system.local. It is required only when the destination cluster name differs from the source cluster name.

Component	Installed on	Provides	Required
`axon-restore-runner`	Bastion / orchestration host	Whole-cluster restore orchestration over SSH	Always
`axon-agent`	Every target Cassandra node	`axon-cassandra-restore` and systemd job execution	Always
`sstable-tools`	Every target Cassandra node	Rewrites `system.local` `cluster_name` when the cluster is renamed	Only for a renamed destination cluster

Orchestration host (bastion)

axon-restore-runner runs on a host with SSH access to every target Cassandra node. It does not run on the Cassandra nodes themselves. The bastion requires:

the OpenSSH ssh client;
network access to the AxonOps API;
an AxonOps API key with read access to organizations, clusters, nodes, and backup history;
SSH access to every target node as root, or through passwordless sudo.

axon-restore-runner is a single self-contained binary, built with the Go standard library; it needs no Ansible or third-party runtime. It is distributed as a DEB or RPM package that installs the binary to /usr/bin/axon-restore-runner. Install the package on the bastion, or copy the binary onto the bastion directly.

Confirm the binary runs:

axon-restore-runner --help

Supply the API key through the environment rather than on the command line:

export AXONOPS_API_KEY='replace-with-your-api-key'

The base URL selects the API-key authorization scheme:

Deployment	Example URL	Authorization scheme
AxonOps Cloud	`https://dash.axonops.cloud`	`Bearer`
AxonOps Self-Hosted	`https://axonops.example.com`	`AxonApi`

API-key authentication is required. Username/password and SAML authentication are not supported by the orchestrator.

Target Cassandra nodes

Each target node runs the per-node restore and, when the cluster is renamed, the system.local rewrite. Every target node requires:

Apache Cassandra, stopped, with empty data directories at restore time;
systemd and systemd-run — the orchestrator submits each restore as a durable systemd unit;
the axon-agent package, which provides axon-cassandra-restore;
sstable-tools, when the destination cluster name differs from the source cluster name.

Install axon-agent

The per-node restore tool axon-cassandra-restore is included in the axon-agent package.

Installing on Debian / Ubuntu

sudo apt-get update
sudo apt-get install -y curl gnupg ca-certificates
curl -L https://packages.axonops.com/apt/repo-signing-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/axonops.gpg
echo "deb [signed-by=/usr/share/keyrings/axonops.gpg] https://packages.axonops.com/apt axonops-apt main" | sudo tee /etc/apt/sources.list.d/axonops-apt.list
sudo apt-get update
sudo apt-get install axon-agent

Installing on CentOS / Red Hat

sudo tee /etc/yum.repos.d/axonops-yum.repo << EOL
[axonops-yum]
name=axonops-yum
baseurl=https://packages.axonops.com/yum/
enabled=1
repo_gpgcheck=0
gpgcheck=0
EOL
sudo yum install axon-agent

After the package is installed, axon-cassandra-restore is at /usr/share/axonops/axon-cassandra-restore. Run it with --help to see the available options:

~# /usr/share/axonops/axon-cassandra-restore --help
ERROR: You must specify exactly one of --restore, --list or --verify
Usage of /usr/share/axonops/axon-cassandra-restore:
  -i, --backup-id string                UUID of the backup to restore
      --cassandra-bin-dir string        Where the Cassandra binary files are stored (e.g. /opt/cassandra/bin)
      --commitlog-end int               End time for commitlog restore as Unix timestamp in milliseconds. Defaults to current time if not specified.
      --commitlog-start int             Start time for commitlog restore as Unix timestamp in milliseconds. Defaults to snapshot time if restoring a snapshot.
      --cqlsh-options string            Options to pass to cqlsh when restoring a table schema
      --dest-table string               The name of the destination table for the restore in keyspace.table format. Requires --tables with a single source table.
      --dir-mode string                 Octal permissions for restored directories (default: 0750)
      --file-mode string                Octal permissions for restored files (default: 0640)
  -h, --help                            Show command-line help
      --legacy-scan-mode                Force using the slower legacy scan mode to find backups created by older versions of AxonOps
  -l, --list                            List backups available in remote storage
      --local-commitlog-dir string      Directory to download commitlog segments into
  -d, --local-sstable-dir string        A local directory in which to store sstables downloaded from backup storage
      --no-host-dir                     Don't create a directory for each host when restoring files. Only valid if downloading the backup for a single host.
      --org-id string                   ID of the AxonOps organisation from which the backup was created
  -r, --restore                         Restore a backup from remote storage
      --restore-commitlogs              Download commitlog segments from remote storage
      --restore-peerlist                Enable restoring the system.peers and system.peers_v2 tables (disabled by default)
      --restore-schema                  Set this when using --use-sstable-loader to restore the CQL schema for each table. Keyspaces must already exist.
      --set-owner string                Set ownership of restored files/directories (user, user.group, or user:group). Requires root.
  -e, --skip-existing-files             Don't download files that already exist in the local destination path
  -c, --source-cluster string           The name of the cluster from which to restore
  -s, --source-hosts string             Comma-separated list containing host IDs for which to restore backups
      --sstable-loader-options string   Options to pass to sstableloader when restoring a backup
      --storage-config string           JSON-formatted remote storage configuration
      --storage-config-file string      Path to a file containing JSON-formatted remote storage configuration
      --table-uuids                     Include table UUIDs as a suffix in the directory name when restoring
  -t, --tables string                   Comma-separated list of keyspace.table to restore. Defaults to all tables if omitted.
      --threads int                     Number of parallel operations to run (default: 1) (default 1)
      --use-sstable-loader              Use sstableloader to restore the backup. Requires --sstable-loader-options and --cassandra-bin-dir.
  -v, --verbose                         Show verbose output when listing backups
      --verify                          Check if all required files are available to restore a backup
      --version                         Show version information and exit

Install sstable-tools

sstable-tools is published in the AxonOps APT and YUM repositories — the same repositories used for axon-agent. Install it on every target node when the destination cluster will be renamed. With the AxonOps repository already configured (as shown above for axon-agent):

Installing on Debian / Ubuntu

sudo apt-get install sstable-tools

Installing on CentOS / Red Hat

sudo yum install sstable-tools

The package installs a single sstable-tools command at /usr/bin/sstable-tools. sstable-tools runs the installed Cassandra release’s own classes and requires a Java runtime; both the Java runtime and Cassandra’s lib directory are already present on a Cassandra node. It selects the correct release adapter from the core JAR in Cassandra’s lib directory.

sstable-tools supports Cassandra 3.11.x, 4.0.x, 4.1.x, and 5.0.x. All release adapters require org.apache.cassandra.dht.Murmur3Partitioner; SSTables produced with another partitioner are not supported.

Finding the source host IDs

The orchestrator discovers source agent IDs automatically through the AxonOps API when it generates the restore mapping, so a whole-cluster restore does not require looking them up by hand. The IDs are still useful for a direct axon-cassandra-restore invocation.

The --source-hosts IDs used by axon-cassandra-restore are the IDs AxonOps assigns to each host and do not relate to the Cassandra host ID seen in nodetool info.

The AxonOps host ID is shown by selecting the node on the Cluster Overview page of the AxonOps dashboard and reading the Agent ID field.

Storage config and storage config file options

The remote storage configuration is passed the same way to both tools: axon-restore-runner reads it from --storage-config-file, and it is the same JSON that axon-cassandra-restore accepts through --storage-config-file or --storage-config.

The --storage-config-file option was added to v1.0.95 and later of axon-agent

The axon-cassandra-restore tool has two options for passing the remote storage configuration:

reference a pre-populated JSON file with --storage-config-file
pass a JSON-formatted string to --storage-config

NOTE: Only one of the options may be used when restoring a backup.

Storage config file examples

This method suits restoring across multiple nodes in a cluster and running verification tests, since the file transfers easily across nodes and simplifies the command-line arguments. The orchestrator uses this form.

Most parameters match the settings in the AxonOps Backup GUI, including the path parameter, which matches the Base Remote Path setting.

These example files can be passed to the --storage-config-file argument.

Local filesystem

{
  "type": "local",
  "path": "/backups/cassandra"
}

Amazon S3

{
  "type": "s3",
  "path": "/axonops-cassandra-backups",
  "access_key_id": "MY_AWS_ACCESS_KEY",
  "secret_access_key": "MY_AWS_SECRET_ACCESS_KEY",
  "region": "eu-west-3"
}

Amazon S3 Using EC2 IAM Credentials

{
  "type": "s3",
  "path": "/axonops-cassandra-backups",
  "region": "eu-west-3",
  "env_auth": "true"
}

Azure Blob Storage

{
  "type": "azureblob",
  "account": "MY_AZURE_ACCOUNT_NAME",
  "key": "MY_AZURE_STORAGE_KEY"
}

Google Cloud Storage

{
  "type": "googlecloudstorage",
  "location": "us",
  "service_account_credentials": "ESCAPED_JSON_PRIVATE_KEY"
}

SSH/SFTP

{
  "type": "sftp",
  "host": "<sftp_server_hostname>",
  "port": "22",
  "path": "/backup/path",
  "user": "<sftp_username>",
  "key_file": "~/private/key/file"
}

SSH/SFTP with password

{
  "type": "sftp",
  "host": "<sftp_server_hostname>",
  "port": "22",
  "path": "/backup/path",
  "user": "<sftp_username>",
  "pass": "<sftp_password>"
}

Storage config examples

This legacy command-line option configures the type of remote storage and the access credentials in an inline format.

Below are examples of the most common storage types that can be passed to --storage-config.