Restore individual nodes with axon-cassandra-restore

axon-cassandra-restore, included in the axon-agent package, restores an AxonOps backup directly on a single node without the orchestrator. It connects to remote storage (for example S3 or GCS) and does not require an AxonOps server or an active AxonOps Cloud account to run.

Use it directly to:

• restore a single node from a snapshot;
• restore one table into a differently named table or keyspace;
• restore to the latest archived state, or to a specific point in time, by replaying commit logs.

For a coordinated whole-cluster restore that maps every source node to a target node and runs each restore as a durable job, use Orchestrated cluster restore instead.

Install axon-agent on the node first, and prepare the remote storage configuration. See the setup guide.

Listing available backups

List the backups in the remote storage bucket with axon-cassandra-restore --list after preparing a --storage-config-file (see the setup guide):

/usr/share/axonops/axon-cassandra-restore \
    --storage-config-file /path/to/remote_storage_config_file.json \
    --org-id myaxonopsorg \
    --list

The tool scans the bucket for AxonOps backups and displays the date and backup ID of each:

Org ID:    myaxonopsorg
Cluster:   testcluster
Time                   Backup ID
2023-09-14 14:30 UTC   c67cea2a-5310-11ee-b686-bed50b9335ec
2023-09-15 14:31 UTC   2c1d9aca-5312-11ee-b686-bed50b9335ec
2023-09-16 14:30 UTC   91be5007-5313-11ee-b686-bed50b9335ec

Adding --verbose lists the nodes and tables in each backup. Scanning can take a long time depending on the storage type and data volume. The search can be restricted to a specific cluster, set of hosts, backup, and tables:

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts 026346a0-dc89-4235-ae34-552fcd453b42 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --tables keyspace1.table1,keyspace1.table2 \
  --verbose \
  --list

The --source-hosts values are AxonOps Agent IDs. Finding them is covered in the setup guide.

Restore a node from a snapshot

A snapshot restore recovers the data captured when the backup was taken. No commit logs are involved.

Download the backup to a local directory

This command downloads the backup for one host into a local directory:

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts 026346a0-dc89-4235-ae34-552fcd453b42 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --local-sstable-dir /opt/cassandra/axonops-restore \
  --restore

The SSTables are written into directories named {local-sstable-dir}/{host-id}/keyspace/table/. From there they can be copied elsewhere or imported into a cluster with sstableloader.

Download and import with sstableloader

Passing --use-sstable-loader, --cassandra-bin-dir, and --sstable-loader-options imports the downloaded files into a target cluster. The command below imports into a cluster with contact points 10.0.0.1, 10.0.0.2, and 10.0.0.3:

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts 026346a0-dc89-4235-ae34-552fcd453b42 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --local-sstable-dir /opt/cassandra/axonops-restore \
  --restore \
  --use-sstable-loader \
  --cassandra-bin-dir /opt/cassandra/bin \
  --sstable-loader-options "-d 10.0.0.1,10.0.0.2,10.0.0.3 -u cassandra -pw cassandra"

NOTE: For a Cassandra installed from the Debian or Red Hat package, use --cassandra-bin-dir /usr/bin.

sstableloader assumes the destination tables already exist and skips any that are undefined. AxonOps stores the table schema with each backup, so adding --restore-schema and --cqlsh-options creates any missing tables during the restore. The destination keyspaces must already exist.

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts 026346a0-dc89-4235-ae34-552fcd453b42 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --local-sstable-dir /opt/cassandra/axonops-restore \
  --restore \
  --use-sstable-loader \
  --cassandra-bin-dir /opt/cassandra/bin \
  --sstable-loader-options "-d 10.0.0.1,10.0.0.2,10.0.0.3 -u cassandra -pw cassandra" \
  --restore-schema \
  --cqlsh-options "-u cassandra -p cassandra 10.0.0.1"

Restore to a different table

This feature is available in AxonOps Agent v1.0.61 or later.

When restoring a single table, --dest-table loads the restored data into a table with a different name or keyspace. Adding --restore-schema creates the new table as part of the restore. The destination keyspace must already exist.

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts 026346a0-dc89-4235-ae34-552fcd453b42 \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --local-sstable-dir /opt/cassandra/axonops-restore \
  --restore \
  --use-sstable-loader \
  --cassandra-bin-dir /opt/cassandra/bin \
  --sstable-loader-options "-d 10.0.0.1,10.0.0.2,10.0.0.3 -u cassandra -pw cassandra" \
  --restore-schema \
  --cqlsh-options "-u cassandra -p cassandra 10.0.0.1" \
  --tables keyspace1.table1 \
  --dest-table restoreks.table1_restored

Restore to latest state or a point in time

A commit-log restore replays the commit logs archived since a snapshot, either to the latest archived state or to a specific timestamp (point-in-time restore, PITR). Both require:

• AxonOps Agent 2.0.12 or later on all source cluster nodes.
• A backup that includes all system tables as well as the target keyspaces and tables.
• The Base Remote Path for both Backup and Commitlog Archiving to be identical, because the storage configuration has a single path parameter.

Step 1: Prepare the target node

Shut down Cassandra on the target and remove all on-disk data:

nodetool flush
nodetool drain
service cassandra stop

# CAUTION: this deletes all Cassandra data on the TARGET node
cd /var/lib/cassandra/
rm -rf commitlog/* data/* hints/* saved_caches/*

Set cluster_name in /etc/cassandra/cassandra.yaml to match the source cluster backups.

Shared cluster name

When the source and target clusters share a cluster_name, isolate the target from the source on the network. The system.peers and system.peers_v2 tables are not restored, which avoids token-ownership conflicts, but network isolation is still required so the clusters cannot gossip.

Step 2: Download the snapshot and commit logs

Run once per node, substituting that node's Agent ID.

For a restore to the latest state, download commit logs directly into Cassandra's active commitlog directory; all segments will be replayed:

AGENT_ID=026346a0-dc89-4235-ae34-552fcd453b42

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts "${AGENT_ID}" \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --restore \
  --threads 20 \
  --table-uuids \
  --set-owner cassandra \
  --no-host-dir \
  --local-sstable-dir /var/lib/cassandra/data \
  --restore-commitlogs \
  --local-commitlog-dir /var/lib/cassandra/commitlog

For a point-in-time restore, add --commitlog-end (a Unix timestamp in milliseconds) and download commit logs to a directory separate from Cassandra's active commitlog directory. The tool downloads only the segments needed to reach the target timestamp:

AGENT_ID=026346a0-dc89-4235-ae34-552fcd453b42
TARGET_TS="1705318800000"  # UNIX timestamp in milliseconds

/usr/share/axonops/axon-cassandra-restore \
  --storage-config-file /path/to/remote_storage_config_file.json \
  --org-id myaxonopsorg \
  --source-cluster testcluster \
  --source-hosts "${AGENT_ID}" \
  --backup-id 2c1d9aca-5312-11ee-b686-bed50b9335ec \
  --restore \
  --threads 20 \
  --table-uuids \
  --set-owner cassandra \
  --no-host-dir \
  --local-sstable-dir /var/lib/cassandra/data \
  --restore-commitlogs \
  --local-commitlog-dir /var/lib/cassandra/tmp/commitlog \
  --commitlog-end "${TARGET_TS}"

Step 3: Configure commitlog_archiving.properties (point-in-time restore)

For a point-in-time restore, edit /etc/cassandra/commitlog_archiving.properties on each node so Cassandra stops replaying at the target point:

restore_directories=/var/lib/cassandra/tmp/commitlog
restore_command=cp -f %from %to
restore_point_in_time=2024:01:15 14:00:00

restore_point_in_time must be in GMT. Accepted precisions:

Precision	Format	Example
Seconds	yyyy:MM:dd HH:mm:ss	2024:01:15 14:00:00
Milliseconds	yyyy:MM:dd HH:mm:ss.SSS	2024:01:15 14:00:00.633
Microseconds	yyyy:MM:dd HH:mm:ss.SSSSSS	2024:01:15 14:00:00.633222

Timestamp separator

The date separator is a colon (:), not a dash. An incorrect format such as 2024-01-15 14:00:00 causes Cassandra to fail to start with Unable to parse restore target time. This applies to Cassandra 4.0, 4.1, and 5.0.

Step 4: Start Cassandra

service cassandra start

For a restore to latest state, Cassandra replays all mutations from the commit log segments in its active commitlog directory. For a point-in-time restore, Cassandra reads commitlog_archiving.properties, copies each archived segment in with restore_command, and replays mutations up to restore_point_in_time.

Step 5: Clear the restore configuration (point-in-time restore)

After a point-in-time restore completes and the cluster is healthy, clear the restore properties and perform a rolling restart so commit log archiving resumes. On each node, empty the restore settings:

restore_directories=
restore_command=
restore_point_in_time=

Then restart each node in sequence, waiting for each to rejoin before moving on:

nodetool drain
service cassandra stop
service cassandra start
nodetool status

Cassandra requires the restore mode to be cleared before normal backup operation resumes.