Point in Time Recovery

The aim of the AxonOps Cassandra Commitlog Archiving (PITR) feature is to provide an easy-to-use graphical interface instead of requiring manual configuration on every Cassandra node in a cluster.

Features

• UI to configure commitlog archiving per data center (DC) in your cluster.
• UI that uses backups and commitlogs to select a point in time to restore your cluster state to.
• UI for viewing current commitlog archiving status.
• Local or remote storage locations to store commitlog archives.
• Retention periods for how long to keep commitlog archives.

Understanding Commitlog Archiving

In Apache Cassandra, the commitlog is a vital component of the database that records every write operation before it is applied to the data files (SSTables). This mechanism ensures durability and helps recover data in case of a node failure.

Commitlog Archiving takes this a step further by continuously saving these logs to a secure location. This process involves:

• Capturing Every Change

  Every modification to the database, including inserts, updates, and deletions, is recorded in the commitlog.

• Archiving Logs

  These logs are periodically copied to an external storage location, creating a history of all database operations.

• Ensuring Data Durability

  In the event of a hardware failure or data corruption, the archived commitlogs can be used to reconstruct the state of the database.

This archiving process is essential for maintaining data integrity, enabling disaster recovery, and supporting compliance with data retention policies.

Point-in-Time Restore (PITR)

Point-in-Time Restore (PITR) is a feature that allows restoration of a database to a specific moment in time.

PITR must be applied consistently across all nodes in the cluster. Restoring a single node to a past point while other nodes remain ahead of that point in time risks data inconsistency: the restored node may have writes replayed onto it from hints or repairs originating from the other nodes, overwriting the restored state. If only one node requires recovery, contact AxonOps to discuss the appropriate approach for the specific scenario. This is particularly useful for recovering from data corruption, accidental deletions, or other operational errors.

Here’s how PITR works in Apache Cassandra:

• Continuous Archiving

  As mentioned, commitlogs are continuously archived, creating a comprehensive record of all database operations.

• Restore Process

  When a restore is needed, the archived commitlogs are replayed from the last known good snapshot up to the desired point in time. This involves:

  • Stopping the Database: Halting operations to ensure data consistency.
  • Applying Logs: Reapplying the archived commitlogs to reconstruct the database state up to the specified timestamp.
  • Restarting Operations: Bringing the database back online, now restored to the desired point in time.

PITR is invaluable for maintaining business continuity and minimizing data loss in critical situations. It provides a granular level of control over data recovery, allowing enterprises to revert their databases to any precise moment before an issue occurred.

Commitlog Archiving and PITR in Production

Commitlog archiving and PITR are not just advanced database features; they are essential tools for enterprise-grade data management. They ensure that:

• Data Integrity

  Your data remains accurate and consistent, even in the face of failures.

• Disaster Recovery

  You can recover quickly from unforeseen disasters with minimal data loss.

• Regulatory Compliance

  You meet stringent data retention and auditing requirements.

• Operational Resilience

  You can handle accidental data modifications or deletions without significant downtime.

The Traditional Challenges of Cassandra Commitlog Archiving and Point-in-Time Restore

Typically, setting up Commitlog Archiving and Point-in-Time Restore in Cassandra is a complex and time-consuming process. It involves configuring various components, ensuring compatibility between different systems, and maintaining an intricate setup that can often be fragile and prone to errors. These configurations require in-depth knowledge and continuous monitoring to ensure everything runs smoothly.

Set up Commitlog (PITR) in a couple of easy steps here.

Setting data directory permissions

When AxonOps performs a Point-in-Time Restore it needs temporary write access to Cassandra’s data directories. How this is handled depends on how axon-agent is installed:

• Default installation — axon-agent is granted the Linux filesystem capabilities (CAP_CHOWN and CAP_FOWNER) it needs to adjust the data-directory permissions automatically. No manual action is required.
• Agent runs as the Cassandra user — axon-agent already owns the data directories, so no manual action is required.
• Hardened installation (capabilities removed) — if those capabilities have been removed and axon-agent does not run as the Cassandra OS user, the agent cannot grant itself write access. AxonOps detects this and the Prepare for Restore step of the PITR wizard shows these instructions. In this case the permissions must be set manually (see below). Restricting axon-agent Privileges covers how the capabilities are removed and what removing them costs.

When manual setup is required, grant the Cassandra OS group write access to the data directory before starting the restore. Once the restore has completed, restore ownership of the data files to the Cassandra user and revoke the group write access:

# Before the restore — grant the cassandra group write access
sudo chmod -R g+w /var/lib/cassandra/data

# After the restore — restore ownership to cassandra and revoke group write access
sudo chown -R cassandra:cassandra /var/lib/cassandra/data
sudo chmod -R g-w /var/lib/cassandra/data

The standard AxonOps installation adds the axonops user to the Cassandra group, so that axon-agent can write the restored files while group write is enabled. After the restore the chown returns ownership of those files to Cassandra.

cassandra:cassandra is the default Cassandra user and group, and /var/lib/cassandra/data is the default data directory. If you run Cassandra as a different user/group, or your nodes use a different path (or more than one path under data_file_directories), adjust the commands accordingly and run them for each directory.