Intro to Cassandra Repair
Repair in Apache Cassandra™ is still a mythical process for many engineers operating this highly scalable distributed database. The documentations are sparse, the nodetool repair command line options are difficult to understand, the theories behind repairs with Merkle trees and how that translates to the log entries, the performance impact of running the repair, etc etc and the list goes on. I feel your pain because I had to learn all these theories myself and understand the behaviours of numerous command line options and the impact they have on Cassandra clusters.
We have seen in a lot of organisations they do not bother running repairs because they don’t understand it, or running repairs impact their queries.
Many engineers operating Cassandra are either using a self-built scripted solution or Reaper – a tool initially developed by the Spotify engineering team and later maintained by The Last Pickle. We also tried using Reaper but in the end, we moved away from it due to some trickiness with it.
Reaper
- Reaper uses JMX to communicate with Cassandra – JMX is an inbound network protocol, and by default bound to localhost for security reasons. To use Reaper we need to bind the JMX port to a public IP address by public I mean a network address accessible from a Reaper server. This requires inbound access from the Reaper service and can get a little tricky with the multi-DC setup of Cassandra.
- Reaper does offer Sidecar mode – this requires a Java application to be running alongside Cassandra and we feel this approach is a little heavy.
- Reaper recommends one Reaper cluster per Cassandra cluster. We had to jump between browser tabs to manage multiple clusters…
- Reaper is not aware of the Cassandra cluster’s performance metrics and doesn’t worry about the repair process resource utilisation impacting the queries during busy periods.
- Not fully integrated with other Cassandra management tools
For these reasons we decided to build our own repair system integrated into AxonOps in order to provide the following:
- Lightweight agent.
- Single agent for metrics/logs collection, and operations such as repairs and backups.
- HTTPS outbound connection from the agent, and not inbound.
- The repair process is aware of metrics such as query latencies, blocked threads etc so the repair can dynamically slow down and speed up based on not only the node level performance but the cluster performance, as well with the goal to complete within the GC Grace Seconds.
AxonOps Adaptive Repair
We have been using AxonOps Adaptive Repair to look after many Cassandra clusters. All we do is simply switch it on and it has been looking after the repairs for the users of AxonOps without any hassle!
Architecture
AxonOps agent-server communication is implemented using the HTTPS protocol, outbound from the lightweight AxonOps agent installed on the Cassandra nodes. We have made the agent extremely lightweight, almost invisible to the servers for the metrics collection.
The HTTPS outbound connection is persisted and bi-directional. This single socket connection is used to send the payloads for performance metrics, configurations, logs and events, and configurations to the AxonOps service. The AxonOps service sends control signals down to the agent using the same HTTPS connections, simplifying the network architecture.
Adaptive Repair Logic
The AxonOps repair logic resides on the server side in the AxonOps architecture. The logic constantly monitors the performance metrics that are available and adjusts the rate at which the repairs are carried out in order to minimise the impact of repairs on the Cassandra queries.
The adaptive repair is based on a sophisticated control equation we have come up with, and this is based on our experience of managing many Cassandra clusters from small to very large.
The logic determines the gc_grace_seconds configured for each table, and prepares repair plans based on these values. The Adaptive Repair logic generates a list of sub-ranges for the entire cluster, and repairs those segments over time period scheduled at the planning stage.
The rate at which the token sub-ranges are repaired slow down if it detects pending and blocked threads increasing, increased latencies, high CPU, high IOWait etc during the busy periods, but speeds back up again when the resources become available to do so in order to complete the repair schedule within the gc_grace_seconds time.
Adaptive Repair Options
We wanted to make Cassandra repair extremely simple even my ma and pap could be expert Cassandra operators!
In the Adaptive Repair section of AxonOps UI you will see the following:
All you need to do is to switch on the Adaptive Repairs to Active and that’s it!
However, if you would like a bit more control over the Adaptive Repairs then you can expand the advanced settings section and you will see the options below:
Adaptive Repair will repair all of your tables that are relevant and automatically excludes TWCS tables. However, you can specify any other tables from the pulldown list of tables by blacklisting them.
You can also control the concurrency depending on the number of tables you may have. Leave it as default if you’re not sure.
Most Cassandra clusters these days are using vnodes these days but the number of vnodes you have is determined by you. The sub-range repair size by default is the vnode range. However, you can opt to make the segments much smaller based on the number of vnodes and Cassandra nodes you have, by increasing this number.
GC Grace Seconds is 10 days by default but you can opt to change this. AxonOps Adaptive Repair will adhere the velocity of repair to the GC Grace Seconds for you!
Multi-DC Cassandra Deployment
You can connect multiple Cassandra datacenters to AxonOps to manage your Cassandra repairs. All you need are the agents installed on your Cassandra servers and pointing them at the AxonOps service. Simple! AxonOps Adaptive Repair will repair your tables across all of your replicas across all datacenters.
Get Started with Cassandra Repair Using AxonOps
AxonOps is free for up to 5 nodes and includes the Adaptive Repair functionality so why don’t you give it a go? It’ll make your life easier and let you focus your time on the real work.
