Nodetool Scheduler

The Nodetool Scheduler lets you run nodetool commands across your Cassandra cluster directly from AxonOps — either immediately or on a recurring cron schedule. Instead of connecting to individual nodes over SSH, you select a command, configure how it rolls out across the cluster, and AxonOps orchestrates the execution through the axon-agents already connected to each node.

The feature is accessible via Operations > Nodetool.

Note

The Nodetool Scheduler requires a Cassandra 4.0 or later axon-agent. Clusters running 3.x agents will see a "Current agent doesn't support this feature" message. If 3.x support is required, contact AxonOps to discuss a custom agent build.

Common uses include:

• Running maintenance commands such as cleanup, compact, or clearsnapshot in a controlled, node-by-node fashion.
• Collecting diagnostic information (info, gossipinfo, status, compactionstats) on demand or on a schedule.
• Standardising routine operational tasks across the whole cluster without manual SSH access to each node.

Note

The list of available commands is served by axon-server from a per-version command catalog, selected using the Cassandra version the cluster reports. That catalog is then filtered through a server-side allowlist before it is returned to the UI, so the commands offered reflect both the cluster's Cassandra version and this curation. Destructive or operationally disruptive commands — among them assassinate, removenode, move, join, drain, and stopdaemon — are excluded from the selectable list to prevent accidental, cluster-wide execution. repair is likewise excluded; use the Adaptive Repair Service instead.

Prerequisites

The Nodetool Scheduler requires a coordinated set of component versions and one manual host-level configuration step. Both must be satisfied before the feature is usable.

Minimum component versions

Support for the scheduler was introduced across axon-server, the Go axon-agent, and the Cassandra Java agent together. The minimum versions are:

Component	Minimum version
axon-server	2.0.31
axon-agent	2.0.24
axon-cassandra5.0-agent	1.0.14
axon-cassandra4.1-agent	1.0.16
axon-cassandra4.0-agent	1.0.18

The scheduling controls are hidden until every node in the cluster advertises the nodetool-scheduler feature flag in both agent_features (reported by the Go axon-agent) and comp_feature_flags (reported by the Cassandra Java agent). A single node still running an older agent suppresses the controls for the entire cluster, so agents must be upgraded on all nodes.

Cassandra service user group membership

The Cassandra service user must be added to the `axonops` group

Nodetool tasks are passed between the Cassandra Java agent and the Go axon-agent through a SQLite disk queue at /var/lib/axonops/queues/queue.db. The agent packages create /var/lib/axonops/queues owned by axonops:axonops with mode 2770. The setgid bit ensures the queue files the JVM creates (queue.db, queue.db-wal, queue.db-shm) inherit the axonops group and remain readable and writable by both processes.

The Cassandra Java agent runs inside the Cassandra JVM, and therefore under the Cassandra service user — commonly cassandra. Package installation does not add that user to the axonops group. It must be added manually:

sudo usermod -aG axonops cassandra

Supplementary group membership is evaluated when a process starts, so Cassandra must be restarted for the change to take effect. Substitute the actual service account where Cassandra does not run as cassandra.

Membership can be confirmed with:

id cassandra

The output must list axonops among the groups.

Symptom when this prerequisite is missed

Without group membership the JVM cannot create the queue database, and the scheduler is silently non-functional: jobs are accepted and appear in the Scheduled list, but no command ever executes on any node.

The axon-agent log identifies the condition, repeating every five seconds:

waiting for database /var/lib/axonops/queues/queue.db to be created by the component agent

Where the database exists but its permissions are wrong, the agent instead logs database /var/lib/axonops/queues/queue.db exists but is not accessible.

Supported Cassandra versions

Support is not uniform across Cassandra versions:

Cassandra version	Nodetool Scheduler
3.11	❌
4.0	✅
4.1	✅
5.0	✅

Legend: ✅ Supported | ❌ Not Supported

Cassandra 3.11 is not supported. No release of the axon-cassandra3.11-agent carries the nodetool-scheduler feature flag, so the scheduling controls never appear for a 3.11 cluster.

The set of selectable commands also differs between the supported versions, because the command catalog tracks the nodetool surface of each Cassandra release. Cassandra 4.0 exposes fewer commands than 4.1 and 5.0:

• Available on 4.1 and 5.0, but not on 4.0: getauthcacheconfig, getcolumnindexsize, getdefaultrf, invalidatecredentialscache, invalidatejmxpermissionscache, invalidatenetworkpermissionscache, invalidatepermissionscache, invalidaterolescache, listpendinghints, recompress_sstables, setcolumnindexsize, and setdefaultrf.
• Available only on 5.0: dropcidrgroup, forcecompact, invalidatecidrpermissionscache, and reloadcidrgroupscache.

Execution Modes

At the top of the form you choose how the job is triggered:

• Immediate — the command runs once, as soon as you start it.
• Cron Schedule — the command runs repeatedly, according to a cron expression you define.

Both modes share the same configuration options described below; the only difference is the trigger.

Configuration

Concurrency (Parallelism)

Parallelism controls how many nodes run the command at the same time, giving you fine-grained control over the blast radius of an operation. It is defined across three nested levels:

Setting	Description
DC Parallelism	Number of data centers processed simultaneously.
Rack Parallelism (within DC)	Number of racks processed simultaneously within each active data center.
Node Parallelism (within rack)	Number of nodes processed simultaneously within each active rack.

The default value of 1 for each level produces a fully sequential rollout — one node at a time — which is the safest option for impactful commands.

To process more nodes at once, increase the relevant level. Setting a value higher than the number of items at that level effectively means "all". For example, to run the command on an entire rack at a time across the cluster:

DC Parallelism:   1
Rack Parallelism: 1
Node Parallelism: 999

Tip

For resource-intensive commands such as compact or cleanup, keep parallelism low (sequential) to avoid placing the whole cluster under load at once. For lightweight, read-only commands such as info or status, higher parallelism is usually safe.

Targeting (Data Centers, Racks, Nodes)

By default the command runs against the entire cluster. You can narrow the scope using the optional selectors on the right of the form:

• Data Centers (optional) — restrict execution to specific data centers.
• Racks (optional) — restrict execution to specific racks.
• Nodes (optional) — restrict execution to specific nodes.

When left empty, each selector defaults to All. Targeting works in combination with parallelism: the selectors decide which nodes are included, while parallelism decides how many run concurrently.

Total Execution Timeout

The Total Execution Timeout is the maximum time allowed for the entire job to complete across all targeted nodes (for example, 15m). If the job exceeds this duration it is stopped. Use it to ensure an operation stays within a maintenance window.

Sleep Time

Sleep Time — Delay between node batches is the pause AxonOps waits between each concurrency batch. Accepted values use a duration format such as 0s, 10s, 5m, or 2h.

A sleep time gives the cluster room to recover between batches — for example, allowing compactions triggered by one batch to settle before the next batch begins. The default 0s means batches run back-to-back with no pause.

Command

The Command field (mandatory) selects the nodetool command to run. Click the field to open the searchable list of available commands.

When you select a command, the form updates to show:

• The argument fields specific to that command (on the left).
• An Arguments information panel (on the right) describing each argument — its description, type, and whether it is required.

The arguments panel also includes a View documentation link for the selected command. This opens the corresponding AxonOps Cassandra nodetool reference page, where the command and its options are explained in detail. See the Nodetool command reference for the full set of documented commands.

Argument fields vary by command. For example, compact exposes options such as args, --split-output, --user-defined, --start-token, --end-token, and --partition, each annotated in the information panel so you can see its purpose and whether it is required before running the command.

Cron Schedule

When you choose Cron Schedule, a Cron Expression field becomes available. Click it to open the Cron Schedule Details builder, which helps you construct and validate the expression.

The builder lets you set each field of the schedule:

• Minute (0–59)
• Hour (0–23)
• Day of the month (1–31)
• Month (1–12) — selectable by name (JAN–DEC)
• Day of the week (0–6, where Sunday is 0 or 7) — selectable by name (SUN–SAT)

The builder validates your input and reports Invalid cron string format until a valid expression is entered. Once the schedule is valid, the job will recur automatically at the specified times.

Starting a Job

• In Immediate mode, click START NODETOOL to run the command once.
• In Cron Schedule mode, click SCHEDULE NODETOOL to register the recurring job.

Schedules and Execution History

Beneath the configuration form, two lists track your jobs:

• Scheduled — currently scheduled and in-progress jobs, including any cron schedules and immediate runs that are still executing.
• History — completed executions.

Both lists share a common set of columns:

Column	Description
Time	When the job ran or is scheduled to run.
Command	The nodetool command executed.
Status	Current state — for example Running or Done.
Duration	How long the execution took.
User	The AxonOps user who created the job.
DCs / Racks / Nodes	The targeting scope applied to the job.
Actions	Per-item actions (available in the Scheduled list).

You can filter the Command and Status columns and sort by the sortable columns to quickly find a specific job.

Execution Details and Output

Click any item in the Scheduled or History list to open a Nodetool Details modal with the full execution context.

The modal summarises:

• User, Scheduled (whether it came from a schedule), Command, and Args.
• Sleep Time and Time of execution.
• The Parallelism applied at DC, Rack, and Node levels.
• The Selected DCs, Selected racks, and Selected nodes for the job.
• A progress indicator (for example 1 / 3) and a per-node Nodes status table showing Host ID, Status (Done, Pending, etc.), Start Time, Duration, and a Message column.

Viewing command output

Each node row includes a Show output toggle. Expand it to view the raw output the command produced on that node.

This is where you read the results of diagnostic commands — for example the full gossipinfo or info output for each node — and confirm that maintenance commands completed successfully. Output can be copied directly from the modal.

Permissions

Access to the Nodetool Scheduler is controlled by AxonOps roles. Only users with the Administrator or DBA role can run or schedule nodetool commands; other roles do not have access to this feature.

Note

The axon-agent does not shell out to the nodetool command-line utility. Instead it invokes the underlying nodetool command classes directly, bypassing the JMX network port entirely.

Related

• Nodetool command reference — detailed documentation for individual nodetool commands.
• Scheduling nodetool Commands Across Clusters — the orchestration concepts behind coordinated, topology-aware execution.
• Adaptive Repair Service — automated repair scheduling and management.
• Rolling Restarts — orchestrated, parallelism-aware cluster restarts.