Cassandra Thread Pools
Cassandra divides the work a node does into stages and runs each stage on its own thread pool. Each pool reports the tasks it is running, the tasks queued behind them, and the tasks it has completed. Saturation normally shows as a queue on one pool before it becomes a client-visible timeout, and the pool that is queueing identifies which part of the node is behind.
A stage is one category of work, such as executing a local read. A thread pool is the fixed set of worker threads that runs the tasks of one stage. In front of each pool sits a queue, holding the tasks that are waiting for a thread. An executor is the object that owns a pool and its queue. Not every executor serves a stage: Cassandra builds executors for compaction, hint delivery and other work that is not a stage, and those sit alongside the stages wherever they report metrics.
Staged Event-Driven Architecture covers why work is divided this way, and what a node does with the work a saturated stage cannot reach in time.
A node reports its pools on three surfaces, each one a way of reading the same pool state:
nodetool tpstats, a subcommand ofnodetool, the command-line administration tool shipped with Cassandra.system_views.thread_pools, a virtual table: a read-only table that exposes node internals to CQL, the Cassandra Query Language, without storing any data.- The thread pool metrics the node publishes over JMX (Java Management Extensions), the management interface the JVM exposes. The other two read those metrics, and a collector can gather them continuously from every node.
To look up a pool name seen on any surface, go to Operator-visible pools.
The first two answer what is happening on one node at the instant of the call. That answer says little on its own: a pool carrying a queue now may be normal for that cluster at that hour. The metrics surface is the same counters, captured continuously. Reading them over long periods establishes the cluster's normal: which pools habitually queue, how deep, and when.
A pool first looked at during an incident has nothing to be compared against. Collect and alert on these counters before an incident, not during one.
Reading the numbers
Section titled “Reading the numbers”All three surfaces report the same counters. nodetool tpstats and system_views.thread_pools name them differently.
nodetool tpstats column | system_views.thread_pools column | Meaning |
|---|---|---|
| Active | active_tasks | Tasks executing now |
(shown by tpstats -v) | active_tasks_limit | Maximum concurrent tasks, which is the pool size |
| Pending | pending_tasks | Tasks queued and waiting for a thread |
| Completed | completed_tasks | Tasks completed since the node started |
| Blocked | blocked_tasks | Submissions waiting for queue space at the instant of the call. Reads as zero on every pool on 5.0, 4.1 and 4.0 |
| All time blocked | blocked_tasks_all_time | Cumulative count of submissions a pool held or refused since the node started. Three pools can move it on 5.0 and 4.1, none on 4.0 |
active near the pool size with pending at zero shows a busy pool working normally. A short-lived pending count on a background pool is normal too, since work is submitted in bursts. What matters is persistence: sustained pending above zero means the pool is not keeping up with what is being handed to it, and a queue that holds across successive samples is the reading to act on. The two blocked counters do not serve that purpose on these releases, for the reasons below.
A backlog is an effect. Sustained pending narrows a bottleneck down to one type of work. The cause usually sits downstream of the pool: disk service time, garbage collection pauses, a slow peer, or another queue further along. The pool that queues first is where diagnosis starts, and seldom where the fix belongs.
What the blocked counters report
Section titled “What the blocked counters report”blocked reads as zero on every pool on 5.0, 4.1 and 4.0. Where the count moves at all, it is incremented and decremented inside one synchronous call on the submitting thread, so no sample observes it above zero, whether taken by nodetool tpstats, by a query against the virtual table, or by a metrics poll.
all time blocked only rises, and on 5.0 and 4.1 three pools can move it:
TracingStage, in its default configuration, when sustained request tracing overflows its 1000-deep queue.Sampler, in its default configuration, when sampling started bynodetool profileloadornodetool toppartitionsbacks up past 1000.Repair-Task, only whererepair_command_pool_full_strategy: rejectis set. The default,queue, uses an unbounded queue and never moves the counter.
On 4.0 the counter stays at zero on every reported pool, including those three.
On none of the three releases can ReadStage, MutationStage, CounterMutationStage, ViewMutationStage, RequestResponseStage, Native-Transport-Requests, or any compaction, validation, flush or index pool move either counter. Those are the pools watched for saturation, and they report zero blocked tasks under any load. The blocking path was removed from the shared executor before 4.0 was released, by CASSANDRA-16186, whose changelog entry reads "Remove the SEPExecutor blocking behavior" (3.0.24, 3.11.10, 4.0-beta4, 4.0). An alert defined on a blocked counter for any of those pools cannot fire.
The difference between 4.0 and the later releases is where the increment sits. On 4.0 it is inside the executor, in JMXEnabledThreadPoolExecutor.onInitialRejection, which is reached only through the default blocking rejection handler; every pool with a bounded queue replaces that handler, so the increment never runs. The rewrite of the executor framework between 4.0 and 4.1 (CASSANDRA-16925) moved the increment into ThreadPoolExecutorJMXAdapter, which wraps whatever rejection handler the pool was built with and increments before delegating to it. A pool that drops work on overflow is therefore counted as blocked on 5.0 and 4.1, which is why tracing and sampling move the counter there and not on 4.0.
A rising all time blocked on TracingStage or Sampler means that pool's queue overflowed and the trace or sample tasks were dropped. Both are best-effort work, so the trace session or the sampling output is incomplete and the read and write paths are unaffected. It reports nothing about back-pressure on requests.
What to watch for saturation instead
Section titled “What to watch for saturation instead”- Sustained
pendingon the pool, read over time and compared with the same pool on the other nodes. - Dropped messages, counted per verb in the second section of
nodetool tpstats; see Dropped messages. A message dropped past its verb timeout is the downstream evidence that a stage did not keep up. - On the client request path, the rate of
OverloadedExceptionreturned to clients, and on 5.0 and on 4.1.6 and later the queue-time overload path described under Native transport. No thread pool counter reports transport overload.
Where the numbers come from
Section titled “Where the numbers come from”nodetool tpstatsnodetool tpstats -v # adds core and maximum pool sizesSELECT name, active_tasks, active_tasks_limit, pending_tasks, completed_tasksFROM system_views.thread_pools;JMX is the metrics surface. A collector polls it on a fixed interval and keeps the result. Over JMX, each counter is an attribute of org.apache.cassandra.metrics:type=ThreadPools,path=<path>,scope=<pool>,name=<counter>. On 5.0, 4.1 and 4.0, path is request for ReadStage, MutationStage, CounterMutationStage, ViewMutationStage and RequestResponseStage, transport for the native transport pools, and internal for every other reported pool.
Diagnosing a backlog needs the shape of the queue over time and the same pool on the other nodes, which means a collector polling those attributes and keeping the series.
AxonOps collects these counters from every node and charts active, pending, completed and blocked per pool on a Thread Pools dashboard. The charts can be grouped across nodes, racks and data centres. Three metric families carry the counters: cas_ThreadPools_request, cas_ThreadPools_internal and cas_ThreadPools_transport. In each, the pool name arrives as the scope attribute and the counter as key. AxonOps Thread Pools Dashboard Metrics Mapping states which family carries which pools, and lists the attributes and the queries the dashboard uses. The Cassandra dashboards show those charts with the node's logs and events on the same time range, where the cause downstream of a queueing pool can be visible. An alert rule set on a charted counter raises a notification on a sustained pending count; Set up alert rules covers the thresholds and the duration a rule checks over.
Operator-visible pools
Section titled “Operator-visible pools”The tables below cover every pool reported in nodetool tpstats and system_views.thread_pools on 5.0, 4.1 and 4.0. They are grouped by what the pool does: request and coordination, native transport, compaction and index building, memtable flush, and a final table for the repair, hints, indexing, sampling, ring and archiving pools. A single-disk 5.0 node with no repair running reports roughly 30 rows. A node with several data directories, one per disk (JBOD, just a bunch of disks), adds a PerDiskMemtableFlushWriter_N row per data directory, and each running repair adds a row.
Legend: ✅ Present | ⚠️ Present only in later patch releases | ❌ Not present
Request and coordination stages
Section titled “Request and coordination stages”| Pool | 5.0 | 4.1 | 4.0 | Sizing (default) | Purpose, and what a backlog means |
|---|---|---|---|---|---|
ReadStage | ✅ | ✅ | ✅ | concurrent_reads (32), resizable at runtime | Executes local reads. A backlog points to disk-bound reads, wide partitions or tombstone scanning |
MutationStage | ✅ | ✅ | ✅ | concurrent_writes (32), resizable at runtime | Applies local writes, batch and hint replay, and read repair mutations. A backlog points to the commit log and memtable, the in-memory write buffer, falling behind |
CounterMutationStage | ✅ | ✅ | ✅ | concurrent_counter_writes (32), resizable at runtime | Runs the read-then-write leader step of a counter update. A backlog points to contention on hot counter partitions |
ViewMutationStage | ✅ | ✅ | ✅ | concurrent_materialized_view_writes (32), resizable at runtime | Applies materialized view updates. A backlog means view maintenance lags base table writes |
RequestResponseStage | ✅ | ✅ | ✅ | One thread per available processor, resizable at runtime | Handles replies to client-driven coordinator requests. A backlog means the coordinator cannot drain replica replies |
InternalResponseStage | ✅ | ✅ | ✅ | One thread per available processor, resizable at runtime | Carries almost nothing on these releases. The generic internal response verb it was built for was deprecated in 4.0 (CASSANDRA-15066) when each request gained its own response verb and stage, so repair replies now run on RequestResponseStage and AntiEntropyStage, and schema work on MigrationStage. A count above zero here is unexpected |
GossipStage | ✅ | ✅ | ✅ | 1 | Handles gossip SYN, ACK and ACK2 messages. Sustained pending means membership and state propagation has stalled |
MigrationStage | ✅ | ✅ | ✅ | 1 | Pulls and pushes schema, deliberately serialised. Pending means schema agreement is not progressing |
AntiEntropyStage | ✅ | ✅ | ✅ | 1 | Handles repair coordination messages. Sustained pending means those messages are waiting behind the single thread and repair on the node has slowed |
MiscStage | ✅ | ✅ | ✅ | 1 | Runs background housekeeping, including snapshots and Paxos cleanup. Pending means housekeeping is waiting behind the single thread; it is not on the read or write path |
TracingStage | ✅ | ✅ | ✅ | 1, queue of 1000 | Writes request tracing sessions. Overflow drops the trace message and does not affect the data path. On 5.0 and 4.1 an overflow also raises all time blocked |
PaxosRepairStage | ✅ | ✅ | ❌ | One thread per available processor, resizable at runtime | Runs Paxos repair tasks under Paxos v2 (Cassandra Enhancement Proposal CEP-14). The pool is active on clusters that make heavy use of lightweight transactions, and sustained pending there means Paxos repair is not keeping up |
The stages can be resized on a running node with nodetool setconcurrency, and the executor behind a stage decides what the command accepts. On 5.0, 4.1 and 4.0 there are three behaviours:
ReadStage,MutationStage,CounterMutationStage,ViewMutationStageandRequestResponseStagecome from the shared executor pool and take a maximum size only. An attempt to set a core size is rejected withIllegalArgumentException. For the first four, the new maximum is also applied to the matchingconcurrent_*setting in the running configuration.InternalResponseStage, andPaxosRepairStageon 5.0 and 4.1, are backed by a plain pooled executor that passes both sizes to the thread pool of the Java Development Kit (JDK) unchanged, so they accept a core size and a maximum size without error.GossipStage,AntiEntropyStage,MigrationStage,MiscStageandTracingStageare single-threaded and cannot be resized at all. The attempt is rejected withUnsupportedOperationException.
No runtime change survives a restart, which returns every pool to the value in cassandra.yaml.
ImmediateStage is in the stage enum on all three releases, but it runs its tasks on the calling thread, has no threads of its own, and registers no metrics. It is not a pool and does not appear as a row.
ReadRepairStage does not exist on 5.0, 4.1 or 4.0. The name appears in older documentation and in the upstream metrics list, and a dashboard that queries it reports nothing on these releases. Read repair sends READ_REPAIR_REQ mutations, which execute on MutationStage, so read repair volume appears there and has no pool of its own.
Native transport
Section titled “Native transport”| Pool | 5.0 | 4.1 | 4.0 | Sizing (default) | Purpose, and what a backlog means |
|---|---|---|---|---|---|
Native-Transport-Requests | ✅ | ✅ | ✅ | native_transport_max_threads (128) | Handles CQL requests from client connections. A backlog means the client-facing request path is saturated |
Native-Transport-Auth-Requests | ✅ | ⚠️ | ⚠️ | native_transport_max_auth_threads: 4 on 5.0, 0 on 4.1.8 and 4.0.15 | Runs authentication work, mainly password hashing, keeping it off the request pool |
The auth pool was added in 5.0 (CASSANDRA-17812), where native_transport_max_auth_threads defaults to 4 and authentication is therefore split off by default. It was backported to 4.1.8 and 4.0.15 with a default of 0 (CASSANDRA-20057); 0 means authentication runs on Native-Transport-Requests unless the setting is raised. The pool is absent from 4.1 and 4.0 patch releases before those.
In 5.0 and in 4.1.6 and later, each native transport request carries a deadline (CASSANDRA-19534). native_transport_timeout, 12000ms by default, sets the age at which a request is rejected with an OverloadedException that names queue time as the reason. The setting is not listed in the shipped cassandra.yaml and has to be added to change it. The mechanism sheds on time, not on queue length: the queue itself is not bounded, and the rejection does not move either blocked counter. Requests shed this way are visible as OverloadedException at the client, not on this pool.
Cassandra reads and writes the client sockets on a worker group of Netty, the network I/O library it uses for socket I/O. That group is separate from these pools and is not reported; see Internode messaging and Netty groups.
Compaction and index building
Section titled “Compaction and index building”| Pool | 5.0 | 4.1 | 4.0 | Sizing (default) | Purpose, and what a backlog means |
|---|---|---|---|---|---|
CompactionExecutor | ✅ | ✅ | ✅ | concurrent_compactors, computed as min(8, max(2, min(processors, data directories))) | Runs compaction, cleanup, scrub and upgradesstables. A backlog means compaction is behind the write rate. A sustained backlog raises SSTable count, which costs read latency, and leaves obsolete data on disk that cannot be reclaimed |
ValidationExecutor | ✅ | ✅ | ✅ | concurrent_validations, which in the shipped configuration takes the value of concurrent_compactors | Builds the Merkle trees, the hash summaries of a token range that repair compares between replicas to find where they differ. A backlog means repairs are slow and validation trees are holding heap |
ViewBuildExecutor | ✅ | ✅ | ✅ | concurrent_materialized_view_builders (1) | Builds and rebuilds materialized views |
CacheCleanupExecutor | ✅ | ✅ | ✅ | 1 | Cleans the key and row caches after data is dropped |
SecondaryIndexExecutor | ✅ | ❌ | ❌ | concurrent_index_builders (2) | Builds secondary and storage-attached (SAI) indexes (CASSANDRA-17781). The pool is resizable at runtime (CASSANDRA-19266) |
On 4.1 and 4.0 there is no separate index build pool: index builds run on CompactionExecutor and compete with compaction for it.
Raising concurrent_validations above concurrent_compactors takes a second step
The shipped cassandra.yaml leaves concurrent_validations at 0 on 5.0, 4.1 and 4.0, and a value below 1 is read as the value of concurrent_compactors, both at startup and when the value is set on a running node. A value above concurrent_compactors is refused unless the limit is lifted first: at startup the system property cassandra.allow_unlimited_concurrent_validations must be set to true, and on a running node the bypassConcurrentValidatorsLimit operation of the org.apache.cassandra.db:type=StorageService MBean, a managed object exposed over JMX, must be invoked, otherwise the change is rejected with IllegalArgumentException. Once the limit is lifted, validation competes with every other form of compaction for disk and heap.
Lifting the limit does not set the value. On 5.0, 4.1 and 4.0 no nodetool subcommand sets concurrent_validations, and nodetool setconcurrency does not reach ValidationExecutor, which is not a stage. The runtime handle is the setConcurrentValidators operation of the same StorageService MBean, read back through getConcurrentValidators; a value set that way is lost on restart, so concurrent_validations in cassandra.yaml must be updated for the change to survive, together with the system property where the value exceeds concurrent_compactors.
Memtable flush
Section titled “Memtable flush”| Pool | 5.0 | 4.1 | 4.0 | Sizing (default) | Purpose, and what a backlog means |
|---|---|---|---|---|---|
MemtableFlushWriter | ✅ | ✅ | ✅ | memtable_flush_writers: 2 with a single data directory, 1 per directory with more than one | Coordinates flushes. A backlog means flushes are queueing and memtable memory is under pressure. A pending count that persists on any Memtable pool warrants checking device write latency and the memtable_flush_writers setting |
PerDiskMemtableFlushWriter_N | ✅ | ✅ | ✅ | memtable_flush_writers threads, one pool per entry in data_file_directories, numbered from _0 | Performs the flush writes for one data directory |
MemtablePostFlush | ✅ | ✅ | ✅ | 1 | Runs post-flush bookkeeping, including commit log discard. A backlog means commit log segments cannot be recycled |
MemtableReclaimMemory | ✅ | ✅ | ✅ | 1 | Reclaims memtable memory once readers release it |
A node with one data directory reports only PerDiskMemtableFlushWriter_0. A JBOD node reports one per directory, and that is the point of the family: a backlog on one _N while the others stay idle identifies a single slow disk, which no aggregate flush metric shows. A node that gives its local system keyspaces their own location also reports LocalSystemKeyspacesDiskMemtableFlushWriter.
Repair, hints, indexing, sampling, ring and archiving
Section titled “Repair, hints, indexing, sampling, ring and archiving”| Pool | 5.0 | 4.1 | 4.0 | Sizing (default) | Purpose, and what a backlog means |
|---|---|---|---|---|---|
Repair-Task | ✅ | ✅ | ✅ | repair_command_pool_size, which defaults to concurrent_validations and so in the shipped configuration to concurrent_compactors | Runs one task per nodetool repair on the coordinating node |
Repair#<n> | ✅ | ✅ | ✅ | Repair job_threads (1) | Coordinates one repair command, with one pool per running command |
HintsDispatcher | ✅ | ✅ | ✅ | max_hints_delivery_threads (2) | Replays stored hints to recovered replicas. A backlog means hints are accumulating and hint files are growing |
SecondaryIndexManagement | ✅ | ✅ | ✅ | 1 | Runs secondary index lifecycle tasks |
SASI-General | ✅ | ✅ | ✅ | One thread per available processor | Flushes on-disk index files for SASI, the SSTable Attached Secondary Index |
SASI-Memtable | ✅ | ✅ | ✅ | One thread per available processor | Flushes the SASI memtable index |
Sampler | ✅ | ✅ | ✅ | 1, queue of 1000 | Samples for nodetool toppartitions and nodetool profileload. Overflow drops the sample, which is best-effort, and on 5.0 and 4.1 raises all time blocked |
PendingRangeCalculator | ✅ | ✅ | ✅ | 1, with at most one queued recalculation | Recomputes pending token ranges when the ring changes |
CommitLogArchiver | ✅ | ✅ | ✅ | 1 | Runs archive_command against commit log segments |
StatusPropagationExecutor | ✅ | ❌ | ❌ | 1 | Propagates index status across the cluster |
repair_command_pool_full_strategy decides what happens when Repair-Task is saturated: queue, the default, queues the repair command without limit, and reject fails it immediately. With reject set, a refused command raises all time blocked on 5.0 and 4.1; with the default it stays at zero. The repair pool names on a live node carry the identifier of the repair that created them, so the rows present depend on what is running.
Dropped messages
Section titled “Dropped messages”nodetool tpstats prints a second section, headed by message type, after the pool table. Every internode message carries a verb, the message type that names the work it asks for, such as MUTATION or READ. A message that waits in a stage queue beyond the timeout for its verb is dropped rather than executed, since the coordinator that sent it has already given up. The timeout comes from the matching request timeout in cassandra.yaml, for example read_request_timeout at 5s and write_request_timeout at 2s. The 4.0 names for those two are read_request_timeout_in_ms and write_request_timeout_in_ms, with the same values.
Drops on MUTATION and READ are the downstream symptom of the same saturation that shows as pending tasks on MutationStage and ReadStage, so the two sections of tpstats should be read together.
The consequence differs by verb:
- A dropped mutation leaves the coordinator without a response, and when the callback for that response expires the coordinator writes a hint for the replica, provided hinted handoff is enabled and the replica is within
max_hint_window. The mutation is replayed to that replica later, so the drop alone does not lose the write, but the replica did not hold it at the time the client was given a result. - A dropped read is not retried by the node. It is not answered at all, and the coordinator returns a read timeout to the client if too few replicas answer.
Repeated drops of either kind mean the node accepted more work than it completed within the timeout. That is a capacity or latency problem, and resizing a thread pool does not address it.
Sizing the pools
Section titled “Sizing the pools”Four request stages are sized directly in cassandra.yaml. The setting names and defaults are the same on 5.0, 4.1 and 4.0.
| Setting | Pool | Default | Resizable at runtime |
|---|---|---|---|
concurrent_reads | ReadStage | 32 | Yes |
concurrent_writes | MutationStage | 32 | Yes |
concurrent_counter_writes | CounterMutationStage | 32 | Yes |
concurrent_materialized_view_writes | ViewMutationStage | 32 | Yes |
The shipped cassandra.yaml gives the same reasoning for those four on all three releases:
- Reads are usually disk-bound, so
concurrent_readsis set at about 16 per drive, which keeps enough operations queued for the operating system and the drive to reorder them. - Writes are rarely I/O-bound, so
concurrent_writesfollows core count, at about 8 per core. - Counter writes read before they write, so they follow the read rule.
- Materialized view writes involve a read, so they should not exceed the lower of the read and write settings.
Other pools take their size from settings documented in the cassandra.yaml reference: concurrent_compactors, concurrent_validations, concurrent_materialized_view_builders, concurrent_index_builders on 5.0, memtable_flush_writers, repair_command_pool_size with repair_command_pool_full_strategy, max_hints_delivery_threads, native_transport_max_threads and native_transport_max_auth_threads.
Compaction threads take CPU and device bandwidth from foreground reads and writes, so raising concurrent_compactors trades foreground latency for compaction throughput.
These settings rarely need changing. The defaults suit most hardware, and a pool that is queueing is usually waiting on a resource that more threads will not create.
Do not raise a pool's size to clear its own backlog
Problem: Pending tasks on ReadStage are read as "not enough read threads", and concurrent_reads is raised. The service time of a read is set by the storage layer, not by the size of the pool, so more threads issue more concurrent I/O against a device that is already saturated.
Symptoms: Queue depth and service time rise on the data device, read latency percentiles get worse, garbage collection pressure increases as more reads hold objects at once, and the pending count shifts elsewhere without falling.
Instead: Establish what the pool is waiting on before resizing it. Check device latency and queue depth, garbage collection pauses, partition sizes and tombstones per read, then correct that. Raise concurrent_reads only where the devices have measured headroom, for example after adding drives.
Version differences
Section titled “Version differences”| Pool | 5.0 | 4.1 | 4.0 | Reference |
|---|---|---|---|---|
PaxosRepairStage | ✅ | ✅ | ❌ | CEP-14 |
Native-Transport-Auth-Requests | ✅ (default 4) | ⚠️ 4.1.8+ (default 0) | ⚠️ 4.0.15+ (default 0) | CASSANDRA-17812, CASSANDRA-20057 |
SecondaryIndexExecutor | ✅ | ❌ | ❌ | CASSANDRA-17781, CASSANDRA-19266 |
StatusPropagationExecutor | ✅ | ❌ | ❌ | Index status propagation |
PaxosUncommittedMerge, PaxosRepairRetries | ✅ | ✅ | ❌ | Internal, Paxos v2 |
SnapshotCleanup, DenylistCache | ✅ | ✅ | ❌ | Internal |
AutoRepair-Repair-<type>, AutoRepair-RepairRunnable-<type> | ⚠️ 5.0.8+ | ❌ | ❌ | CEP-37, CASSANDRA-21138 |
UserDefinedScriptFunctions | ❌ | ✅ | ✅ | CASSANDRA-18252 |
HadoopInput | ❌ | ✅ | ✅ | CASSANDRA-18323 |
Legend: ✅ Present | ⚠️ Present from the named patch release | ❌ Not present
Two further differences apply to visibility, and leave the set of pools unchanged:
- The
system_views.thread_poolsvirtual table was added in 4.0 (CASSANDRA-14523). On 5.0, 4.1 and 4.0 it reports the same set of pools asnodetool tpstats. - The executor framework was rewritten between 4.0 and 4.1 (CASSANDRA-16925). Pool names, sizes and the set of pools reported did not change, so a comparison of
tpstatsoutput across that boundary is valid.
All three releases dispatch work stage by stage. The proposed thread-per-core model (CASSANDRA-10989) has not shipped in any of them.
Release alone does not settle which rows a given node prints, since a pool appears only if its executor registers metrics.
Which pools a node reports
Section titled “Which pools a node reports”A pool appears on any surface only if it registers thread pool metrics with JMX, under the domain org.apache.cassandra.metrics:type=ThreadPools. Each registered pool is an MBean, and those MBeans are the metrics surface. nodetool tpstats reads them, and system_views.thread_pools lists the same registered metrics. The virtual table is present in 4.0 and later (CASSANDRA-14523); on earlier releases, nodetool tpstats and JMX are the only surfaces.
No surface reads the Stage enum. The reported set is therefore the stages plus every other executor that registers metrics. A list built from the stages alone is incomplete, and the reported set is still short of every executor in the process.
Three kinds of executor register. The request stages and the native transport pools register because they are taken from the server's shared executor pool. On 5.0 and 4.1, an executor also registers if it is built with an explicit JMX path. The equivalent on 4.0 is a set of JMX-enabled executor classes.
Other executors register nothing on any of the three releases: plain JDK executors, the debuggable executor classes, and the Netty event loop groups. All of them still run work; no surface shows it.
Internal executors that are not reported
Section titled “Internal executors that are not reported”The executors below register no thread pool metrics, so on 5.0, 4.1 and 4.0 they appear on none of the three surfaces. They are listed here because their thread names appear in thread dumps, in log lines and in profiler output, and a reader who finds one there will not find a matching row.
Legend: ✅ Present | ⚠️ Present, with the qualification stated below the table | ❌ Not present. The ⚠️ mark carries a different meaning here from the one it has in the operator-visible tables above, where it marks a pool present only in later patch releases.
| Executor | 5.0 | 4.1 | 4.0 | Sizing | Purpose |
|---|---|---|---|---|---|
ScheduledFastTasks | ✅ | ✅ | ✅ | 1 scheduled | Runs high-frequency periodic tasks |
ScheduledTasks | ✅ | ✅ | ✅ | 1 scheduled | Runs general periodic tasks |
NonPeriodicTasks | ✅ | ✅ | ✅ | 1 scheduled | Runs one-shot delayed tasks |
OptionalTasks | ✅ | ✅ | ✅ | 1 scheduled | Runs best-effort periodic work, such as saving caches; not awaited on shutdown |
GossipTasks | ✅ | ✅ | ✅ | 1 scheduled | Drives the one-second gossip round |
BatchlogTasks | ✅ | ✅ | ✅ | 1 scheduled | Replays the batchlog |
<AuthCache>Refresh | ✅ | ✅ | ✅ | 1 sequential | Refreshes the roles, permissions and credentials caches asynchronously |
loadSavedCache | ✅ | ✅ | ⚠️ | 1 sequential | Loads the saved key and row caches at startup |
CDCSizeCalculationExecutor | ✅ | ✅ | ⚠️ | 1, discards when busy | Recomputes the size of the change data capture (CDC) raw directory |
CompactionLogger | ✅ | ✅ | ⚠️ | 1 sequential | Writes compaction-*.log |
HintsWriteExecutor | ✅ | ✅ | ✅ | 1 | Writes and flushes hint buffers to hint files |
IndexSummaryManager | ✅ | ✅ | ✅ | 1 scheduled, minimum priority | Redistributes index summaries periodically |
read-hotness-tracker | ✅ | ✅ | ✅ | 1 scheduled | Persists per-SSTable read rates |
SSTableBatchOpen | ✅ | ✅ | ✅ | One thread per available processor | Opens SSTables in parallel at startup |
Callback-Map-Reaper | ✅ | ✅ | ✅ | 1 scheduled | Expires internode request callbacks |
Messaging-SynchronousWork | ✅ | ✅ | ✅ | Unbounded cached | Runs blocking work off the Netty event loops |
RepairJobTask | ✅ | ✅ | ✅ | Unbounded cached | Runs per-session repair job steps |
Repair-<sessionID> | ✅ | ✅ | ⚠️ | One thread per table in the session | Finalises an incremental repair session locally |
RepairSnapshotExecutor | ✅ | ✅ | ✅ | 1, one-hour keepalive | Takes repair snapshots |
StreamReceiveTask | ✅ | ✅ | ✅ | Unbounded | Applies received streamed data during bootstrap, repair and rebuild |
StreamingTransferTaskTimeouts | ✅ | ✅ | ✅ | 1 scheduled | Tracks timeouts for outbound stream transfers |
NettyStreaming-Outbound-<peer> | ✅ | ✅ | ✅ | Parallel transfer limit, one pool per peer | Streams files outbound |
DiagnosticSnapshot | ✅ | ✅ | ✅ | 1 sequential | Takes automatic diagnostic snapshots on a read mismatch |
BinLogArchiver | ✅ | ✅ | ✅ | 1 sequential | Archives full query log and audit log files |
Strong-Reference-Leak-Detector | ✅ | ✅ | ✅ | 1 scheduled, only with -Dcassandra.debugrefcount | Detects reference counting leaks |
JMX (JMX broadcast) | ✅ | ✅ | ✅ | 1 sequential | Broadcasts JMX progress notifications |
SASI <n>-SEARCH- | ✅ | ✅ | ✅ | Per-query concurrency factor | Iterates terms in parallel for a SASI query |
UserDefinedFunctions | ✅ | ✅ | ✅ | Bounded | Runs Java user-defined functions under the UDF sandbox |
UserDefinedScriptFunctions | ❌ | ✅ | ✅ | Bounded | Ran scripted user-defined functions, removed in 5.0 (CASSANDRA-18252) |
HadoopInput | ❌ | ✅ | ✅ | 128 | Reads Hadoop input format splits, removed in 5.0 (CASSANDRA-18323) |
PaxosUncommittedMerge | ✅ | ✅ | ❌ | 1 sequential | Merges uncommitted Paxos state under Paxos v2 |
PaxosRepairRetries | ✅ | ✅ | ❌ | 1 scheduled | Retries Paxos repair rounds |
SnapshotCleanup | ✅ | ✅ | ❌ | 1 scheduled | Expires snapshots that have a TTL |
DenylistCache | ✅ | ✅ | ❌ | 2 | Refreshes the partition denylist |
AutoRepair-Repair-<type> | ⚠️ | ❌ | ❌ | 1 scheduled per repair type | Schedules automatic repair (CEP-37) |
AutoRepair-RepairRunnable-<type> | ⚠️ | ❌ | ❌ | 1 scheduled per repair type | Runs automatic repair (CEP-37) |
The ⚠️ rows on 4.0 mark executors that exist there but are created without a name, so their threads carry default JDK names rather than the name given above; they became named, and still unreported, in 4.1. The ⚠️ rows on 5.0 are present from 5.0.8 (CASSANDRA-21138) and absent from 5.0.0 through 5.0.7.
Internode messaging and Netty groups
Section titled “Internode messaging and Netty groups”Internode and client socket I/O runs on Netty event loop groups and plain executors, none of which register thread pool metrics. No surface reports them, and their saturation is visible only in thread dumps and process-level CPU metrics.
| Group | Sizing | Purpose |
|---|---|---|
Messaging-EventLoop | Internode I/O threads, by default one per available processor | Handles internode socket I/O |
Messaging-AcceptLoop | 1 | Accepts inbound internode connections |
Streaming-EventLoop | Event threads | Handles outbound streaming |
| Native transport worker group | Netty default for the selected transport | Handles CQL socket I/O, which feeds the reported Native-Transport-Requests pool |
Listen-<address> | 1 | Listens for inbound connections; 4.0 only |
Related documentation
Section titled “Related documentation”- nodetool tpstats for the command, its options and its output format
- thread_pools virtual table for the table schema and queries against it
- Staged Event-Driven Architecture for why work is split into stages and how a node behaves when one cannot keep up
- nodetool setconcurrency for resizing a stage on a running node
- cassandra.yaml reference for the settings that size the pools
- Key Metrics for thread pool counters alongside the rest of the node metrics