nodetool tpstats
Prints the task counters for every thread pool the node registers, followed by the messages the node has dropped by message type.
Synopsis
Section titled “Synopsis”nodetool [connection_options] tpstats [options]See connection options for connection options.
Description
Section titled “Description”nodetool tpstats reads the thread pool metrics the node registers over JMX and prints one row per pool, then a second section counting dropped messages by type. It reports the node it connects to, at the moment it runs. system_views.thread_pools returns the same pools over CQL; see thread_pools virtual table.
This page covers the command, its options and how to read its output. A pool name printed in that output can be looked up in Thread Pools, which records what each pool does and which of 5.0, 4.1 and 4.0 report it.
Options
Section titled “Options”| Option | Description |
|---|---|
-F, --format | Output format (json, yaml) |
-v, --verbose | Show detailed thread pool sizes (core, max, active) |
Output Format
Section titled “Output Format”Pool Name Active Pending Completed Blocked All time blockedReadStage 2 0 1234567 0 0MutationStage 4 3 9876543 0 0CounterMutationStage 0 0 1234 0 0GossipStage 0 0 567890 0 0RequestResponseStage 0 0 8765432 0 0AntiEntropyStage 0 0 123 0 0MigrationStage 0 0 12 0 0MiscStage 0 0 5678 0 0CompactionExecutor 2 0 234567 0 0MemtableFlushWriter 1 0 12345 0 0MemtablePostFlush 0 0 12345 0 0MemtableReclaimMemory 0 0 12345 0 0PendingRangeCalculator 0 0 5 0 0Sampler 0 0 0 0 0SecondaryIndexManagement 0 0 50 0 0HintsDispatcher 0 0 100 0 0Native-Transport-Requests 8 12 12345678 0 0ViewMutationStage 0 0 0 0 0ViewBuildExecutor 0 0 0 0 0
Message type Dropped Latency waiting in queue (micros) 50% 75% 95% 98% 99% MaxREAD 0 125.0 234.0 567.0 890.0 1234.0 5678.0RANGE_SLICE 0 234.0 345.0 678.0 901.0 1345.0 6789.0MUTATION 0 89.0 123.0 345.0 456.0 567.0 2345.0READ_REPAIR 0 456.0 567.0 789.0 1012.0 1234.0 7890.0COUNTER_MUTATION 0 123.0 234.0 456.0 567.0 678.0 3456.0HINT 0 45.0 67.0 123.0 178.0 234.0 1234.0REQUEST_RESPONSE 0 34.0 45.0 89.0 123.0 156.0 890.0Output Fields
Section titled “Output Fields”Thread Pool Statistics
Section titled “Thread Pool Statistics”| Field | Description |
|---|---|
| Pool Name | Name of the thread pool |
| Active | Threads currently executing tasks |
| Pending | Tasks waiting for a thread |
| Completed | Total tasks completed |
| Blocked | Submissions waiting for queue space at the instant of the call. Zero on every pool on 5.0, 4.1 and 4.0 |
| All time blocked | Submissions the pool held or refused since the node started. Only TracingStage, Sampler and Repair-Task under repair_command_pool_full_strategy: reject can move it, and only on 5.0 and 4.1 |
Message Statistics
Section titled “Message Statistics”| Field | Description |
|---|---|
| Message type | Type of inter-node message |
| Dropped | Messages dropped due to timeout |
| Latency percentiles | Time spent waiting in queue |
Reading a Row
Section titled “Reading a Row”Pool Name Active Pending Completed Blocked All time blockedReadStage 1 0 1234567 0 0MutationStage 2 0 9876543 0 0CompactionExecutor 1 3 234567 0 0Active close to the pool size with Pending at zero is a busy pool rather than a problem, and a short queue on a background pool such as CompactionExecutor is expected. A Pending count that persists across successive runs means the pool is not keeping up with what is handed to it, and is the reading to act on. The Blocked and All time blocked columns are not the saturation signal they appear to be: on 5.0, 4.1 and 4.0 both stay at zero for ReadStage, MutationStage, Native-Transport-Requests and every compaction, validation and flush pool, since the blocking path was removed from the shared executor before 4.0 was released (CASSANDRA-16186). What the blocked counters report states the three pools that can move All time blocked on 5.0 and 4.1. The counters reset when the node restarts.
tpstats -v adds the core and maximum size of each pool, which gives the denominator for the Active column.
What a backlog on a particular pool indicates, and which settings size it, is covered per pool in Thread Pools.
Dropped Messages
Section titled “Dropped Messages”The second section counts messages the node discarded rather than executed, by message type.
Message type DroppedREAD 125MUTATION 50A message is dropped when it has waited in a stage queue past the timeout of its verb, because the coordinator that sent it has already given up. Drops on READ and MUTATION are therefore the downstream symptom of the saturation that the pool section shows as pending tasks on ReadStage and MutationStage, and the two sections should be read together. Dropped mutations are recorded as hints and replayed later; dropped reads are not retried by the node. Thread Pools covers the consequences by verb.
What pushes a verb past its timeout differs by verb:
| Message type | What causes drops |
|---|---|
READ | Slow data devices, large partitions, garbage collection pauses |
MUTATION | Write overload, memtable flush pressure |
READ_REPAIR | Saturated or slow replicas, leaving the repair mutation unapplied within write_request_timeout |
REQUEST_RESPONSE | Network latency, cross data centre round trips |
Every verb the node counts, with the metric that carries it, is listed in AxonOps Dropped Messages Dashboard Metrics Mapping.
# Latency of the requests the coordinator did servenodetool proxyhistogramsExamples
Section titled “Examples”Basic Usage
Section titled “Basic Usage”nodetool tpstatsJSON Output for Parsing
Section titled “JSON Output for Parsing”nodetool tpstats -F jsonVerbose Output with Thread Pool Sizes
Section titled “Verbose Output with Thread Pool Sizes”nodetool tpstats -vMonitor Thread Pools Continuously
Section titled “Monitor Thread Pools Continuously”watch -n 2 'nodetool tpstats | head -20'Check for Queued Tasks
Section titled “Check for Queued Tasks”nodetool tpstats | awk '$3 > 0 {print}'Field 3 is the Pending column. The header row prints nothing, since awk compares its text as zero.
Compare Across Nodes
Section titled “Compare Across Nodes”#!/bin/bash# Compare thread pool stats across all nodes using SSH
for node in node1 node2 node3; do echo "=== $node ===" ssh "$node" "nodetool tpstats" | grep -E "ReadStage|MutationStage|Pending"doneThread Pool Sizing
Section titled “Thread Pool Sizing”The pools are sized in cassandra.yaml, and the four request stages can be resized on a running node with nodetool setconcurrency. The setting behind each pool is listed in Thread Pools, which also covers when resizing is the wrong response to a backlog.
Related Commands
Section titled “Related Commands”| Command | Relationship |
|---|---|
| info | General node information |
| proxyhistograms | Request latency histograms |
| tablestats | Table-level statistics |
| compactionstats | Active compaction details |
| netstats | Network/streaming statistics |
| gcstats | GC statistics |