AxonOps Table Dashboard Metrics Mapping

This document maps the metrics used in the AxonOps Table dashboard.

Dashboard Overview

The Table dashboard provides comprehensive table-level metrics including coordinator and replica statistics, latency metrics, throughput, data distribution, and performance indicators. It helps identify table-specific issues and optimization opportunities.

Metrics Mapping

Data Distribution Metrics

Dashboard Metric	Description	Attributes
cas_Table_LiveDiskSpaceUsed	Live data size per table	keyspace, scope (table), function=Count, dc, rack, host_id

Coordinator Metrics (Table-level)

Dashboard Metric	Description	Attributes
cas_Table_CoordinatorReadLatency	Read latency at coordinator level	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id
cas_Table_CoordinatorWriteLatency	Write latency at coordinator level	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id
cas_Table_CoordinatorScanLatency	Range scan latency at coordinator	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id

Replica Metrics (Local)

Dashboard Metric	Description	Attributes
cas_Table_ReadLatency	Local read latency	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id
cas_Table_WriteLatency	Local write latency	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id
cas_Table_RangeLatency	Local range query latency	keyspace, scope (table), function (percentiles/Count), axonfunction (rate), dc, rack, host_id

Partition and SSTable Metrics

Dashboard Metric	Description	Attributes
cas_Table_MeanPartitionSize	Average partition size	keyspace, scope (table), dc, rack, host_id
cas_Table_MaxPartitionSize	Maximum partition size	keyspace, scope (table), dc, rack, host_id
cas_Table_EstimatedPartitionCount	Estimated number of partitions	keyspace, scope (table), dc, rack, host_id
cas_Table_LiveSSTableCount	Number of live SSTables	keyspace, scope (table), dc, rack, host_id
cas_Table_SSTablesPerReadHistogram	SSTables accessed per read	keyspace, scope (table), function (percentiles), dc, rack, host_id

Performance Indicators

Dashboard Metric	Description	Attributes
cas_Table_TombstoneScannedHistogram	Tombstones scanned per query	keyspace, scope (table), function (percentiles), dc, rack, host_id
cas_Table_SpeculativeRetries	Speculative retry attempts	keyspace, scope (table), function=Count, axonfunction (rate), dc, rack, host_id
cas_Table_BloomFilterFalseRatio	Bloom filter false positive ratio	keyspace, scope (table), dc, rack, host_id
cas_Table_BloomFilterDiskSpaceUsed	Disk space used by bloom filters	keyspace, scope (table), dc, rack, host_id

Memory Metrics

Dashboard Metric	Description	Attributes
cas_Table_AllMemtablesHeapSize	Heap memory used by memtables	keyspace, scope (table), dc, rack, host_id
cas_Table_AllMemtablesOffHeapSize	Off-heap memory used by memtables	keyspace, scope (table), dc, rack, host_id

JVM Garbage Collection Metrics

Dashboard Metric	Description	Attributes
jvm_GarbageCollector_G1_Young_Generation	G1 young generation GC	function (CollectionTime/CollectionCount), axonfunction (rate), dc, rack, host_id
jvm_GarbageCollector_Shenandoah_Cycles	Shenandoah GC cycles	function (CollectionCount), axonfunction (rate), dc, rack, host_id
jvm_GarbageCollector_Shenandoah_Pauses	Shenandoah GC pauses	function (CollectionTime), axonfunction (rate), dc, rack, host_id
jvm_GarbageCollector_ZGC	ZGC garbage collector	function (CollectionTime/CollectionCount), axonfunction (rate), dc, rack, host_id

Query Examples

Tables Overview Section

// Table Data Size Distribution (Pie Chart)
sum by (keyspace,scope) (cas_Table_LiveDiskSpaceUsed{function='Count',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'})

// Coordinator Table Reads Distribution (Pie Chart)
sum by (keyspace,scope) (cas_Table_CoordinatorReadLatency{axonfunction='rate',dc=~'$dc',rack=~'$rack',host_id=~'$host_id',function='Count'})

// Coordinator Table Writes Distribution (Pie Chart)
sum by (keyspace, scope) (cas_Table_CoordinatorWriteLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',axonfunction='rate',function=~'Count'})

Coordinator Table Statistics

// Max Coordinator Read Latency
max(cas_Table_CoordinatorReadLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',function='$percentile',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

// Max Coordinator Write Latency
max(cas_Table_CoordinatorWriteLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',function='$percentile',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

// Max Coordinator Range Read Latency
max(cas_Table_CoordinatorScanLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',function=~'$percentile',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

// Total Coordinator Reads/sec
sum (cas_Table_CoordinatorReadLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',axonfunction='rate',function=~'Count',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

// Average Coordinator Range Reads/sec
avg(cas_Table_CoordinatorScanLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',axonfunction='rate',function=~'Count',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

// Total Coordinator Writes/sec
sum (cas_Table_CoordinatorWriteLatency{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',axonfunction='rate',function='Count',keyspace=~'$keyspace',scope=~'$scope'}) by ($groupBy,keyspace,scope)

Table Replica Statistics

// Average Replica Read Latency
avg(cas_Table_ReadLatency{scope=~'$scope',scope!='',keyspace=~'$keyspace',function='$percentile',function!='Min|Max',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

// Average Replica Range Read Latency
avg(cas_Table_RangeLatency{scope=~'$scope',scope!='',keyspace=~'$keyspace',function='$percentile',function!='Min|Max',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

// Average Replica Write Latency
avg(cas_Table_WriteLatency{scope=~'$scope',scope!='',keyspace=~'$keyspace',function='$percentile',function!='Min|Max',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

// Total Replica Reads/sec
sum(cas_Table_ReadLatency{axonfunction='rate',scope=~'$scope',scope!='',keyspace=~'$keyspace',function='Count',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

// Total Replica Range Reads/sec
sum(cas_Table_RangeLatency{axonfunction='rate',scope=~'$scope',scope!='',keyspace=~'$keyspace',function='Count',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

// Total Replica Writes/sec
sum(cas_Table_WriteLatency{axonfunction='rate',scope=~'$scope',scope!='',keyspace=~'$keyspace',function='Count',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}) by ($groupBy,keyspace,scope)

Latency Causes

// Average Mean Partition Size
avg(cas_Table_MeanPartitionSize{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',scope=~'$scope',scope!=''}) by ($groupBy,keyspace,scope)

// Total Estimated Partitions
sum(cas_Table_EstimatedPartitionCount{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',keyspace='$keyspace',scope='$scope'}) by ($groupBy,keyspace,scope)

// Max Partition Size
max(cas_Table_MaxPartitionSize{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',scope=~'$scope',scope!=''}) by ($groupBy,keyspace,scope)

// SSTables Per Read
cas_Table_SSTablesPerReadHistogram{scope=~'$scope',scope!='',keyspace=~'$keyspace',function='$percentile',function!='Min|Max',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

// Max Live SSTables
max(cas_Table_LiveSSTableCount{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',scope=~'$scope',scope!=''}) by ($groupBy,keyspace,scope)

// Tombstones Scanned
cas_Table_TombstoneScannedHistogram{scope=~'$scope',scope!='',keyspace=~'$keyspace',function='$percentile',function!='Count|Min|Max',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

// Speculative Retries
cas_Table_SpeculativeRetries{axonfunction='rate',scope=~'$scope',scope!='',keyspace=~'$keyspace',function='Count',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

// Bloom Filter False Ratio
cas_Table_BloomFilterFalseRatio{scope=~'$scope',scope!='',keyspace=~'$keyspace',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

// Max Bloom Filter Disk
max(cas_Table_BloomFilterDiskSpaceUsed{dc=~'$dc',rack=~'$rack',host_id=~'$host_id',keyspace='$keyspace',scope='$scope'}) by ($groupBy,keyspace,scope)

Memory Statistics

// Total Table Heap Memory
sum(cas_Table_AllMemtablesHeapSize{dc=~'$dc',rack='$rack',host_id=~'$host_id',keyspace='$keyspace',scope='$scope'}) by ($groupBy,keyspace,scope)

// Total Table Off-Heap Memory
sum(cas_Table_AllMemtablesOffHeapSize{dc=~'$dc',rack='$rack',host_id=~'$host_id',keyspace='$keyspace',scope='$scope'}) by ($groupBy,keyspace,scope)

// GC Duration - G1 YoungGen
jvm_GarbageCollector_G1_Young_Generation{axonfunction='rate',function='CollectionTime',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

// GC Count per sec - G1 YoungGen
jvm_GarbageCollector_G1_Young_Generation{axonfunction='rate',function='CollectionCount',dc=~'$dc',rack=~'$rack',host_id=~'$host_id'}

Panel Organization

Tables Overview

• Table Data Size % Distribution - Pie chart showing relative data size per table

• Coordinator Table Reads % Distribution - Read request distribution by table

• Coordinator Table Writes % Distribution - Write request distribution by table

Coordinator Table Statistics

• Max Coordinator Table Read Latency - Maximum read latency at coordinator

• Max Coordinator Table Write Latency - Maximum write latency at coordinator

• Max Coordinator Table Range Read Latency - Maximum range query latency

• Total Coordinator Table Reads/Sec - Read throughput at coordinator

• Average Coordinator Table Range Reads/Sec - Range query throughput

• Total Coordinator Table Writes/Sec - Write throughput at coordinator

Table Replica Statistics

• Average Replica Read Latency - Local read latency

• Average Replica Range Read Latency - Local range query latency

• Average Replica Write Latency - Local write latency

• Total Replica Reads/sec - Local read throughput

• Total Replica Table Range Reads/sec - Local range query throughput

• Total Replica Writes/sec - Local write throughput

Latency Causes

• Average Mean Table Partition Size - Average partition size indicator

• Total Estimated Table Partitions Count - Partition count per table

• Max Table Partition Size - Largest partition (hotspot indicator)

• SSTables Per Read - SSTable access efficiency

• Max Live SSTables per Table - SSTable count (compaction indicator)

• Tombstones Scanned per Table - Tombstone impact on reads

• SpeculativeRetries By Node For Table Reads - Retry attempts

• Bloom Filter False Positive Ratio - Filter efficiency

• Max Table Bloom Filter Disk - Bloom filter storage

Memory Statistics

• Total Table Heap Memory - Memtable heap usage

• Total Table Off-Heap Memory - Memtable off-heap usage

• GC duration - G1 YoungGen - Young generation GC time

• GC count per sec - G1 YoungGen - Young generation GC frequency

• GC duration - Shenandoah - Shenandoah GC time

• GC Count per sec - Shenandoah - Shenandoah GC frequency

• GC duration - ZGC - ZGC time

• GC Count per sec - ZGC - ZGC frequency

Filters

• data center (dc) - Filter by data center

• rack - Filter by rack

• node (host_id) - Filter by specific node

• groupBy - Dynamic grouping (dc, rack, host_id)

• percentile - Select latency percentile (50th, 75th, 95th, 98th, 99th, 999th)

• keyspace - Filter by keyspace

• table (scope) - Filter by table

Understanding Table Metrics

Coordinator vs Replica Metrics

• Coordinator: Metrics from the node coordinating the request

• Replica: Metrics from nodes storing the data

• Coordinator latency includes network and replica time

• Replica latency is local operation time only

Performance Indicators

Partition Size:

• Large partitions (>100MB) cause performance issues
• Monitor max size for hotspot detection
• Mean size indicates data distribution

SSTable Metrics:

• High SSTable count impacts read performance
• More SSTables = more files to check
• Indicates compaction strategy effectiveness

Tombstones:

• Deleted data markers
• High tombstone counts slow reads
• Indicates need for compaction or TTL review

Bloom Filters:

• Probabilistic data structure for SSTable lookups
• False positive ratio should be <0.1
• Higher ratios mean unnecessary SSTable reads

Speculative Retries:

• Proactive retry mechanism for slow reads
• High rates indicate inconsistent performance
• May need tuning or investigation

Best Practices

Table Design

• Partition Size: Keep <100MB, ideally <10MB

• Even Distribution: Avoid hotspots

• Appropriate TTL: Manage tombstone creation

• Compression: Choose based on workload

Performance Monitoring

Latency Percentiles:

• p50: Median performance
• p99: Tail latency
• Large p50-p99 gap indicates issues

Throughput Balance:

• Even distribution across tables
• Identify heavy tables
• Plan capacity accordingly

Resource Usage:

• Monitor memory per table
• Track GC impact
• Balance heap/off-heap usage

Troubleshooting

High Latency:

• Check partition sizes
• Review SSTable counts
• Monitor tombstones
• Verify bloom filter efficiency

Memory Issues:

• Check memtable sizes
• Monitor GC frequency
• Review flush thresholds

Throughput Problems:

• Analyze coordinator distribution
• Check speculative retries
• Review consistency levels

Units and Display

• Latency: microseconds

• Throughput: ops/sec (rps/wps)

• Size: bytes (binary units)

• Ratio: decimal/percentage

• Count: short (absolute numbers)

• GC: milliseconds/count per sec

Legend Format:

• Overview: $keyspace $scope
• Details: $groupBy - $keyspace $scope
• Node-specific: $dc - $host_id

Notes

• The scope!='' filter excludes empty table names
• function!='Min|Max' excludes extreme values for percentiles
• Coordinator metrics show client-facing performance
• Replica metrics show storage-layer performance
• GC metrics help correlate latency with JVM behavior
• Some queries use special operations like diff or specific rack filtering