AxonOps Kafka ZooKeeper Dashboard Metrics Mapping

Overview

The Kafka ZooKeeper Dashboard monitors the health and performance of ZooKeeper ensemble used by Kafka for cluster coordination (in non-KRaft mode). It tracks connections, request latency, node statistics, and session management to ensure ZooKeeper is functioning properly.

Metrics Mapping

Dashboard Metric	Description	Attributes
ZooKeeper Health Metrics
zk_NumAliveConnections	Number of active client connections	port={port}
zk_NodeCount	Total number of znodes	port={port}
zk_WatchCount	Total number of watches	port={port}
zk_OutstandingRequests	Number of queued requests	port={port}
Request Latency Metrics
zk_MinRequestLatency	Minimum request latency	port={port}
zk_AvgRequestLatency	Average request latency	port={port}
zk_MaxRequestLatency	Maximum request latency	port={port}
Packet Metrics
zk_PacketsSent	Number of packets sent	port={port}
zk_PacketsReceived	Number of packets received	port={port}
Kafka-Reported ZooKeeper Metrics
kaf_ZooKeeperClientMetrics_ZooKeeperRequestLatencyMs	ZooKeeper request latency from Kafka perspective	-
kaf_SessionExpireListener_ZooKeeperExpiresPerSec	Rate of ZooKeeper session expirations	-
kaf_SessionExpireListener_ZooKeeperAuthFailuresPerSec	Rate of ZooKeeper authentication failures	-
kaf_SessionExpireListener_ZooKeeperSyncConnectsPerSec	Rate of ZooKeeper connections	-
kaf_SessionExpireListener_ZooKeeperDisconnectsPerSec	Rate of ZooKeeper disconnections	-

Query Examples

Health Check Metrics

// Alive connections
zk_NumAliveConnections{rack='$rack',host_id=~'$host_id'}

// Total znode count
sum(zk_NodeCount{host_id=~'$host_id',type='kafka',node_type='zookeeper'})

// Total watch count
sum(zk_WatchCount{host_id=~'$host_id',type='kafka',node_type='zookeeper'})

// Outstanding requests
sum(zk_OutstandingRequests{host_id=~'$host_id',type='kafka',node_type='zookeeper'})

Request Latency

// Minimum request latency
zk_MinRequestLatency{host_id=~'$host_id',type='kafka',node_type='zookeeper'}

// Average request latency
zk_AvgRequestLatency{host_id=~'$host_id',node_type='zookeeper',type='kafka'}

// Maximum request latency
zk_MaxRequestLatency{host_id=~'$host_id',node_type='zookeeper',type='kafka'}

// Kafka-reported ZooKeeper latency
kaf_ZooKeeperClientMetrics_ZooKeeperRequestLatencyMs{rack=~'$rack',host_id=~'$host_id'}

Traffic Metrics

// Packets sent rate
sum(zk_PacketsSent{host_id=~'$host_id', axonfunction='rate', type='kafka',node_type='zookeeper'})

// Packets received rate
sum(zk_PacketsReceived{host_id=~'$host_id', axonfunction='rate', type='kafka',node_type='zookeeper'})

// Znode creation rate
avg(zk_NodeCount{host_id=~'$host_id', axonfunction='rate',type='kafka',node_type='zookeeper'})

Connection Management

// Session expiration rate
kaf_SessionExpireListener_ZooKeeperExpiresPerSec{axonfunction='rate',rack=~'$rack',host_id=~'$host_id'}

// Authentication failure rate
kaf_SessionExpireListener_ZooKeeperAuthFailuresPerSec{axonfunction='rate',rack=~'$rack',host_id=~'$host_id'}

// Connection rate
kaf_SessionExpireListener_ZooKeeperSyncConnectsPerSec{axonfunction='rate',rack=~'$rack',host_id=~'$host_id'}

// Disconnection rate
kaf_SessionExpireListener_ZooKeeperDisconnectsPerSec{axonfunction='rate',rack=~'$rack',host_id=~'$host_id'}

Panel Organization

Overview Section

• Empty row for spacing/organization

Health Check

• Alive Connections
• Outstanding Requests
• Number of Watchers
• Number of ZNodes

Request Latency

• Packets (sent/received rates)
• Znode Creation Rate
• Request Latency - Minimum
• Request Latency - Average
• Request Latency - Maximum
• Kafka Reported Request Latency

Connections

• Zookeeper expired connections per sec
• Zookeeper auth failures per sec
• Zookeeper disconnect per sec
• Zookeeper connections per sec

Filters

• host_id: Filter by specific ZooKeeper node

• rack: Filter by rack location

Best Practices

Health Monitoring

• Monitor alive connections for capacity planning
• Outstanding requests should remain low
• High watch count may impact performance
• Monitor znode count growth

Latency Analysis

• Average latency should be below tickTime
• High max latency indicates potential issues
• Compare ZK-reported vs Kafka-reported latency

Connection Management

• Monitor session expirations for client issues
• Auth failures indicate security problems
• High disconnect rate suggests network issues

Performance Tuning

• Adjust tickTime based on latency requirements
• Monitor packet rates for network saturation
• Balance connections across ensemble members

Troubleshooting

• High outstanding requests: Check ZK performance
• Session expirations: Review session timeout settings
• Auth failures: Check SASL/ACL configurations

Capacity Planning

• Monitor znode growth rate
• Track connection count trends
• Plan for watch count scaling

ZooKeeper Ensemble Health

• Ensure all ensemble members are responsive
• Monitor for leader elections
• Check fsync latency on ZK data directory