AxonOps Kafka Replication Dashboard Metrics Mapping

Overview

The Kafka Replication Dashboard provides comprehensive monitoring of Kafka’s data replication health and performance. It tracks partition states, ISR (In-Sync Replica) changes, and request latencies for both leaders and followers to ensure data durability and availability.

Metrics Mapping

Dashboard Metric	Description	Attributes
Partition State Metrics
`kaf_ReplicaManager_UnderReplicatedPartitions`	Number of under-replicated partitions	-
`kaf_KafkaController_OfflinePartitionsCount`	Number of offline partitions	-
`kaf_ReplicaManager_PartitionCount`	Total number of partitions on the broker	-
`kaf_ReplicaManager_UnderMinIsrPartitionCount`	Partitions with ISR count below min.insync.replicas	-
ISR Change Metrics
`kaf_ReplicaManager_IsrShrinksPerSec`	Rate of ISR shrinks per second	-
`kaf_ReplicaManager_IsrExpandsPerSec`	Rate of ISR expansions per second	-
Leader Request Metrics
`kaf_RequestMetrics_LocalTimeMs` (request=‘FetchFollower’)	Time leader spends processing follower fetch requests	request=FetchFollower
`kaf_RequestMetrics_LocalTimeMs` (request=‘Fetch’)	Time leader spends processing consumer fetch requests	request=Fetch
`kaf_RequestMetrics_LocalTimeMs` (request=‘FetchConsumer’)	Time leader spends processing consumer fetch requests	request=FetchConsumer
`kaf_RequestMetrics_LocalTimeMs` (request=‘Produce’)	Time leader spends processing produce requests	request=Produce
Follower Request Metrics
`kaf_RequestMetrics_RemoteTimeMs` (request=‘Produce’)	Time follower waits for produce replication	request=Produce
`kaf_RequestMetrics_RemoteTimeMs` (request=‘Fetch’)	Time follower waits for fetch requests	request=Fetch
`kaf_RequestMetrics_RemoteTimeMs` (request=‘FetchConsumer’)	Time follower waits for consumer fetch	request=FetchConsumer
`kaf_RequestMetrics_RemoteTimeMs` (request=‘FetchFollower’)	Time follower waits for follower fetch	request=FetchFollower

Query Examples

Partition Health

// Under-replicated partitions
kaf_ReplicaManager_UnderReplicatedPartitions{rack=~'$rack',host_id=~'$host_id'}

// Offline partitions
kaf_KafkaController_OfflinePartitionsCount{rack=~'$rack',host_id=~'$host_id'}

// Total partition count
kaf_ReplicaManager_PartitionCount{rack=~'$rack',host_id=~'$host_id'}

// Under min ISR partitions
kaf_ReplicaManager_UnderMinIsrPartitionCount{rack=~'$rack',host_id=~'$host_id'}

ISR Changes

// ISR shrink rate
kaf_ReplicaManager_IsrShrinksPerSec{function='MeanRate',rack=~'$rack',host_id=~'$host_id'}

// ISR expand rate
kaf_ReplicaManager_IsrExpandsPerSec{function='MeanRate', rack=~'$rack',host_id=~'$host_id'}

Leader Performance

// Leader processing time for follower fetch requests
kaf_RequestMetrics_LocalTimeMs{request='FetchFollower',function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

// Leader processing time for consumer fetch requests
kaf_RequestMetrics_LocalTimeMs{request='Fetch',function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

// Leader processing time for produce requests
kaf_RequestMetrics_LocalTimeMs{request='Produce',function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

Follower Performance

// Follower wait time for produce replication
kaf_RequestMetrics_RemoteTimeMs{request='Produce', function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

// Follower wait time for fetch requests
kaf_RequestMetrics_RemoteTimeMs{request='Fetch',function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

// Follower wait time for follower fetch
kaf_RequestMetrics_RemoteTimeMs{request='FetchFollower',function=~'$percentile',rack=~'$rack',host_id=~'$host_id'}

Panel Organization

Overview Section

Empty row for spacing/organization

Replication

Under Replicated Partitions
Online Partitions
Offline Partitions
Under Min ISR Partitions

Leader Performance

Leader FetchFollower Requests
Leader Fetch Requests
Leader FetchConsumer Requests
Leader Produce Requests

Follower Performance

Follower Produce Requests Time
Follower Fetch Requests Time
Follower FetchConsumer Request Time
Follower FetchFollower Request Time

ISR Shrinks / Expands

IsrShrinks per Sec by Host
IsrExpands per Sec By Host

Filters

rack: Filter by rack location
host_id: Filter by specific host/broker
percentile: Select percentile for latency metrics (50th, 95th, 99th, etc.)

Best Practices

Partition Health Monitoring

Under-replicated partitions should be 0
Offline partitions indicate serious issues
Monitor under min ISR for potential data loss risk

ISR Monitoring

Frequent ISR shrinks indicate replication lag
High ISR churn suggests network or performance issues
ISR expansions should follow shrinks during recovery

Leader Performance

Monitor leader request processing times
High FetchFollower times indicate replication bottlenecks
Compare produce vs fetch latencies

Follower Performance

High RemoteTimeMs indicates replication delays
Monitor follower fetch times for lag issues
Ensure followers can keep up with leaders

Replication Tuning

Adjust replica.lag.time.max.ms for ISR membership
Tune num.replica.fetchers for better throughput
Monitor min.insync.replicas compliance

Troubleshooting

Under-replicated partitions: Check broker health and network
ISR shrinks: Investigate disk I/O and network latency
High follower lag: Check replication thread count
Offline partitions: Critical issue requiring immediate attention