Kafka clients maintain metadata about the cluster topology, including broker addresses, topic partitions, and partition leaders. Proper metadata management is essential for efficient request routing and handling topology changes.
Field Description cluster_idUnique cluster identifier controller_idCurrent controller broker ID brokersList of all brokers with host/port/rack
Field Description node_idUnique broker identifier hostBroker hostname or IP portBroker port (default 9092) rackRack identifier (optional)
Field Description nameTopic name partitionsNumber of partitions is_internalInternal topic flag
Replication factor is derived from the partition replica lists, not returned directly in the metadata response.
Field Description partitionPartition index leaderCurrent leader broker ID leader_epochLeader election epoch replicasAll replica broker IDs isrIn-sync replica broker IDs offline_replicasOffline replica broker IDs
# Multiple brokers for redundancy
bootstrap.servers =kafka1:9092,kafka2:9092,kafka3:9092
# Client will try brokers in random order
# Only needs ONE successful connection for discovery
Best Practices:
Guideline Rationale List 3+ brokers Redundancy during broker failures Use DNS names Easier maintenance than IPs Include brokers from different racks Survive rack failures Don’t list all brokers Unnecessary, any broker returns full metadata
topics: [TopicName] | null // v1+: null = all topics; [] = no topics
allow_auto_topic_creation: bool // v4+
include_cluster_authorized_operations: bool // v8–10
include_topic_authorized_operations: bool // v8+
In request version 0, an empty topics array requests all topics. In version 1 and later, use a null array for all topics; an empty array requests no topic metadata. Field availability also depends on the request version.
partitions: [PartitionMetadata]
offline_replicas: [int32]
# Maximum age before forced refresh
metadata.max.age.ms =300000 # 5 minutes (default)
Event Action metadata.max.age.ms expiryMark stale, refresh NOT_LEADER_OR_FOLLOWERInvalidate partition UNKNOWN_TOPIC_OR_PARTITIONInvalidate topic Node disconnect Invalidate node
// Find leader for partition
public Node leader ( TopicPartition partition ) {
PartitionInfo info = partitionsByTopicPartition . get ( partition ) ;
// Find all partitions for topic
public List< PartitionInfo > partitionsForTopic ( String topic ) {
return partitionsByTopic . get ( topic ) ;
Error Code Name Cause Client Action 3 UNKNOWN_TOPIC_OR_PARTITIONTopic/partition doesn’t exist Refresh only if the topic is expected to exist 5 LEADER_NOT_AVAILABLELeader election in progress Wait and retry 6 NOT_LEADER_OR_FOLLOWERStale leader info Refresh metadata 15 COORDINATOR_NOT_AVAILABLEGroup coordinator unavailable Retry FindCoordinator
Leader epoch is a monotonically increasing number that identifies the term of a partition leader. It prevents issues from stale leadership information.
# Client rack (for follower fetching)
# Enables rack-aware replica selection
Optimization Configuration Increase refresh interval metadata.max.age.ms=600000Request specific topics Don’t request all topics Cache locally Avoid redundant lookups
Kafka batches metadata requests when multiple threads need updates:
Producer client metrics are reported under the producer-metrics group.
Metric Description Alert Threshold metadata-ageAge in seconds of the current producer metadata > 2 × metadata.max.age.ms / 1000 metadata-wait-time-ns-totalCumulative time spent waiting for metadata (ns) Sustained increase in rate
# Enable metadata debug logging
log4j.logger.org.apache.kafka.clients.Metadata =DEBUG
log4j.logger.org.apache.kafka.clients.NetworkClient =DEBUG
Common issues:
Symptom Cause Solution Frequent refreshes Many NOT_LEADER errors Check cluster stability Stale metadata Long metadata.max.age.ms Reduce refresh interval Missing topics Authorization issues Check ACLs Wrong broker count Partial visibility Check bootstrap servers
Feature Minimum Version Basic metadata 0.8.0 Rack information 0.10.0 Leader epoch 0.11.0 Offline replicas 1.0.0 Authorized operations 2.3.0