Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Metadata Management

Kafka clients maintain metadata about the cluster topology, including broker addresses, topic partitions, and partition leaders. Proper metadata management is essential for efficient request routing and handling topology changes.

Client metadata cache contents and refresh triggers

FieldDescription
cluster_idUnique cluster identifier
controller_idCurrent controller broker ID
brokersList of all brokers with host/port/rack
FieldDescription
node_idUnique broker identifier
hostBroker hostname or IP
portBroker port (default 9092)
rackRack identifier (optional)
FieldDescription
nameTopic name
partitionsNumber of partitions
is_internalInternal topic flag

Replication factor is derived from the partition replica lists, not returned directly in the metadata response.

FieldDescription
partitionPartition index
leaderCurrent leader broker ID
leader_epochLeader election epoch
replicasAll replica broker IDs
isrIn-sync replica broker IDs
offline_replicasOffline replica broker IDs

Bootstrap discovery with fallback to a second bootstrap broker

# Multiple brokers for redundancy
bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092
# Client will try brokers in random order
# Only needs ONE successful connection for discovery

Best Practices:

GuidelineRationale
List 3+ brokersRedundancy during broker failures
Use DNS namesEasier maintenance than IPs
Include brokers from different racksSurvive rack failures
Don’t list all brokersUnnecessary, any broker returns full metadata

MetadataRequest {
topics: [TopicName] | null // v1+: null = all topics; [] = no topics
allow_auto_topic_creation: bool // v4+
include_cluster_authorized_operations: bool // v8–10
include_topic_authorized_operations: bool // v8+
}

In request version 0, an empty topics array requests all topics. In version 1 and later, use a null array for all topics; an empty array requests no topic metadata. Field availability also depends on the request version.

MetadataResponse {
throttle_time_ms: int32
brokers: [Broker]
cluster_id: string
controller_id: int32
topics: [TopicMetadata]
}
Broker {
node_id: int32
host: string
port: int32
rack: string (nullable)
}
TopicMetadata {
error_code: int16
name: string
is_internal: bool
partitions: [PartitionMetadata]
}
PartitionMetadata {
error_code: int16
partition_index: int32
leader_id: int32
leader_epoch: int32
replica_nodes: [int32]
isr_nodes: [int32]
offline_replicas: [int32]
}

Metadata refresh triggers by category

# Maximum age before forced refresh
metadata.max.age.ms=300000 # 5 minutes (default)

Client metadata cache structure

EventAction
metadata.max.age.ms expiryMark stale, refresh
NOT_LEADER_OR_FOLLOWERInvalidate partition
UNKNOWN_TOPIC_OR_PARTITIONInvalidate topic
Node disconnectInvalidate node
// Find leader for partition
public Node leader(TopicPartition partition) {
PartitionInfo info = partitionsByTopicPartition.get(partition);
if (info == null) {
return null;
}
return info.leader();
}
// Find all partitions for topic
public List<PartitionInfo> partitionsForTopic(String topic) {
return partitionsByTopic.get(topic);
}

Error CodeNameCauseClient Action
3UNKNOWN_TOPIC_OR_PARTITIONTopic/partition doesn’t existRefresh only if the topic is expected to exist
5LEADER_NOT_AVAILABLELeader election in progressWait and retry
6NOT_LEADER_OR_FOLLOWERStale leader infoRefresh metadata
15COORDINATOR_NOT_AVAILABLEGroup coordinator unavailableRetry FindCoordinator

Leader epoch is a monotonically increasing number that identifies the term of a partition leader. It prevents issues from stale leadership information.

Leader Epoch Timeline

# Client rack (for follower fetching)
client.rack=rack-a
# Enables rack-aware replica selection

Producer metadata lookup and partition selection

OptimizationConfiguration
Increase refresh intervalmetadata.max.age.ms=600000
Request specific topicsDon’t request all topics
Cache locallyAvoid redundant lookups

Kafka batches metadata requests when multiple threads need updates:

Producer client metrics are reported under the producer-metrics group.

MetricDescriptionAlert Threshold
metadata-ageAge in seconds of the current producer metadata> 2 × metadata.max.age.ms / 1000
metadata-wait-time-ns-totalCumulative time spent waiting for metadata (ns)Sustained increase in rate
# Enable metadata debug logging
log4j.logger.org.apache.kafka.clients.Metadata=DEBUG
log4j.logger.org.apache.kafka.clients.NetworkClient=DEBUG

Common issues:

SymptomCauseSolution
Frequent refreshesMany NOT_LEADER errorsCheck cluster stability
Stale metadataLong metadata.max.age.msReduce refresh interval
Missing topicsAuthorization issuesCheck ACLs
Wrong broker countPartial visibilityCheck bootstrap servers

FeatureMinimum Version
Basic metadata0.8.0
Rack information0.10.0
Leader epoch0.11.0
Offline replicas1.0.0
Authorized operations2.3.0