Kafka Migration

Migration guides for Apache Kafka clusters.

---

Migration Types

Migration	Complexity	Downtime
Version upgrade	Low-Medium	Zero (rolling)
ZooKeeper to KRaft	Medium	Zero (rolling)
Cluster migration	High	Depends on approach
Cloud migration	High	Depends on approach

---

ZooKeeper to KRaft Migration

Overview

Migration Steps

1. Prepare cluster

# Verify all brokers are healthy
kafka-broker-api-versions.sh --bootstrap-server kafka:9092

# Check for under-replicated partitions
kafka-topics.sh --bootstrap-server kafka:9092 \
  --describe --under-replicated-partitions

2. Deploy KRaft controllers

controller.properties

process.roles=controller
node.id=100
controller.quorum.voters=100@ctrl1:9093,101@ctrl2:9093,102@ctrl3:9093
listeners=CONTROLLER://0.0.0.0:9093
controller.listener.names=CONTROLLER

3. Enable migration mode on controllers

# Add to controller.properties
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
zookeeper.metadata.migration.enable=true

4. Start migration

# Format controller storage
kafka-storage.sh format -t $(kafka-storage.sh random-uuid) \
  -c controller.properties

# Start controllers
kafka-server-start.sh controller.properties

5. Roll brokers to enable migration

# Add to server.properties
controller.quorum.voters=100@ctrl1:9093,101@ctrl2:9093,102@ctrl3:9093
controller.listener.names=CONTROLLER

6. Complete migration

# Verify migration status
kafka-metadata.sh --snapshot /var/kafka/__cluster_metadata-0/*.log \
  --command "migration-state"

# Disable ZooKeeper on controllers
# Remove: zookeeper.metadata.migration.enable=true
# Restart controllers

7. Remove ZooKeeper dependency from brokers

# Remove from server.properties
# zookeeper.connect=...

# Rolling restart brokers

---

Version Upgrades

Rolling Upgrade Process

Pre-Upgrade Checklist

• Review release notes for breaking changes
• Verify client compatibility
• Backup configurations
• Test upgrade in non-production
• Plan rollback procedure

Upgrade Commands

# Set protocol version before upgrade
# server.properties
inter.broker.protocol.version=3.6
log.message.format.version=3.6

# Upgrade each broker
for broker in broker1 broker2 broker3; do
  # Stop
  ssh $broker "sudo systemctl stop kafka"

  # Install new version
  ssh $broker "sudo yum install kafka-3.7"

  # Start
  ssh $broker "sudo systemctl start kafka"

  # Wait for recovery
  sleep 60
  kafka-topics.sh --bootstrap-server kafka:9092 \
    --describe --under-replicated-partitions
done

# Update protocol version
# server.properties
inter.broker.protocol.version=3.7
log.message.format.version=3.7

# Rolling restart

---

Cluster Migration

Using MirrorMaker 2

Migration Steps

1. Set up target cluster

# Create topics on target with same configuration
kafka-topics.sh --bootstrap-server target:9092 \
  --create --topic my-topic \
  --partitions 12 --replication-factor 3

2. Deploy MirrorMaker 2

mm2.properties

clusters=source,target
source.bootstrap.servers=source-kafka:9092
target.bootstrap.servers=target-kafka:9092

source->target.enabled=true
source->target.topics=.*

sync.group.offsets.enabled=true
emit.checkpoints.enabled=true

3. Verify replication

# Check lag
kafka-consumer-groups.sh --bootstrap-server target:9092 \
  --describe --group mm2-source-target

# Verify data integrity
kafka-console-consumer.sh --bootstrap-server target:9092 \
  --topic my-topic --from-beginning --max-messages 10

4. Migrate consumers

# Translate offsets
# Consumers can read checkpoints topic to find equivalent offset

5. Migrate producers

# Update bootstrap.servers to target cluster
# Deploy updated configuration

6. Decommission source

# Stop MirrorMaker
# Verify no traffic to source
# Decommission source cluster

---

Rollback Procedures

Version Rollback

# Stop broker
sudo systemctl stop kafka

# Install previous version
sudo yum downgrade kafka-3.6

# Ensure protocol version matches
# server.properties
inter.broker.protocol.version=3.6

# Start broker
sudo systemctl start kafka

Migration Rollback

# For ZK to KRaft migration:
# 1. Re-enable ZooKeeper on brokers
# 2. Rolling restart brokers
# 3. Shut down KRaft controllers
# 4. Verify cluster operates with ZooKeeper

---

Related Documentation

→ Version Compatibility - JDK, client-broker, and KRaft compatibility matrices

• Operations - Operational procedures
• Cluster Management - Cluster operations
• Architecture - System architecture
• Backup and Restore - DR procedures