Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Performance Internals

Deep dive into Kafka's performance architecture and optimization techniques.

Performance Design Principles

Section titled “Performance Design Principles”
Layers of the Kafka performance stackLayers of the Kafka performance stackKafka Performance StackBatchingCompressionSequential I/OZero-CopyPage CacheEfficient SerializationEach layer multipliesperformance benefitsreducesmessagessmallerwritesoptimaldisk usememoryefficiencyminimalcopies

Sequential I/O

Section titled “Sequential I/O”

For complete log segment structure, indexes, and retention policies, see Storage Engine.

Write Path

Section titled “Write Path”

Kafka appends all writes to the end of log segments, achieving sequential disk access.

Random database writes compared with Kafka sequential appendRandom database writes compared with Kafka sequential appendTraditional DatabaseRandom WritesKafkaSequential AppendPage 1Page 5Page 2Page 8Offset 0Offset 1Offset 2Offset 3Seeks between pages~100 IOPS on HDDSingle sequential stream~100+ MB/s on HDD

Performance Comparison

Section titled “Performance Comparison”
Access PatternHDD Performance (illustrative)SSD Performance (illustrative)
Random 4KB~100 IOPS~100K IOPS
Sequential~100 MB/s~500 MB/s
Kafka advantageOrder-of-magnitude on HDDSmaller delta on SSD

Zero-Copy Transfers

Section titled “Zero-Copy Transfers”

Traditional Copy Path

Section titled “Traditional Copy Path”
Traditional read and send path with four data copiesTraditional read and send path with four data copiesApplication Read/SendDiskKernelBufferUserBufferSocketBufferNIC4 data copies2 context switchesCPU overhead1. DMA read2. CPU copy3. CPU copy4. DMA send

Zero-Copy Path (sendfile)

Section titled “Zero-Copy Path (sendfile)”
Zero-copy send path using sendfile()Zero-copy send path using sendfile()Kafka sendfile()DiskPageCacheNIC0 CPU copiesDirect kernel transferMinimal CPU usage1. DMA read2. DMA send

Implementation

Section titled “Implementation”
// Kafka uses FileChannel.transferTo()
// which maps to sendfile() system call
fileChannel.transferTo(position, count, socketChannel);

Limitations

Section titled “Limitations”
ConditionZero-Copy Available
PlaintextYes
TLS/SSL enabledNo (encryption requires user-space)
CompressionYes (independent of compression)

TLS Impact

Enabling TLS disables zero-copy; throughput impact depends on CPU and workload.

Batching

Section titled “Batching”

Producer Batching

Section titled “Producer Batching”
Producer record accumulator batching per topic partitionProducer record accumulator batching per topic partitionProducerRecordAccumulatorTopic-PartitionBatchesSenderP0 BatchP1 BatchP2 BatchBrokerbatch readybatch readysingle requestmultiple batches

Batching Configuration

Section titled “Batching Configuration”
ParameterDefaultEffect
batch.size16384Maximum bytes per batch
linger.ms0Time to wait for more records
buffer.memory33554432Total memory for batching

Batching Benefits

Section titled “Batching Benefits”
Without batching (1000 messages):
  1000 network round-trips
  1000 small disk writes
  High overhead per message


With batching (1000 messages in 10 batches):
  10 network round-trips
  10 larger disk writes
  Amortized overhead

Consumer Fetch Batching

Section titled “Consumer Fetch Batching”
# Minimum data to fetch
fetch.min.bytes=1


# Maximum time to wait
fetch.max.wait.ms=500


# Maximum data per request
fetch.max.bytes=52428800

Compression

Section titled “Compression”

Compression Algorithms

Section titled “Compression Algorithms”
AlgorithmIndicative Ratio (varies by data)CPU UsageSpeed
none1.0xNoneFastest
gzip~5-8xHighSlow
snappy~2-3xLowFast
lz4~3-4xLowFastest
zstd~4-6xMediumFast

Compression Flow

Section titled “Compression Flow”
Compression flow from producer through broker to consumerCompression flow from producer through broker to consumerProducerBrokerConsumerBatch(uncompressed)CompressedBatchStored(compressed)DecompressedBatchBroker stores compressed dataRecompresses only if configuredcompressstore as-isdecompress

Configuration

Section titled “Configuration”
# Producer compression
compression.type=lz4


# Topic-level compression (broker will recompress if different)
compression.type=producer  # Keep producer compression
compression.type=gzip      # Force specific compression

Compression Selection Guide

Section titled “Compression Selection Guide”
Use CaseRecommended
High throughput, low latencylz4
Balancedzstd
Maximum compressiongzip
Minimal CPUsnappy
Already compressed datanone

Request Pipelining

Section titled “Request Pipelining”

In-Flight Requests

Section titled “In-Flight Requests”
Pipelined in-flight producer requestsProducerBrokerProducerProducerBrokerBrokerRequest 1Request 2Request 3Response 1Response 2Response 3Multiple requests in flightwithout waiting for responses
# Maximum in-flight requests
max.in.flight.requests.per.connection=5

Ordering Considerations

Section titled “Ordering Considerations”
SettingOrderingThroughput
max.in.flight=1GuaranteedLower
max.in.flight=5May reorder on retryHigher
max.in.flight=5 + idempotentGuaranteedHigher

Thread Model

Section titled “Thread Model”

Broker Threads

Section titled “Broker Threads”
Broker network and I/O thread modelBroker network and I/O thread modelBroker Thread ModelNetwork Threads(num.network.threads)I/O Threads(num.io.threads)Request QueueResponse QueueAcceptReadWriteIO-1IO-2IO-Nenqueueprocessenqueuesend

Thread Configuration

Section titled “Thread Configuration”
ParameterDefaultRecommendation
num.network.threads3CPU cores / 4
num.io.threads8CPU cores
num.replica.fetchers12-4 for high partition counts
num.recovery.threads.per.data.dir12-4 for faster recovery

Page Cache Optimization

Section titled “Page Cache Optimization”

Warm Cache Benefits

Section titled “Warm Cache Benefits”
Read latency by page cache stateRead latency by page cache stateRead ScenariosHot Read(cache hit)Warm Read(partial hit)Cold Read(cache miss)~sub-ms~1-5 ms~10+ msRecent data served from memoryConsumer lag = more cache misses

Optimal Memory Allocation

Section titled “Optimal Memory Allocation”
Total RAM: 64 GB


JVM Heap: 6 GB (enough for metadata)
OS/System: 2 GB
Page Cache: 56 GB (for log data)


If hourly throughput = 50 GB
Page cache covers ~1 hour of data

Benchmarking

Section titled “Benchmarking”

Producer Performance Test

Section titled “Producer Performance Test”
Terminal window
# Throughput test
kafka-producer-perf-test.sh \
  --topic test-topic \
  --num-records 10000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props \
    bootstrap.servers=kafka:9092 \
    batch.size=65536 \
    linger.ms=10 \
    compression.type=lz4

Consumer Performance Test

Section titled “Consumer Performance Test”
Terminal window
# Throughput test
kafka-consumer-perf-test.sh \
  --bootstrap-server kafka:9092 \
  --topic test-topic \
  --messages 10000000 \
  --threads 4

End-to-End Latency Test

Section titled “End-to-End Latency Test”
Terminal window
kafka-run-class.sh kafka.tools.EndToEndLatency \
  kafka:9092 \
  test-topic \
  10000 \
  all \
  1024

Performance Metrics

Section titled “Performance Metrics”

Key Metrics

Section titled “Key Metrics”
MetricExample Range
RequestsPerSecDepends on workload
TotalTimeMs (P99)< 100 ms
RequestQueueTimeMs< 10 ms
LocalTimeMs< 50 ms
RemoteTimeMs< 50 ms
ThrottleTimeMs0

Bottleneck Identification

Section titled “Bottleneck Identification”
SymptomLikely Bottleneck
High RequestQueueTimeMsNetwork threads saturated
High LocalTimeMsDisk I/O or I/O threads saturated
High RemoteTimeMsReplication lag
High ResponseQueueTimeMsNetwork threads saturated
Section titled “Related Documentation”