Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Streams

Kafka Streams is a client library for building stream processing applications that transform, aggregate, and analyze data stored in Kafka topics.

This guide covers current Kafka Streams APIs and operations. Features introduced in a specific Kafka release are identified where they affect configuration, compatibility, or APIs.


Stream processing is the continuous processing of data as it arrives, rather than collecting data into batches for periodic processing.

AspectBatch ProcessingStream Processing
Data modelFinite dataset processed as a wholeUnbounded sequence of events processed incrementally
LatencyMinutes to hours (wait for batch to complete)Milliseconds to seconds (process on arrival)
Processing triggerSchedule (hourly, daily) or manualEach record arrival or micro-batch
StateRecomputed from scratch each runMaintained incrementally across records
ResultsComplete after batch finishesContinuously updated

Traditional batch architectures introduce inherent latency—data must accumulate before processing begins, and results are only available after the batch completes. For many applications, this delay is unacceptable:

DomainStream Processing Application
Fraud detectionScore transactions in real-time before authorization
MonitoringDetect anomalies and alert within seconds of occurrence
PersonalizationUpdate recommendations based on current session behavior
IoTReact to sensor readings as they arrive
Financial marketsProcess market data with minimal latency

Stream processing enables applications to react to events as they happen rather than discovering them hours or days later.

Processing unbounded data streams introduces challenges that batch systems avoid:

ChallengeDescription
Unbounded dataNo defined end; must process incrementally
Out-of-order eventsNetwork delays cause events to arrive out of sequence
Late arrivalsEvents may arrive after their time window has closed
State managementAggregations require persistent, fault-tolerant state
Exactly-once semanticsFailures must not cause duplicates or data loss
BackpressureMust handle bursts without losing data

Kafka Streams addresses these challenges with built-in primitives for windowing, state management, and exactly-once processing.


Raw Kafka consumers and producers provide low-level access to topics but require manual implementation of common stream processing concerns:

ConcernRaw Consumer/ProducerKafka Streams
Stateful processingManual state management, external storageBuilt-in state stores with automatic persistence
Fault toleranceApplication must handle failures, replayAutomatic state recovery from changelog topics
Exactly-once semanticsComplex transaction coordinationSingle configuration option
Windowed aggregationsManual time tracking, expiration logicDeclarative window definitions
Stream-table joinsCustom implementation, consistency challengesNative join operations with co-partitioning
ScalingManual partition assignment coordinationAutomatic partition rebalancing

Kafka Streams solves these problems by providing a high-level abstraction over the consumer/producer APIs while maintaining Kafka's scalability and fault-tolerance guarantees.


Kafka Streams is designed around several core principles that distinguish it from other stream processing systems:

Kafka Streams is a library that runs within a standard Java application—not a framework requiring a dedicated cluster.

AspectCluster-Based SystemsKafka Streams
DeploymentDedicated cluster (YARN, Kubernetes, Mesos)Standard application deployment
Resource managementCluster manager allocates resourcesApplication controls its own resources
Operational complexitySeparate system to monitor and maintainSame as any JVM application
ScalingCluster-level configurationAdd/remove application instances
DependenciesRequires cluster infrastructureRequires only Kafka brokers

This design means Kafka Streams applications can be packaged as microservices, deployed in containers, or embedded within existing applications without infrastructure changes.

Kafka Streams requires no external systems beyond Kafka itself:

  • State storage: Uses RocksDB locally, backed by Kafka changelog topics
  • Coordination: Uses either the classic client-driven protocol or the broker-driven Streams group protocol
  • Checkpointing: Commits offsets to Kafka's __consumer_offsets topic
  • Fault tolerance: Replays from Kafka topics to recover state

This architecture eliminates the operational burden of managing separate storage or coordination systems (ZooKeeper, HDFS, external databases) that other stream processors require.

Kafka Streams parallelism is determined by input topic partitions:

max_parallelism = max(partitions across all input topics)

Each stream task processes one partition from each input topic. Scaling works by:

  1. Adding application instances—partitions automatically rebalance
  2. Removing instances—remaining instances absorb partitions
  3. No manual partition assignment required
Input PartitionsApplication InstancesTasks per Instance
616
623
632
661
6120-1 (6 idle)

Partition Count Limit

Running more instances than input partitions results in idle instances. The partition count must be chosen to accommodate expected maximum parallelism.


Use CaseWhy Kafka Streams
Event enrichmentJoin streams with reference data tables
Real-time aggregationsWindowed counts, sums, averages with exactly-once
Stream-table joinsEnrich events with current entity state
Microservice event processingEmbedded library, no external cluster
Stateful transformationsDeduplication, sessionization, pattern detection
CDC processingProcess database change streams
ScenarioConsideration
Sub-millisecond latencyKafka Streams adds overhead; consider direct consumer
Non-JVM languagesLimited to Java/Scala; use native consumers or alternative systems
Complex event processing (CEP)Pattern matching across streams may require specialized CEP engines
Batch processingKafka Streams is designed for continuous streaming, not batch windows
Multi-cluster topologiesKafka Streams operates within a single cluster

Kafka Streams provides configurable processing semantics:

GuaranteeConfigurationBehavior
At-least-onceAT_LEAST_ONCERecords may be reprocessed on failure; duplicates possible
Exactly-onceEXACTLY_ONCE_V2Each record processed exactly once; no duplicates

Exactly-once processing (Kafka 2.5+) coordinates:

  • Consumer offset commits
  • State store updates
  • Producer writes to output topics

All three operations succeed or fail atomically via Kafka transactions.

Exactly-Once Requirements

Exactly-once semantics require:

  • All input and output topics on the same Kafka cluster
  • Kafka broker version 2.5+ for EXACTLY_ONCE_V2
  • processing.guarantee set to EXACTLY_ONCE_V2

Kafka Streams topology from source processor to sink processor

FeatureDescription
Library, not frameworkEmbed in any Java/Scala application
No cluster requiredNo separate processing cluster needed
Exactly-onceFull EOS support
ScalableElastic scaling via partitions
Fault-tolerantAutomatic state recovery

Unbounded stream of records (event stream).

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> events = builder.stream("events");
events
.filter((key, value) -> value.contains("important"))
.mapValues(value -> value.toUpperCase())
.to("filtered-events");

Changelog stream representing current state (table semantics).

KTable<String, Long> counts = builder.table("user-counts");
// Updates replace previous values for same key
// null value = tombstone (delete)

Fully replicated table for broadcast joins.

GlobalKTable<String, String> config = builder.globalTable("config");
// Every instance has complete copy
// Useful for reference data

Operations that process records independently.

OperationDescriptionExample
filterKeep matching recordsstream.filter((k, v) -> v > 0)
mapTransform key and valuestream.map((k, v) -> KeyValue.pair(k, v * 2))
mapValuesTransform value onlystream.mapValues(v -> v.toUpperCase())
flatMapOne-to-many transformationstream.flatMap((k, v) -> splitToMultiple(v))
branchSplit stream by conditionstream.branch(isA, isB, other)
mergeCombine streamsstream1.merge(stream2)
KStream<String, Event> events = builder.stream("raw-events");
events
.filter((key, event) -> event.getType().equals("click"))
.mapValues(event -> new ClickEvent(event))
.to("click-events");

Operations that maintain state across records.

KStream<String, Purchase> purchases = builder.stream("purchases");
KTable<String, Long> purchaseCounts = purchases
.groupBy((key, purchase) -> purchase.getCustomerId())
.count();
KTable<String, Double> purchaseTotals = purchases
.groupBy((key, purchase) -> purchase.getCustomerId())
.aggregate(
() -> 0.0,
(key, purchase, total) -> total + purchase.getAmount(),
Materialized.with(Serdes.String(), Serdes.Double())
);
Join TypeLeftRightOutput
InnerKStreamKStreamMatches only
LeftKStreamKStreamAll left + matches
OuterKStreamKStreamAll records
KStream-KTableKStreamKTableEnrich stream
KStream-GlobalKTableKStreamGlobalKTableBroadcast join
KStream<String, Order> orders = builder.stream("orders");
KTable<String, Customer> customers = builder.table("customers");
KStream<String, EnrichedOrder> enriched = orders.join(
customers,
(order, customer) -> new EnrichedOrder(order, customer),
Joined.with(Serdes.String(), orderSerde, customerSerde)
);

Group records by time windows for temporal aggregations.

Window TypeExample BoundariesOverlapDescription
Tumbling[0-5), [5-10), [10-15)NoFixed-size, non-overlapping
Hopping (size=5, advance=2)[0-5), [2-7), [4-9)YesFixed-size, overlapping
Sliding (difference=5)Per-recordYesWindow created for each record; includes all records within the time difference
Session (gap=5)[0-3], [10-12]NoDynamic boundaries; inactivity gap exceeding threshold creates a new session

Fixed-size, non-overlapping windows.

KStream<String, Click> clicks = builder.stream("clicks");
KTable<Windowed<String>, Long> clicksPerMinute = clicks
.groupBy((key, click) -> click.getPageId())
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
.count();

Fixed-size, overlapping windows.

KTable<Windowed<String>, Long> clicksHopping = clicks
.groupBy((key, click) -> click.getPageId())
.windowedBy(TimeWindows.ofSizeAndGrace(
Duration.ofMinutes(5),
Duration.ofMinutes(1)
).advanceBy(Duration.ofMinutes(1)))
.count();

Dynamic windows based on activity gaps.

KTable<Windowed<String>, Long> sessions = clicks
.groupBy((key, click) -> click.getUserId())
.windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)))
.count();

StoreBuilder<KeyValueStore<String, Long>> storeBuilder =
Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("my-store"),
Serdes.String(),
Serdes.Long()
);
builder.addStateStore(storeBuilder);

Query state stores from external applications.

ReadOnlyKeyValueStore<String, Long> store =
streams.store(
StoreQueryParameters.fromNameAndType(
"my-store",
QueryableStoreTypes.keyValueStore()
)
);
Long value = store.get("key");

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.String().getClass());
// State directory
props.put(StreamsConfig.STATE_DIR_CONFIG, "/var/kafka-streams");
// Exactly-once
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE_V2);
// Commit interval
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100);
// Threading
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 4);
// Buffering
props.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 10 * 1024 * 1024);
// Commit frequency
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000);

cache.max.bytes.buffering has been deprecated since Kafka 3.4. Use statestore.cache.max.bytes (STATESTORE_CACHE_MAX_BYTES_CONFIG).

The broker-driven Streams rebalance protocol is generally available and can reduce rebalance disruption:

group.protocol=streams

Kafka 4.4 still defaults the client to classic. With group.protocol=streams, heartbeat, session, and standby-replica settings are broker or group configurations; client settings such as session.timeout.ms, heartbeat.interval.ms, num.standby.replicas, task.assignor.class, warmup-task settings, and rack-aware assignment settings are ignored.

The Streams protocol does not support online migration, regex topic subscriptions, topology changes within an existing group, warmup tasks, or the classic high-availability assignor. Stop every instance before migrating between classic and streams. Kafka 4.4 adds static membership through group.instance.id, but both clients and brokers must be Kafka 4.4 or later.


StreamsBuilder builder = new StreamsBuilder();
// ... define topology ...
KafkaStreams streams = new KafkaStreams(builder.build(), props);
// Handle shutdown gracefully
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
// Start processing
streams.start();

Kafka Streams client state transitions

Out-of-order records are common in distributed systems due to network delays, partition lag, or event-time vs ingestion-time differences.

CauseDescription
Within partitionRecords with larger timestamps may have smaller offsets
Across partitionsDifferent partitions have different processing progress
Late arrivalsRecords arrive after their time window has passed
OperationOut-of-Order Handling
StatelessNo impact—each record processed independently
Windowed aggregationsConfigure grace period to wait for late arrivals
Stream-Stream joinsAll types (inner, outer, left) handle correctly
Stream-Table joinsDefault: not handled; With versioned stores: timestamp-based lookup
Table-Table joinsDefault: not handled; With versioned stores: timestamp-based semantics

Control how long to wait for out-of-order records:

// Tumbling window with 5-minute grace period
TimeWindows.ofSizeAndGrace(
Duration.ofMinutes(10), // window size
Duration.ofMinutes(5) // grace period
)
// Session window with grace period
SessionWindows.ofInactivityGapAndGrace(
Duration.ofMinutes(30), // inactivity gap
Duration.ofMinutes(5) // grace period
)

Records arriving after the grace period are discarded.

For handling out-of-order data in joins, use versioned state stores (Kafka Streams 3.5+):

// Create versioned store
StoreBuilder<VersionedKeyValueStore<String, String>> storeBuilder =
Stores.versionedKeyValueStoreBuilder(
Stores.persistentVersionedKeyValueStore("my-store", Duration.ofDays(1)),
Serdes.String(),
Serdes.String()
);

Versioned stores enable timestamp-based lookups instead of offset-based, properly handling out-of-order data in stream-table and table-table joins.


ContextTimestamp Assignment
Processing input recordInherits input record timestamp
Punctuator callbackCurrent stream time of the task
AggregationsMaximum timestamp of contributing records
Joins (stream-stream, table-table)max(left.ts, right.ts)
Stream-table joinsStream record timestamp
Stateless operations (map, filter)Input record timestamp passed through
flatMap and siblingsAll output records inherit input timestamp

Implement TimestampExtractor for custom timestamp logic:

public class EventTimeExtractor implements TimestampExtractor {
@Override
public long extract(ConsumerRecord<Object, Object> record, long partitionTime) {
// Extract timestamp from record value
Event event = (Event) record.value();
return event.getTimestamp();
}
}
// Use in configuration
props.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
EventTimeExtractor.class);

KafkaStreams streams = new KafkaStreams(topology, props);
// Register shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
streams.close(Duration.ofSeconds(30));
}));
streams.start();

Monitor application state transitions:

streams.setStateListener((newState, oldState) -> {
if (newState == KafkaStreams.State.ERROR) {
// Handle error state
log.error("Streams entered ERROR state");
}
});

Handle unrecoverable errors:

streams.setUncaughtExceptionHandler(exception -> {
log.error("Uncaught exception in streams", exception);
// Return action: SHUTDOWN_CLIENT, REPLACE_THREAD, or SHUTDOWN_APPLICATION
return StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.SHUTDOWN_CLIENT;
});

In Kafka 4.2+ (KIP-1034), Streams exception handlers can return DLQ records, and built-in handlers can use errors.dead.letter.queue.topic.name. This provides a structured path for poison pills and processing failures. A DLQ reduces silent loss only when it is correctly provisioned, authorized, monitored, and its production succeeds.

In Kafka 4.2+ (KIP-1153), KafkaStreams.close() supports a fluent CloseOptions API. Kafka 4.4 adds the protocol-aware DEFAULT operation (KIP-1284):

streams.close(CloseOptions.timeout(Duration.ofSeconds(30))
.withGroupMembershipOperation(
CloseOptions.GroupMembershipOperation.DEFAULT));

DEFAULT keeps classic members in the group. Under the Streams protocol, dynamic members leave and static members remain until their session timeout. Use LEAVE_GROUP or REMAIN_IN_GROUP only when overriding that protocol-aware behavior intentionally. The older nested KafkaStreams.CloseOptions and its leaveGroup(boolean) method are deprecated.


The Processor API provides low-level control for custom stream processing logic.

public class WordCountProcessor implements Processor<String, String, String, Long> {
private KeyValueStore<String, Long> kvStore;
private ProcessorContext<String, Long> context;
@Override
public void init(ProcessorContext<String, Long> context) {
this.context = context;
this.kvStore = context.getStateStore("counts");
// Schedule punctuation (periodic callback)
context.schedule(
Duration.ofSeconds(10),
PunctuationType.STREAM_TIME,
this::forwardCounts
);
}
@Override
public void process(Record<String, String> record) {
String[] words = record.value().toLowerCase().split("\\W+");
for (String word : words) {
Long count = kvStore.get(word);
kvStore.put(word, (count == null) ? 1L : count + 1);
}
}
private void forwardCounts(long timestamp) {
try (KeyValueIterator<String, Long> iter = kvStore.all()) {
while (iter.hasNext()) {
KeyValue<String, Long> entry = iter.next();
context.forward(new Record<>(entry.key, entry.value, timestamp));
}
}
}
@Override
public void close() {
// Clean up resources (but not state stores)
}
}
TypeTriggerUse Case
STREAM_TIMEEvent timestamps advanceEmit results based on event progress
WALL_CLOCK_TIMEReal clock timePeriodic actions regardless of data flow

In Kafka 4.2+ (KIP-1146), wall-clock punctuation supports an optional startTime parameter for anchored scheduling. This enables punctuation callbacks to fire at predictable wall-clock intervals rather than relative to the application start time.

Topology topology = new Topology();
topology.addSource("source", "input-topic");
topology.addProcessor("processor", WordCountProcessor::new, "source");
StoreBuilder<KeyValueStore<String, Long>> storeBuilder =
Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("counts"),
Serdes.String(),
Serdes.Long()
);
topology.addStateStore(storeBuilder, "processor");
topology.addSink("sink", "output-topic", "processor");
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> stream = builder.stream("input");
// Inject custom processor into DSL topology
stream.process(WordCountProcessor::new, Named.as("word-count"), "counts-store");

Kafka Streams requires serializers/deserializers (SerDes) for keys and values.

Data TypeSerDe
byte[]Serdes.ByteArray()
ByteBufferSerdes.ByteBuffer()
StringSerdes.String()
IntegerSerdes.Integer()
LongSerdes.Long()
DoubleSerdes.Double()
BooleanSerdes.Boolean()
UUIDSerdes.UUID()
VoidSerdes.Void()
List<T>Serdes.ListSerde()
// Explicit SerDes for to()
stream.to("output", Produced.with(Serdes.String(), Serdes.Long()));
// Explicit SerDes for groupBy
stream.groupBy(
(key, value) -> value.getCategory(),
Grouped.with(Serdes.String(), orderSerde)
);
// Time-windowed SerDe
WindowedSerdes.TimeWindowedSerde<String> timeWindowedSerde =
new WindowedSerdes.TimeWindowedSerde<>(Serdes.String());
// Session-windowed SerDe
WindowedSerdes.SessionWindowedSerde<String> sessionWindowedSerde =
new WindowedSerdes.SessionWindowedSerde<>(Serdes.String());

Explicit naming prevents topology incompatibilities during application upgrades.

Without explicit names, changes to the topology can generate different internal names for:

  • Repartition topics
  • Changelog topics
  • State stores
  • Processor nodes

This causes incompatible state stores, new repartition topics, and potential data loss during upgrades.

StreamsBuilder builder = new StreamsBuilder();
// Name the source
KStream<String, String> stream = builder.stream(
"input",
Consumed.as("input-source")
);
// Name transformations
KStream<String, String> filtered = stream.filter(
(k, v) -> v != null,
Named.as("null-filter")
);
// Name repartition
KStream<String, Long> repartitioned = stream.selectKey(
(k, v) -> v.getUserId(),
Named.as("rekey-by-user")
).repartition(Repartitioned.as("user-repartition"));
// Name aggregation (state store)
KTable<String, Long> counts = stream
.groupByKey(Grouped.as("group-by-key"))
.count(Named.as("count-op"), Materialized.as("counts-store"));
// Name output
counts.toStream().to("output", Produced.as("output-sink"));

Kafka 4.2 includes several Kafka Streams improvements:

FeatureKIPDescription
Streams Rebalance Protocol GAKIP-1071Server-side rebalance protocol with broker-coordinated task assignment, promoted from early access (4.1) to generally available
Dead letter queue supportKIP-1034Exception handlers can route failed records to a dead letter topic
Anchored punctuationKIP-1146Optional startTime parameter for predictable wall-clock punctuation scheduling
Fluent CloseOptions APIKIP-1153CloseOptions with explicit leave-group control
Rebalance callback latency metricsKIP-1216Thread-level latency metrics for rebalance listener callbacks
application-id metric tagKIP-1221application-id tag added to client state metric
State directory permissionsKIP-1230Optional allow.os.group.write.access configuration for state directory file permissions

Kafka 4.3 adds the following Kafka Streams capabilities:

FeatureReferenceDescription
Record headers in state storesKAFKA-20056State stores can persist record headers alongside keys and values
DSL opt-in for header-aware storesKAFKA-20194DSL support for header-aware KeyValueStore, WindowStore, and SessionStore implementations
State-store managed changelog offsets—Changelog offsets managed within the state store rather than tracked externally, reducing recovery-path complexity
streams-scala deprecationKAFKA-19976The streams-scala module is deprecated and slated for removal in a future release

Header-aware state stores

Prior to Kafka 4.3, state stores discarded record headers during materialization. Applications requiring headers downstream had to embed them in the value payload. With the 4.3 DSL opt-in, headers can be stored and retrieved as a first-class property of the stored record.


Kafka 4.4 adds the following Kafka Streams capabilities:

FeatureKIPDescription
Transactional state storesKIP-892Opt-in state-store transactions preserve committed data after an error and replay only writes since the last successful commit
Topology description pluginKIP-1331Brokers can store a human-readable topology description and expose it through the Admin API and kafka-streams-groups.sh --describe --topology
Broker-side custom task assignorsKIP-1357Streams groups can select a custom assignor registered with group.streams.assignors
Static membership with the Streams protocol—group.instance.id is supported when both brokers and clients run Kafka 4.4 or later
Header-aware interactive queriesKIP-1356New IQv2 query types return record headers from header-aware state stores
Header-aware StreamPartitionerKIP-1321The partitioner can receive record headers when selecting a partition
GlobalKTable join keysKIP-1340Joiners can access both the mapped key and the original stream record key
Protocol-aware close defaultKIP-1284CloseOptions.GroupMembershipOperation.DEFAULT adapts group-leave behavior to the active protocol and static membership
Cleanup semanticsKIP-1283KafkaStreams.cleanUp() no longer warns when only process metadata and lock files prevent complete state-directory removal
application.server validationKIP-1245Streams configuration now rejects values that are neither empty nor a valid host:port endpoint

Transactional state stores are opt-in

Set enable.transactional.statestores=true to enable the feature. Review statestore.uncommitted.max.bytes and default.interactive.query.isolation.level before enabling it in production.

Kafka 4.4-only behavior

Every feature in this section requires Kafka Streams 4.4. Static Streams-group membership and broker-side topology descriptions also require compatible 4.4 brokers.