Skip to content
AxonOps Documentation

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Architecture Patterns

Common architectural patterns for building event-driven systems with Apache Kafka.

Pattern Overview

Section titled “Pattern Overview”
PatternUse CaseComplexity
Event NotificationDecoupled service communicationLow
Event-Carried State TransferAvoid synchronous queriesMedium
Event SourcingAudit, replay, temporal queriesHigh
CQRSSeparate read/write modelsHigh
SagaDistributed transactionsHigh

Event Notification

Section titled “Event Notification”

Services publish events when state changes occur. Consumers react to events independently.

Event notification pattern with minimal event payloadsEvent notification pattern with minimal event payloadsOrder ServiceKafkaInventory ServiceShipping ServiceNotification ServiceEvent contains minimal data(e.g., order_id only)Consumers query for detailsOrderCreatedconsumeconsumeconsume

Characteristics

Section titled “Characteristics”
AspectDescription
CouplingLoose—services only share event contracts
DataMinimal—typically just identifiers
Query patternConsumers may need to query producer for details
ConsistencyEventually consistent

When to Use

Section titled “When to Use”
  • Simple notification requirements
  • Consumers need fresh data on demand
  • Low event volume

Event-Carried State Transfer

Section titled “Event-Carried State Transfer”

Events carry complete state needed by consumers, eliminating synchronous queries.

Event-carried state transfer with full state in the eventEvent-carried state transfer with full state in the eventOrder ServiceKafkaShipping ServiceNo need to query Order ServiceAll required data in eventOrderCreated {order_idcustomer_nameshipping_addressitems[]}consume

Characteristics

Section titled “Characteristics”
AspectDescription
CouplingMedium—larger event contracts
DataComplete—all data consumers need
Query patternNo synchronous queries required
ConsistencyEventually consistent with local caching

When to Use

Section titled “When to Use”
  • Consumers need complete data for processing
  • Reducing inter-service dependencies
  • Building local read models

Event Sourcing

Section titled “Event Sourcing”

Store all state changes as an immutable sequence of events. Current state is derived by replaying events.

Event sourcing write path and read pathEvent sourcing write path and read pathWrite PathRead PathCommandAggregateProjectionRead ModelEvent Store(Kafka topic)Events are immutableTopic uses log compactionor infinite retentionvalidateappend eventsconsumeupdate

Event Store Design

Section titled “Event Store Design”
Topic: orders (compacted or infinite retention)


Key: order_id
Events:
  OrderCreated { order_id, customer_id, items }
  ItemAdded { order_id, item }
  ItemRemoved { order_id, item_id }
  OrderSubmitted { order_id, submitted_at }
  OrderCancelled { order_id, reason }

Characteristics

Section titled “Characteristics”
AspectDescription
AuditComplete history of all changes
ReplayRebuild state from any point in time
Temporal queriesQuery state at any historical moment
ComplexityHigh—requires event versioning, snapshots

Implementation Considerations

Section titled “Implementation Considerations”
ConcernSolution
Event versioningInclude version in events, use upcasters
SnapshotsPeriodically snapshot state to reduce replay time
CompactionUse log compaction to retain latest per key
Schema evolutionUse Schema Registry with compatibility rules

CQRS (Command Query Responsibility Segregation)

Section titled “CQRS (Command Query Responsibility Segregation)”

Separate models for reading and writing data. Kafka connects the two.

CQRS command side and query side connected by KafkaCQRS command side and query side connected by KafkaCommand SideQuery SideCommand APIDomain ModelWrite StoreQuery APIProjectionRead Store(Cassandra)KafkaEvents synchronizeread and write modelscommandspersistpublish eventsconsumeupdatequery

Read Model Optimization

Section titled “Read Model Optimization”
Read PatternOptimized StoreExample
Key-value lookupCassandraUser by ID
Full-text searchElasticsearchProduct search
AggregationsClickHouseAnalytics
Graph queriesNeo4jRecommendations

Characteristics

Section titled “Characteristics”
AspectDescription
ScalabilityIndependent scaling of read/write
OptimizationEach model optimized for its access pattern
ConsistencyEventually consistent between models
ComplexityHigh—multiple data stores, synchronization

Saga Pattern

Section titled “Saga Pattern”

Coordinate distributed transactions across services using events.

Saga choreography across order, payment, and inventory servicesSaga choreography across order, payment, and inventory servicesOrder SagaOrder ServicePayment ServiceInventory ServiceKafkaChoreography: Services react to eventsNo central coordinatorOrderCreatedcomplete orderreserve paymentPaymentReservedreserve itemsItemsReserved

Saga Types

Section titled “Saga Types”
TypeDescriptionProsCons
ChoreographyServices react to eventsLoose couplingHard to track flow
OrchestrationCentral coordinatorClear flowSingle point of failure

Compensation

Section titled “Compensation”

When a step fails, compensating events undo previous steps.

Happy Path:
  OrderCreated -> PaymentReserved -> ItemsReserved -> OrderCompleted


Failure (inventory unavailable):
  OrderCreated -> PaymentReserved -> ItemsUnavailable -> PaymentReleased -> OrderFailed

Implementation with Kafka

Section titled “Implementation with Kafka”
ConcernSolution
OrderingUse order_id as partition key
IdempotencyInclude saga_id, step_id in events
TimeoutUse Kafka Streams punctuators or external scheduler
Dead lettersRoute failed events to DLQ for investigation

Streaming Patterns

Section titled “Streaming Patterns”

Stateless Stream Processing

Section titled “Stateless Stream Processing”

Transform events without maintaining state.

Stateless stream processing topologyStateless stream processing topologyInput TopicKafka Streams(stateless)Output TopicOperations:- filter- map- flatMap- brancheventstransformed

Stateful Stream Processing

Section titled “Stateful Stream Processing”

Aggregate events maintaining state.

Stateful stream processing with a RocksDB state storeStateful stream processing with a RocksDB state storeInput TopicKafka Streams(stateful)State Store(RocksDB)Output TopicOperations:- count- reduce- aggregate- joineventsread/writeaggregated

Pattern Selection Guide

Section titled “Pattern Selection Guide”
RequirementRecommended Pattern
Simple decouplingEvent Notification
Avoid inter-service queriesEvent-Carried State Transfer
Complete audit trailEvent Sourcing
Separate read optimizationCQRS
Distributed transactionsSaga
Real-time transformationsKafka Streams
Section titled “Related Documentation”