Skip to content

AxonOps — AI-Native Control Plane for Open Source Data Platforms

Kafka Connect Exactly-Once

Kafka Connect provides framework-managed exactly-once semantics (EOS) for compatible source connectors in distributed mode. Sink delivery is at-least-once at the Kafka boundary; end-to-end exactly-once behavior depends on the sink connector and external system.


Available in Kafka 3.3+. Connect writes the source records and their source offsets in one Kafka transaction and fences tasks from obsolete generations.

# Enable exactly-once support
exactly.once.source.support=enabled

For a new Connect cluster, set enabled on every worker. For an existing distributed cluster, roll out preparing on every worker first, then perform a second rolling update to enabled. The default is disabled.

Transaction BoundaryBehavior
pollTransaction per poll() call (default)
connectorConnector defines boundaries
intervalTransaction every N milliseconds

Source task committing records and offsets in one transaction

Transaction boundaries and the connector capability check are connector settings, not worker settings:

{
"name": "eos-source",
"config": {
"connector.class": "...",
"tasks.max": "1",
"exactly.once.support": "required",
"transaction.boundary": "poll"
}
}

exactly.once.support=required rejects the configuration unless the connector reports that it can provide EOS. The default, requested, permits connectors that may support EOS but do not advertise the capability. Use transaction.boundary=connector only when the connector implements its own transaction boundaries. With interval, set transaction.boundary.interval.ms; otherwise it inherits the worker’s offset.flush.interval.ms.


The Connect framework does not make arbitrary sink writes transactional with Kafka offset commits. A sink can still provide an effectively exactly-once outcome through connector-specific support or an idempotent external data model.

StrategyImplementation
UpsertPrimary key ensures idempotent updates
DeduplicationTrack offsets in sink system
External transactionsCommit offset with sink transaction

Use natural keys for idempotent writes:

{
"name": "cassandra-sink",
"config": {
"connector.class": "com.datastax.oss.kafka.sink.CassandraSinkConnector",
"topics": "events",
"topic.events.keyspace.table.mapping": "analytics.events"
}
}

Cassandra’s upsert semantics ensure duplicates overwrite with same data.

Store Kafka offsets alongside data:

-- PostgreSQL example
CREATE TABLE events (
kafka_topic VARCHAR(255),
kafka_partition INT,
kafka_offset BIGINT,
event_data JSONB,
PRIMARY KEY (kafka_topic, kafka_partition, kafka_offset)
);

AspectSource EOSSink EOS
MechanismKafka transactionsIdempotent writes
Kafka version3.3+Any
ConfigurationWorker and connectorConnector/sink-specific
OverheadTransaction coordinationSink-dependent

LimitationDescription
Source connector supportConnector must be EOS-compatible
Sink external systemsMust support idempotent writes
PerformanceTransaction overhead on source side
Distributed modeFramework-managed source EOS is not available in standalone mode
Zombie fencingRequires every worker in the Connect cluster to enable source EOS

Validate the connector configuration before deployment:

Terminal window
curl -X PUT \
http://connect:8083/connector-plugins/com.example.MySourceConnector/config/validate \
-H 'Content-Type: application/json' \
-d @connector-config.json | jq

The status endpoint does not report whether records were committed exactly once. Test failure and task-fencing scenarios, then consume the output with isolation.level=read_committed and verify the external source positions against the committed records.