Kafka Connect Exactly-Once
Kafka Connect provides framework-managed exactly-once semantics (EOS) for compatible source connectors in distributed mode. Sink delivery is at-least-once at the Kafka boundary; end-to-end exactly-once behavior depends on the sink connector and external system.
Source Connector EOS
Section titled “Source Connector EOS”Available in Kafka 3.3+. Connect writes the source records and their source offsets in one Kafka transaction and fences tasks from obsolete generations.
Worker Configuration
Section titled “Worker Configuration”# Enable exactly-once supportexactly.once.source.support=enabledFor a new Connect cluster, set enabled on every worker. For an existing distributed cluster, roll out preparing on every worker first, then perform a second rolling update to enabled. The default is disabled.
| Transaction Boundary | Behavior |
|---|---|
poll | Transaction per poll() call (default) |
connector | Connector defines boundaries |
interval | Transaction every N milliseconds |
How It Works
Section titled “How It Works”Connector Configuration
Section titled “Connector Configuration”Transaction boundaries and the connector capability check are connector settings, not worker settings:
{ "name": "eos-source", "config": { "connector.class": "...", "tasks.max": "1", "exactly.once.support": "required", "transaction.boundary": "poll" }}exactly.once.support=required rejects the configuration unless the connector reports that it can provide EOS. The default, requested, permits connectors that may support EOS but do not advertise the capability. Use transaction.boundary=connector only when the connector implements its own transaction boundaries. With interval, set transaction.boundary.interval.ms; otherwise it inherits the worker’s offset.flush.interval.ms.
Sink Connector Delivery
Section titled “Sink Connector Delivery”The Connect framework does not make arbitrary sink writes transactional with Kafka offset commits. A sink can still provide an effectively exactly-once outcome through connector-specific support or an idempotent external data model.
Strategies
Section titled “Strategies”| Strategy | Implementation |
|---|---|
| Upsert | Primary key ensures idempotent updates |
| Deduplication | Track offsets in sink system |
| External transactions | Commit offset with sink transaction |
Upsert Pattern
Section titled “Upsert Pattern”Use natural keys for idempotent writes:
{ "name": "cassandra-sink", "config": { "connector.class": "com.datastax.oss.kafka.sink.CassandraSinkConnector", "topics": "events", "topic.events.keyspace.table.mapping": "analytics.events" }}Cassandra’s upsert semantics ensure duplicates overwrite with same data.
Offset Tracking Pattern
Section titled “Offset Tracking Pattern”Store Kafka offsets alongside data:
-- PostgreSQL exampleCREATE TABLE events ( kafka_topic VARCHAR(255), kafka_partition INT, kafka_offset BIGINT, event_data JSONB, PRIMARY KEY (kafka_topic, kafka_partition, kafka_offset));Comparison
Section titled “Comparison”| Aspect | Source EOS | Sink EOS |
|---|---|---|
| Mechanism | Kafka transactions | Idempotent writes |
| Kafka version | 3.3+ | Any |
| Configuration | Worker and connector | Connector/sink-specific |
| Overhead | Transaction coordination | Sink-dependent |
Limitations
Section titled “Limitations”| Limitation | Description |
|---|---|
| Source connector support | Connector must be EOS-compatible |
| Sink external systems | Must support idempotent writes |
| Performance | Transaction overhead on source side |
| Distributed mode | Framework-managed source EOS is not available in standalone mode |
| Zombie fencing | Requires every worker in the Connect cluster to enable source EOS |
Verification
Section titled “Verification”Validate the connector configuration before deployment:
curl -X PUT \ http://connect:8083/connector-plugins/com.example.MySourceConnector/config/validate \ -H 'Content-Type: application/json' \ -d @connector-config.json | jqThe status endpoint does not report whether records were committed exactly once. Test failure and task-fencing scenarios, then consume the output with isolation.level=read_committed and verify the external source positions against the committed records.
Related Documentation
Section titled “Related Documentation”- Kafka Connect - Connect overview
- Error Handling - DLQ configuration
- Exactly-Once Semantics - EOS concepts