Build vs Buy - Kafka Integration

When integrating external systems with Kafka, organizations face a fundamental choice: use Kafka Connect with pre-built connectors, or write custom integration code using the Producer and Consumer APIs.

---

The Integration Challenge

Every Kafka deployment requires moving data between Kafka and external systems—databases, cloud storage, APIs, legacy applications. This integration code must handle:

Concern	Description
Connectivity	Establishing and maintaining connections to external systems
Serialization	Converting between external formats and Kafka records
Error handling	Retries, dead letter queues, failure recovery
Offset management	Tracking position to enable resume after failures
Scaling	Parallelizing work across partitions and workers
Monitoring	Metrics, logging, alerting on integration health
Exactly-once processing	Connector-dependent duplicate avoidance

Building this infrastructure from scratch for each integration requires significant engineering effort. Kafka Connect exists to provide this infrastructure as a reusable framework.

---

The Two Approaches

Kafka Connect (Buy)

Use the Kafka Connect framework with pre-built or custom connectors:

• Connectors handle system-specific logic (database queries, API calls, file operations)
• Framework handles common concerns (offset management, fault tolerance, scaling)
• Configuration-driven—deploy integrations without writing code
• Ecosystem of 200+ production-grade connectors

Custom Code (Build)

Write application code using the Producer/Consumer APIs directly:

• Full control over behavior and performance characteristics
• No framework overhead—direct API access
• Application-specific logic integrated with business code
• Responsibility for all infrastructure concerns

---

When This Decision Matters

The build vs buy decision has long-term implications:

Factor	Impact
Initial velocity	Connect deploys in hours; custom takes weeks
Maintenance burden	Connect offloads to maintainers; custom requires ongoing ownership
Flexibility	Custom can do anything; Connect has framework constraints
Team expertise	Connect needs configuration skills; custom needs Kafka development expertise
Operational model	Connect standardizes operations; custom varies per integration

Neither approach is universally better—the right choice depends on specific requirements, team capabilities, and organizational context.

---

Decision Framework

---

Kafka Connect Advantages

Operational Benefits

Benefit	Description
Standardized deployment	Same process for all integrations
Built-in fault tolerance	Automatic task redistribution
Offset management	Framework handles position tracking
Monitoring	Standard JMX metrics
Configuration	REST API, no code deployment
Scaling	Add workers, increase tasks

Feature Benefits

Benefit	Description
Schema Registry integration	Automatic serialization
Single Message Transforms	Lightweight transformations
Dead letter queues	Standardized error handling
Exactly-once	Supported for compatible connectors and configurations
Converters	Multiple serialization formats

Cost Benefits

Aspect	Connect	Custom
Initial development	Configuration only	Full implementation
Maintenance	Connector updates	Full ownership
Testing	Connector tested by community	Full test suite needed
Documentation	Provided by maintainer	Must create

---

When to Use Kafka Connect

Strong Indicators

Scenario	Rationale
Standard sinks (object storage, search, databases)	Mature, tested connectors exist
CDC from databases	Debezium connectors are excellent
Cloud service integration	Vendor-maintained connectors
Operational simplicity required	Same management for all integrations
Team lacks Kafka expertise	Connect abstracts complexity

Example: S3 Integration

Using Connect (Recommended):

{
  "name": "s3-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "events",
    "s3.bucket.name": "data-lake",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "flush.size": "10000"
  }
}

Custom Implementation Required:

• S3 client setup
• Batch management
• File rotation logic
• Offset tracking
• Error handling
• Retry logic
• Exactly-once semantics
• Parquet serialization
• Partitioning logic
• Monitoring metrics

The connector encapsulates thousands of lines of production-tested code.

---

When to Build Custom

Strong Indicators

Scenario	Rationale
Complex business logic	Kafka Streams better suited
Sub-millisecond latency	Direct producer has less overhead
No suitable connector	Novel or proprietary system
Connector missing critical feature	Custom may be faster than contribution
Deep system integration	Application-specific behavior

Example: Complex Event Processing

Custom Code Better:

// Business logic interleaved with Kafka operations
streams.stream("orders")
    .filter(order -> validateOrder(order))
    .mapValues(order -> enrichWithInventory(order))
    .filter(order -> checkFraudRules(order))
    .mapValues(order -> calculatePricing(order))
    .to("validated-orders");

This logic doesn’t fit the Connect model of simple record transformation.

Example: Proprietary Protocol

Custom Code Required:

// No connector exists for legacy system
LegacyClient client = new LegacyClient(config);
Producer<String, byte[]> producer = new KafkaProducer<>(props);

while (true) {
    LegacyMessage msg = client.receive();
    ProducerRecord<String, byte[]> record =
        new ProducerRecord<>("legacy-events", msg.toBytes());
    producer.send(record);
}

---

Hybrid Approaches

Connect + Kafka Streams

Use Connect for I/O, Streams for processing.

Connect with Custom SMT

Write custom Single Message Transform for specialized logic.

public class CustomTransform implements Transformation<SinkRecord> {
    @Override
    public SinkRecord apply(SinkRecord record) {
        // Custom transformation logic
        return record.newRecord(
            record.topic(),
            record.kafkaPartition(),
            record.keySchema(),
            record.key(),
            transformedSchema,
            transformedValue,
            record.timestamp()
        );
    }
}

Configuration:

{
  "transforms": "custom",
  "transforms.custom.type": "com.example.CustomTransform"
}

---

Decision Matrix

Factor	Connect	Custom	Hybrid
Simple source/sink	✅	❌	⚠️
Complex processing	❌	✅	✅
Sub-ms latency	❌	✅	❌
Operational simplicity	✅	❌	⚠️
Schema management	✅	⚠️	✅
Exactly-once	✅	⚠️	✅
Novel integration	❌	✅	⚠️
Team expertise needed	Low	High	Medium

Legend: ✅ Recommended | ⚠️ Possible | ❌ Not Recommended

---

Total Cost of Ownership

Connect Approach

Phase	Effort
Initial setup	Hours to days
Configuration	JSON/REST
Testing	Connector validation
Maintenance	Connector upgrades
Monitoring	Standard metrics
Scaling	Configuration change

Custom Approach

Phase	Effort
Initial development	Weeks to months
Implementation	Full code base
Testing	Unit, integration, load
Maintenance	Bug fixes, feature additions
Monitoring	Custom instrumentation
Scaling	Code changes possibly needed

Break-Even Analysis

---

Connector Evaluation Checklist

When evaluating whether a connector meets requirements:

Criteria	Questions
Functionality	Does it support required features?
Performance	Can it handle expected throughput?
Reliability	What’s the production track record?
Maintainer	Is it actively maintained?
License	Compatible with deployment model?
Support	What support options exist?
Documentation	Is configuration well-documented?
Community	Are issues addressed promptly?

---

Migration Paths

Custom to Connect

1. Deploy connector alongside custom code
2. Validate connector output matches custom
3. Gradually shift traffic to connector
4. Decommission custom code

Connect to Custom

1. Implement custom code with same behavior
2. Deploy alongside connector
3. Validate output equivalence
4. Switch over
5. Remove connector

---

Related Documentation

• Kafka Connect Concepts - Connect overview
• Connector Ecosystem - Available connectors
• Cloud Storage - Data lake patterns
• Kafka Connect Guide - Implementation reference