9. Kafka Connect
Chapters 4 and 5 showed how to write producers and consumers by hand. Most data that enters or leaves Kafka, though, lives in systems that already exist (relational databases, object stores, search indexes, SaaS APIs), and writing a custom client for each one means reimplementing offset tracking, retries, serialization, scaling, and failover every time. Kafka Connect is the framework that does this work once: you supply a configuration, a connector plugin does the system-specific reading or writing, and Connect runs it reliably across a cluster of workers. This chapter covers the runtime architecture, the most common connector families (JDBC, Debezium change data capture, object storage), the transformation and error-handling pipeline, and how to manage, extend, and scale Connect clusters.
<a id="91-connect-architecture"></a>
9.1 Connect Architecture
This section explains the runtime that executes connectors: the processes that host them (workers), the units of parallel work (tasks), the two deployment modes, the Kafka topics a distributed cluster uses to store its state, and the converters that turn records into bytes and back.
Workers and Tasks
Theory
Imagine copying every new row of a PostgreSQL orders table into a Kafka topic with a hand-written producer. The first version is twenty lines. Then the real requirements arrive: remember which rows were already copied so a restart does not start over, retry when the database or a broker is briefly unavailable, serialize rows in a format downstream teams agree on, run on more than one machine so a host failure does not stop the pipeline, split the work when there are fifty tables instead of one, and expose health and metrics. None of this is specific to PostgreSQL, yet every team that writes its own pipeline rebuilds all of it.
Kafka Connect splits the problem in two:
- What to copy and how to talk to the external system: a connector plugin, written once (usually by a vendor or the community) and reused by everyone.
- How to run it reliably: the Connect framework, which provides the processes, offset storage, serialization, scaling, failover, configuration, and a REST API.
Analogy: Connect is to data integration what a servlet container (Tomcat, Jetty) is to web applications. You deploy a plugin and give it a configuration, and the container handles threads, lifecycle, configuration, and restarts. The plugin only contains the logic that is unique to it.
The runtime is built from a small set of concepts:
| Concept | What it is | Example |
|---|---|---|
| Worker | A JVM process running the Connect runtime. Hosts connector instances and tasks | connect-distributed.sh on three VMs or three Kubernetes pods |
| Connector plugin | The code (JARs) implementing a SourceConnector or SinkConnector | Debezium PostgreSQL connector, Confluent S3 sink |
| Connector instance | A named, configured use of a plugin. Coordinates work but copies no data itself | orders-cdc using the Debezium PostgreSQL plugin |
| Task | A unit of work created by a connector instance. Tasks do the actual copying | orders-cdc-0, orders-cdc-1 |
| Converter | Translates between Connect's internal data model and the bytes stored in Kafka | JsonConverter, AvroConverter |
| Transform (SMT) | A lightweight per-record modification applied in the pipeline | Rename a field, mask a column, change the topic |
Connector instances and tasks
A connector instance does not move data. Its job is to look at its configuration (and sometimes at the external system) and decide how to divide the work into at most tasks.max tasks. It returns one configuration map per task, and the framework starts those tasks on the workers of the cluster.
Connector "jdbc-shop" (tasks.max = 3), configured for 7 tables
Connector instance: lists tables, then calls taskConfigs(maxTasks = 3)
|
+--> task config 0: tables = customers, orders, order_items
+--> task config 1: tables = products, stock
+--> task config 2: tables = invoices, payments
Framework: starts jdbc-shop-0, jdbc-shop-1, jdbc-shop-2 on whatever workers are available
tasks.max is a ceiling, not a target. A connector reading one sequential log (most CDC connectors) creates one task no matter how high tasks.max is. A sink connector's tasks share the partitions of its input topics through a consumer group, so tasks beyond the number of partitions sit idle (Section 5.2).
When the external system changes shape (a new table appears), the connector instance can call context.requestTaskReconfiguration(). The framework then asks it for new task configurations and restarts the tasks with them.
Where data flows
Every task runs in its own thread inside a worker and follows a fixed pipeline. The direction depends on the connector type:
SOURCE task (external system -> Kafka)
external --poll()--> SourceRecord --SMTs--> converter --bytes--> producer --> topic
system (Connect data) |
v
source offsets committed periodically to the offsets topic (Section 9.1)
SINK task (Kafka -> external system)
topic --> consumer --bytes--> converter --SinkRecord--> SMTs --put()--> external system
|
consumer offsets committed to __consumer_offsets after flush <----+
The connector code only implements poll() (source) or put() (sink). The producer, the consumer, conversion, transforms, error handling, and offset commits all belong to the framework, which is why the same connector can produce JSON in one deployment and Avro in another without changing a line of its code.
Connect also gives its embedded clients safe defaults. Source tasks' producers use acks=all, max.in.flight.requests.per.connection=1, and effectively unlimited delivery.timeout.ms, so records are not lost or reordered by retries. Sink tasks' consumers disable auto-commit and default to auto.offset.reset=earliest, so a new sink reads a topic from the beginning.
Plugin isolation
Connector plugins bring their own dependencies: one connector may need Jackson 2.13 and another Jackson 2.17. Workers load each plugin in its own class loader, discovered from the directories listed in plugin.path:
/opt/connect/plugins/ <- plugin.path
|-- debezium-connector-postgres/ <- one directory = one isolated plugin
| |-- debezium-connector-postgres-3.x.jar
| |-- debezium-core-3.x.jar
| `-- postgresql-42.x.jar
|-- confluentinc-kafka-connect-jdbc/
| `-- lib/
| |-- kafka-connect-jdbc-10.x.jar
| `-- postgresql-42.x.jar <- JDBC drivers go INSIDE the plugin directory
`-- my-http-source.jar <- an uber JAR placed directly is also a plugin
Classes from connect-api, kafka-clients, and the JDK always come from the worker, which is why plugins declare them as provided dependencies (Section 9.4). Kafka 4.x workers require Java 17.
Example: inspecting a worker and its tasks
# connect-distributed.properties (excerpt)
bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092
group.id=connect-shop # workers with the same group.id form one cluster
plugin.path=/opt/connect/plugins # comma-separated list of plugin directories
listeners=http://0.0.0.0:8083 # REST API
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
# Which plugins did the worker discover?
curl -s localhost:8083/connector-plugins | jq -r '.[] | "\(.type)\t\(.class)\t\(.version)"'
# source io.debezium.connector.postgresql.PostgresConnector 3.x
# sink io.confluent.connect.jdbc.JdbcSinkConnector 10.x
# source org.apache.kafka.connect.mirror.MirrorSourceConnector 4.1.0
# Where does each task of a connector run?
curl -s localhost:8083/connectors/jdbc-shop/status | jq '{connector: .connector, tasks: .tasks}'
# {
# "connector": { "state": "RUNNING", "worker_id": "10.0.1.11:8083" },
# "tasks": [
# { "id": 0, "state": "RUNNING", "worker_id": "10.0.1.12:8083" },
# { "id": 1, "state": "RUNNING", "worker_id": "10.0.1.13:8083" },
# { "id": 2, "state": "RUNNING", "worker_id": "10.0.1.11:8083" }
# ]
# }
The connector instance and its tasks are scheduled independently: the instance can run on one worker while its tasks run on others.
Key Takeaways
- Connect separates system-specific logic (connector plugins) from reliable execution (the framework: workers, offsets, conversion, scaling, failover).
- A connector instance plans the work and splits it into at most
tasks.maxtasks. Tasks do the copying, each in its own thread on some worker. - The framework owns the producer or consumer, converters, transforms, error handling, and offset commits. Connector code implements only
poll()orput(). - Each plugin is loaded in an isolated class loader from
plugin.path. Its dependencies, including JDBC drivers, must sit inside the plugin's directory.
Practice
- (Beginner) A sink connector reads a topic with 6 partitions and is configured with
tasks.max=10. How many tasks do useful work, and why?- (Intermediate) Install two plugins in
plugin.path, start a worker, and list them withGET /connector-plugins. Then move one plugin's JAR out of its directory into the worker'slibs/folder and observe which errors or warnings appear, and explain them in terms of class loader isolation.- Interview question: "What is the difference between a connector and a task in Kafka Connect, and why does Connect need both?" Hint: think about who decides how work is divided, who performs it, and what must happen when the external system gains a new table.
Standalone vs Distributed Mode
Theory
The same connectors can run in two modes. The difference is where Connect keeps its state (connector configurations, source offsets, statuses) and whether more than one process cooperates.
- Standalone mode runs a single worker. Connector configurations come from files passed on the command line, and source offsets are stored in a local file. It is easy to start and has nothing to coordinate, but if the process or its disk is lost, the pipeline stops and its position may be lost with it.
- Distributed mode runs any number of workers sharing the same
group.id. They form a group through the broker's group coordinator (the same mechanism consumer groups use, Section 5.2), elect a leader worker, and divide connectors and tasks among themselves. All state lives in Kafka topics, so any worker can take over any task.
Analogy: standalone mode is a single cook working from recipe cards on the kitchen counter. Distributed mode is a restaurant kitchen where recipes and progress notes are pinned to a shared board (the Kafka topics), so when one cook goes home, another picks up the half-finished dish and knows exactly where it stood.
| Aspect | Standalone | Distributed |
|---|---|---|
| Processes | One worker | One or more workers with the same group.id |
| Connector configuration | .properties or .json files on the command line (REST changes are not persisted) | Created and changed through the REST API, stored in the config topic |
| Source offsets | Local file (offset.storage.file.filename) | Offsets topic in Kafka |
| Sink offsets | Consumer group offsets in Kafka (both modes) | Consumer group offsets in Kafka |
| Fault tolerance | None: the process is a single point of failure | Tasks of a failed worker are reassigned to the others |
| Scaling | Vertical only | Add workers; tasks are rebalanced |
| Exactly-once source support | Not available | Available (KIP-618, Section 9.2) |
| Typical use | Local development, demos, an agent pinned to one host (for example tailing that host's files) | Production |
How distributed workers share work
When workers join or leave, the group rebalances. Since Kafka 2.3 (KIP-415), Connect uses incremental cooperative rebalancing: instead of stopping every task in the cluster and reassigning everything, the leader computes a new assignment and only the tasks that actually move are stopped and started.
A worker that disappears is not replaced immediately. The leader waits up to scheduled.rebalance.max.delay.ms (default 5 minutes) for it to come back before handing its tasks to the others. This prevents a rolling restart from shuffling every task several times, at the cost of a delay before tasks of a crashed worker resume.
t=0 worker A: c1-0, c2-0 worker B: c1-1, c2-1 worker C: c1-2
t=1 worker C crashes. c1-2 is now unassigned. A and B keep running their tasks
t=1..5m leader waits (scheduled.rebalance.max.delay.ms) in case C restarts with the same tasks
t=5m C has not returned: c1-2 is assigned to A (or B). Nothing else is stopped
The REST API is available on every worker. Requests that change state (create, update, delete a connector) are forwarded internally to the leader, which writes them to the config topic. Every worker then reads the change from that topic. This is why all workers must be able to reach each other on their REST addresses (rest.advertised.host.name, rest.advertised.port).
Example: the same file pipeline in both modes
The FileStreamSource and FileStreamSink connectors ship with Kafka for demonstrations. Since Kafka 3.2 their JAR is not on the default classpath, so add it to plugin.path.
# config/connect-standalone.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
offset.storage.file.filename=/tmp/connect.offsets # where source offsets survive restarts
offset.flush.interval.ms=10000
plugin.path=libs/connect-file-4.1.0.jar
# config/file-source.properties
name=file-source
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
file=/tmp/input.txt
topic=lines
# Standalone: worker config first, then one or more connector configs
bin/connect-standalone.sh config/connect-standalone.properties config/file-source.properties
echo "hello connect" >> /tmp/input.txt
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic lines --from-beginning
# hello connect
# config/connect-distributed.properties
bootstrap.servers=localhost:9092
group.id=connect-local # identifies the Connect cluster
config.storage.topic=connect-local-configs
offset.storage.topic=connect-local-offsets
status.storage.topic=connect-local-status
config.storage.replication.factor=1 # only because this is a one-broker dev cluster
offset.storage.replication.factor=1
status.storage.replication.factor=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
listeners=http://0.0.0.0:8083
plugin.path=libs/connect-file-4.1.0.jar
# Distributed: start the worker (repeat on more hosts with the same file to add workers)
bin/connect-distributed.sh config/connect-distributed.properties
# Connectors are created over REST, not on the command line
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/file-source/config \
-d '{
"connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"tasks.max": "1",
"file": "/tmp/input.txt",
"topic": "lines"
}'
Note that FileStreamSource reads a file local to whichever worker runs its task, which is exactly the kind of connector that is awkward in distributed mode: if the task moves to another host, the file is not there.
Key Takeaways
- Standalone mode: one process, configuration from files, source offsets in a local file, no fault tolerance. Use it for development or host-bound agents.
- Distributed mode: workers with the same
group.idcooperate, store all state in Kafka, accept configuration over REST, and take over each other's tasks. - Incremental cooperative rebalancing moves only the tasks that must move. Tasks of a vanished worker wait up to
scheduled.rebalance.max.delay.ms(5 minutes) before reassignment. - Any worker accepts REST requests; changes are forwarded to the leader, so workers must reach each other over REST.
Practice
- (Beginner) Run the file source in standalone mode, append lines to the input file, stop the worker, append more lines, and restart. Explain why only the new lines are produced, and which file made that possible.
- (Intermediate) Start two distributed workers on different REST ports with the same
group.id, create a connector withtasks.max=2, and kill one worker. Record how long it takes for its task to move, then repeat withscheduled.rebalance.max.delay.ms=10000.- Interview question: "Your team runs a critical CDC pipeline with
connect-standalone.shon one VM because it is simpler. What risks would you point out, and what would change if you moved to distributed mode?" Hint: consider where the source offsets live, what happens when the VM or its disk is lost, and how a config change is deployed.
Connect Internal Topics
Theory
A distributed Connect cluster has no database and no local state worth keeping. Everything it needs to survive a restart of every worker is stored in three Kafka topics, which together act as the cluster's shared memory:
| Topic (worker setting) | Holds | Required settings | Defaults |
|---|---|---|---|
config.storage.topic | Connector configurations, task configurations, target states (running, paused, stopped), session keys | Exactly 1 partition, compacted | Replication factor 3 (config.storage.replication.factor) |
offset.storage.topic | Source connectors' offsets: how far each source partition has been read | Compacted | 25 partitions (offset.storage.partitions), RF 3 |
status.storage.topic | Current state of every connector and task (RUNNING, FAILED, ...) and the topics each connector uses | Compacted | 5 partitions (status.storage.partitions), RF 3 |
All three are compacted (Section 7.2) because only the latest value per key matters: the latest configuration of a connector, the latest offset of a source partition, the latest state of a task. The config topic must have a single partition because workers rely on a total order of configuration changes. A task configuration must never be read before the connector configuration it belongs to.
Workers create the topics on startup with the right settings if they do not exist. If you create them yourself (common when automatic topic creation is restricted), you must set cleanup.policy=compact and the partition counts above. Setting a replication factor of -1 uses the broker's default.
What the keys look like
config topic
connector-orders-cdc -> {"properties": {connector.class, tasks.max, ...}}
task-orders-cdc-0 -> {"properties": {... task 0 configuration ...}}
commit-orders-cdc -> {"tasks": 1} <- task configs above are now complete
target-state-orders-cdc -> {"state": "STARTED"} <- or PAUSED / STOPPED
session-key -> key used to sign internal REST requests between workers
offsets topic (source connectors only)
["file-source",{"filename":"/tmp/input.txt"}] -> {"position":1024}
^ connector name ^ source partition ^ source offset
status topic
status-connector-orders-cdc -> {"state":"RUNNING","trace":null,"worker_id":"10.0.1.11:8083","generation":7}
status-task-orders-cdc-0 -> {"state":"FAILED","trace":"org.apache.kafka...","worker_id":"10.0.1.12:8083",...}
status-topic-shop.public.orders:connector-orders-cdc -> {"topic":{"name":"shop.public.orders",...}}
The commit-<connector> record is how Connect makes a set of task configurations appear atomically: workers only act on task configurations once the matching commit record has been written.
Sink offsets are elsewhere
Sink connectors do not use the offsets topic. Each sink connector consumes through an ordinary consumer group named connect-<connector name>, and its positions live in __consumer_offsets (Section 5.3). This means standard tools work on them:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group connect-s3-archive
Rules that prevent corrupted clusters
group.idand the three topic names identify a Connect cluster together. Two clusters must never share any of the topics: they would overwrite each other's configurations and statuses. Prefix all four with the cluster name (connect-shop,connect-shop-configs, ...).- Never produce to these topics yourself. To change source offsets, use the offsets REST API (Section 9.4).
- With exactly-once source support (Section 9.2), the leader writes to the config topic transactionally, and each connector may use its own offsets topic (
offsets.storage.topicset in the connector configuration). The worker principal then needs transactional ACLs as well (Chapter 11).
Example: reading the internal topics
# Configuration history, with keys
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic connect-local-configs --from-beginning --property print.key=true
# connector-file-source {"properties":{"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector",
# "tasks.max":"1","file":"/tmp/input.txt","topic":"lines","name":"file-source"}}
# task-file-source-0 {"properties":{"file":"/tmp/input.txt","topic":"lines","task.class":"...FileStreamSourceTask",...}}
# commit-file-source {"tasks":1}
# Source offsets: the file connector stores the byte position it has read up to
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic connect-local-offsets --from-beginning --property print.key=true
# ["file-source",{"filename":"/tmp/input.txt"}] {"position":28}
# Verify the settings the worker used when creating them
bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type topics \
--entity-name connect-local-configs
# cleanup.policy=compact ...
Key Takeaways
- A distributed Connect cluster stores configurations, source offsets, and statuses in three compacted topics, so workers are stateless and replaceable.
- The config topic must have exactly one partition to preserve the order of configuration changes. Defaults: offsets 25 partitions, status 5, all with RF 3.
- Sink connectors keep their positions in
__consumer_offsetsunder the groupconnect-<connector name>, not in the offsets topic. - Each Connect cluster needs its own
group.idand its own three topics. Never write to them directly.
Practice
- (Beginner) Start a distributed worker, create a connector, pause it, and resume it. Read the config topic with
print.key=trueand identify every record those three operations produced.- (Intermediate) Two teams accidentally started Connect clusters with different
group.idvalues but the sameconfig.storage.topic. Predict the symptoms each team would see, then describe how you would separate them without losing source offsets.- Interview question: "Why does Kafka Connect require its configuration topic to have exactly one partition, while the offsets topic can have many?" Hint: compare which records depend on the order of other records, and which only depend on the latest value of their own key.
Converters
Theory
A connector should not care whether the data ends up as JSON, Avro, or Protobuf in Kafka. A JDBC source knows it read a row with an integer id, a string customer, and a decimal amount; how those values become bytes is a decision for the team that operates the pipeline, not for the connector author.
Connect therefore gives connectors a format-neutral, typed data model in org.apache.kafka.connect.data:
Schemadescribes a value: primitive types (INT8toINT64,FLOAT32,FLOAT64,BOOLEAN,STRING,BYTES), structured types (STRUCT,ARRAY,MAP), optionality, defaults, and logical types such asDecimal,Date,Time, andTimestamp.Structholds a structured value that conforms to a schema.
A converter is the bridge between that model and the bytes in Kafka. Source tasks' records pass through a converter just before the producer. Sink tasks' records pass through one just after the consumer. Keys, values, and headers each have their own converter (key.converter, value.converter, header.converter).
Analogy: converters are the power adapters of Connect. The connector's appliance always uses the same internal plug (Connect's data model). The adapter you choose decides which socket (wire format) it fits in Kafka.
SOURCE: row from DB --> Struct{id=42, customer="ada", amount=99.90} --SMTs--> value.converter --> bytes --> topic
SINK: topic --> bytes --> value.converter --> Struct{...} --SMTs--> put() --> row in target DB
^ must match the converter the producer of the topic used
Common converters
| Converter | Class | Wire format | Schema handling |
|---|---|---|---|
| String | org.apache.kafka.connect.storage.StringConverter | UTF-8 text | None. Structured values become their toString() form |
| Byte array | org.apache.kafka.connect.converters.ByteArrayConverter | Raw bytes, passed through | None. Ideal for copying data you do not need to interpret |
| Numeric | LongConverter, IntegerConverter, DoubleConverter, ... (org.apache.kafka.connect.converters) | Kafka's numeric serializers | Primitive only. Typical for numeric keys |
| JSON | org.apache.kafka.connect.json.JsonConverter | JSON | With schemas.enable=true (default) every message embeds its schema; with false, plain JSON without type information |
| Avro | io.confluent.connect.avro.AvroConverter | Avro + schema ID | Schemas stored in Schema Registry (Section 6.2) |
| Protobuf | io.confluent.connect.protobuf.ProtobufConverter | Protobuf + schema ID | Schema Registry |
| JSON Schema | io.confluent.connect.json.JsonSchemaConverter | JSON + schema ID | Schema Registry |
The Avro, Protobuf, and JSON Schema converters are not part of Apache Kafka. They are installed as plugins (Confluent's, or equivalents from Apicurio Registry). Header values default to SimpleHeaderConverter, which stores them as strings.
JsonConverter and the embedded schema
With schemas.enable=true, JsonConverter wraps each value in an envelope containing its full schema. This lets sinks that need types (a JDBC sink creating columns) work without a registry, at the cost of messages several times larger than the data:
{
"schema": {
"type": "struct", "name": "com.shop.Order", "optional": false,
"fields": [
{ "field": "id", "type": "int64", "optional": false },
{ "field": "customer", "type": "string", "optional": false },
{ "field": "amount", "type": "bytes", "optional": false,
"name": "org.apache.kafka.connect.data.Decimal", "parameters": { "scale": "2" } }
]
},
"payload": { "id": 42, "customer": "ada", "amount": "JwY=" }
}
The amount value is base64 because decimals are serialized as bytes by default. Set decimal.format=NUMERIC on the converter to write 99.90 instead.
With schemas.enable=false, the same record is {"id":42,"customer":"ada","amount":"JwY="}: compact, readable, but a sink can no longer know that amount is a decimal with scale 2.
Where converters are configured
Converters are set on the worker as defaults and can be overridden per connector. Converter-specific options use the converter setting as a prefix:
# Worker default
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false
# Connector override (in the connector's configuration)
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081
key.converter=org.apache.kafka.connect.storage.StringConverter
Example: building a record with Connect's data API
This is what a source connector does internally. The converter chosen in the configuration decides how the result is written.
import org.apache.kafka.connect.data.*;
import org.apache.kafka.connect.source.SourceRecord;
import java.math.BigDecimal;
import java.util.Map;
Schema orderSchema = SchemaBuilder.struct()
.name("com.shop.Order") // record name, used by Avro/Protobuf converters
.field("id", Schema.INT64_SCHEMA)
.field("customer", Schema.STRING_SCHEMA)
.field("amount", Decimal.schema(2)) // logical type: BigDecimal with scale 2
.field("note", Schema.OPTIONAL_STRING_SCHEMA)
.build();
Struct order = new Struct(orderSchema)
.put("id", 42L)
.put("customer", "ada")
.put("amount", new BigDecimal("99.90")); // "note" omitted: optional fields may be null
order.validate(); // throws DataException if a required field is missing
SourceRecord record = new SourceRecord(
Map.of("table", "orders"), // source partition
Map.of("id", 42L), // source offset
"shop.orders", // topic
Schema.INT64_SCHEMA, 42L, // key schema and key
orderSchema, order); // value schema and value
Typical converter mistakes
Symptom (in the task's trace) | Cause | Fix |
|---|---|---|
JsonConverter with schemas.enable requires "schema" and "payload" fields | Sink reads plain JSON but the converter expects the envelope | value.converter.schemas.enable=false for that connector |
Unknown magic byte! | AvroConverter (or another registry converter) reads data not written in the registry format | Use the converter matching what the producers wrote |
| Sink fails because it needs a schema | Data is schemaless JSON or strings, but the sink (JDBC, Parquet) needs types | Produce with a registry format, or use JSON with embedded schemas |
Values in Kafka look like Struct{id=42,...} | StringConverter used for structured values | Use a structured converter |
Key Takeaways
- Connectors work with Connect's typed data model (
Schema,Struct). Converters translate it to and from bytes, so the wire format is a deployment choice. - Source pipeline: SMTs run before the converter. Sink pipeline: the converter runs before SMTs.
JsonConverterembeds a schema in every message whenschemas.enable=true(the default). Registry-based converters (Avro, Protobuf, JSON Schema) store schemas once in Schema Registry.- A sink's converter must match how the topic was written. Most "Connect is broken" tickets are converter mismatches.
Practice
- (Beginner) Run the file source twice, once with
StringConverterand once withJsonConverter(schemas.enable=true), and compare the bytes in the topic withkafka-console-consumer.sh.- (Intermediate) A JDBC sink connector reading a topic of plain JSON fails at startup of every task. Explain why, and list two different fixes, with the trade-offs of each.
- Interview question: "Why does Kafka Connect have converters instead of letting each connector serialize its own output?" Hint: think about how many connectors and how many formats exist, and who should decide the format of a topic.
<a id="92-connectors"></a>
9.2 Connectors
This section covers what source and sink connectors are responsible for and which guarantees they give, then the connector families found in almost every Kafka deployment: JDBC, change data capture with Debezium, and object storage sinks.
Source Connectors
Theory
A source connector brings data from an external system into Kafka. The hard part is not reading the data; it is resuming correctly. After a crash, a redeploy, or a task moving to another worker, the new task must know exactly where the previous one stopped, without the external system having to remember anything about Kafka.
Connect solves this with two small maps attached to every record a source task returns from poll():
- Source partition: identifies which stream in the external system the record came from (a file name, a table, a database server, a shard).
- Source offset: identifies how far into that stream the record is (a byte position, the highest ID read, a log sequence number).
The framework writes the latest offset for each source partition to the offsets topic. When a task starts, it asks the framework for the last committed offset of each of its source partitions and resumes from there. Only the connector understands what the values mean; Connect just stores them.
Analogy: a bookmark. The source partition is which book you are reading, the source offset is the page number. Connect keeps everyone's bookmarks in a drawer (the offsets topic). Whoever picks up the book next, on whichever worker, opens it at the bookmarked page.
Anatomy of a SourceRecord
| Field | Purpose |
|---|---|
sourcePartition | Map<String, ?> naming the source stream, for example {"table": "orders"} |
sourceOffset | Map<String, ?> with the position in that stream, for example {"id": 1042} |
topic | Destination topic |
kafkaPartition | Optional explicit partition. If null, the producer's partitioner uses the key (Section 4.2) |
keySchema, key | Record key and its Connect schema |
valueSchema, value | Record value and its Connect schema |
timestamp | Optional record timestamp |
headers | Optional Connect headers |
At-least-once by default
Source offsets are committed periodically, every offset.flush.interval.ms (default 60 seconds), and only for records the producer has acknowledged. If a worker dies between commits, the replacement task resumes from the last committed offset and produces again everything after it:
poll -> records r1..r500 sent, acked
offset commit at t=60s: {"id": 500}
poll -> records r501..r800 sent, acked
worker crashes at t=95s (no commit since t=60s)
new task resumes from {"id": 500} -> r501..r800 are produced a second time
Duplicates are bounded by the commit interval, and consumers must tolerate them (Section 5.4), unless the worker and connector support exactly-once delivery.
Exactly-once source connectors (KIP-618)
Since Kafka 3.3, distributed workers can write a batch of source records and the corresponding source offsets in one Kafka transaction (Chapter 8). Either both become visible or neither does, so a restart never replays records whose offsets were already committed, and never skips records whose offsets were not.
| Setting | Where | Values |
|---|---|---|
exactly.once.source.support | Worker | disabled (default), preparing (first step of a rolling upgrade), enabled |
exactly.once.support | Connector | requested (default: use it if possible) or required (fail validation if the connector cannot guarantee it) |
transaction.boundary | Connector | poll (default: one transaction per poll() batch), interval (every transaction.boundary.interval.ms), connector (the connector decides) |
offsets.storage.topic | Connector | Optional per-connector offsets topic, so its offsets can be committed in the same transaction |
It only works if the connector's source offsets truly identify positions it can resume from deterministically, so the connector must declare support (exactlyOnceSupport(), Section 9.4). Downstream consumers must use isolation.level=read_committed to benefit.
Creating topics automatically
A source connector may write to topics that do not exist yet (Debezium creates one per table). Rather than relying on the broker's auto.create.topics.enable with cluster defaults, Connect can create topics itself with settings chosen per connector (KIP-158, enabled on the worker with topic.creation.enable=true, the default):
topic.creation.default.replication.factor=3
topic.creation.default.partitions=6
topic.creation.default.cleanup.policy=compact # any topic-level config can be set this way
# Named groups apply different settings to topics matching a pattern
topic.creation.groups=audit
topic.creation.audit.include=shop\\.audit\\..*
topic.creation.audit.partitions=1
topic.creation.audit.retention.ms=31536000000
Tuning the embedded producer
Worker-wide producer settings use the producer. prefix in the worker configuration. Per-connector overrides use producer.override. in the connector configuration, if the worker's connector.client.config.override.policy allows them (Section 9.4):
producer.override.compression.type=zstd
producer.override.linger.ms=50
producer.override.batch.size=262144
Example: a source connector with topic creation and exactly-once delivery
# Worker (all workers, after a rolling upgrade through "preparing")
exactly.once.source.support=enabled
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/file-source/config \
-d '{
"connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"tasks.max": "1",
"file": "/data/events.log",
"topic": "app-events",
"topic.creation.default.replication.factor": "3",
"topic.creation.default.partitions": "3",
"exactly.once.support": "requested",
"transaction.boundary": "poll"
}'
# "requested" because FileStreamSource does not declare exactly-once support:
# with "required" the request would be rejected by validation.
Pull, not push
The source API is a polling loop: the framework calls poll() repeatedly and the task returns whatever is new. This fits databases, logs, and APIs with cursors. Push-based sources (webhooks, devices sending to an endpoint) fit poorly because Connect cannot acknowledge an HTTP request only after the record is safely in Kafka; for those, an HTTP proxy that produces directly (for example a REST proxy or a small service with an idempotent producer) is usually a better fit.
Key Takeaways
- Every
SourceRecordcarries a source partition (which stream) and source offset (how far). Connect stores the latest offset per partition and hands it back on restart. - By default source connectors are at-least-once: records produced after the last offset commit (every
offset.flush.interval.ms, 60 s) are produced again after a crash. - Exactly-once source delivery (Kafka 3.3+, KIP-618) needs
exactly.once.source.support=enabledon workers and a connector that supports it. - Connect can create destination topics with per-connector settings (
topic.creation.*) and tune its producer per connector (producer.override.*).
Practice
- (Beginner) For a connector that copies rows from a table with an auto-increment
id, propose a source partition and a source offset. What must be true aboutidfor resuming to be correct?- (Intermediate) Run the file source with
offset.flush.interval.ms=60000, append 100 lines, kill the worker withkill -9after 30 seconds, and restart it. Count the records in the topic and explain the result.- Interview question: "How does a Kafka Connect source connector achieve exactly-once delivery, and what must the connector itself guarantee for it to work?" Hint: think about what must be committed together, and whether reading from the same offset twice always yields the same records.
Sink Connectors
Theory
A sink connector moves data from Kafka into an external system. Reading from Kafka is the easy part, because the framework does it with a normal consumer. The hard part is writing to a system Kafka does not control in a way that survives failures without losing or duplicating data.
Each sink connector is a consumer group named connect-<connector name>. Its tasks are the group members, so Kafka divides the partitions of the input topics among them (Section 5.2). Input topics are configured with topics (a list) or topics.regex (a pattern).
The framework drives each task with a simple contract:
| Method | Called when | Task's responsibility |
|---|---|---|
open(partitions) | Partitions are assigned | Prepare writers or buffers for them |
put(records) | Each time the consumer returns a batch | Write, or buffer, the records |
preCommit(offsets) / flush(offsets) | Before each offset commit (every offset.flush.interval.ms) | Make everything received so far durable in the target. Return the offsets that are safe to commit |
close(partitions) | Partitions are revoked | Flush and release resources |
The framework commits to Kafka only what preCommit declares safe. A crash therefore causes records after the last commit to be delivered to put() again.
consumer.poll() --> [converter + SMTs] --> task.put(batch 1) --> buffered
consumer.poll() --> [converter + SMTs] --> task.put(batch 2) --> buffered
...every offset.flush.interval.ms:
task.preCommit(...) --> writes buffers to the target, returns safe offsets
framework commits those offsets for group connect-<name>
Delivery guarantees of a sink
Out of the box, a sink is at-least-once. Whether duplicates matter depends on how the connector writes:
| Strategy | How it works | Effective result | Examples |
|---|---|---|---|
| Plain append | INSERT or append each record | At-least-once: duplicates after a crash | JDBC sink with insert.mode=insert, HTTP sinks |
| Idempotent write | Records are written by key, so rewriting gives the same final state | Effectively exactly-once for current state | JDBC upsert, Elasticsearch with document ID from the key |
| Deterministic output | Output names and contents depend only on Kafka offsets, so a retry overwrites the same objects | Exactly-once | S3 sink with a deterministic partitioner (see Object Storage Sink Connectors) |
| Offsets stored in the target | The task writes data and offsets in one target transaction, commits nothing to Kafka, and on open() seeks to the offsets stored in the target | Exactly-once | Some database and lakehouse sinks |
Backpressure and retries
If the target is temporarily unavailable, put() can throw RetriableException. The framework then pauses consumption and redelivers the same batch later (after the delay the task set with context.timeout(ms)). A task can also pause and resume individual partitions through its context. Any other exception from put() fails the task, unless the connector reports the bad record instead (Section 9.3).
Tuning the embedded consumer
As with sources, worker-wide settings use the consumer. prefix and per-connector overrides use consumer.override.:
consumer.override.max.poll.records=2000 # larger batches for bulk-loading sinks
consumer.override.fetch.min.bytes=1048576
consumer.override.auto.offset.reset=latest # default for sinks is earliest
Example: a sink connector and its consumer group
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/file-sink/config \
-d '{
"connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"tasks.max": "2",
"topics": "lines",
"file": "/tmp/output.txt"
}'
# The sink is an ordinary consumer group: lag is visible with standard tools
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group connect-file-sink
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
# connect-file-sink lines 0 1520 1520 0 connector-consumer-file-sink-0-...
# connect-file-sink lines 1 1498 1610 112 connector-consumer-file-sink-1-...
Consumer lag of the connect-* groups is the single most useful health signal for sinks (Section 9.4).
Key Takeaways
- A sink connector is a consumer group named
connect-<connector name>; its tasks share the input partitions, so useful parallelism is bounded by the partition count. - The framework commits offsets only after
preCommit/flushconfirms the data is durable in the target. Delivery is at-least-once by default. - Duplicates are harmless when the connector writes idempotently (upserts by key, deterministic file names) or stores offsets in the target.
RetriableExceptionfromput()makes the framework retry the same batch. Other exceptions fail the task.
Practice
- (Beginner) Create a sink connector named
orders-esand find its consumer group withkafka-consumer-groups.sh --list. What happens to the group's committed offsets when you delete and recreate the connector with the same name?- (Intermediate) A JDBC sink writes order events with plain
INSERT. After a worker crash, the target table has duplicate rows. Describe two changes, one to the data model and one to the connector configuration, that make the pipeline idempotent.- Interview question: "When does a Kafka Connect sink commit consumer offsets, and why can't it simply commit after every
put()call?" Hint: consider connectors that buffer records (files, bulk loads) and what "safe to commit" means.
JDBC Connector
Theory
Relational databases are the most common source and target of Kafka pipelines, and JDBC is the universal way to talk to them from the JVM. The JDBC connector (confluentinc/kafka-connect-jdbc) uses plain SQL queries over JDBC, so it works with PostgreSQL, MySQL, Oracle, SQL Server, DB2, SQLite, and most other databases that have a JDBC driver. It is released under the Confluent Community License: free to use and modify, but not to offer as a competing hosted service.
The connector contains a source and a sink.
JDBC source: query-based polling
The source periodically runs a SELECT against each table (every poll.interval.ms, default 5 seconds) and produces the rows it has not seen yet. How it recognizes new rows depends on mode:
mode | Detects | Requirement | Weakness |
|---|---|---|---|
bulk | Nothing: copies the whole table on every poll | None | Repeats all data each time. Only for small reference tables |
incrementing | New rows | A strictly increasing column (incrementing.column.name), for example an auto-increment id | Misses updates to existing rows |
timestamp | New and updated rows | A last-modified column (timestamp.column.name) updated on every change | Rows sharing the same timestamp can be missed if the poll lands between them |
timestamp+incrementing | New and updated rows, reliably | Both columns | The recommended mode for tables with a modification timestamp |
In timestamp+incrementing mode, the source offset holds the last timestamp and ID seen, and each poll runs a query shaped like this:
SELECT * FROM orders
WHERE updated_at < ? -- upper bound: now minus timestamp.delay.interval.ms
AND ((updated_at = ? AND id > ?) -- same timestamp as the last row seen, higher id
OR updated_at > ?) -- or a newer timestamp
ORDER BY updated_at, id ASC
Query-based capture has limits that no configuration removes:
- Deletes are invisible. A deleted row simply stops appearing in query results. (Soft deletes with an
is_deletedcolumn work around it.) - Intermediate states are lost. If a row changes three times between two polls, only the last version is captured.
- Late commits can be skipped. A transaction that started earlier but commits after a later one can carry an older timestamp than rows already read.
timestamp.delay.interval.mswaits a fixed time before reading recent rows to reduce (not eliminate) this risk. - Load on the database. Every poll is a query; the timestamp and ID columns must be indexed.
These limits are the reason log-based change data capture exists (next topic).
JDBC sink: writing rows
The sink turns each record into a row. Because it must generate SQL with column names and types, it requires records with schemas (a registry converter, or JsonConverter with schemas.enable=true).
| Setting | Values | Meaning |
|---|---|---|
insert.mode | insert (default), upsert, update | upsert generates the database's native upsert (INSERT ... ON CONFLICT in PostgreSQL, MERGE in others) and makes the sink idempotent |
pk.mode | none, kafka, record_key, record_value | Where the primary key comes from. kafka uses topic, partition, and offset as the key |
pk.fields | Column names | Which fields form the primary key |
auto.create / auto.evolve | true/false | Create the table, or add columns, from the record schema |
delete.enabled | true/false | Treat tombstones (null values) as DELETE. Requires pk.mode=record_key |
table.name.format | Default ${topic} | Target table name |
Example: replicate a table between two databases
# Source: orders table in the shop database, new and updated rows every 2 seconds
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/jdbc-orders-source/config \
-d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:postgresql://shop-db:5432/shop",
"connection.user": "connect",
"connection.password": "${file:/etc/connect/secrets.properties:shop.db.password}",
"table.whitelist": "public.orders",
"mode": "timestamp+incrementing",
"timestamp.column.name": "updated_at",
"incrementing.column.name": "id",
"timestamp.delay.interval.ms": "5000",
"poll.interval.ms": "2000",
"topic.prefix": "jdbc.",
"tasks.max": "1",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema-registry:8081"
}'
# Produces to topic "jdbc.orders". The ${file:...} placeholder is explained in Section 9.4.
# Sink: upsert into the reporting database, keyed by the order id from the record value
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/jdbc-orders-sink/config \
-d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:postgresql://reporting-db:5432/reporting",
"connection.user": "connect",
"connection.password": "${file:/etc/connect/secrets.properties:reporting.db.password}",
"topics": "jdbc.orders",
"table.name.format": "orders",
"insert.mode": "upsert",
"pk.mode": "record_value",
"pk.fields": "id",
"auto.create": "true",
"auto.evolve": "true",
"tasks.max": "3",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema-registry:8081"
}'
The PostgreSQL JDBC driver must be placed in the JDBC connector's plugin directory, not in the worker's classpath; otherwise the connector fails with No suitable driver found (Section 9.1, plugin isolation). The source runs one task per table at most, so tasks.max above 1 only helps with several tables. Debezium also ships an Apache-licensed JDBC sink connector that understands Debezium change events directly, including deletes.
Key Takeaways
- The JDBC source polls tables with SQL. Use
timestamp+incrementingfor tables with a reliable last-modified column and an increasing ID. - Query-based capture cannot see deletes, loses intermediate updates, and can miss late-committing transactions. Use log-based CDC when these matter.
- The JDBC sink needs schemas.
insert.mode=upsertwith a primary key makes it idempotent, anddelete.enabled=trueturns tombstones into deletes. - JDBC drivers belong inside the connector's plugin directory.
Practice
- (Beginner) For each table, choose a JDBC source mode and justify it: a
countriestable with 250 rows that rarely changes; an append-onlyaudit_logwith an identity column; acustomerstable withidandupdated_at.- (Intermediate) Configure a JDBC source in
timestamp+incrementingmode, then update a row twice within one poll interval and delete another row. Inspect the topic and explain exactly what was and was not captured.- Interview question: "Why can a timestamp-based JDBC source connector miss rows even when every row has a correct
updated_atcolumn?" Hint: think about when a transaction's timestamp is assigned versus when its rows become visible to other sessions.
Change Data Capture with Debezium
Theory
Applications often need to tell the rest of the company that data changed: a new order, an updated address, a cancelled subscription. Two obvious approaches both fail:
- Dual writes: the application writes to its database and then produces to Kafka. If it crashes between the two, or one succeeds and the other fails, the database and Kafka disagree, and nothing repairs it.
- Polling the tables (the JDBC source above): misses deletes and intermediate states, and adds query load.
Every relational database already keeps an exact, ordered record of every committed change for its own replication and crash recovery: the MySQL binlog, the PostgreSQL write-ahead log (WAL), SQL Server's CDC tables, Oracle's redo log, MongoDB's oplog and change streams. Log-based change data capture (CDC) reads that log, just as a replica would, and turns each committed change into an event.
Debezium is the most widely used open-source CDC platform (Apache 2.0). Its connectors run on Kafka Connect as source connectors.
Analogy: polling a table is like photographing a whiteboard every five seconds. You never see what was written and erased in between, and you cannot tell what was wiped. CDC is like reading the notebook of the person who writes on the whiteboard, where every stroke and every erasure is recorded in order.
application --INSERT/UPDATE/DELETE--> PostgreSQL
|
| WAL (logical decoding, plugin pgoutput)
v
replication slot "debezium" <- remembers the position Debezium confirmed
|
Debezium PostgreSQL connector (1 task, on Connect)
|
+-----------------------------+-----------------------------+
v v v
shop.public.orders shop.public.customers shop.public.payments
(one topic per table: <topic.prefix>.<schema>.<table>, keyed by primary key)
Snapshot, then stream
When a Debezium connector starts for the first time, the log no longer contains the full history of the tables, so it first takes a consistent snapshot of the existing rows (events with op = r, for "read"), then switches to streaming changes from the exact log position at which the snapshot was taken. Its source offset is a log position (an LSN in PostgreSQL, a binlog file and position or GTID set in MySQL), so after a restart it resumes from the log without snapshotting again.
snapshot.mode (common values) | Behavior |
|---|---|
initial (default) | Snapshot on first start, then stream |
no_data | Capture the schema only, stream changes from now on |
when_needed | Snapshot whenever the stored offset is missing or no longer available in the log |
initial_only | Snapshot, then stop |
Large tables can also be re-snapshotted later while streaming continues, using incremental snapshots triggered through a signaling table (signal.data.collection).
The change event
Each event's key is the row's primary key. The value is an envelope describing the change:
| Field | Meaning |
|---|---|
before | Row state before the change (null for inserts). For PostgreSQL, full old rows require REPLICA IDENTITY FULL on the table |
after | Row state after the change (null for deletes) |
op | c create, u update, d delete, r snapshot read, t truncate |
source | Metadata: connector, database, schema, table, transaction ID, log position, commit time |
ts_ms | When Debezium processed the event |
A delete produces a delete event (op = d) followed by a tombstone (same key, null value), so that log compaction can eventually remove the key entirely (Section 7.2). Disable it with tombstones.on.delete=false if no consumer needs it.
Example: capturing a PostgreSQL database
-- postgresql.conf: wal_level = logical (requires a restart)
CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '...';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium; -- needed for snapshots
CREATE PUBLICATION dbz_publication FOR TABLE public.orders, public.customers;
ALTER TABLE public.orders REPLICA IDENTITY FULL; -- optional: full "before" images
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/shop-cdc/config \
-d '{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "shop-db",
"database.port": "5432",
"database.user": "debezium",
"database.password": "${file:/etc/connect/secrets.properties:shop.cdc.password}",
"database.dbname": "shop",
"topic.prefix": "shop",
"plugin.name": "pgoutput",
"slot.name": "debezium_shop",
"publication.name": "dbz_publication",
"table.include.list": "public.orders,public.customers",
"heartbeat.interval.ms": "10000",
"decimal.handling.mode": "string",
"tasks.max": "1"
}'
# decimal.handling.mode=string emits NUMERIC columns as "99.90". The default ("precise")
# uses Connect's Decimal logical type, which JsonConverter writes as base64 bytes (Section 9.1).
An UPDATE orders SET status = 'SHIPPED' WHERE id = 42 produces this value on shop.public.orders (shown with JsonConverter and schemas.enable=false):
{
"before": { "id": 42, "customer_id": 7, "status": "PAID", "amount": "99.90" },
"after": { "id": 42, "customer_id": 7, "status": "SHIPPED", "amount": "99.90" },
"source": {
"connector": "postgresql", "name": "shop", "db": "shop",
"schema": "public", "table": "orders",
"txId": 78123, "lsn": 24023128, "ts_ms": 1761210942000
},
"op": "u",
"ts_ms": 1761210942153
}
Flattening the envelope
Many sinks (a JDBC sink, a search index) want only the new row state. Debezium's ExtractNewRecordState transform (a Single Message Transform, Section 9.3) replaces the envelope with after, and can turn deletes into tombstones for the sink:
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.delete.tombstone.handling.mode=tombstone # deletes become null-value records
transforms.unwrap.add.fields=op,source.ts_ms # keep some metadata as __op, __source_ts_ms
The transactional outbox
CDC also solves the dual-write problem for events that are not simply table rows. The application writes its business data and an event row into an outbox table in the same database transaction. Debezium captures the outbox table, and the io.debezium.transforms.outbox.EventRouter transform routes each outbox row to a topic named after its aggregate type, using the aggregate ID as the key. The event is published if and only if the transaction committed. The pattern is described in Section 15.1.
Operating CDC safely
| Concern | Why it matters | Mitigation |
|---|---|---|
| Replication slot retains WAL | PostgreSQL keeps all WAL the slot has not confirmed. A stopped connector can fill the database's disk | Alert on slot lag; set max_slot_wal_keep_size (PostgreSQL 13+); drop slots of decommissioned connectors |
| Idle captured tables | If captured tables are quiet but others are busy, the slot's confirmed position does not advance | heartbeat.interval.ms (and optionally heartbeat.action.query) |
| One task per connector | The log is a single ordered stream | Scale by splitting databases across connectors, not by tasks.max |
| Schema changes | Columns added or removed change the event schema | Use a schema registry with compatible evolution (Section 6.2). MySQL connectors also keep a schema history topic (schema.history.internal.kafka.topic) |
| Lost log position | If the log was purged past the stored offset, streaming cannot resume | snapshot.mode=when_needed, or a new snapshot |
Key Takeaways
- Log-based CDC reads the database's own replication log, so it captures every committed insert, update, and delete in commit order, with little load on the database.
- Debezium snapshots existing data first (
op = r), then streams changes. Its source offset is a log position. - Events carry
before,after,op, andsource. Deletes are followed by a tombstone for compaction.ExtractNewRecordStateflattens events for sinks. - CDC plus an outbox table removes dual writes. Monitor replication slots: an unconsumed slot can fill the database's disk.
Practice
- (Beginner) Insert, update, and delete a row in a table captured by Debezium. Write down the
op,before, andafterof each event, and explain the extra record after the delete.- (Intermediate) Stop a Debezium PostgreSQL connector for an hour on a busy database. Query
pg_replication_slotsandpg_current_wal_lsn()to measure how much WAL is retained, then explain what would happen if the connector stayed down for a week.- Interview question: "A service must update its database and publish an event to Kafka. Why is writing to both from application code unreliable, and how does CDC with an outbox table fix it?" Hint: list the failure points between the two writes, then consider what a single database transaction guarantees.
Object Storage Sink Connectors
Theory
Kafka is not meant to keep every event forever at the lowest possible cost, and analytical tools (Spark, Trino, Athena, BigQuery, Snowflake) read files from object storage far more efficiently than they read topics. Object storage sinks continuously copy topics into Amazon S3, Google Cloud Storage, or Azure Blob Storage as files, turning a stream into a data lake that can be queried, archived, or replayed.
Object stores have properties that shape how these connectors work:
- Objects are immutable and written whole. A file cannot be appended to, so the connector buffers records per output file and uploads the file when it is complete.
- Many small files are expensive. Query engines slow down and request costs grow with the number of objects, so files should be large (tens to hundreds of megabytes).
- The layout is the index. Query engines prune data by path (
year=2026/month=10/day=23/), so the directory structure matters as much as the content.
The configuration reflects these concerns:
| Concern | Settings (Confluent S3 sink naming) |
|---|---|
| Where | s3.bucket.name, s3.region, topics.dir (default topics) |
| File format | format.class: JSON, Avro, Parquet, or raw bytes. Parquet and Avro need schemas |
| Directory layout | partitioner.class: DefaultPartitioner (by Kafka partition), FieldPartitioner (by record fields), TimeBasedPartitioner (by time, with path.format, partition.duration.ms, timestamp.extractor) |
| When to close a file | flush.size (records per file), rotate.interval.ms (span of record timestamps per file), rotate.schedule.interval.ms (wall-clock schedule) |
How files are named and why it gives exactly-once
Each file's name contains the topic, the Kafka partition, and the offset of its first record:
s3://shop-lake/topics/orders/year=2026/month=10/day=23/hour=14/orders+0+0000004200.snappy.parquet
^ ^ ^ partitioner output ^ ^ ^ first offset in the file
topics.dir topic topic Kafka partition
The connector commits Kafka offsets only after a file is uploaded. If a task crashes mid-file, it restarts from the last committed offset, rebuilds the same file from the same records, and uploads it under the same name, overwriting any partial attempt. As long as the file boundaries depend only on the data (record counts, record timestamps, record fields), the result is exactly-once: every record appears in exactly one file.
That condition fails when boundaries depend on something outside the data: rotate.schedule.interval.ms (wall clock) or a TimeBasedPartitioner with timestamp.extractor=Wallclock. After a retry, the same records can land in differently named files, producing duplicates. Use them only when duplicates are acceptable.
Memory sizing
Each task keeps an upload buffer for every output file it has open. With a time-based layout, a task assigned 10 partitions writing to 2 hourly directories at once (late data from the previous hour) has 20 open files. With multipart uploads of s3.part.size (25 MB by default for the Confluent S3 sink), that is about 500 MB of heap for one task. Many partitions per task, high-cardinality FieldPartitioner fields, or large part sizes are a common cause of OutOfMemoryError in sink workers.
Example: hourly Parquet files on S3
curl -s -X PUT -H "Content-Type: application/json" \
localhost:8083/connectors/s3-orders/config \
-d '{
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"topics": "orders",
"tasks.max": "6",
"s3.bucket.name": "shop-lake",
"s3.region": "eu-west-1",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
"parquet.codec": "snappy",
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"path.format": "'\''year'\''=YYYY/'\''month'\''=MM/'\''day'\''=dd/'\''hour'\''=HH",
"partition.duration.ms": "3600000",
"timestamp.extractor": "Record",
"locale": "en-US",
"timezone": "UTC",
"flush.size": "100000",
"rotate.interval.ms": "600000",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema-registry:8081"
}'
# path.format is the Java date pattern 'year'=YYYY/'month'=MM/...; the '\'' sequences only escape
# single quotes for the shell. A file closes after 100,000 records or 10 minutes of record time.
AWS credentials come from the standard provider chain (environment, instance profile, or IRSA on Kubernetes), not from the connector configuration.
Choosing a connector
| Option | License | Notes |
|---|---|---|
| Confluent S3, GCS, Azure Blob sinks | Confluent Community License (S3), commercial (some others) | Most widely deployed; the settings above |
| Aiven S3 and GCS connectors | Apache 2.0 | Similar concepts, different setting names |
Apache Iceberg sink (org.apache.iceberg.connect.IcebergSinkConnector) | Apache 2.0 | Writes Iceberg tables instead of loose files: atomic commits, schema evolution, and table metadata coordinated through a control topic |
Writing to a table format (Iceberg, Delta Lake, Hudi) instead of plain files is increasingly common, because query engines then see a consistent table with snapshots, rather than having to discover files and tolerate partially written directories.
Key Takeaways
- Object storage sinks turn topics into files for analytics and archiving. They buffer per output file and upload complete objects.
- The partitioner decides the directory layout;
flush.sizeandrotate.interval.msdecide file size. Aim for large files and a layout matching common query filters. - File names contain topic, partition, and start offset. With data-determined boundaries, retries overwrite the same files, which gives exactly-once. Wall-clock rotation breaks this.
- Memory grows with open files per task times the upload part size. Table-format sinks (Iceberg) add atomic, queryable commits on top.
Practice
- (Beginner) Given
topics.dir=archive, topicclicks,DefaultPartitioner, and a file whose first record is offset 1,500,000 on partition 3, write the expected object key for a JSON file.- (Intermediate) A sink uses
TimeBasedPartitionerwithtimestamp.extractor=Wallclockandrotate.schedule.interval.ms=600000. After a worker restart, analysts report duplicate rows. Explain the mechanism and change the configuration so it is exactly-once again, stating what you give up.- Interview question: "How does the S3 sink connector achieve exactly-once delivery without transactions in S3?" Hint: think about what determines a file's name and contents, and what happens when the same records are processed twice.
<a id="93-data-transformation-and-error-handling"></a>
9.3 Data Transformation and Error Handling
This section covers how records are modified in flight with Single Message Transforms, how predicates apply transforms selectively, and how Connect behaves when a record cannot be converted, transformed, or written: failing fast, skipping, retrying, or routing the record to a dead letter queue.
Single Message Transforms
Theory
Data rarely arrives in exactly the shape the destination needs. A CDC event has an envelope the search index does not want, a column holds a credit card number that must not leave the database, the topic name shop.public.orders should be orders, the key should be the order ID instead of null. Writing a Kafka Streams application (Chapter 10) for each such adjustment means another service to deploy and another topic to store.
Single Message Transforms (SMTs) are small, configurable functions that Connect applies to each record inside the connector pipeline. They take one record and return one record (modified), or null to drop it. They run in the worker, so they add no topics, no services, and almost no latency.
Analogy: SMTs are the stations on a conveyor belt at a packaging plant. Each station does one simple thing to each parcel (stick on a label, remove a leaflet, divert to another belt), and the order of the stations matters. Anything that requires looking at several parcels together belongs in a different department.
Where SMTs run
SOURCE: poll() -> SourceRecord -> [SMT 1] -> [SMT 2] -> ... -> converter -> producer
SINK: consumer -> converter -> SinkRecord -> [SMT 1] -> [SMT 2] -> ... -> put()
Transforms always see Connect records (with schemas and structured values), never raw bytes. Each transform receives the output of the previous one.
Configuring a chain
transforms=route,mask,addSource # names, applied in this order
transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.route.regex=shop\\.public\\.(.*)
transforms.route.replacement=$1 # shop.public.orders -> orders
transforms.mask.type=org.apache.kafka.connect.transforms.MaskField$Value
transforms.mask.fields=card_number
transforms.addSource.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.addSource.static.field=source_system
transforms.addSource.static.value=shop-db
Many transforms come in two variants: $Key operates on the record key and $Value on the record value.
Built-in transforms (Apache Kafka)
| Transform | Does | Typical use |
|---|---|---|
InsertField | Adds a field with a static value or record metadata (topic, partition, offset, timestamp) | Lineage: record where data came from |
ReplaceField | Keeps (include), drops (exclude), or renames (renames) fields | Remove internal columns, align names |
MaskField | Replaces field values with a null-equivalent or a fixed replacement | Hide PII before it reaches Kafka |
ValueToKey | Builds the key from fields of the value | Set a key for partitioning or compaction |
ExtractField | Replaces the key or value with one of its fields | Turn a single-field struct key into a plain ID |
HoistField | Wraps the key or value in a struct with one field | Give a primitive value a name |
Flatten | Flattens nested structs to parent.child fields | Sinks that do not support nesting |
Cast | Changes field or whole-value types | int64 to string, decimals to float64 |
TimestampConverter | Converts between Unix epoch, string, and Connect date/time types | Normalize date columns |
SetSchemaMetadata | Changes the schema name or version | Control the Avro record name |
RegexRouter | Rewrites the topic name with a regular expression | Drop prefixes, merge topics |
TimestampRouter | Builds the topic name from the topic and record timestamp | Daily indexes in Elasticsearch (logs-20261023) |
InsertHeader, DropHeaders, HeaderFrom | Add a static header, remove headers, copy or move fields into headers | Metadata in headers instead of the payload |
Filter | Drops records (used with a predicate, next topic) | Skip tombstones or unwanted topics |
Connector projects add their own, such as Debezium's ExtractNewRecordState and EventRouter (Section 9.2).
Example: shaping CDC events for a search index
A record from shop.public.customers after Debezium's unwrap transform:
topic: shop.public.customers
key: Struct{id=7}
value: Struct{id=7, name="Ada", email="ada@example.com", tax_id="RSSMRA80A01H501U", created_at=1761210000000}
transforms=unwrap,route,dropPii,keyId
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.route.regex=shop\\.public\\.(.*)
transforms.route.replacement=search-$1
transforms.dropPii.type=org.apache.kafka.connect.transforms.ReplaceField$Value
transforms.dropPii.exclude=tax_id
transforms.keyId.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.keyId.field=id # Struct{id=7} -> 7, used as the document ID
topic: search-customers
key: 7
value: Struct{id=7, name="Ada", email="ada@example.com", created_at=1761210000000}
Writing a custom transform
A transform implements Transformation<R extends ConnectRecord<R>>. This one lowercases a configurable string field, handling both records with schemas (Struct) and schemaless records (Map):
package com.shop.connect;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.connect.connector.ConnectRecord;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.transforms.Transformation;
import org.apache.kafka.connect.transforms.util.SimpleConfig;
import java.util.HashMap;
import java.util.Map;
public class LowercaseField<R extends ConnectRecord<R>> implements Transformation<R> {
public static final ConfigDef CONFIG_DEF = new ConfigDef()
.define("field", ConfigDef.Type.STRING, ConfigDef.NO_DEFAULT_VALUE,
ConfigDef.Importance.HIGH, "Name of the string field to lowercase");
private String field;
@Override
public void configure(Map<String, ?> props) {
field = new SimpleConfig(CONFIG_DEF, props).getString("field");
}
@Override
public R apply(R record) {
if (record.value() == null) {
return record; // leave tombstones untouched
}
Object newValue;
if (record.valueSchema() == null) { // schemaless: value is a Map
@SuppressWarnings("unchecked")
Map<String, Object> copy = new HashMap<>((Map<String, Object>) record.value());
copy.computeIfPresent(field, (k, v) -> v.toString().toLowerCase());
newValue = copy;
} else { // with schema: value is a Struct
Struct original = (Struct) record.value();
Struct copy = new Struct(original.schema()); // same schema, field type unchanged
original.schema().fields().forEach(f -> copy.put(f, original.get(f)));
String current = original.getString(field);
copy.put(field, current == null ? null : current.toLowerCase());
newValue = copy;
}
// newRecord keeps topic, partition, key, and timestamp, replacing only the value
return record.newRecord(record.topic(), record.kafkaPartition(),
record.keySchema(), record.key(), record.valueSchema(), newValue,
record.timestamp(), record.headers());
}
@Override
public ConfigDef config() { return CONFIG_DEF; }
@Override
public void close() { }
}
Never modify the incoming record or its value in place: other components may hold references to it. Package the class like a connector plugin (Section 9.4) and reference it as transforms.lower.type=com.shop.connect.LowercaseField.
When not to use SMTs
| Need | Why an SMT is the wrong tool | Use instead |
|---|---|---|
| Joins, lookups against other data | SMTs see one record at a time and should not call external services | Kafka Streams or Flink (Chapter 10) |
| Aggregation, deduplication, windowing | Requires state across records | Stream processing |
| Splitting one record into several | An SMT returns at most one record | Stream processing |
| Heavy business logic | Hidden inside configuration, hard to test and version, runs in the shared worker JVM | A dedicated processing service |
A useful rule: SMTs are for structural, stateless adjustments (renaming, routing, masking, type casting). If you need to explain the transform's business meaning, it probably belongs in code.
Key Takeaways
- SMTs modify one record at a time inside the Connect pipeline: after
poll()and before the converter for sources, after the converter and beforeput()for sinks. transformslists the chain in order. Each transform's settings use the prefixtransforms.<name>., and$Key/$Valueselect which part of the record it touches.- Built-in SMTs cover routing, field selection and renaming, masking, key extraction, type casts, timestamps, headers, and filtering.
- Custom SMTs implement
Transformation, must handle schema and schemaless records, and must return new records. Stateful or multi-record logic belongs in stream processing.
Practice
- (Beginner) Write the transform configuration that renames topic
legacy_orders_v2toorders, drops the fieldinternal_flags, and adds a fieldingested_fromwith valuelegacy-erp.- (Intermediate) A sink connector needs the key to be the
customer_idfield of the value, as a plain string, so that a compacted topic keeps one record per customer. Write the chain usingValueToKey,ExtractField$Key, and, if needed,Cast$Key. Explain why the order matters.- Interview question: "When would you use a Single Message Transform and when would you write a Kafka Streams application instead?" Hint: think about state, the number of input and output records, external lookups, and where the logic is tested and deployed.
Predicates
Theory
A transform in the chain applies to every record by default. Real pipelines often need it for only some of them: unwrap only the topics that contain CDC events, skip tombstones that would crash a transform, route only records carrying a particular header. Before Kafka 2.6, the only options were separate connectors or custom transforms with hard-coded conditions.
Predicates (KIP-585) are named conditions that can be attached to any transform. If the predicate is true for a record, the transform is applied; otherwise the record passes through unchanged. Setting negate=true inverts the condition.
Analogy: a predicate is a sensor in front of a conveyor-belt station. The station acts only on parcels the sensor flags, and the others glide past untouched.
Built-in predicates
| Predicate | True when | Setting |
|---|---|---|
org.apache.kafka.connect.transforms.predicates.TopicNameMatches | The record's topic matches a regular expression | pattern |
org.apache.kafka.connect.transforms.predicates.HasHeaderKey | The record has at least one header with the given name | name |
org.apache.kafka.connect.transforms.predicates.RecordIsTombstone | The record's value is null | (none) |
Configuration
Predicates are declared once and referenced by name from transforms:
predicates=isTombstone,isOrders # declare predicates
predicates.isTombstone.type=org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
predicates.isOrders.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches
predicates.isOrders.pattern=shop\\.public\\.orders
transforms=dropTombstones,maskCard
transforms.dropTombstones.type=org.apache.kafka.connect.transforms.Filter
transforms.dropTombstones.predicate=isTombstone # Filter drops records for which the predicate is true
transforms.maskCard.type=org.apache.kafka.connect.transforms.MaskField$Value
transforms.maskCard.fields=card_number
transforms.maskCard.predicate=isOrders # only orders have card_number
Filter: predicates as a filter
Filter is a transform that drops every record it is applied to. On its own it would drop everything; combined with a predicate, it becomes a conditional filter:
| Configuration | Effect |
|---|---|
Filter + predicate P | Drop records for which P is true |
Filter + predicate P + negate=true | Keep only records for which P is true |
Example: one connector, different rules per topic
A sink reads all topics of the shop CDC connector into a data warehouse. Only the payments table needs masking, debug records marked with a header must never reach the warehouse, and tombstones must be dropped because the warehouse sink does not process deletes.
topics.regex=shop\\.public\\..*
predicates=isPayments,isDebug,isTombstone
predicates.isPayments.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches
predicates.isPayments.pattern=shop\\.public\\.payments
predicates.isDebug.type=org.apache.kafka.connect.transforms.predicates.HasHeaderKey
predicates.isDebug.name=debug
predicates.isTombstone.type=org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
transforms=dropTombstones,dropDebug,unwrap,maskIban
transforms.dropTombstones.type=org.apache.kafka.connect.transforms.Filter
transforms.dropTombstones.predicate=isTombstone
transforms.dropDebug.type=org.apache.kafka.connect.transforms.Filter
transforms.dropDebug.predicate=isDebug
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.maskIban.type=org.apache.kafka.connect.transforms.MaskField$Value
transforms.maskIban.fields=iban
transforms.maskIban.replacement=****
transforms.maskIban.predicate=isPayments
record dropTombstones dropDebug unwrap maskIban result
shop.public.orders, value=envelope pass pass flatten skip written
shop.public.payments, value=envelope pass pass flatten mask iban written, iban=****
shop.public.orders, value=envelope, header debug pass DROP - - dropped
shop.public.orders, value=null (tombstone) DROP - - - dropped
Placing the Filter transforms first avoids running later transforms on records that will be discarded, and protects transforms that would fail on a null value.
Combining conditions
Each transform references at most one predicate, and there are no AND/OR operators. Combine conditions through the structure of the chain instead:
- OR for dropping: two
Filtertransforms, one per predicate (as above). - AND: express both conditions in one predicate where possible (one regular expression covering both topic sets), or write a custom predicate.
A custom predicate implements org.apache.kafka.connect.transforms.predicates.Predicate<R> with boolean test(R record), config(), configure(), and close(), and is packaged like any other plugin.
Key Takeaways
- Predicates make a transform conditional: the transform runs only when the predicate is true (or false, with
negate=true). - Built-in predicates test the topic name, the presence of a header, or whether the record is a tombstone.
Filterplus a predicate is Connect's way to drop (or, negated, keep) records selectively.- One predicate per transform. Build combinations with several
Filtersteps or a custom predicate, and put filters early in the chain.
Practice
- (Beginner) Write a configuration that keeps only records from topics starting with
prod.and drops everything else, using one predicate and one transform.- (Intermediate) A Debezium sink with
ExtractNewRecordStatefails on topics that are not CDC topics (they come from a regular producer and have no envelope). Use a predicate so that the transform applies only to CDC topics, and explain how you would test the configuration before deploying it.- Interview question: "How would you apply a transform only to records that come from topic A and carry header
x-source?" Hint: look at what one transform can reference, and at which part of the plugin API you can extend.
Dead Letter Queues
Theory
Sooner or later a record will arrive that a pipeline cannot handle: a producer wrote plain text into an Avro topic, a field is missing that a transform requires, a value exceeds the target column's size. By default the task fails, and the whole connector stops on one bad record among millions, which is a "poison pill" (Section 5.4). Skipping the record silently loses data nobody may ever notice is missing.
A dead letter queue (DLQ) is the middle ground: the bad record is set aside in a separate Kafka topic, together with the reason it failed, and the pipeline continues. Someone can inspect, fix, and replay the set-aside records later.
Analogy: a postal sorting center that receives a letter with an unreadable address does not shut down the building, and does not throw the letter away. It puts it in a "dead letter office" bin with a note explaining the problem, and keeps sorting.
Scope: what a Connect DLQ catches
Connect's DLQ exists for sink connectors only. A source record that fails has no original Kafka bytes to preserve, so source errors can only be tolerated and logged (next topic).
| Stage of a sink pipeline | Goes to the DLQ with errors.tolerance=all? |
|---|---|
| Key, value, or header converter (deserializing) | Yes |
| Transformations (SMTs, predicates) | Yes |
put(), when the connector reports the record through the errant record reporter (KIP-610, Kafka 2.6) | Yes, if the connector supports it |
put(), when the connector throws an exception | No: the task fails (or retries, for RetriableException) |
| Consumer errors (broker unavailable, authorization) | No |
What reaches the DLQ is the original record as consumed from Kafka: the same key, value, and header bytes, before conversion. This matters because a record that failed conversion has no other representation, and because replaying it later gives the original data back to a fixed pipeline.
+------------------ ok -------------------> put() ---> target
topic --> consumer --> converter --> SMTs --+
| |
+--- error --+--> original bytes + error headers --> DLQ topic
(task keeps running)
Configuration
errors.tolerance=all # required: without it, errors fail the task
errors.deadletterqueue.topic.name=dlq.s3-orders
errors.deadletterqueue.topic.replication.factor=3 # default 3; created automatically if missing
errors.deadletterqueue.context.headers.enable=true # default false: add error details as headers
With context headers enabled, each DLQ record carries:
| Header | Content |
|---|---|
__connect.errors.topic, __connect.errors.partition, __connect.errors.offset | Where the original record came from |
__connect.errors.connector.name, __connect.errors.task.id | Which connector and task failed |
__connect.errors.stage | KEY_CONVERTER, VALUE_CONVERTER, HEADER_CONVERTER, TRANSFORMATION, TASK_PUT, ... |
__connect.errors.class.name | The converter, transform, or task class involved |
__connect.errors.exception.class.name, __connect.errors.exception.message, __connect.errors.exception.stacktrace | The error itself |
Example: catching and inspecting bad records
# A producer writes plain text into a topic the sink reads with AvroConverter
echo 'not-avro' | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
# The sink keeps running; the record is in the DLQ with headers explaining why
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic dlq.s3-orders \
--from-beginning --property print.headers=true --max-messages 1
# __connect.errors.topic:orders,__connect.errors.partition:2,__connect.errors.offset:91873,
# __connect.errors.connector.name:s3-orders,__connect.errors.task.id:1,
# __connect.errors.stage:VALUE_CONVERTER,__connect.errors.class.name:io.confluent.connect.avro.AvroConverter,
# __connect.errors.exception.class.name:org.apache.kafka.connect.errors.DataException,
# __connect.errors.exception.message:Failed to deserialize data for topic orders to Avro: ...
# not-avro
Working with a DLQ
A DLQ that nobody reads is just silent data loss with extra steps. Treat it as an operational queue:
- Alert on it. Monitor the DLQ topic's size or the task metric
deadletterqueue-produce-requests(Section 9.4), and alert on any increase for pipelines where loss matters. - Separate DLQs per connector. One shared DLQ mixes records from different topics and schemas, making replay hard.
- Replay deliberately. After fixing the cause (a converter setting, a transform, the producer), replay the records: either with a small consumer that re-produces them to the original topic, or with a second connector reading the DLQ topic using the corrected configuration.
- Expect broken order. Replayed records arrive after newer records for the same key. For sinks that keep current state (upserts), check timestamps or versions before overwriting.
- Mind sensitive data. The DLQ holds the original, unmasked bytes even if an SMT would have masked them. Protect it with the same ACLs and retention as the source topic.
Key Takeaways
- A DLQ diverts records that fail conversion or transformation (and records the connector reports as errant) to a separate topic, so one bad record does not stop a sink connector.
- Connect DLQs exist only for sink connectors and require
errors.tolerance=all. - DLQ records keep the original bytes.
errors.deadletterqueue.context.headers.enable=trueadds the source location, stage, and exception as headers. - Monitor DLQs, use one per connector, replay after fixing the cause, and secure them like the source data.
Practice
- (Beginner) Configure a sink with a DLQ and context headers, produce one valid and one invalid record, and identify the stage and exception of the failed record from its headers.
- (Intermediate) Write a small consumer (in the language of your choice) that reads a DLQ topic, filters records by
__connect.errors.stage, and re-produces converter failures to their original topic and partition after a fix. What must it do to avoid replaying the same record twice?- Interview question: "Your sink connector has a DLQ. Which kinds of failures will still stop the connector, and why can't Connect send those records to the DLQ as well?" Hint: compare errors in the framework's stages with errors raised inside the connector's own
put()and with errors reaching Kafka itself.
Error Tolerance Settings
Theory
How a pipeline should react to errors is a business decision, not a technical one. For a payments ledger, stopping on the first bad record and paging someone is correct: a skipped record is a wrong balance. For clickstream analytics, losing one malformed event in a billion is acceptable, and a stopped pipeline is not. Connect lets each connector state its policy with a small group of errors.* settings.
Errors fall into two categories:
- Transient: the operation might succeed if tried again (a broker leader moving, a schema registry briefly unreachable). These are signaled with
RetriableException. - Permanent: retrying will not help (a malformed record, a value that cannot be cast). These must be either tolerated or allowed to fail the task.
The settings
| Setting | Default | Meaning |
|---|---|---|
errors.retry.timeout | 0 | How long (ms) to keep retrying a failed operation that raised a retriable error. 0 means no retries, -1 means retry forever |
errors.retry.delay.max.ms | 60000 | Maximum delay between retries. Delays grow exponentially up to this value |
errors.tolerance | none | none: any error that is not resolved by retrying fails the task. all: skip the problematic record and continue |
errors.log.enable | false | Log each tolerated error (the failed operation and the exception) |
errors.log.include.messages | false | Include the record (topic, partition, offset, key, value) in those logs |
errors.deadletterqueue.* | (unset) | Sinks only: send skipped records to a DLQ (previous topic) |
What the settings govern
These settings apply to the stages the framework controls: converting keys, values, and headers, and applying transforms and predicates. For source connectors, since Kafka 3.2 (KIP-779), errors.tolerance=all also skips records the producer fails to write (for example, a record larger than max.request.size) instead of failing the task.
They do not change how exceptions thrown by connector code are handled:
Thrown from poll() or put() | Effect |
|---|---|
RetriableException | The framework retries the same call (sinks: redelivers the same batch), independently of errors.retry.timeout |
| Any other exception | The task fails, whatever errors.tolerance says. Connectors that want per-record tolerance in put() must use the errant record reporter |
What a failure looks like
A failed task stays in the FAILED state, with the stack trace in its status. It does not restart on its own; an operator or automation must restart it after fixing the cause (Section 9.4).
curl -s localhost:8083/connectors/jdbc-orders-sink/status | jq '.tasks[0]'
# {
# "id": 0,
# "state": "FAILED",
# "worker_id": "10.0.1.12:8083",
# "trace": "org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n
# ... Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data
# for topic jdbc.orders to Avro ..."
# }
# After fixing the cause: restart only the failed tasks
curl -s -X POST "localhost:8083/connectors/jdbc-orders-sink/restart?includeTasks=true&onlyFailed=true"
"Tolerance exceeded in error handler" is the message Connect uses when an error occurs with errors.tolerance=none; the real cause is in the Caused by lines. Worker log lines include the connector and task in their context ([jdbc-orders-sink|task-0]), which makes them easy to filter.
Example: three error-handling profiles
# 1. Strict (financial data): retry transient problems for 10 minutes, then stop and alert
errors.retry.timeout=600000
errors.retry.delay.max.ms=30000
errors.tolerance=none
# 2. Tolerant with a safety net (most sink pipelines): never stop for a bad record, but keep it
errors.retry.timeout=300000
errors.tolerance=all
errors.log.enable=true
errors.log.include.messages=false # avoid writing record contents (PII) to logs
errors.deadletterqueue.topic.name=dlq.orders-es
errors.deadletterqueue.context.headers.enable=true
# 3. Best effort (high-volume telemetry): skip and log, no DLQ
errors.tolerance=all
errors.log.enable=true
errors.log.include.messages=true # acceptable only if records contain no sensitive data
| Pipeline | Profile | Reason |
|---|---|---|
| Ledger, payments, billing | Strict | A missing record is a correctness bug. Better to stop than to drift |
| Search index, data warehouse sinks | Tolerant with DLQ | Availability matters, but bad records must be repairable |
| Metrics, logs, clickstream | Best effort | Volume is high, value per record is low |
| Source connectors | Strict or best effort | No DLQ exists for sources; decide between stopping and skipping-and-logging |
Combine tolerant profiles with monitoring of the error metrics (total-record-errors, total-records-skipped, total-errors-logged in the task-error-metrics group): an error rate that suddenly rises usually means a producer changed its format, and the DLQ is filling with everything rather than with exceptions.
Key Takeaways
errors.tolerance=none(default) fails the task on the first unrecoverable error;allskips the record and continues.errors.retry.timeout(default0) anderrors.retry.delay.max.mscontrol retries of transient errors before the tolerance policy applies.- The settings govern conversion and transformation (and, for sources since 3.2, producer write failures). Exceptions from connector code in
poll()/put()follow their own rules. - A failed task stays
FAILEDuntil restarted. Choose the profile per pipeline from the business cost of a missing record versus a stopped pipeline, and monitor error metrics.
Practice
- (Beginner) Configure a connector with
errors.tolerance=none, send it a malformed record, and read the failure from the status endpoint. Then switch toerrors.tolerance=allwitherrors.log.enable=trueand find the logged error in the worker log.- (Intermediate) A source connector fails every night with
RecordTooLargeExceptioncaused by a few huge rows. Compare three fixes: raising size limits,errors.tolerance=all, and an SMT that drops or truncates the field. Which would you choose for an orders table, and which for an application log table?- Interview question: "How do you decide between failing fast and tolerating errors in a Kafka Connect pipeline, and what do you put in place so that a tolerant pipeline does not silently lose data?" Hint: start from the cost of a missing record versus a stopped pipeline, then think about DLQs, metrics, and alerts.
<a id="94-connect-development-and-management"></a>
9.4 Connect Development and Management
This section covers the day-to-day work of running Connect: managing connectors through the REST API, writing a connector when none exists, keeping configurations and secrets under control, and scaling a Connect cluster as the number of pipelines grows.
Connect REST API
Theory
A distributed Connect cluster has no configuration files for connectors. Everything an operator does (create, inspect, pause, restart, delete, reset offsets) goes through an HTTP API that every worker exposes, by default on port 8083 (listeners=http://0.0.0.0:8083). This makes Connect easy to automate: shell scripts, CI pipelines, Kubernetes operators, and UIs all use the same endpoints.
Any worker can receive any request. Read requests are answered locally from the state every worker reads from the internal topics; requests that change configuration are forwarded to the leader worker (Section 9.1).
Connector and task states
create
|
v
UNASSIGNED ----> RUNNING <----------------+
| ^ |
pause | | resume | restart
v | |
PAUSED FAILED -----+
| ^ ^
stop | | resume | unrecoverable error
v | |
STOPPED (from RUNNING)
| State | Meaning |
|---|---|
UNASSIGNED | Not yet assigned to a worker (just created, or during a rebalance) |
RUNNING | Working normally |
PAUSED | Tasks exist and keep their resources, but do not poll or put records |
STOPPED | Tasks are shut down and their configurations removed, but the connector configuration and offsets are kept (Kafka 3.5+, KIP-875). Required before changing offsets |
FAILED | Stopped by an error. The trace field contains the stack trace. Stays failed until restarted |
RESTARTING | A restart has been requested and is in progress |
The most used endpoints
| Method and path | Purpose |
|---|---|
GET / | Worker version, commit, and Kafka cluster ID |
GET /connector-plugins | Installed connector plugins (?connectorsOnly=false also lists transforms, converters, predicates) |
PUT /connector-plugins/{class}/config/validate | Validate a configuration without creating anything |
GET /connectors | Connector names. ?expand=status&expand=info adds statuses and configurations |
POST /connectors | Create a connector from {"name": ..., "config": {...}}. Fails if it exists. Kafka 3.7+ accepts "initial_state": "STOPPED" or "PAUSED" |
PUT /connectors/{name}/config | Create or replace the configuration. Idempotent, so preferred for automation |
PATCH /connectors/{name}/config | Change only the given keys (Kafka 3.8+). A null value removes a key |
GET /connectors/{name}/config | Current configuration (secrets shown as placeholders, see Connector Configuration Management) |
GET /connectors/{name}/status | State of the connector and each task, with worker IDs and traces |
PUT /connectors/{name}/pause, /resume, /stop | Change the target state |
POST /connectors/{name}/restart?includeTasks=true&onlyFailed=true | Restart the connector and its failed tasks (Kafka 3.0+, KIP-745) |
POST /connectors/{name}/tasks/{id}/restart | Restart one task |
GET, PATCH, DELETE /connectors/{name}/offsets | Read, alter, or reset offsets (Kafka 3.6+). The connector must be STOPPED to alter or reset |
GET /connectors/{name}/topics, PUT /connectors/{name}/topics/reset | Topics the connector has used (KIP-558) |
DELETE /connectors/{name} | Delete the connector (its source offsets remain in the offsets topic) |
GET /admin/loggers, PUT /admin/loggers/{logger} | Read or change log levels at runtime. ?scope=cluster applies to all workers (Kafka 3.7+) |
Example: a connector's lifecycle from the command line
CONNECT=http://localhost:8083
# 1. Validate before deploying: error_count must be 0
curl -s -X PUT -H "Content-Type: application/json" \
$CONNECT/connector-plugins/FileStreamSinkConnector/config/validate \
-d '{"connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"tasks.max": "1", "topics": "lines"}' \
| jq '{error_count, errors: [.configs[] | select(.value.errors | length > 0) | {name: .value.name, errors: .value.errors}]}'
# { "error_count": 0, "errors": [] }
# 2. Create (or update) idempotently
curl -s -X PUT -H "Content-Type: application/json" $CONNECT/connectors/file-sink/config \
-d '{"connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"tasks.max": "1", "topics": "lines", "file": "/tmp/output.txt"}'
# 3. Overview of all connectors and their task states
curl -s "$CONNECT/connectors?expand=status" \
| jq -r 'to_entries[] | "\(.key)\t\(.value.status.connector.state)\t\([.value.status.tasks[].state] | join(","))"'
# file-sink RUNNING RUNNING
# 4. Change one setting without resending the whole config (3.8+)
curl -s -X PATCH -H "Content-Type: application/json" $CONNECT/connectors/file-sink/config \
-d '{"tasks.max": "2"}'
# 5. Turn on debug logging for one connector package, cluster-wide, then turn it off
curl -s -X PUT -H "Content-Type: application/json" \
"$CONNECT/admin/loggers/org.apache.kafka.connect.file?scope=cluster" -d '{"level": "DEBUG"}'
curl -s -X PUT -H "Content-Type: application/json" \
"$CONNECT/admin/loggers/org.apache.kafka.connect.file?scope=cluster" -d '{"level": "INFO"}'
Example: rewinding offsets
Reprocessing data used to require producing into the internal offsets topic by hand. Since Kafka 3.6 the offsets endpoints do it safely:
# Stop the connector: tasks shut down, configuration and offsets are kept
curl -s -X PUT $CONNECT/connectors/file-source/stop
# Read current source offsets
curl -s $CONNECT/connectors/file-source/offsets | jq
# { "offsets": [ { "partition": { "filename": "/tmp/input.txt" }, "offset": { "position": 28 } } ] }
# Source connector: set a new offset (re-read the file from the start)
curl -s -X PATCH -H "Content-Type: application/json" $CONNECT/connectors/file-source/offsets \
-d '{"offsets": [ { "partition": { "filename": "/tmp/input.txt" }, "offset": { "position": 0 } } ]}'
# Sink connectors use Kafka coordinates instead
curl -s -X PATCH -H "Content-Type: application/json" $CONNECT/connectors/file-sink/offsets \
-d '{"offsets": [ { "partition": { "kafka_topic": "lines", "kafka_partition": 0 },
"offset": { "kafka_offset": 1000 } } ]}'
# Or forget all offsets (the connector starts as if new)
curl -s -X DELETE $CONNECT/connectors/file-sink/offsets
curl -s -X PUT $CONNECT/connectors/file-source/resume
Operational details
409 Conflictis returned while the cluster is rebalancing or when the request raced another configuration change. Automation should retry with a short backoff.- Security: the REST API has no authentication by default, and anyone who can reach it can read configurations and run arbitrary connector code on the workers. Expose it only on a private network, use
listeners=https://...with TLS settings prefixedlisteners.https., and add authentication through a REST extension (rest.extension.classes; Kafka shipsBasicAuthSecurityRestExtensionfor HTTP Basic auth backed by a JAAS login module).admin.listenersmoves the/adminendpoints to a separate listener, or disables them when set to an empty value. - Deleting a connector does not delete its source offsets or its sink consumer group. Recreating a connector with the same name resumes from them, which is either exactly what you want or a surprising source of missing data. Reset the offsets first if you want a fresh start.
Key Takeaways
- Every worker exposes the REST API (port 8083 by default). Reads are served locally; configuration changes are forwarded to the leader.
- Use
PUT /connectors/{name}/configfor idempotent create-or-update,validatebefore deploying, and?expand=statusfor fleet overviews. STOPPED(3.5+) keeps configuration and offsets but releases tasks. Offsets can be read, altered, or reset through/offsets(3.6+) while stopped.- The API is unauthenticated by default and allows running code on the workers: restrict network access, enable TLS, and add an authentication extension.
Practice
- (Beginner) Using only
curlandjq, write a one-liner that prints the name of every connector with at least oneFAILEDtask, and another that restarts only the failed tasks of each.- (Intermediate) A sink connector must reprocess the last 24 hours of a topic. Write the sequence of REST calls (and any CLI command to find the right offsets, for example
kafka-get-offsets.shwith a timestamp) needed to do it safely.- Interview question: "Why is an unauthenticated Kafka Connect REST API a serious security risk, beyond someone deleting connectors?" Hint: consider what a newly created connector can make the worker do, which classes it can load, and which credentials it can see.
Writing a Custom Connector
Theory
Write a connector only after looking for an existing one: Debezium, the Confluent Hub catalog, Aiven, Apache Camel Kafka connectors, and Lenses Stream Reactor together cover hundreds of systems. Custom connectors make sense for internal systems, proprietary APIs, or when existing connectors lack a required guarantee.
Writing one is less work than writing a robust standalone producer or consumer, because the framework handles everything except the conversation with the external system. The connector author's real job is to design resumable positions (source) or safe writes (sink).
Classes to implement
| Class | Responsibility | Key methods |
|---|---|---|
SourceConnector / SinkConnector | Validate configuration and split the work into task configurations. No data handling | start, taskClass, taskConfigs(maxTasks), stop, config, version |
SourceTask | Read from the external system and return records with source partitions and offsets | start, poll, stop, optionally commit/commitRecord |
SinkTask | Write records to the external system and say which offsets are safe | start, put, preCommit/flush, open, close, stop |
ConfigDef | Declare settings with types, defaults, validators, and documentation | Used by validation and the REST API |
Optional capabilities are declared by overriding methods on the connector: exactlyOnceSupport(config) and canDefineTransactionBoundaries(config) for exactly-once sources (Section 9.2), alterOffsets(config, offsets) to validate or apply offset changes from the REST API, and validate(config) for cross-field checks.
Design rules that follow from how the framework works:
- Tasks must be restartable anywhere. All position information must be in the source offset or in the external system, never only in memory or on local disk.
poll()should not block for long. Returnnullor an empty list when nothing is available, so the framework can commit offsets and react tostop().- Signal transient failures with
RetriableException. The framework retries; other exceptions fail the task. - Offsets must be deterministic for exactly-once. Reading from the same offset must return the same records.
Example: an HTTP polling source connector
The connector reads events from one or more HTTP endpoints that support a cursor (GET /events?since_id=N returns a JSON array of events ordered by id). Each endpoint is a source partition; the highest ID seen is its offset. Endpoints are spread across tasks.
package com.shop.connect.http;
import org.apache.kafka.common.config.AbstractConfig;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.common.config.ConfigDef.Importance;
import org.apache.kafka.common.config.ConfigDef.Type;
import org.apache.kafka.connect.connector.Task;
import org.apache.kafka.connect.source.ExactlyOnceSupport;
import org.apache.kafka.connect.source.SourceConnector;
import org.apache.kafka.connect.util.ConnectorUtils;
import java.util.*;
public class HttpPollSourceConnector extends SourceConnector {
static final String ENDPOINTS = "endpoints";
static final String TOPIC = "topic";
static final String POLL_INTERVAL_MS = "poll.interval.ms";
static final String TASK_ENDPOINTS = "task.endpoints"; // internal: set per task
static final ConfigDef CONFIG_DEF = new ConfigDef()
.define(ENDPOINTS, Type.LIST, ConfigDef.NO_DEFAULT_VALUE, Importance.HIGH,
"Comma-separated URLs exposing GET ?since_id=N")
.define(TOPIC, Type.STRING, ConfigDef.NO_DEFAULT_VALUE, Importance.HIGH,
"Destination topic")
.define(POLL_INTERVAL_MS, Type.LONG, 10_000L, ConfigDef.Range.atLeast(1_000),
Importance.MEDIUM, "Delay between polls of each endpoint");
private Map<String, String> props;
@Override
public void start(Map<String, String> props) {
new AbstractConfig(CONFIG_DEF, props); // throws ConfigException if invalid
this.props = props;
}
@Override
public Class<? extends Task> taskClass() {
return HttpPollSourceTask.class;
}
@Override
public List<Map<String, String>> taskConfigs(int maxTasks) {
List<String> endpoints = new AbstractConfig(CONFIG_DEF, props).getList(ENDPOINTS);
int numTasks = Math.min(maxTasks, endpoints.size()); // never create idle tasks
List<Map<String, String>> configs = new ArrayList<>();
for (List<String> group : ConnectorUtils.groupPartitions(endpoints, numTasks)) {
Map<String, String> taskProps = new HashMap<>(props);
taskProps.put(TASK_ENDPOINTS, String.join(",", group));
configs.add(taskProps);
}
return configs;
}
@Override
public ExactlyOnceSupport exactlyOnceSupport(Map<String, String> config) {
return ExactlyOnceSupport.SUPPORTED; // since_id=N always yields the same events
}
@Override public void stop() { }
@Override public ConfigDef config() { return CONFIG_DEF; }
@Override public String version() { return "1.0.0"; }
}
package com.shop.connect.http;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.errors.RetriableException;
import org.apache.kafka.connect.source.SourceRecord;
import org.apache.kafka.connect.source.SourceTask;
import java.io.IOException;
import java.net.URI;
import java.net.http.*;
import java.util.*;
public class HttpPollSourceTask extends SourceTask {
private final HttpClient http = HttpClient.newHttpClient();
private final ObjectMapper json = new ObjectMapper();
private final Map<String, Long> lastIds = new HashMap<>(); // in-memory cursor per endpoint
private List<String> endpoints;
private String topic;
private long intervalMs;
private long nextPollAt;
@Override
public void start(Map<String, String> props) {
endpoints = List.of(props.get(HttpPollSourceConnector.TASK_ENDPOINTS).split(","));
topic = props.get(HttpPollSourceConnector.TOPIC);
intervalMs = Long.parseLong(props.getOrDefault(HttpPollSourceConnector.POLL_INTERVAL_MS, "10000"));
for (String endpoint : endpoints) {
// Resume from the last committed offset; null on the very first start
Map<String, Object> offset = context.offsetStorageReader().offset(sourcePartition(endpoint));
lastIds.put(endpoint, offset == null ? 0L : ((Number) offset.get("last_id")).longValue());
}
}
@Override
public List<SourceRecord> poll() throws InterruptedException {
long waitMs = nextPollAt - System.currentTimeMillis();
if (waitMs > 0) {
Thread.sleep(Math.min(waitMs, 1_000)); // short sleeps keep stop() responsive
return null; // "nothing yet": framework calls poll() again
}
nextPollAt = System.currentTimeMillis() + intervalMs;
List<SourceRecord> records = new ArrayList<>();
for (String endpoint : endpoints) {
for (JsonNode event : fetch(endpoint, lastIds.get(endpoint))) {
long id = event.get("id").asLong();
records.add(new SourceRecord(
sourcePartition(endpoint), // which stream
Map.of("last_id", id), // how far: committed by the framework
topic,
Schema.STRING_SCHEMA, String.valueOf(id), // key: event id
Schema.STRING_SCHEMA, event.toString())); // value: raw JSON event
lastIds.put(endpoint, id); // advance the cursor for the next poll only
}
}
return records;
}
private JsonNode fetch(String endpoint, long sinceId) {
HttpRequest request = HttpRequest.newBuilder(URI.create(endpoint + "?since_id=" + sinceId)).GET().build();
try {
HttpResponse<String> response = http.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() >= 500 || response.statusCode() == 429) {
throw new RetriableException("HTTP " + response.statusCode() + " from " + endpoint);
}
if (response.statusCode() != 200) {
throw new ConnectException("HTTP " + response.statusCode() + " from " + endpoint); // fails the task
}
return json.readTree(response.body());
} catch (JsonProcessingException e) {
throw new ConnectException("Invalid JSON from " + endpoint, e); // permanent: retrying won't help
} catch (IOException e) {
throw new RetriableException("GET " + endpoint + " failed", e); // network problem: retry
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RetriableException("Interrupted", e);
}
}
private static Map<String, String> sourcePartition(String endpoint) {
return Map.of("endpoint", endpoint);
}
@Override public void stop() { }
@Override public String version() { return "1.0.0"; }
}
Note that lastIds is only the cursor for the next request. If the worker crashes, the in-memory value is lost and the replacement task starts from the committed offset, which is exactly what makes at-least-once (or, with transactions, exactly-once) delivery work.
The sink side in brief
public class AuditSinkTask extends SinkTask {
private AuditClient client; // hypothetical client for the target system
private final List<SinkRecord> buffer = new ArrayList<>();
@Override public void start(Map<String, String> props) { client = AuditClient.connect(props); }
@Override
public void put(Collection<SinkRecord> records) {
buffer.addAll(records); // buffer; write in bulk at flush time
if (buffer.size() >= 5_000) writeBuffer();
}
@Override
public void flush(Map<TopicPartition, OffsetAndMetadata> offsets) {
writeBuffer(); // after this returns, the framework commits offsets
}
private void writeBuffer() {
try {
client.upsertAll(buffer); // idempotent: keyed by record key
buffer.clear();
} catch (AuditUnavailableException e) {
context.timeout(10_000); // redeliver the same batch in 10 s; re-buffered
// duplicates are harmless because writes are upserts
throw new RetriableException(e);
}
}
@Override public void stop() { if (client != null) client.close(); }
@Override public String version() { return "1.0.0"; }
}
Packaging and installation
<!-- pom.xml (excerpt) -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>connect-api</artifactId>
<version>4.1.0</version>
<scope>provided</scope> <!-- supplied by the worker; never bundle it -->
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.17.2</version> <!-- bundled: isolated in the plugin's class loader -->
</dependency>
src/main/resources/META-INF/services/org.apache.kafka.connect.source.SourceConnector
com.shop.connect.http.HttpPollSourceConnector
The service file lets workers discover the plugin through Java's ServiceLoader instead of scanning every class in every JAR. With plugin.discovery=service_load on the worker (KIP-898, Kafka 3.6+), startup is much faster; the bin/connect-plugin-path.sh list and sync-manifests commands check and add these files for third-party plugins. Install the plugin by copying its JAR and its dependencies into one directory under plugin.path (or an uber JAR directly into plugin.path), then restart the workers. Plugins are not loaded at runtime.
Testing
- Unit tests: instantiate the task, mock
SourceTaskContextandOffsetStorageReader(orSinkTaskContext), and assert on the records returned bypoll()or the writes made byput(). - Integration tests: run Kafka and a Connect worker with the plugin mounted, using Testcontainers, create the connector over REST, and assert on the topic contents. Include a restart of the worker to verify that offsets resume correctly.
Key Takeaways
- Reuse existing connectors when possible. A custom connector needs a connector class (plans tasks), a task class (moves data), and a
ConfigDef. - Source tasks resume only from source offsets; keep no essential state in memory. Sink tasks must make data durable in
flush/preCommitbefore offsets are committed. - Return
nullfrompoll()when idle, throwRetriableExceptionfor transient problems, and declareexactlyOnceSupportonly if offsets are deterministic. - Package with
connect-apiasprovided, bundle other dependencies, add aMETA-INF/servicesfile, and install intoplugin.path.
Practice
- (Beginner) In
HttpPollSourceConnector.taskConfigs, what configurations are produced for 5 endpoints withtasks.max=2, and for 1 endpoint withtasks.max=4?- (Intermediate) Extend
HttpPollSourceTaskso that endpoints returning429 Too Many Requestswith aRetry-Afterheader are not polled again before that time, without blocking the other endpoints of the same task.- Interview question: "You are writing a source connector for an API that only offers
GET /items?page=Nover a list that changes while you page through it. How would you design the source partition and offset, and can the connector support exactly-once?" Hint: ask whether re-reading from the same offset returns the same records, and whether the API exposes anything stable, such as an ID or a modification time, to use instead of page numbers.
Connector Configuration Management
Theory
A team with three connectors can manage them by hand with curl. A platform with three hundred, spread across environments, needs the same discipline as application code: configurations under version control, reviewed changes, reproducible deployments, and secrets that never appear in plain text in Git, logs, or Kafka topics.
Layers of configuration
worker configuration (connect-distributed.properties)
- cluster settings: bootstrap.servers, group.id, internal topics, plugin.path, listeners
- defaults for every connector: key.converter, value.converter, producer.*, consumer.*
- config providers and the client override policy
|
v connector settings override worker defaults
connector configuration (REST, stored in the config topic)
- name, connector.class, tasks.max, plugin-specific settings
- key.converter / value.converter (+ their options), transforms, predicates, errors.*
- producer.override.*, consumer.override.*, admin.override.* (if the policy allows)
Settings every connector shares
| Setting | Meaning |
|---|---|
name | Unique connector name. Also determines the sink's consumer group (connect-<name>) and the key of its source offsets, so renaming means a new connector without history |
connector.class | Plugin class (fully qualified, or the simple class name if unambiguous) |
tasks.max | Maximum number of tasks |
key.converter, value.converter, header.converter | Converters, overriding worker defaults |
transforms, predicates | Transform chain and predicates (Section 9.3) |
errors.* | Error handling (Section 9.3) |
topics, topics.regex | Sinks: input topics |
config.action.reload | restart (default): restart the connector when a referenced config provider value changes, if the provider supports change notifications. none: ignore |
Client overrides and the override policy
Each connector can override the settings of its embedded producer, consumer, or admin client with producer.override.*, consumer.override.*, and admin.override.*. The worker decides which overrides are allowed through connector.client.config.override.policy:
| Policy | Allows |
|---|---|
All (default since Kafka 3.0) | Any client setting |
Principal | Only security.protocol, sasl.mechanism, and sasl.jaas.config, so each connector can authenticate as its own principal |
None | No overrides |
Per-connector principals are good practice in shared clusters: each connector gets only the ACLs it needs (Chapter 11), instead of all connectors sharing the worker's identity.
Secrets with config providers
Configurations are stored in the config topic and returned by GET /connectors/{name}/config. A password written there in plain text is visible to anyone with access to either. Config providers (KIP-297) replace secret values with placeholders that the worker resolves only when starting the connector and its tasks:
# Worker configuration
config.providers=file,dir,env
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.providers.dir.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider
config.providers.env.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider
config.providers.file.param.allowed.paths=/etc/connect/secrets # restrict readable locations
| Placeholder | Resolved from |
|---|---|
${file:/etc/connect/secrets/db.properties:password} | Key password in a properties file |
${dir:/etc/connect/secrets:db-password} | Contents of the file db-password in a directory (fits Kubernetes secrets mounted as files) |
${env:DB_PASSWORD} | Environment variable of the worker process (Kafka 3.5+) |
{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:postgresql://reporting-db:5432/reporting",
"connection.user": "connect",
"connection.password": "${dir:/etc/connect/secrets:reporting-db-password}"
}
The config topic and REST responses only ever contain the placeholder. Providers for external vaults (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) are available as plugins and are configured the same way. The secret must exist at the same path or variable on every worker, because tasks can run anywhere.
Configuration as code
Keep one JSON file per connector in Git, without secrets, and let CI apply them. Because PUT /connectors/{name}/config creates or replaces, applying the same files twice is harmless:
#!/usr/bin/env bash
# deploy-connectors.sh <connect-url> <dir>: validate and apply every connector config in <dir>
set -euo pipefail
CONNECT="$1"; DIR="$2"
for file in "$DIR"/*.json; do
name="$(basename "$file" .json)" # file name = connector name
class="$(jq -r '."connector.class"' "$file")"
errors="$(curl -sf -X PUT -H "Content-Type: application/json" \
"$CONNECT/connector-plugins/$class/config/validate" \
-d "$(jq --arg n "$name" '. + {name: $n}' "$file")" | jq '.error_count')"
if [ "$errors" != "0" ]; then
echo "invalid configuration: $file" >&2; exit 1 # stop before changing anything
fi
done
for file in "$DIR"/*.json; do
name="$(basename "$file" .json)"
curl -sf -X PUT -H "Content-Type: application/json" \
"$CONNECT/connectors/$name/config" -d @"$file" > /dev/null
echo "applied $name"
done
On Kubernetes, the Strimzi operator expresses the same idea declaratively: a KafkaConnector resource per connector, reconciled against the REST API by the operator.
apiVersion: kafka.strimzi.io/v1beta2 # check which API version your Strimzi release serves
kind: KafkaConnector
metadata:
name: shop-cdc # connector name
labels:
strimzi.io/cluster: connect-shop # the KafkaConnect cluster that runs it
spec:
class: io.debezium.connector.postgresql.PostgresConnector
tasksMax: 1
config:
database.hostname: shop-db
database.dbname: shop
database.user: debezium
database.password: ${secrets:shop/shop-db-credentials:password} # Kubernetes secrets provider, set up in the KafkaConnect resource
topic.prefix: shop
plugin.name: pgoutput
What a configuration change does
Changing a connector's configuration restarts the connector and regenerates its task configurations; tasks whose configuration changed are restarted. Expect a short pause in data flow. Changing name is not a change: it creates a different connector with no offsets, no consumer group history, and no status.
Key Takeaways
- Worker settings provide cluster settings and defaults; connector settings override them;
*.override.*settings tune each connector's clients within the worker's override policy. - Never put secrets in connector configurations. Use config providers (
file,dir,env, or vault plugins) so only placeholders are stored and returned. - Store connector configurations in Git and apply them with
validateplus idempotentPUT, or declaratively with an operator such as Strimzi. - A configuration change restarts the connector and affected tasks. A new
namemeans a new connector without offsets or history.
Practice
- (Beginner) Move the database password of a JDBC connector from plain text into a file read by
FileConfigProvider. Verify withGET /connectors/{name}/configand by reading the config topic that the password no longer appears.- (Intermediate) Extend
deploy-connectors.shso that it also deletes connectors that exist in the cluster but have no file in Git, with a dry-run mode. What safeguards would you add before letting CI delete connectors in production?- Interview question: "Where can a password in a Kafka Connect connector configuration leak, and how do config providers prevent it?" Hint: follow the configuration from the REST request through the config topic, the REST responses, the logs, and the task.
Scaling Connect Clusters
Theory
A Connect deployment scales along two independent dimensions:
- Parallelism within a connector: how many tasks it runs (
tasks.max), bounded by how much the source or sink can be split. - Capacity of the cluster: how many workers exist to run all the tasks of all connectors.
Adding workers only helps if there are tasks to give them, and raising tasks.max only helps if the connector can actually divide its work. Knowing which limit you are hitting is most of capacity planning for Connect.
Analogy: tasks are lanes on a road and workers are the toll booths that serve them. Opening more booths does nothing if there are only two lanes, and painting more lanes does nothing if the bridge at the end (the external system) has only one.
What bounds the number of useful tasks
| Connector type | Unit of parallelism | Practical limit |
|---|---|---|
| Sink connectors (any) | Topic partitions | Total partitions of the input topics |
| JDBC source | Tables (or one custom query) | Number of tables; one task per table at most |
| Debezium and most CDC sources | The database's change log | One task per connector; scale by splitting databases across connectors |
| Object storage sinks | Topic partitions | Partitions, and memory per task (Section 9.2) |
| MirrorMaker 2 source | Topic partitions being replicated | Partitions (Chapter 14) |
| HTTP or API sources | Endpoints or shards defined by the connector | Number of endpoints, and the API's rate limits |
How tasks are spread over workers
The leader assigns connectors and tasks so that every worker has about the same number of them. It knows nothing about how heavy each task is: a task copying 50,000 records per second counts the same as one copying 5 per minute. When a worker joins, incremental cooperative rebalancing moves only enough tasks to even out the counts:
Before: 2 workers, 6 tasks After adding worker C
A: s3-0 s3-1 s3-2 A: s3-0 s3-1
B: cdc-0 jdbc-0 jdbc-1 B: cdc-0 jdbc-0
C: s3-2 jdbc-1 <- only these two tasks were stopped and moved
If a few tasks dominate the load, count-based balancing can still leave one worker overloaded. The remedies are to give heavy connectors more, smaller tasks (if they can be split), or to move them to a dedicated Connect cluster.
One shared cluster or several
| Topology | Advantages | Disadvantages |
|---|---|---|
| One shared cluster | Fewer processes to run, efficient use of resources | A misbehaving connector (memory leak, CPU-hungry converter, plugin dependency conflict) affects all of them; upgrades and plugin changes are coordinated across every team; rebalances touch everyone |
| Cluster per team or domain | Isolation, independent upgrades and plugin sets, clear ownership and quotas | More clusters, internal topics, and monitoring to operate |
| Dedicated cluster per critical or heavy connector | Strongest isolation, sizing tailored to one workload | Highest overhead; most workers mostly idle for small connectors |
A common middle ground is a cluster per domain plus dedicated clusters for the few pipelines with strict availability requirements (CDC for core databases) or unusual resource needs (large object storage sinks). Each cluster needs its own group.id and internal topics (Section 9.1).
Signals to watch
JMX MBean (kafka.connect:type=...) | Metrics | What they tell you |
|---|---|---|
connect-worker-metrics | connector-count, task-count, connector-startup-failure-total, task-startup-failure-total | Load per worker and failures to start |
connect-worker-rebalance-metrics | rebalancing, completed-rebalances-total, rebalance-avg-time-ms, time-since-last-rebalance-ms | Rebalance storms (frequent rebalances stop moved tasks repeatedly) |
connector-task-metrics,connector=*,task=* | status, running-ratio, batch-size-avg, offset-commit-success-percentage | Per-task health; low running-ratio means paused or stuck |
source-task-metrics,connector=*,task=* | source-record-poll-rate, source-record-write-rate, poll-batch-avg-time-ms | Source throughput; poll time shows a slow external system |
sink-task-metrics,connector=*,task=* | sink-record-read-rate, sink-record-send-rate, put-batch-avg-time-ms | Sink throughput; high put time means the target is the bottleneck |
task-error-metrics,connector=*,task=* | total-record-errors, total-records-skipped, deadletterqueue-produce-requests | Error handling activity (Section 9.3) |
For sinks, also monitor consumer lag of the connect-<name> groups (Section 12.3): growing lag with a high put-batch-avg-time-ms means the target system is the bottleneck, and more tasks will not help unless the target can absorb more parallel writes. Growing lag with low CPU on the workers and idle tasks means the partition count is the limit.
Example: scaling a sink and checking the result
CONNECT=http://localhost:8083
# How many partitions can the sink use?
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders | head -1
# Topic: orders TopicId: ... PartitionCount: 12 ReplicationFactor: 3 ...
# Raise parallelism from 4 to 12 tasks (one per partition)
curl -s -X PATCH -H "Content-Type: application/json" $CONNECT/connectors/s3-orders/config \
-d '{"tasks.max": "12"}'
# Tasks per worker across the whole cluster: are they evenly spread?
curl -s "$CONNECT/connectors?expand=status" \
| jq -r '.[].status.tasks[].worker_id' | sort | uniq -c
# 5 10.0.1.11:8083
# 5 10.0.1.12:8083
# 4 10.0.1.13:8083
# Is the lag shrinking?
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group connect-s3-orders \
| awk 'NR > 1 { lag += $6 } END { print "total lag:", lag }'
Sizing workers and autoscaling
- Heap: dominated by buffering sinks (object storage, bulk loaders) and large batches. Set it explicitly (
KAFKA_HEAP_OPTS="-Xms4g -Xmx4g") and size from the open-file arithmetic in Section 9.2. - CPU: dominated by conversion (Avro, Protobuf, JSON parsing), transforms, and compression, not by the connector logic.
- Autoscaling on Kubernetes (for example on consumer lag): useful for sink-heavy clusters, with two caveats. Scaling beyond the total number of tasks adds idle workers, and every scale event triggers a rebalance that pauses the moved tasks, so use generous stabilization windows to avoid oscillation.
- Planned restarts: rolling restarts are cheap thanks to
scheduled.rebalance.max.delay.ms, as long as each worker comes back within that window with the same identity.
Key Takeaways
- Scale tasks within the limit of the connector (partitions for sinks, tables for JDBC, one log for CDC), and scale workers to host all tasks. Neither helps beyond the other's limit.
- Tasks are balanced by count, not by load. Isolate heavy or critical connectors in dedicated clusters when count-based balancing is not enough.
- Choose between shared, per-domain, and dedicated clusters by weighing isolation and ownership against operational overhead.
- Watch task throughput, put and poll times, rebalance metrics, error metrics, and sink consumer lag to find whether Connect, the partitions, or the external system is the bottleneck.
Practice
- (Beginner) A cluster has 3 workers. It runs one Debezium connector, a JDBC source over 4 tables with
tasks.max=8, and a sink reading a 6-partition topic withtasks.max=6. How many tasks run in total, and roughly how many does each worker get?- (Intermediate) A sink connector's lag keeps growing. CPU on the workers is at 15%, and
put-batch-avg-time-mshas risen from 50 ms to 2 seconds. What is the bottleneck, which change would you try first, and which change would make it worse?- Interview question: "How would you organize Kafka Connect for a company where twenty teams each run several connectors?" Hint: weigh blast radius, plugin and version conflicts, upgrade coordination, security principals, and operational cost.
<a id="10-stream-processing"></a>