Inside Kafka Connect: How Workers, Plugins, Connectors and Tasks Actually Work

The runtime that does the heavy lifting behind every Kafka data pipeline

From Experiment to Core Platform

Like many teams, we split our platform into microservices, each with its own database. It gave us the scale we wanted. It also scattered our data across dozens of databases, and rich reporting quietly became hard. Our BI team started struggling: every report meant stitching together data from several services, and querying production databases directly was never a safe option.

In my previous article, How a Unified Data Pipeline Powers the Modern Enterprise, I described the architecture we built to fix that: change data capture from each service’s database into Kafka, and from there into warehouses designed for reporting.

It started as an experiment, so we wanted something that wouldn’t cost much. It didn’t stay an experiment. Today the pipeline streams around 50 million events a day from hundreds of Postgres tables, MongoDB collections and Elasticsearch. It has become the foundation for BI, rich reporting, ETL, our semantic layer and data sources for AI.

The piece doing the actual work is Kafka Connect. The Kafka broker only stores data. It doesn’t know how to read a Postgres WAL or write to Elasticsearch. Connect does. Instead of writing code for each integration, you configure a connector: source connectors bring data into Kafka, and sink connectors take it out to warehouses, databases and S3. It’s built on Kafka’s producer and consumer APIs, but it adds the things that make custom integration code painful to maintain at scale: running many connectors in parallel, handling data formats, and moving work to healthy machines when one fails.

We first ran it on AWS MSK Connect, then moved to Strimzi on EKS, which saved us a lot of cost. That migration is the subject of my next article. This one covers what’s inside Connect: Workers, Plugins, Connectors and Tasks, and how they work together at runtime.

Building Blocks

Connect has four main components, and each does one job.

1. The Worker

A Worker is the actual software process, a JVM, running on your server or pod. Run one Worker and you have Connect in standalone mode. Run several with the same group.id and they form a distributed cluster.

Its core responsibilities:

• Provides the compute: It supplies the memory, CPU, and threads your pipelines run on.
• Hosts the REST API: It listens for HTTP commands to create, pause, update, or delete connectors.
• Runs connectors and tasks: It instantiates them, gives each its own thread, and restarts them on assignment changes.
• Handles all Kafka I/O: It owns the producers and consumers, and it applies converters and transforms.
• Reports health: It publishes the status of every connector and task it runs back to the cluster.

2. The Plugin

Out of the box, a Worker doesn’t know how to talk to PostgreSQL or Amazon S3. A plugin is a code bundle (one or more JARs, from a vendor or built by you) that you install on the Worker to teach it how. Its core responsibilities:

• Provides the integration code: Supplies the logic and drivers for a specific database, storage service, or API.
• Maps external data into Connect records: It turns rows, files, or events into Connect’s standard record format, so the Worker can handle every connector’s data the same way.

Plugins are discovered only when a Worker starts, and they must be installed on every Worker, because any Worker may be handed a connector’s tasks.

In practice, this was our biggest source of friction. A connector plugin rarely works on its own. To authenticate with AWS IAM, we had to add a separate JAR from outside the plugin. To fetch credentials from Vault, another one. Then came compatibility problems, with library versions that clashed with each other or with the Connect version we were running. Getting a set of JARs that worked together took real effort.

3. The Connector

When you submit a configuration to start moving data, the Worker spins up a Connector. Think of this as the manager thread. The Connector manages the plan, but it does not move the data itself. Its core responsibilities:

• Reads your configuration: Validates the exact rules, database names, and credentials you submitted.
• Inspects the external system: Works out what there is to copy, for example “I see 10 database tables”.
• Splits the work: Breaks that work into individual task configurations, up to tasks.max.
• Owns the lifecycle: Handles paused states, reconfiguration, and graceful shutdowns.

4. The Task

Tasks are the actual execution threads. Once the Connector plans the work, the Worker creates the Tasks that move the data. These run continuously in the background. Their core responsibilities:

• Executes the data loops: Runs the continuous loops to pull data from a source or push data to a destination.
• Holds the connections: Manages the actual physical network connection to your database or API.
• Tracks its position: Reports how far it has read.

Source Tasks (Ingestion): Run a continuous poll() loop that reads external data and hands the records to the Worker, which writes them to Kafka.

Get Jatin Bansal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Sink Tasks (Consumption): Receive records from the Worker through a continuous put() loop and write them to your external destination.

How the Pieces Fit Inside a Worker

The three layers are a simplification for this article. The official docs call the whole runtime the Worker.

Plugin, Connector, and Task are all definitions. None of them run on their own. The Worker is what brings them to life, and inside it the work splits into three layers:

• Management is where your intent lands. You tell Connect what to do through a REST API. Workers find each other through Kafka, and one of them is elected leader to decide which worker runs which piece of work.
• Engine is the shared machinery every connector gets. It reads and writes Kafka, converts data formats, applies small transforms, and tracks progress.
• Plugin is the only layer that knows anything about the outside world. The Connector plans and splits the work into tasks, and the Tasks copy the data in or out.

A connector doesn’t live on one Worker: its Connector instance runs on one, while its Tasks are spread across all of them.

Cluster Coordination

To scale, multiple Workers form a cluster. They coordinate almost entirely through your Kafka brokers, which act as a shared whiteboard. The one exception: if you send a config change to a follower, it forwards the request to the Leader over REST.

• Leader election: The group coordinator, running on a Kafka broker, tracks which Workers are alive and elects one as Leader. The Leader is an ordinary Worker with one extra job, and there is no dedicated master node.
• Work distribution: The Leader decides which Worker runs which connector and task. It hands out that assignment through Kafka’s group protocol, the same mechanism consumer groups use, while the connector and task configs themselves live in an internal topic.
• Self-healing: When a Worker disappears, the group rebalances and a new Leader is elected if needed. By default, Connect waits up to scheduled.rebalance.max.delay.ms (5 minutes) before moving the lost Worker's tasks, so a quick restart doesn't cause needless churn. If you test failover and see a pause of a few minutes, that is by design.

The Three Internal Topics

Kafka Connect runs without an external database by keeping its state in three compacted Kafka topics. Their names are configurable, and the defaults vary by distribution.

• Config topic (config.storage.topic): Stores connector configs, task configs, and whether each connector is running or paused. It must have exactly one partition, so every change is applied in order.
• Status topic (status.storage.topic): Records the state of every connector and task, including error traces.
• Offset topic (offset.storage.topic): Records where each source task left off. Sink connectors track their progress as a normal consumer group in __consumer_offsets.

Because all of this lives in Kafka rather than in any Worker, a Worker can die without taking any state with it. Another Worker picks up its tasks and resumes from the last committed offset.

Putting It All Together

The diagram above shows the same pieces as they run in production.

Every Worker is the JVM process described earlier. Each one exposes the REST API, so you can manage connectors through any of them, and each runs a share of the Connectors and Tasks.

The Kafka cluster plays two roles. It holds your application topics. Through the three internal topics and the group coordinator, it also keeps the Workers in sync: which ones are alive, which one leads, and where each task runs.

Running Connect in Production

Everything above describes the runtime. Operating it is a different problem, because open-source Kafka Connect gives you the runtime and very little around it.

• Connectors are managed through HTTP calls: There’s no built-in declarative way to manage connectors. Creating or updating one means a POST or PUT to the REST API. You have to build your own way to version configs, review changes and apply them consistently across environments.
• Restarts need care: Connector configs live in the config topic, so connectors come back when the Workers restart. But a task that has failed stays failed until someone restarts it through the API. Connect won’t retry it for you.
• Health checks are up to you: The REST API reports the status of every connector and task, but nothing acts on it. Alerting, automatic restarts and dashboards all need custom tooling.
• Plugins must match on every Worker: Every Worker needs the same plugins and dependency JARs. Adding or upgrading one means rebuilding and restarting all of them.
• Scaling is manual: Adding capacity means starting more Workers and waiting for a rebalance, and it only helps if connectors have enough tasks to spread out. Debezium’s Postgres connector, for example, always runs a single task. There’s no built-in autoscaling, and because connectors share Workers, one heavy connector can slow down the rest.

Two Ways to Solve This

We’ve run both.

AWS MSK Connect is a managed service. You upload plugins to S3, define connectors through the AWS console, API or infrastructure-as-code tools, and AWS runs the Workers. It scales Workers automatically within limits you set and sends logs to CloudWatch. Each connector gets its own Workers, so one connector can’t slow down another. The trade-off is cost and control: you pay for capacity per connector, and you’re limited to the Connect versions and settings AWS supports.

Strimzi on Kubernetes is an open-source operator that runs Kafka Connect as Kubernetes resources. The Workers are defined as a KafkaConnect resource, and each connector is a KafkaConnector resource: plain YAML you can keep in Git. The operator calls the REST API for you, can restart failed connectors and tasks automatically, and can build a Worker image with every plugin and extra JAR you list, which addresses the JAR problem described earlier. And because it all runs on Kubernetes, you keep full control over how it operates: scale Workers automatically with KEDA, deploy Workers and connectors through GitOps tools such as Argo CD, and plug in the monitoring you already use. This is where we landed, and it saved us a lot of cost.

Wrapping Up

What started for us as a low-cost experiment now feeds BI, reporting, ETL and AI data sources, and this architecture is the reason it scaled that far. You deploy Workers, install Plugins on them, and create Connectors through the API. The cluster decides where the Tasks run, and that’s what lets it survive losing a Worker. What it doesn’t give you is the operational layer around it: declarative deployments, automatic restarts, health checks and autoscaling. That’s the gap MSK Connect and Strimzi fill.

In the next article, I’ll cover how we moved from MSK Connect to Strimzi on EKS, and how we handle plugin JARs there.