Member-only story
Kafka Connect: Connectors, Tasks, and Workers
The Internal Wiring of Kafka Connect That Makes Everything Else Make Sense
I once watched a senior engineer spend two hours debugging a Connect performance problem that turned out to be a single line in the connector config.
He had set tasks.max=10, expecting ten parallel workers pulling from his database. What he actually had was one task. The connector did not log a warning. The REST API showed the connector as RUNNING. Everything looked fine — it was just slow. When he finally checked the actual task count via the status endpoint, there was one task. Not ten.
Understanding why requires understanding what tasks.max actually means, which requires understanding the relationship between connectors, tasks, and workers. The Kafka Connect documentation explains each concept in isolation. It rarely explains how they interact, and it almost never explains what happens when things go wrong at runtime.
This article is that explanation. No connector setup, no code -just the mental framework.
The Three Layers, separated clearly
Kafka Connect has three distinct layers of abstraction. They are easy to confuse because the word “connector” is overloaded in everyday conversation to mean all three at once.
