Kubernetes Controllers Explained: How the Reconciliation Loop Actually Works

Understand how Kubernetes continuously reconciles desired and actual state, how controllers watch resources, use work queues, and drive the cluster toward the state you declared.

Kubernetes is often described as a declarative system: You describe what you want, and Kubernetes makes it happen. But what actually makes Kubernetes continuously move the cluster toward the state you declared? The answer is controllers.

Controllers are the control loops that watch Kubernetes resources, compare the desired state with the current state, and take action when they detect a difference. Understanding controllers is one of the keys to understanding how Kubernetes actually works internally.

1. Kubernetes Is a Control System

Consider a simple Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3

This doesn’t mean: “Create three Pods right now.” It describes a desired state:

Desired state
    ↓
3 replicas of the application

Kubernetes then continuously works to make the actual cluster state converge toward that desired state.

Conceptually:

             Desired State
                   │
                   ▼
             ┌───────────┐
             │Controller │
             └─────┬─────┘
                   │
                   ▼
              Actual State
                   │
                   │
              Difference?
                /      \
              Yes       No
               │         │
               ▼         ▼
            Take       Nothing
            Action      to do
               │
               └──────► Observe again

The controller pattern is the foundation of Kubernetes automation. The official Kubernetes documentation describes controllers as control loops that watch cluster state and make changes to move the current state closer to the desired state.

2. Desired State vs Actual State

Kubernetes resources generally describe desired state through their spec.

For example:

spec:
  replicas: 3

The controller observes the cluster and determines what is actually happening. Imagine the current situation is:

Desired:
3 Pods

Actual:
2 Pods

There is a difference:

3 desired - 2 actual = 1 missing Pod

The controller takes action to correct that difference. After the replacement Pod is created:

Desired:
3 Pods

Actual:
3 Pods

The system has converged.

This distinction between desired and observed state is fundamental to Kubernetes’ declarative model.

Many Kubernetes resources also expose a status field containing observations about their current state. For example, a ReplicaSet reports information such as the number of currently observed replicas and ready replicas.

3. The Reconciliation Loop

The reconciliation loop is the heart of the controller pattern.

A simplified version looks like this:

          ┌─────────────────────┐
          │   Observe State     │
          └──────────┬──────────┘
                     ↓
          ┌─────────────────────┐
          │ Compare Desired vs  │
          │     Actual State    │
          └──────────┬──────────┘
                     ↓
          ┌─────────────────────┐
          │     Take Action     │
          └──────────┬──────────┘
                     ↓
          ┌─────────────────────┐
          │   State Changes     │
          └──────────┬──────────┘
                     │
                     └──────────────► Observe again

The important point is that reconciliation is not a one-time operation. Controllers continuously work toward convergence. If the desired state is already satisfied, reconciliation may result in no changes.

Desired = 3
Actual  = 3

→ Nothing to change

If the state changes:

Desired = 3
Actual  = 2

→ Take corrective action

4. How Does a Controller Know Something Changed?

A controller needs a way to observe resources.

At a high level:

                  API Server
                        │
                     Watch
                        │
                        ▼
                    Informer
                        │
                        ▼
                  Local Cache
                        │
                        ▼
                   Work Queue
                        │
                        ▼
                 Controller Worker
                        │
                        ▼
                   Reconcile

This is where Kubernetes controller internals become more interesting. Controllers don’t normally need to continuously poll the API server for every object.

Instead, Kubernetes’ client libraries provide mechanisms such as informers that maintain a local cache and receive changes through the API server’s watch mechanism.

The informer can notify interested controllers when resources are added, updated, or deleted. Current Kubernetes documentation describes this architecture using components such as the Reflector, local store/indexer, event handlers, and work queues.

A controller doesn’t necessarily modify the resource it watches. It may create or update other resources whose state collectively moves the system toward the desired state.

5. From an Event to Reconciliation

Suppose a ReplicaSet expects three Pods.

ReplicaSet
replicas: 3

Pods:
web-abc
web-def
web-ghi

Everything is healthy. Now someone deletes:

web-def

The sequence is approximately:

Pod deleted
    ↓
API Server records the change
    ↓
Watch delivers the event
    ↓
Informer updates its local cache
    ↓
Controller receives the event
    ↓
Work item enters the queue
    ↓
Controller worker processes it
    ↓
Reconciliation
    ↓
ReplicaSet has 2 Pods
but wants 3
    ↓
Create replacement Pod

The controller doesn’t need to be manually told:

kubectl create pod ...

The difference between desired and actual state is enough to trigger corrective action.

A ReplicaSet’s purpose is specifically to maintain a stable number of matching Pods, creating or deleting Pods as necessary to reach its desired replica count.

6. Why the Work Queue Matters

There is an important step between:

Event
  ↓
Reconcile

That step is the work queue. Conceptually:

Resource Event
      ↓
Work Queue
      ↓
Controller Worker
      ↓
Reconcile

The queue provides several useful properties:

• Events can be processed asynchronously.
• Multiple events can be handled without blocking the API watch.
• Work can be retried after failures.
• Duplicate work can be consolidated.
• Multiple workers can process reconciliation requests concurrently.

In controller-runtime, for example, events are transformed into reconciliation requests and placed into a work queue. The worker then retrieves the request and invokes the reconciler.

The important mental model is: Events tell the controller that something may need attention. The reconciliation logic determines what the actual state should be.

An event itself is not the desired state.

7. Deployment → Replica Set → Pod

Now let’s follow a real Kubernetes workload.

Get Subhra Jyoti Paul’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When you create:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx

You don’t directly get three running containers. The control flow looks more like:

Deployment
     │
     │ Deployment Controller
     ▼
ReplicaSet
     │
     │ ReplicaSet Controller
     ▼
   Pods
     │
     │ Scheduler
     ▼
Assigned Nodes
     │
     │ Kubelet
     ▼
Running Containers

A Deployment manages ReplicaSets, while ReplicaSets maintain the desired number of Pods. Kubernetes documentation explicitly describes this relationship: a Deployment creates and manages ReplicaSets, and the ReplicaSet creates Pods.

This is a good example of multiple reconciliation loops cooperating.

8. Multiple Controllers, Multiple Responsibilities

Kubernetes doesn’t have one giant controller responsible for everything. Instead, different controllers manage different resources and responsibilities.

For example:

                API Server
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
 Deployment       Job         DaemonSet
 Controller      Controller    Controller
        │            │            │
        ▼            ▼            ▼
   ReplicaSet       Pods         Pods
   Controller
        │
        ▼
       Pods

Other controllers manage things such as:

• Nodes
• StatefulSets
• Jobs
• Namespaces
• EndpointSlices
• Service accounts
• PersistentVolumes
• PersistentVolumeClaims
• Cloud-provider integration

The kube-controller-manager embeds many of the core Kubernetes control loops in a single process.

The important distinction is:

kube-controller-manager
        ≠
one controller

It is a process that runs multiple controllers.

9. What Happens When Reconciliation Fails?

Controllers operate in an environment where failures are normal.

For example:

Controller
    ↓
Try to create resource
    ↓
API request fails
    ↓
Reconciliation fails

A failed reconciliation can cause the work to be requeued and retried, typically with backoff. Conceptually:

          Reconcile
              │
              ▼
          Action fails
              │
              ▼
           Requeue
              │
              ▼
        Try again later

This is important because Kubernetes controllers are designed around eventual convergence, rather than assuming every operation succeeds immediately.

A temporary API error, network failure, or unavailable dependency should not permanently leave the system in the wrong state.

10. Reconciliation Doesn’t Mean Constant Polling

A common misconception is: “The controller must constantly check the API server.”

Not exactly.

Controllers primarily react to changes delivered through mechanisms such as watches and informers. They can also reconcile periodically or requeue work when appropriate. This gives the system two useful properties:

Event happens
     ↓
Reconcile

Periodic/rescheduled reconciliation
     ↓
Reconcile

Why is this useful? Because not every state change necessarily originates from a Kubernetes API event. External systems can change independently, and a later reconciliation provides another opportunity to detect and correct drift. The controller pattern is therefore more robust than simply saying:

Event → execute command

It is:

Observe state → determine desired outcome → converge toward it

Reconciliation Should Be Idempotent

Controllers may reconcile the same resource multiple times, so reconciliation logic should be safe to run repeatedly. If the desired state has already been achieved, another reconciliation should normally result in no unnecessary changes.

Desired = 3
Actual  = 3

Reconcile
    ↓
No change

This is one reason Kubernetes controllers are designed around continuously reconciling state rather than executing one-time commands.

11. Where Controllers Run

Built-in Kubernetes controllers are generally run by the control plane. One of the core components is:

kube-controller-manager

Its role is to run the core control loops shipped with Kubernetes. A simplified control-plane view is:

Control Plane
────────────────────────────────────────

kube-apiserver
             │
       ┌─────┴─────┐
       │           │
       ▼           ▼
  Scheduler   Controller Manager
                   │
          ┌────────┼─────────┐
          ▼        ▼         ▼
     Deployment   Node      Job
     Controller Controller Controller

The API server remains the central interface through which controllers observe and modify Kubernetes objects.

12. The Controller Pattern Is Extensible

Controllers aren’t limited to Kubernetes’ built-in resources.

Kubernetes provides an extension mechanism through Custom Resource Definitions (CRDs) and custom controllers. For example, you could introduce:

apiVersion: example.com/v1
kind: RedisCluster
spec:
  replicas: 3

A custom controller can watch this resource:

RedisCluster
      │
      ▼
Custom Controller
      │
      ▼
Reconcile
      │
      ├── Create StatefulSet
      ├── Configure Services
      ├── Manage storage
      └── Update status

The controller can continuously work toward the desired state represented by the custom resource. This is the foundation behind many Kubernetes operators.

Kubernetes’ official extension documentation describes the controller pattern as a way for programs to read an object’s spec, perform actions, and update its status.

13. The Complete Mental Model

Putting everything together:

                        User
                          │
                          │ Desired State
                          ▼
                    Kubernetes API
                          │
                          │ Watch
                          ▼
                       Informer
                          │
                          ▼
                     Local Cache
                          │
                          ▼
                     Work Queue
                          │
                          ▼
                  Controller Worker
                          │
                          ▼
                    Reconciliation
                          │
                 ┌────────┴────────┐
                 │                 │
            Desired ≠ Actual   Desired = Actual
                 │                 │
                 ▼                 ▼
              Take Action        No Action
                 │
                 ▼
             State Changes
                 │
                 └──────────────► Observe Again

This is the core idea behind Kubernetes controllers. They don’t simply execute your YAML. They continuously work to make the actual cluster state converge toward the desired state.

Once you understand this loop, many Kubernetes behaviors become easier to explain:

• Why a deleted Pod gets recreated
• Why Deployments create ReplicaSets
• Why ReplicaSets maintain Pod count
• Why Kubernetes can recover from failures
• Why Operators work
• Why Kubernetes is declarative
• Why controllers are central to the control plane

The Kubernetes control plane isn’t a collection of commands that execute once.

It is a collection of control loops continuously driving the cluster toward its desired state.