The Control Loop: Kubernetes’ Hidden Superpower

Part 1 of “Kubernetes Controllers: From Watchers to Builders”

You’ve used kubectl apply hundreds of times. You write a Deployment manifest, apply it, and pods appear. You change the replica count, apply again, and Kubernetes scales up. It feels automatic. Almost magical.

It isn’t magic. It’s a control loop. It’s the single most important idea in Kubernetes. Every Deployment rollout, every crashed pod that respawns, every node that gets drained is driven by small, independent programs called controllers that run the same loop over and over:

1. Watch for changes
2. Compare desired state to actual state
3. Act to close the gap

That’s it. That’s the whole idea. And by the end of this post, you’ll have built one yourself.

Desired state vs. actual state

Most systems you’ve worked with are imperative. You give orders. “Start three pods.” “Kill that container.” “Scale up.” You’re a drill sergeant barking commands.

Kubernetes flips this on its head. It’s declarative. You don’t tell it what to do. You tell it what you want, and it figures out the rest. You write “I want 3 replicas” in a YAML file. You don’t write “start pod-1, then pod-2, then pod-3.” The Deployment controller reads your desired state, checks the actual state, and does whatever is necessary to make them match.

Think of it like a hotel housekeeper. Every morning, the housekeeper walks into a room with the same checklist: bed made, towels folded, trash emptied. They don’t ask the front desk “what happened in this room last night?” They don’t care. They look at the room, compare it to the checklist, and fix whatever doesn’t match. A party happened? Doesn’t matter. A guest barely touched anything? Doesn’t matter. The checklist is the same. The result is the same.

Controllers work exactly like that housekeeper. They’re stateless decision-makers that look at the world, compare it to a spec, and fix the difference. This makes them naturally idempotent: you can run the same reconciliation a hundred times, and the result is the same. It also makes them self-healing. Imagine a mischievous guest sneaks in and messes up the towels. The housekeeper comes back, checks the room, folds the towels again. No drama. No incident report. Just quiet correction. If someone manually deletes a pod, the controller does the same thing. It notices the gap on its next pass and creates a new one.

As someone that learns through metaphors, I’ve created a interactive example on github pages for a simple walkthrough:

The Kubernetes Control Loop

What’s inside kube-controller-manager

When you set up a Kubernetes cluster, a binary called kube-controller-manager starts running on the control plane. Picture it as an office building. Inside that single building are dozens of independent workers, each sitting in their own cubicle, each responsible for one narrow job:

• The Deployment controller manages ReplicaSets when you update a Deployment
• The ReplicaSet controller ensures the right number of pods are running
• The Job controller runs pods to completion
• The Namespace controller cleans up resources when a namespace is deleted
• The ServiceAccount controller creates default service accounts in new namespaces
• The Node controller detects when nodes go down and evicts pods

Each one follows the same pattern. They watch a specific resource type, and when something changes, they reconcile. They don’t chat at the water cooler. They don’t pass notes. They don’t share spreadsheets. They just do their one job in a loop.

This is an underrated aspect of Kubernetes’ design. Because each controller is independent, the system is modular. You can reason about the Deployment controller without understanding the Job controller. You can replace one. You can write your own.

The anatomy of a control loop

Let’s get more specific. Every controller built with client-go (the official Go library for talking to Kubernetes) uses the same four building blocks. Think of them as stations on an assembly line, each one passing work to the next.

Informer. This is the controller’s newspaper subscription. Instead of running down to the API server every morning to check what changed, the informer opens a streaming connection and gets updates pushed to it in real time. It keeps a local, cached mirror of every resource it cares about. This is critical for scalability. A cluster with thousands of controllers all polling the API server would be like every employee in a company calling the CEO directly for status updates. The informer is the internal memo system that keeps everyone informed without melting the phone lines.

Event handlers. When the informer detects a change (a resource was created, updated, or deleted), it calls your handler. But your handler isn’t the one doing the heavy lifting. Think of it as the receptionist at a doctor’s office. A patient walks in, the receptionist writes their name on the clipboard, and they sit down. That’s it. The handler extracts the object’s key (namespace + name) and drops it into a queue.

Work queue. This is the clipboard. It’s a thread-safe, rate-limited queue that buffers keys waiting to be processed. If the same name appears twice before the doctor calls it, the queue deduplicates it. If a patient’s visit fails (the API server was unavailable), the queue reschedules them with exponential backoff, waiting longer each time. This decouples the “noticing someone arrived” step from the “treating them” step.

Reconciler. This is the doctor. It pulls the next name from the clipboard, looks up the patient’s chart in the informer’s cache, compares the desired state (“healthy, labelled, running”) to the actual state, and prescribes whatever is needed to close the gap. If the treatment fails, the name goes back on the clipboard.

This is level-triggered design, not edge-triggered. The reconciler doesn’t care what changed. It doesn’t ask “was this an add or an update?” It’s like that hotel housekeeper again. They don’t need to know whether the guest jumped on the bed or just wrinkled the sheets. They see the bed is messy, and they make it. This matters because events can be lost, duplicated, or arrive out of order. A level-triggered reconciler handles all of those correctly because it only depends on the current state.

Let’s build one

Enough theory. Let’s write a controller that does something simple: it watches ConfigMaps in a namespace and ensures every one of them has a managed-by: configmap-controller label. If a ConfigMap is created without the label, the controller adds it. If someone removes the label, the controller puts it back.

Get Update-later’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It’s trivial on purpose. The point isn’t the business logic. It’s the plumbing. This is the “Hello, World” of Kubernetes controllers.

Project setup

The full code is on GitHub: k8s-controller-series. Clone it if you want to follow along.

.
├── cmd/configmap-controller/
│   └── main.go              # The complete controller
├── deploy/
│   └── manifests.yaml       # RBAC + Deployment for in-cluster
├── Dockerfile
├── Makefile
└── go.mod

Building the client

First, we connect to the cluster. client-go supports both in-cluster authentication (using the service account token mounted in a pod) and out-of-cluster via a kubeconfig file:

config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
    klog.Fatalf("Building kubeconfig: %v", err)
}
clientset, err := kubernetes.NewForConfig(config)

When kubeconfig is empty, clientcmd.BuildConfigFromFlags falls back to in-cluster config automatically. One line of code, both environments covered.

Setting up the informer

We create a SharedInformerFactory scoped to a single namespace and grab the ConfigMap informer:

factory := informers.NewSharedInformerFactoryWithOptions(
    clientset,
    30*time.Second, // resync period
    informers.WithNamespace(*namespace),
)
cmInformer := factory.Core().V1().ConfigMaps().Informer()

The resync period (30 seconds here) controls how often the informer redelivers all objects to your event handlers, even if nothing changed. Think of it as a fire alarm test. Nothing is on fire, but the system runs the drill anyway just to be sure. If a reconciliation was somehow missed, the resync catches it. In production, you’d typically set this to several minutes.

Registering event handlers

Our event handlers are deliberately boring. They just extract the object key and drop it in the queue:

informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, err := cache.MetaNamespaceKeyFunc(obj)
        if err == nil {
            c.queue.Add(key)
        }
    },
    UpdateFunc: func(_, newObj interface{}) {
        key, err := cache.MetaNamespaceKeyFunc(newObj)
        if err == nil {
            c.queue.Add(key)
        }
    },
})

Notice we’re not registering a DeleteFunc. When a ConfigMap is deleted, there's nothing to reconcile. You can't stamp a label on a gravestone. The reconciler handles this case by checking whether the object still exists in the cache.

Also notice that AddFunc and UpdateFunc do the exact same thing. This is level-triggered design in action. We don't care why we were called, only that we need to look at this key. The receptionist doesn't need to know whether this is a new patient or a returning one. They just write the name on the clipboard.

The work queue

The queue is created with rate limiting built in:

queue: workqueue.NewRateLimitingQueue(
    workqueue.DefaultControllerRateLimiter(),
)

The default rate limiter combines two strategies: an exponential backoff per item (starting at 5ms, doubling up to 1000s) and an overall rate limit (10 items/sec, burst of 100). This is like a bouncer at a nightclub. One person keeps getting rejected at the door? They have to wait longer each time before trying again. And no matter what, only 10 people get through the door per second. A single failing key won’t flood the API server with retries.

Inside the Reconciler Function

The reconcile function is the controller's main decision loop, the core process where the desired state is brought to reality. We can visualize this detailed sequence by breaking it down into a single, comprehensive compartment with sequential steps.

func (c *Controller) reconcile(key string) error {
    // Step 1: Split Key
    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        return fmt.Errorf("invalid key %q: %w", key, err)
    }
    // Step 2: Read from Cache (Fast and Free)
    // Local Sticky Note vs. Walking Across Building
    obj, exists, err := c.informer.GetStore().GetByKey(key)
if err != nil {
        return fmt.Errorf("fetching %s from cache: %w", key, err)
    }
    if !exists {
        // Step 3: Check for Existence
        // Deleted between enqueue and now. Nothing to do.
        return nil
    }
    cm := obj.(*corev1.ConfigMap)
    // Step 4: Check Before We Act (Idempotency)
    // Red Door Painter Example
    if cm.Labels != nil && cm.Labels[labelKey] == labelValue {
        // Already labelled? Done.
        return nil
    }
    // Step 5: Deep-copy (Shared Library, No Scribbling)
    // Shared and Read-only Cache. Always DeepCopy().
    updated := cm.DeepCopy()
    if updated.Labels == nil {
        updated.Labels = make(map[string]string)
    }
    updated.Labels[labelKey] = labelValue
    // Step 6: Write Back to API Server (Source of Truth)
    // Stabilizing the Loop (Pendulum)
    _, err = c.clientset.CoreV1().ConfigMaps(namespace).Update(
        context.TODO(), updated, metav1.UpdateOptions{},
    )
    return err
}

A few critical principles to grasp about this process:

• Read from the Cache, Not the API Server: Our reconciler performs a local, in-memory lookup (c.informer.GetStore().GetByKey(key)), avoiding costly network round-trips to the API server. Think of it as checking a sticky note on your desk to confirm a detail rather than walking across the entire building to ask someone. It’s fast and resource-free.
• Check Before You Act (Idempotency): The reconciler function is idempotent: running it a hundred times on the same state yields the exact same result. It does this by checking if the desired state already exists and immediately returning if it does. Picture a painter hired to paint a door red. If they arrive and see the door is already red, they pack up and go home. If called back next week and the door is still red, they go home again. No wasted paint, no double coat.
• Deep-copy Before Mutating: The informer cache is a shared library of resources. While many parts of the system can read from it simultaneously, no one is allowed to scribble in the margins. Direct modifications to cached objects will corrupt the data for all other goroutines. Always DeepCopy() to work on a safe, private duplicate.
• Write Back to the API Server, Not the Cache: After making a change, we update the API server, which is the singular source of truth. The informer will then detect this change, re-enqueue the key, and trigger the reconciler loop again. This time, the idempotency check will pass, and the loop will terminate, just as a pendulum swings once and settles.

Processing the queue

The worker loop is boilerplate that every controller uses:

func (c *Controller) processNextWorkItem() bool {
    key, shutdown := c.queue.Get()
    if shutdown {
        return false
    }
    defer c.queue.Done(key)
    err := c.reconcile(key.(string))
    if err == nil {
        c.queue.Forget(key) // reset retry counter
        return true
    }
    klog.Errorf("Error reconciling %s: %v", key, err)
    c.queue.AddRateLimited(key) // retry with backoff
    return true
}

queue.Get() blocks until a key is available. queue.Done(key) tells the queue we're finished with this key (so it can be re-added if needed). queue.Forget(key) resets the backoff counter on success. queue.AddRateLimited(key) re-enqueues on failure with exponential backoff.

Running it

# Clone the repo
git clone https://github.com/yourusername/k8s-controller-series
cd k8s-controller-series

# Build and run locally (requires a running cluster)
make run

In another terminal:

# Create a ConfigMap without our label
kubectl create configmap hello --from-literal=greeting=world
# Check the labels (give it a second)
kubectl get configmap hello --show-labels
# NAME    DATA   AGE   LABELS
# hello   1      3s    managed-by=configmap-controller

The label appeared. Now try removing it:

kubectl label configmap hello managed-by-
kubectl get configmap hello --show-labels
# The controller puts it right back.

This is what self-healing looks like. You can peel the sticker off, but the controller walks by, notices it’s missing, and slaps a new one on. Every. Single. Time.

What you just built

This is roughly 160 lines of Go, and it contains every structural element found in the controllers that ship with Kubernetes. The Deployment controller is far more complex. It manages rollout strategies, revision history, and multiple ReplicaSets. But it uses the exact same informer, queue, reconcile pipeline. The skeleton is identical. You’ve just built the skeleton.

Here’s what we covered:

Informers cache state locally so controllers don’t hammer the API server. They’re your local newspaper instead of a trip to city hall.

Work queues decouple event detection from processing and handle retries. They’re the clipboard between the receptionist and the doctor.

Level-triggered reconciliation means your reconciler doesn’t need to care about event types. It just fixes the current state, like a housekeeper who doesn’t need a damage report.

Idempotent updates mean the same reconciliation can run repeatedly without side effects. The painter shows up, sees the door is red, goes home.

Deep-copies protect the shared cache from mutation bugs. The library’s books stay clean.

What’s next

In the next post, we’ll look at how Kubernetes’ built-in controllers compose. How does a Deployment create ReplicaSets? How do ReplicaSets create Pods? And how do owner references and garbage collection tie it all together? Picture it like a corporate org chart: the Deployment is the VP, the ReplicaSet is the manager, and the Pods are the individual contributors. Understanding this chain of command is essential before you start building controllers that manage their own child resources.