How We Built a Kubernetes Operator to Handle Rollbacks: Part 1

Introduction: the cost of a slow rollback

One of our core values at GoCardless is to “start with why,” and any engineering decision we make is expected to answer that question. So, why did we need a dedicated rollback mechanism?

The first driver was the absence of a paved path for reverting a bad change once it had reached production. Engineers had two real options: revert the offending pull request and wait for the fix to redeploy, or deploy a previous release of the application using our internal CLI, Utopia. However, the CLI commands had no way of indicating what previous releases existed or which were healthy when deployed, so few engineers felt confident running them in the middle of an active incident, and fewer still could vouch for their behaviour in less common cases.

The second driver was our error budget. GoCardless holds a 99.95% API availability SLA with a number of our partners, and a handful of recent incidents had already consumed a meaningful share of the downtime that budget allows for. Reverting a pull request and waiting for a fresh deployment is slow: it depends on CI, the deployment pipeline, and however long it takes for a fix to be reviewed and merged. None of this is well suited to an incident, where minutes matter.

We needed a mechanism that could scale across hundreds of services, minimise the time between detecting a bad release and returning to a known-good state, and be simple enough that any engineer could reach for it with confidence, under pressure, and without needing to be a specialist in the tooling.

Overview of our CI/CD ecosystem

Before delving into the details of our solution, it’s worth stepping back to outline how services are deployed and managed at GoCardless. At a high level, each service relies on the following components:

• Anu (infrastructure monorepo) — holds the configuration for all of our infrastructure definitions: clusters, DNS zones, services, etc. We use Jsonnet to define our service infrastructure that we later render into Kubernetes manifests. These will hold configuration for Kubernetes primitives (i.e. Services, Deployments, Ingress) and any 3rd party CRDs (e.g. Config Connector resources: SQLInstances, PubSubSubscriptions, etc.). Kubernetes is our orchestrator of choice, and Anu is the source of truth for every service’s manifests. This is where a service’s infrastructure_revision comes from — a term we'll return to shortly.
• Application repository — each service has its own code repository, which is the source of its application_revision. Images are tagged using commit SHAs from that repository during the build process and are stored in an artifact registry.
• ArgoCD — our continuous deployment tool of choice. ArgoCD combines the infrastructure_revision and application_revision for a given service — via a config management plugin (CMP) we've written — to produce the final set of Kubernetes manifests.
• CI — new versions of a service are rolled out through a combination of GitHub deployments and customised Tekton pipelines. We collectively call our CI ecosystem Dispatcher.

With this picture in place, the next question was how to actually perform a rollback against it — and before landing on our own approach, it’s worth going through the solutions we considered first and rejected.

Alternatives Considered

Before getting to the solution we went with, it’s worth covering the ground work we did evaluating existing solutions for performing rollbacks, and why they didn’t fit our needs exactly.

Two separate problems needed solving:

• Deciding if we need to rollback
• Figuring out how to rollback

and we evaluated alternatives for each. This article covers the first; the alternatives we looked at for carrying out the rollback itself are covered in part two.

ArgoCD sync history. ArgoCD already keeps a native record of the last 10 syncs for an application, including the CMP parameters used at sync time — which meant both our infrastructure and application revisions were already captured, seemingly for free. Two things ruled it out, though. First, ArgoCD’s own documentation strongly discourages raising that history limit past 10, since it increases the amount of data stored in etcd; second, there’s no way to append additional information to an existing sync record. That second point was the more serious one: we had nowhere to record whether a given release turned out to be healthy, which is central to how rollback decisions get made, both for a manual rollback and for automated rollbacks.

ArgoCD Source Hydrator. The Source Hydrator is an alpha ArgoCD feature that renders “hydrated” manifests and commits them to a git repository, letting ArgoCD deploy directly from that checked-out revision rather than rendering on the fly. Because every rendered revision ends up in git, it effectively behaves like a ready-made database of releases, which makes it attractive on paper. In practice, it wasn’t compatible with our CMP since the renderer doesn’t pass through parameters like ArgoCD’s regular manifest renderer. We looked at forking ArgoCD to add that support, but decided the ongoing maintenance burden of a fork wasn’t worth it for a component as central to our deployment pipeline as ArgoCD.

Kargo. Kargo is a continuous promotion tool that layers on top of ArgoCD. Its Freight system — a set of references to versioned artifacts promoted between environments — looked like a strong fit for recording release history, and it comes with built-in deployment health verification via Argo Rollouts, which we’d otherwise have to build ourselves. The catch is that Kargo can only track history for deployments that go through its own promotion workflow. Anything triggered outside of it, including ArgoCD’s auto-sync and manual syncs, won’t appear in a stage’s freight history, and as we support these deployment methods at GC, we needed a way of recording them in the history as valid rollback targets. Adopting Kargo properly would have meant fundamentally reworking how we trigger deployments across the board.

Introducing — Utopia Rollbacks

To recap, we needed a mechanism that interacts with our existing CI/CD system and set a service’s two revisions — infrastructure and application — to a specific, healthy pair. We called the initiative Utopia Rollbacks, and split it into three sub-projects:

1. Manual Rollback — gives service owners the ability to roll back to a specific pair of revisions (a release) on demand.
2. Rollback Metrics — provides a way to evaluate the health of a release.
3. Automated Rollback — combines the two previous projects, triggering a rollback automatically once a release is deemed unhealthy.

Since most of our workloads run on Kubernetes, and at the time we were operating around 15 clusters, we needed an approach that would translate cleanly across all of them. The solution we settled on was a set of Kubernetes operators, deployed across every cluster, responsible for managing releases and rollbacks across all three sub-projects.

How do we know what to roll back to?

Before building the Kubernetes operators, we needed to design the APIs they’d operate on. Kubernetes lets you extend its API using Custom Resource Definitions, so that’s where we started.

Get Emil Lozev’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

As we already mentioned, in our release process each service relies on two revisions: an infrastructure_revision and an application_revision. The former is a git SHA from our infrastructure monorepo; the latter is an image tag from our container registry (technically also a git SHA, but from the application repository). At the time, our tooling had no reliable way to record both revisions together. On top of that, the two later sub-projects — Rollback Metrics and Automated Rollback — both needed a way to record and determine whether a release was healthy, so that service owners and our operators alike could act on that signal. No existing system captured all of this in one place.

With that in mind, our custom resource needed to:

• Record multiple revisions
• Carry an indication of health
• Record supporting fields — deployment start/end time, status message, the previous release, and so on

Release CRD

Here’s the Release CRD we landed on:

Release CR

A few fields are worth calling out:

• .config.revisions holds named revisions, each tagged with a type (git, container_image, or helm_chart) so the operator knows how to interpret the id field.
• status.signature is a deterministic hash of the revision names and IDs — a cheap way to compare two releases without re-parsing every revision, which becomes useful once we get to selecting which release to roll back to.
• status.previousRelease points back to the release this one superseded, letting us walk the release history backwards.
• status.conditions captures the two states we care most about: whether a release is currently Active, and whether it's Healthy — the latter being the field that Rollback Metrics and Automated Rollback are ultimately working to set.

Something also worth flagging is why the CRD uses a top-level .config field rather than the conventional .spec. By Kubernetes convention, .spec is meant to hold the desired state of a resource, while .status holds its current observed state. A Release, however, we record after the fact — once ArgoCD has already synced the manifests into the cluster — so there's no meaningful "desired" state to declare; by the time the resource exists, the release has already happened. Using .config instead of .spec avoids implying otherwise, while still keeping the revision data in a field name that reads clearly as configuration.

Recording releases

With a way to store a Release and its metadata in place, the next problem was recording releases across our fleet of clusters. As mentioned earlier, we run around 15 Kubernetes clusters, hosting roughly 1,100 ArgoCD Applications between them. Conveniently, all of this is managed by a single ArgoCD instance. We decided to make use of ArgoCD’s notifications controller and send events out to a Release recording service deployed across the fleet, which would create releases for the services running in that cluster.

Practically, this works through ArgoCD’s notification triggers and templates. We configured a trigger that fires whenever an Application’s sync completes successfully, which prompts a notification to be sent to the release-recorder webhook server running in that application’s target cluster. The payload carries what we need to construct a Release: the application namespace (where the Release will be created), its revisions and some metadata about sync (its start/end time).

On receipt, the release-recorder translates that payload into a Release resource and creates it in the correct namespace. Because there’s a single ArgoCD instance, this fan-out is what lets one central sync event reliably turn into a namespaced, cluster-local record.

Recording Releases

Release controller

Once a Release resource exists, a dedicated release controller takes over its lifecycle. On each reconcile, it initialises a new release’s status if needed, and updates any .status fields like deployment start and end time, conditions (e.g. Active=true), and a reference to the previous release. It's also what triggers the health analysis (covered in a future section), and what keeps release history from growing unbounded.

Keeping this controller scoped to just the Release resource was a deliberate choice that lines up with Kubernetes’ own guidance on controller design: having several small, single-purpose controllers is generally preferred over a large one of interlinked control loops. It’s the same reasoning we used to design the other 2 controllers part of this project.

Next Steps

On its own, the release controller doesn’t give us rollbacks, but it does give us an answer to the question: for a given service, what versions has it been running and what can we roll back to? In part two we cover how we extended the Kubernetes operator to tag releases with their health and perform rollbacks to them.

If you feel that GoCardless appeals to you and that you would like to find out more about Life at GoCardless, you can find our posts on Instagram and LinkedIn.

Are you interested in joining GoCardless? See our jobs board here.

Note about the authors and project

This piece has been jointly co-authored by Emil Lozev, Gusts Jonasts and Isaac James.

The controller and CRDs mentioned in the article above are part of our open-source collection of Kubernetes controllers you can find at github.com/gocardless/theatre.