How I Went Cloud-Agnostic With the Argo Ecosystem

THE ARGO LOG · Entry 01

How I researched a self-managed delivery platform for a multi-cloud SaaS platform I worked on, the options I weighed, and why the design ended up pull-based with zero inbound connections. A personal engineering log, written while the work is still in progress.

Every fragmented CI/CD setup has the same origin story, and it never starts with a bad decision. It starts with a hundred good, small ones.

A Jenkins job for one service, because that’s what was already running. A GitHub Action for another, because the repo lived on GitHub and the workflow file took ten minutes to write. An Azure DevOps pipeline for a third, because that’s where the team building it was comfortable. Each choice was reasonable on its own. Nobody sat down and decided “let’s run three orchestration engines for one product.” It accumulated.

That’s roughly where I found the delivery side of a multi-cloud SaaS platform I work on, which ships to both AWS and Azure, stage and production. Three systems, three mental models, three places to look when a deploy went sideways. It worked. It also meant nobody really owned the delivery lifecycle — three vendors did, three separate ways, and none of them shared a security model, a secrets store, or an audit trail with each other.

This is the first entry of a log I’m keeping while I rebuild that delivery pipeline on Argo Workflows, as an intern project on the SRE team. It’s not a “here’s the finished thing” post. Most of it is the research: what I was actually trying to solve, what I considered, what I threw away, and why. The finished parts come in later entries.

What I was actually trying to solve

Before picking a tool I wrote down what “better” had to mean, because “modern CI/CD” is not a requirement. Four things came out:

1. One place to look. A failed deploy should be debugged in one system, with one way of reading logs and one way of retrying.
2. The same pipeline on every cloud. If AWS and Azure need different pipeline files for the same service, the differences will drift, and I’d be maintaining two things that are supposed to be one.
3. No orchestrator that can reach into production. Most CI systems hold long-lived credentials for every environment they deploy to — a kubeconfig here, an IAM role there. Compromise the CI system and you own everything downstream in one step. I wanted that to be structurally hard, not a policy that says please don’t.
4. Own the trust boundary. Where credentials live, who can trigger what, and what gets audited should be our decision, not an accident of which vendor hosts which pipeline.

Everything after this is me testing ideas against those four.

Decision 1: consolidate, or replace?

The lowest-effort option was to move everything onto Azure DevOps, since most of the pipelines were already there. I took it seriously — it would have removed two of the three systems and reduced the “three places to look” problem to one.

It doesn’t touch the other three requirements, though. AWS deploys would still be driven by a vendor-hosted orchestrator holding credentials into every environment; the lock-in would just be one vendor instead of three. Consolidation fixes the mess without changing who owns the lifecycle. I ruled it out, but kept it as the baseline I compared everything else against — if a new design couldn’t clearly beat “just use ADO for everything,” it wasn’t worth building.

Decision 2: why not just adopt GitOps?

The first question anyone asks is why not Argo CD or Flux, the famous names in the same family.

They solve a different problem. Argo CD and Flux are reconciliation engines: you declare a desired state in Git, and a controller nudges the live cluster toward it, continuously, forever. That’s exactly right when the unit of work is “keep this manifest in sync.”

My unit of work looks like this instead:

deploy → wait for rollout → run verification checks
   ├─ pass → alert "success", update tracking record
   └─ fail → automatic rollback → alert "failed", branch to incident path

That’s orchestration — explicit control flow with branches and retries — not a converging loop. Argo Workflows is built for exactly that shape: DAGs, conditional (when:) steps, retries, and onExit handlers that branch on success or failure, all expressed as native Kubernetes resources instead of a vendor's own YAML dialect. Two details sealed it:

ClusterWorkflowTemplate maps almost directly onto the shared-library pattern every CI system already has — the "download a release," "send an alert," "commit a changelog" building blocks every pipeline calls. Porting them is translation, not a rewrite.

CronWorkflow gives me scheduled jobs (periodic test suites, backups) as a native resource in the same system, instead of reaching for a fourth tool for that one need.

I still use Argo CD — but only for what it’s actually good at: keeping per-environment configuration in sync between Git and a cluster. The pipelines themselves run on Workflows.

Reconciliation (Argo CD/Flux): Git state ⇄ cluster state, continuous loop” vs “Orchestration (Argo Workflows)

Decision 3: how does the orchestrator reach the clusters?

This was the real research, and where my first design was wrong.

My first design: two clusters. One cluster for R&D, one for SaaS, each running Argo Workflows and deploying directly into whatever it managed. It was simple, and I sent it out for review. The feedback — and my own second look — exposed the problem immediately: to deploy anywhere, an orchestrator has to be able to reach that place, and that reach is exactly the thing requirement 3 ruled out. Simple doesn’t mean safe.

Get Yohan Senadheera’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So I compared three ways for an orchestrator to get work onto a remote cluster:

How the orchestrator reaches a data plane — and why pull-through-a-broker won.

Concretely: one control plane that orchestrates and runs no workloads, and independent data planes per cloud and environment tier that execute. The broker is NATS, run with mutual TLS and per-subject permissions — each data plane authenticates with its own certificate and can only read and write its own subjects (tasks.aws-prod, results.aws-prod, and nothing else). I didn't just configure that boundary and assume it held — I tested it: a data plane attempting to publish outside its allowed subjects gets rejected server-side, not silently ignored.

The dispatch mechanic itself is deliberately thin:

1. Control plane:  submit "aws-prod dispatcher" → publish task on tasks.aws-prod → suspend
                    (suspending costs nothing — there is no pod sitting there waiting)
2. Data plane:      Argo Events listener on tasks.aws-prod fires → submits the executor
3. Data plane:      executor runs the real, unmodified pipeline (Helm deploy, checks, rollback)
4. Data plane:      publish result on results.aws-prod
5. Control plane:  Argo Events resumes the exact suspended workflow by name, with the real result

All connections between the data plane and the broker are opened by the data plane. NATS never dials out to a data plane.

One consequence I only appreciated later: because the invariant is “zero inbound, ever,” it constrains every later decision automatically. When I wanted a Git commit to trigger a config sync, the obvious answer was a GitHub webhook into each cluster’s Argo CD. That’s an inbound connection, so it was dropped on sight, and I accepted Argo CD’s default polling instead. A few minutes of extra latency is a cheap price for an invariant I can state in one sentence and never have to re-check per feature.

Decision 4: isolating stage from prod inside one cloud

A single cloud hosts both stage and prod, so “separate data planes per cloud” isn’t enough by itself — I needed stage to be structurally unable to touch prod even while sharing infrastructure underneath.

The options: separate clusters per environment (strongest, most expensive to run), namespaces only (cheap, but shared network and shared nodes underneath), or a middle path. I built the middle path on Azure:

# prod node pool — workloads must opt in to land here
taints:
  - key: "tier"
    value: "prod"
    effect: "NoSchedule"

# a prod workflow's pod spec has to explicitly tolerate the taint
tolerations:
  - key: "tier"
    operator: "Equal"
    value: "prod"
    effect: "NoSchedule"

Separate node pools per environment, the prod pool tainted so only workloads that opt in can land there, separate subnets with a network security group denying stage→prod traffic outright, and a dedicated NAT gateway per subnet so the two don’t even share an outbound IP.

The first version had a real gap that only showed up live: the taint was correctly applied, but nothing had told the actual prod workloads to tolerate it, so they quietly scheduled onto the stage pool anyway — no error, no warning, just a pod landing somewhere I didn’t expect. The mechanism was correct; I just hadn’t opted anything into it. Good reminder that an isolation design isn’t isolation until you’ve watched a workload actually try to cross the line and fail to.

What broke on the way

The first proof of the whole loop ran on two Azure VMs standing in for “different clouds.” It worked — and then I rebuilt it properly on real EKS and AKS with Terraform, and found that “worked” had been hiding a pile of problems:

• The broker’s certificate didn’t list the control plane’s external address, so no data plane’s cross-cluster connection had ever actually passed verification on the new infrastructure. It looked connected; it wasn’t. openssl s_client against the NATS endpoint was the thing that actually showed it: hostname mismatch, not a network problem.
• The “resume” step brought back no real pass or fail data. The dispatcher finished, but I couldn’t tell from it whether the deploy had succeeded or failed — it just said “done.” Fixed by having the data plane patch the workflow’s exit-result annotation directly via the Kubernetes API before triggering resume, instead of trusting whatever Argo Events handed back by default.
• After a broker restart, data-plane listeners stayed silently disconnected. Publishing succeeded, nothing fired, and nothing logged an error anywhere — the kind of failure that only shows up because a deploy that should have happened, didn’t.
• Missing configuration keys, not logic, were the most common reason a run failed at submit time. So I stopped hand-writing dispatch YAML per service and started generating every dispatch file from one registry, checking each required config key against every environment before anything could run — turning a class of “found out at 2am” failures into a lint error.

Where it stands, honestly

This is a migration running alongside real traffic that won’t wait for me, so “done” is the wrong frame — here’s the actual state:

• The docs pipeline runs end to end through the full dispatch chain on three of the four (cloud × environment) targets; the fourth is waiting on access grants, not a bug.
• The core identity service’s deploy has gone through a real Helm deploy on one target, then hit a capacity limit on the test target (2 vCPU available, 4 requested) — an infrastructure sizing issue, not a pipeline issue.
• Most of the rest — about thirty services for stage and a large set for prod — are ported and lint-clean but haven’t run on a cluster yet.
• One correction I owe my own assumptions: I first built the AWS pipelines as copies of the Azure ones. The real AWS pipelines are a genuinely different design — different secrets store, storage, container registry, and identity model — so I threw that version out and rebuilt from the real source. Assuming two clouds were the same shape was the most expensive mistake so far.

The short version, if you’re weighing something similar

• Write down what “better” means before you compare tools. I only got clarity when I turned “modern CI/CD” into four checkable requirements.
• Consolidating onto one vendor removes clutter, not lock-in.
• Reconciliation tools and orchestration tools solve different problems. Pick by the shape of your unit of work, not by which one is more popular.
• Decide your hard invariant early (“no inbound connections, ever”), and let it reject options for you automatically. It saved me from a webhook and from a network peering without a second debate.
• An isolation mechanism isn’t done until you’ve watched a workload try to cross it and fail.
• A proof of concept proves the idea, not the infrastructure. Expect a second, different set of bugs when you rebuild it for real.

The next entries are the how: the control plane for real, the parts that broke, and what it looks like when a commit turns into a running service in two clouds without three consoles in between.

Next entry → the control plane for real…