Everything Was Green. Nothing Was Deploying.

What three days of building a DevSecOps platform taught me about the failures you can’t search for

The dashed lines are the point. Kargo polls the registry and opens merge requests against git; ArgoCD reconciles what gets merged. This is a promotion system with a human in it, not a deploy script.

The promotion reported seven steps, all green. Kargo cloned the repository, rewrote the image tag, committed, pushed a branch, opened a merge request, waited for me to approve it, and told ArgoCD to sync. ArgoCD said Synced. ArgoCD said Healthy.

The pod was running the previous image.

Nothing had failed. No error appeared anywhere in the chain. Every dashboard I had built specifically to catch this said the system was working, and the system was not working.

That failure, and eight or nine others like it, is what I actually want to write about. The architecture is the least interesting part of this project. You can read a hundred posts describing an EKS platform with ArgoCD and GitOps. What you can’t easily find is a list of the things that break in ways that produce no error, and how you notice.

What I built, briefly 🔥

Over three days I built a four-tier EKS platform with an enforced software supply chain. The short version:

• Terraform and Terragrunt for infrastructure, modular, with tiers as thin overrides. Adding a tier is four files averaging twenty lines.
• ArgoCD in an app-of-apps pattern. Terraform creates exactly one Kubernetes object; the root Application and everything after that is reconciled from git.
• GitLab CI building a hardened multi-stage image: FIPS-mode Go on a distroless static base, 7.5 MB, no shell, no package manager, no libc.
• Trivy and Kubescape as blocking gates, syft for SBOMs, cosign signing with an AWS KMS key.
• Kyverno verifying those signatures at admission, in Enforce mode, and rewriting image tags to the verified digest.
• Kargo promoting images between tiers by opening merge requests.
• Velero for backups, Coroot for eBPF observability, Pod Security Admission and NetworkPolicy for workload isolation.

No long-lived cloud credentials anywhere. IRSA for everything in-cluster, GitLab OIDC for CI. The only stored secrets are GitLab tokens in AWS Secrets Manager, pulled at runtime by External Secrets.

All of it built and broken in a development cluster costing about seven dollars a day, destroyed and rebuilt daily.

That last point turns out to be the whole argument, and I’ll come back to it.

72 hours of pure crying and fun 😄

Failure one: the promotion that changed nothing

Back to the green promotion.

The cause was two images: entries in a Kustomize overlay. My base Deployment used a short alias:

image: my-app

with the overlay mapping it to the real registry path:

images:
  - name: my-app
    newName: <account>.dkr.ecr-fips.us-east-1.amazonaws.com/dsp-shared/my-app
    newTag: 7fffedb8

Kargo’s kustomize-set-image step doesn't know about that indirection. It writes an entry keyed on the full registry path:

images:
  - name: <account>.dkr.ecr-fips.us-east-1.amazonaws.com/dsp-shared/my-app
    newTag: 7481984b        # Kargo's edit
  - name: my-app
    newName: <account>.dkr.ecr-fips.us-east-1.amazonaws.com/dsp-shared/my-app
    newTag: 7fffedb8        # the original, which wins

Two entries, both valid, both matching nothing the other matched. Kustomize applied the second. Kargo had done exactly what it was asked. ArgoCD had synced exactly what git contained. The deployed image never changed.

The fix was making the base reference the full path so both tools key on the same string. The lesson was the diagnostic:

kubectl kustomize apps/my-app/overlays/dev | grep "image:"

One line, one tag. Run it before merging a promotion. And after the promotion completes, check the pod rather than the promotion:

kubectl -n my-app get pod -o jsonpath='{.items[0].spec.containers[0].image}'

Green statuses are a claim about process. The running image is a fact.

Failure two: ArgoCD reported Synced while refusing to deploy

This one I caused myself, trying to fix something cosmetic 😢.

Kyverno’s mutateDigest rewrites an image reference at admission so what runs is provably the bytes that passed signature verification:

my-app:7481984b@sha256:095729a0c4e28cf2c331645514718628a91ef06...

Git has only the tag. ArgoCD compares live to desired, sees a difference it can never reconcile, and reports OutOfSync forever. Worse, selfHeal keeps trying to strip the digest that Kyverno keeps re-adding, resulting in two controllers quietly fighting over one field.

The obvious fix is to tell ArgoCD to ignore that field:

ignoreDifferences:
  - group: apps
    kind: Deployment
    jqPathExpressions:
      - '.spec.template.spec.containers[].image'

That works and then boom, the Application goes green.

It also means ArgoCD can no longer see legitimate image changes. When Kargo updated the tag in git, ArgoCD compared the live Deployment to the desired one, found no difference in any field it was allowed to look at, and reported Synced. It deployed nothing. There was no error, no warning, no OutOfSync marker. The dashboard was green and the system had silently stopped working.

I found it by forcing a sync manually and watching the image change.

The resolution was to accept a permanently OutOfSync-but-Healthy Application. A misleading status indicator is annoying. An Application that reports success while refusing to deploy is an outage waiting for someone to notice.

There’s a general principle here that cost me several hours across the project: anything that writes to a field git doesn’t control produces unreconcilable drift. I hit it four separate times before I recognized the pattern.

Each time, something other than git was writing to a field, so ArgoCD compared live state to desired state and found a difference it could never reconcile:

• containers[].image — Kyverno's mutateDigest appends the verified digest. Git only has the tag, because the digest isn't known until verification happens.
• APIService.spec.caBundle — cert-manager injects a CA bundle generated in-cluster. That certificate cannot exist in git; it doesn't exist until the cluster creates it.
• Warehouse.spec.subscriptions — Kargo's admission webhook writes defaulted fields back onto its own custom resource.
• directory.recurse: false — the API server itself, and this one is its own category.

That last one is worth its own sentence. I wrote recurse: false explicitly in an ArgoCD Application. false is the default, so the API server drops the key entirely. Git said false, the live object said nothing, and ArgoCD reported drift that could never be resolved.

Never declare a field at its default value in a GitOps manifest.

Failure three: NetworkPolicy that isn’t enforced

I wrote default-deny NetworkPolicies for every namespace (I know, many wouldn’t do this for shifting things left, but I DO 😀), plus explicit allows for DNS, same-namespace traffic, and the Kubernetes API. Applied them. kubectl get networkpolicy -A listed them all. Kubescape moved the relevant control from 0% to 68%.

None of it was enforced 😄😭.

The AWS VPC CNI accepts NetworkPolicy objects and ignores them unless you explicitly turn on enforcement:

configuration_values = jsonencode({ enableNetworkPolicy = "true" })

This is worse than having no policies. With no policies you know you have no policies. With unenforced policies you have documentation that claims otherwise, a compliance scanner that agrees with the documentation, and no actual isolation.

The check takes one command:

kubectl get ds aws-node -n kube-system \
  -o jsonpath='{.spec.template.spec.containers[*].name}'
# aws-node aws-eks-nodeagent

No aws-eks-nodeagent, no enforcement.

Then, once enforcement was on, the policies broke ArgoCD.

Not immediately, and not in a way that looked like networking. Every ArgoCD Application went to Unknown; not OutOfSync, not Degraded. The application-controller logs showed connection timeouts to two addresses:

dial tcp 172.20.222.51:8081: i/o timeout    # repo-server
dial tcp 172.20.180.188:6379: i/o timeout   # redis

My VPC CIDR is 10.10.0.0/16. I had allowed egress to the VPC and to 0.0.0.0/0 on 443. But ClusterIP Service addresses come from the service CIDR, 172.20.0.0/16 on EKS, which is an entirely separate range. A pod calling a Service in its own namespace still egresses to 172.20.x.x.

Two rules were missing:

egress:
  - to: [podSelector: {}]                       # same namespace, any port
  - to: [ipBlock: {cidr: 172.20.0.0/16}]        # ClusterIP Services

The same class of mistake took down health probes later. Kubelet probes originate from the node, not from a pod, so a policy allowing only same-namespace ingress drops them. Pods sat at 0/1 Ready indefinitely with nothing in the container logs, because the request never arrived.

Blocked connections look like timeouts, not refusals. They present as slow startup, mysterious readiness failures, and controllers that report Unknown. They rarely look like a firewall.

Failure four: thirty minutes of nothing (lol LITERALLY)

I added a launch template to the node group which was needed for Bottlerocket’s SSM control container, and a good opportunity to set the IMDS hop limit to 1 so pods can’t reach node credentials.

The apply ran for twenty-eight minutes. I checked:

aws eks describe-nodegroup --cluster-name dsp-dev-eks \
  --nodegroup-name dsp-dev-eks-ng --query 'nodegroup.{status:status,health:health}'

{ "status": "CREATING", "health": { "issues": [] } }

Creating, no issues. Meanwhile:

{ "desired": 3, "instances": 0 }

Zero instances. Every one was being terminated at launch, and the node group API reported nothing at all.

Get Katoria Henry’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The reason was only visible in one place:

aws autoscaling describe-scaling-activities --auto-scaling-group-name "$ASG" \
  --query 'Activities[].{status:StatusCode,cause:StatusMessage}'

Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

I had encrypted the root volumes with the cluster’s KMS key, the one used for envelope-encrypting Kubernetes Secrets. Encrypting EBS volumes with a customer managed key requires the Auto Scaling service-linked role to hold kms:CreateGrant on it. It doesn't, by default, and nothing tells you.

The fix was to use the AWS-managed EBS key, which encrypts at rest just as effectively. The cluster key’s job is etcd, not block storage.

describe-nodegroup will not tell you why a node group is stuck. describe-scaling-activities will. That single command would have saved twenty-five minutes.

Failure five: a chart pinned to an image that no longer exists

I pinned every Helm chart to an exact version, which is correct practice and which I felt good about.

Then Kyverno’s cleanup jobs started failing with ImagePullBackOff, and an ArgoCD sync operation wedged waiting on a hook that could never complete. It blocked that Application four separate times over two days.

The chart was pinned. The chart hadn’t changed. The image the chart references had been retired from public Docker Hub by its vendor.

A pinned chart guarantees the chart doesn’t change. It says nothing about the images inside it.

The second instance of the same class was subtler. Another job referenced:

image: busybox:1.35

No registry prefix, so it resolves to Docker Hub, and anonymous pulls from AWS IP ranges hit Docker Hub’s rate limit. Intermittent ImagePullBackOff on a chart that worked yesterday and will work again tomorrow.

The answer to both is mirroring third-party images into a registry you control. Which introduced its own trap:

Don’t mirror with docker pull on an Apple Silicon machine. You'll pull arm64 images and push them into a registry serving amd64 nodes, and the failure appears at runtime as an exec format error. crane copy transfers the full multi-arch manifest registry-to-registry without pulling anything locally.

crane manifest <image> | jq -r '.manifests[].platform | "\(.os)/\(.architecture)"'

Failure six: six (6) wrong Helm values paths

This one is embarrassing in aggregate 😭.

Helm silently ignores unknown values keys. A wrong path produces no warning, no error, and no effect. The deployment succeeds. The setting does nothing.

I got six of them wrong across the project:

• Kyverno’s IRSA annotation lives at admissionController.rbac.serviceAccount.annotations, not admissionController.serviceAccount.annotations. The leaf key existed; I had the parent wrong.
• Kubescape’s nodeAgent.enabled doesn't exist at all. The eBPF agent is controlled through capability flags, most of which default to enabled. My attempt to disable it did nothing, and would have deployed a privileged DaemonSet onto two t3.medium nodes.
• Coroot’s communityEdition sounds like a boolean. It's an object; its presence enables the feature. communityEdition: true is rejected; communityEdition: {} is correct.
• Coroot’s logCollector has no enabled field. It takes collectLogEntries and collectLogBasedMetrics.

Each was found by reading the actual values file or CRD schema:

helm show values <repo>/<chart> --version <v> > /tmp/values.yaml
kubectl explain <kind>.spec.<field>
kubectl apply --dry-run=server -f <file>

That last one is the best of the three. Server-side validation, one second, exact field name. Once I started running it before every commit, this entire class of failure disappeared.

There’s a variant when a CRD uses x-kubernetes-preserve-unknown-fields and there's no schema to read. Then you ask the webhook by submitting something deliberately wrong:

spec.subscriptions[0].image.imageSelectionStrategy: Unsupported value:
"DEFINITELY_NOT_VALID": supported values: "Digest", "Lexical",
"NewestBuild", "SemVer"

The rejection enumerates the valid values. Faster than searching documentation, and correct for the version you’re actually running.

The failures that were the system working

Not everything that looked like a problem was one.

Pod Security Admission rejected my test pod with five specific violations — privileged, no allowPrivilegeEscalation: false, capabilities not dropped, not runAsNonRoot, no seccomp profile. That's the API server refusing a non-compliant workload before Kyverno was even consulted.

A LimitRange rejected a pod requesting 16Mi because the namespace sets a 32Mi floor. The platform’s own guardrail catching the person who wrote it.

Kyverno blocked an unsigned image pushed to the same repository as a signed one:

resource Pod/my-app/blocked was blocked due to the following policies
verify-image-signatures:
  verify-signed-by-platform-key: 'failed to verify image ...: no signatures found'

That last test mattered more than the positive one. A policy that passes everything is indistinguishable from a policy that isn’t running. I only knew signature verification worked because I watched it refuse something.

Nothing could be created while Kyverno was down. During a rolling node replacement its admission webhook lost all endpoints, and because the chart uses failurePolicy: Fail, every pod creation cluster-wide was blocked until it recovered.

That’s correct behavior, and it’s the strongest argument I have for three admission controller replicas in production before tightening failurePolicy on my own policies. Enforcement that can be bypassed by an outage is weaker than it looks. Enforcement that causes the outage is worse.

What actually got built

Setting the failures aside, the finished path:

A commit triggers a build of a 7.5 MB distroless image running FIPS-mode Go. Trivy scans it and fails on HIGH or CRITICAL findings that have a fix available. Syft generates SBOMs in two formats. The image is pushed to ECR with an immutable tag, then signed with an AWS KMS key, signing after the push, because a signature binds to a digest and the digest doesn’t exist until the push completes.

Kargo polls ECR, notices the new image, and creates Freight. Promoting that Freight clones the GitOps repository, rewrites the image tag in the target overlay, and opens a merge request. It holds a token that could push directly to a protected branch. It deliberately doesn’t. The promotion blocks until a human merges.

ArgoCD syncs. At admission, Pod Security Admission enforces restricted, Kyverno verifies the cosign signature against a public key committed to git; no KMS call, no AWS dependency in the path that decides whether a pod may start — and rewrites the tag to the verified digest. NetworkPolicy denies everything not explicitly allowed. The pod cannot reach the instance metadata service, blocked both by policy and by an IMDS hop limit of 1.

Velero backs it all up to a versioned S3 bucket via IRSA, and I tested a restore rather than assuming one.

Three FIPS layers: the Go cryptographic module in the application, Bottlerocket FIPS AMIs on staging and production nodes, and FIPS endpoints for AWS API calls. The application reports its own mode at runtime, because a Dockerfile comment claiming FIPS proves nothing:

{ "go": "go1.26.8", "fipsMode": "fips140=on" }

Production was stood up and torn down deliberately, which converted four assumptions into facts; most importantly that EKS merges its bootstrap configuration into Bottlerocket’s TOML user data, which if wrong would have meant nodes that never join. It cost about a dollar.

Kargo promotion, seven green steps

Seven steps, all green, status Succeeded. The deployed image did not change.

Kargo pipeline view with “Waiting for Approval”

Kargo holds a token that could push directly to a protected branch. It opens a merge request instead and blocks until a human merges it.

Kargo MR Submission

MR for promotion submitted via Kargo; all MRs require human approval

GitLab pipeline, seven jobs across five stages

Build, scan, SBOM, publish, sign. Nothing reaches the registry until every gate passes; nothing is signed until it is in the registry, because a signature binds to a digest.

ArgoCD applications tiles

Fifteen Applications, all reconciled from git. Terraform creates exactly one of them; the rest are discovered from the root Application.

Kargo-pipelines remained OutOfSync for hours, despite reporting as “healthy”

Kubescape, cert-manager, and platform-dev were the least problematic ArgoCD Apps

Coroot service map

Coroot builds this from eBPF alone. No instrumentation, no sidecars, no application changes. The SSM agent connection is the private-endpoint access path, discovered rather than declared.

CloudWatch audit log group

All five control plane log types enabled, including audit. Retention is three days in dev and 365 in production.

The part I actually want you to take away

Every failure in this post happened in a development cluster.

Signature verification was configured wrong four times before it worked. The wrong Kyverno values path. Credentials the admission controller couldn’t use. A transparency log setting that had to match on both the signing and verifying side. A mutateDigest incompatibility with Audit mode that Kyverno rejects outright.

Four failures, each diagnosed in minutes, against a cluster nobody depended on and that I destroyed at the end of the day.

The same four failures against production would have been an incident, a change freeze, a rollback, and a meeting.

“Shift security left” usually means scanning earlier in the pipeline. That’s fine, and it’s the easy part. The version I think is more useful: build the enforcement, break it deliberately, and learn its failure modes somewhere the cost of being wrong is a rebuild.

You cannot learn that NetworkPolicy silently isn’t enforced on EKS by reading about NetworkPolicy. You learn it by writing policies, believing they work, and then testing whether a pod can actually reach something it shouldn’t.

You cannot learn that ignoring a field in ArgoCD can silently stop deployments by reading the ArgoCD documentation. You learn it by doing it and noticing your image didn’t change.

The cluster cost seven dollars a day. Every one of these lessons was available for less than the price of lunch, and every one of them would have been considerably more expensive to learn later.

The platform is Terraform, Terragrunt, EKS, ArgoCD, Kargo, Kyverno, Kubescape, Cosign, Velero, Coroot and Git

Follow me on LinkedIn 😄!

https://www.linkedin.com/in/katoria-henry-2018