<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rufaialmansur on Medium]]></title>
        <description><![CDATA[Stories by Rufaialmansur on Medium]]></description>
        <link>https://medium.com/@rufaialmansur3?source=rss-a30ec89f31d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*MbtSALtpremGKxRI</url>
            <title>Stories by Rufaialmansur on Medium</title>
            <link>https://medium.com/@rufaialmansur3?source=rss-a30ec89f31d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:02:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rufaialmansur3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sentinel Part 2: Safe Releases, Policy, CI Gates, and Secrets]]></title>
            <link>https://medium.com/@rufaialmansur3/sentinel-part-2-safe-releases-policy-ci-gates-and-secrets-19be7aae57d7?source=rss-a30ec89f31d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/19be7aae57d7</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Rufaialmansur]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 16:27:01 GMT</pubDate>
            <atom:updated>2026-09-22T16:27:01.856Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1 covered the foundation of the project, a containerized reference app, a local Kind cluster, and GitOps with ArgoCD. This post covers the part of Sentinel I was most looking forward to, actually watching traffic shift between versions during a release, having the cluster reject anything that doesn’t meet policy before it ever runs, gating the pipeline itself on tests and vulnerability scans, and pulling secrets in from a real secrets manager instead of hard-coding them.</p><h3>Replacing the Deployment with a Rollout</h3><p>A plain Kubernetes Deployment gives you rolling updates, but no control over the pace or shape of a release, no traffic percentages, no automated pause-and-check. Argo Rollouts solves that by introducing a Rollout custom resource that&#39;s a near drop-in replacement for a Deployment, but with a canary strategy layered on top.</p><pre>kubectl create namespace argo-rollouts<br>kubectl apply --server-side -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml</pre><p>The Rollout spec defines weighted steps with pauses between them:</p><pre>strategy:<br>  canary:<br>    steps:<br>      - setWeight: 25<br>      - pause: {duration: 60}<br>      - setWeight: 50<br>      - pause: {duration: 60}<br>      - setWeight: 100</pre><p>Every deploy now climbs through 25% → 50% → 100% traffic, rather than replacing every pod at once.</p><h3>Watching it actually work</h3><p>The kubectl argo rollouts plugin gives a live view of a release in progress, old and new ReplicaSets coexisting, weight climbing step by step. To prove it wasn&#39;t just the controller&#39;s internal accounting, I ran a loop hitting the service directly during a rollout and watched the responses mix between the old and new version in roughly the expected proportion before fully cutting over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3a87ulDeIkSf9-KlRr9uIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Scb24IxYWNl2TxVbFQWnAw.png" /></figure><h3>Kyverno: policy that actually blocks things</h3><p>Argo Rollouts controls how a release happens, Kyverno controls what’s allowed to run in the first place. I wrote two policies: one requiring every pod to run as non-root, and the other rejecting any image tagged :latest in favor of pinned versions.</p><pre>helm install kyverno kyverno/kyverno -n kyverno --create-namespace<br>kubectl apply -f policies/kyverno/require-non-root.yaml<br>kubectl apply -f policies/kyverno/disallow-latest-tag.yaml</pre><p>To prove enforcement rather than just assume it, I deployed a deliberately non-compliant pod:</p><pre>kubectl run bad-pod --image=nginx:latest -n sentinel</pre><p>It was rejected outright, with both violations listed in a single admission error — the :latest tag and the missing non-root context. That&#39;s the kind of proof that matters more than the policy YAML itself.</p><h3>CI: failing the build before it ever reaches the cluster</h3><p>Everything above governs what happens once something is deployed. None of it matters if a vulnerable dependency or a leaked credential gets built into the image in the first place, so the pipeline itself needed to be a gate, not just a build step.</p><p>The GitHub Actions workflow runs a chain of checks in sequence, and any failure stops the pipeline before an image goes anywhere:</p><pre>- name: Run tests<br>  run: npm test<br><br>- name: Secrets scan<br>  uses: gitleaks/gitleaks-action@v2<br><br>- name: Build image<br>  run: docker build -t sentinel-api:${{ github.sha }} .<br><br>- name: Trivy vulnerability scan<br>  uses: aquasecurity/trivy-action@master<br>  with:<br>    image-ref: sentinel-api:${{ github.sha }}<br>    severity: HIGH,CRITICAL<br>    exit-code: 1<br><br>- name: Generate SBOM<br>  uses: anchore/sbom-action@v0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v8zyymYe8BVegjRgugkd0A.png" /></figure><p>Tests run first, then gitleaks scans for anything that looks like a committed secret, then the image builds, then Trivy scans it and fails the build outright on any HIGH or CRITICAL CVE, then an SBOM gets generated and archived as a build artifact, a full inventory of everything that went into the image.</p><p>A pipeline that’s never seen a real failure isn’t proven. So I deliberately broke it: pushed a throwaway branch with a hardcoded fake AWS key and, separately, a known-vulnerable dependency version. Both were caught — gitleaks flagged the fake key before the image even built, and Trivy failed the build red on the CVE. Reverted both afterward. The gate isn’t decorative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dv-JX_y9tE7r6_q_kqfXTw.png" /></figure><h3>Secrets: pulling from a real secrets manager, not hardcoding</h3><p>The same pipeline that now blocks leaked secrets from ever being committed still needed somewhere real to pull legitimate secrets from at runtime, rather than baking them into manifests. That’s External Secrets Operator (ESO). It syncs a secret from an external store, AWS Secrets Manager, in this case, into a native Kubernetes Secret, so nothing sensitive lives in Git.</p><p>I built and proved this twice. First entirely offline against LocalStack (a local AWS emulator, free, no account needed) to prove the sync mechanics, then once for real against AWS Secrets Manager with a least-privilege IAM policy, since that’s the piece that’s actually interview-relevant, proving the sync alone only shows “it connects,” not that it’s scoped correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gOallvGRyA1FZAJ6I8ht3w.png" /></figure><p><strong>Local pass, offline:</strong></p><pre>docker run -d --name localstack -p 4566:4566 localstack/localstack:4.4.0<br>aws --endpoint-url=http://localhost:4566 secretsmanager create-secret \<br>  --name sentinel/api-key --secret-string &#39;{&quot;apiKey&quot;:&quot;local-test-value&quot;}&#39;<br><br>helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace</pre><p>A SecretStore points ESO at the provider, and an ExternalSecret declares what to pull and where to put it:</p><pre>apiVersion: external-secrets.io/v1<br>kind: SecretStore<br>metadata:<br>  name: localstack-store<br>spec:<br>  provider:<br>    aws:<br>      service: SecretsManager<br>      region: eu-north-1<br>      auth:<br>        secretRef:<br>          accessKeyIDSecretRef: {name: localstack-creds, key: access-key}<br>          secretAccessKeySecretRef: {name: localstack-creds, key: secret-key}</pre><pre>kubectl -n sentinel get secret sentinel-api-secret -o jsonpath=&#39;{.data.apiKey}&#39; | base64 -d<br># local-test-value</pre><p><strong>Real AWS pass, least privilege:</strong> an IAM user scoped to exactly one action on exactly one resource, nothing broader:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;secretsmanager:GetSecretValue&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:sentinel/api-key-XXXXXX&quot;<br>    }<br>  ]<br>}</pre><p>Swapping the SecretStore&#39;s credentials to that user and repeating the sync against real AWS Secrets Manager produced the same clean result real-test-value decoded out of a native Kubernetes secret, this time backed by a real secrets manager with a policy that grants nothing beyond what&#39;s needed. Both stores stayed live side by side, LocalStack and real AWS, same mechanism proven twice at two different trust levels.</p><h3>What broke</h3><p><strong>An orphaned Deployment kept serving stale traffic.</strong> After swapping the Deployment for a Rollout in Kustomize, I assumed ArgoCD would clean up the old resource automatically. It didn’t kubectl apply and ArgoCD&#39;s default sync don&#39;t prune resources removed from Git unless pruning is explicitly enabled. The old Deployment&#39;s pods were still running, still labeled identically to the new ones, and the Service happily load-balanced across both — which meant my &quot;canary&quot; test was silently serving 50% stale v1.0.0 traffic from pods that weren&#39;t even part of the Rollout anymore. Caught it by comparing kubectl get deployments against what Kustomize actually declared, and fixed it with a direct kubectl delete deployment.</p><p><strong>A named Dockerfile </strong><strong>USER isn&#39;t verifiably non-root.</strong> Adding runAsNonRoot: true to the Rollout&#39;s pod spec, the &quot;correct&quot; way to declare intent, rather than relying on the Dockerfile alone immediately broke every new pod with CreateContainerConfigError. The actual error: container has runAsNonRoot and image has non-numeric user (sentinel), cannot verify user is non-root. Kubernetes can&#39;t resolve a named user against a security policy; it needs a literal numeric UID it can check is non-zero. The fix was assigning an explicit UID in the Dockerfile (adduser -u 1001 ... &amp;&amp; USER 1001) instead of a name. The irony: my image was already running as non-root and had been the whole time — it just wasn&#39;t <em>provably</em> non-root until this change, which is exactly the gap Kyverno&#39;s policy exists to catch.</p><p><strong>A valid setup that was silently talking to the wrong place.</strong> This one didn’t throw an error until the very last step. The SecretStore validated fine. The ExternalSecret applied fine, no complaints, no events. Then nothing synced. The controller logs told the real story: UnrecognizedClientException: The security token included in the request is invalid straight from <em>real</em> AWS Secrets Manager, not my LocalStack container. My SecretStore spec had no endpoint override anywhere in it, so ESO&#39;s AWS SDK defaulted to the actual secretsmanager.&lt;region&gt;.amazonaws.com and dutifully tried to authenticate with throwaway LocalStack credentials of course real AWS rejected them. ESO&#39;s AWS provider doesn&#39;t take a per-SecretStore endpoint field; the fix lives one level up, on the controller deployment itself:</p><pre>helm upgrade external-secrets external-secrets/external-secrets -n external-secrets \<br>  --reuse-values \<br>  --set-json &#39;extraEnv=[{&quot;name&quot;:&quot;AWS_ENDPOINT_URL&quot;,&quot;value&quot;:&quot;http://host.docker.internal:4566&quot;}]&#39;</pre><p>Everything that looked configured wasn’t actually pointed anywhere near where I thought the kind of bug that hides behind a clean-looking Ready: True status until reconciliation actually runs.</p><h3>What’s next</h3><p>Sentinel now has real progressive delivery, real policy enforcement, a CI pipeline that actually blocks bad builds, and secrets pulled from a real secrets manager with least-privilege access. Part 3 covers observability with Prometheus and Grafana wired in so releases can be gated on real metrics, not just a fixed timer between canary steps.</p><p><em>This is Part 2 of the Sentinel series — a Kubernetes-native platform combining GitOps, progressive delivery, policy enforcement, security-gated CI, secrets management, and observability-driven releases. Repo: </em><a href="https://github.com/amrufai/sentinel"><em>github.com/amrufai/sentinel</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19be7aae57d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentinel: Building a Self-Governing Kubernetes Platform]]></title>
            <link>https://medium.com/@rufaialmansur3/sentinel-building-a-self-governing-kubernetes-platform-795004e23710?source=rss-a30ec89f31d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/795004e23710</guid>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[Rufaialmansur]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 13:44:09 GMT</pubDate>
            <atom:updated>2026-09-16T13:44:09.786Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1: Local Dev &amp; GitOps Foundations</p><h3>Why I’m building this</h3><p>I’ve spent the last few months going through Linux, Docker, AWS, Terraform, and Kubernetes on a structured DevOps path. Along the way I studied a few existing DevOps reference projects, solid individually, but their best ideas never combined. Real progressive delivery and observability existed only in a serverless setup, while the Kubernetes-focused work never got that same maturity. None had automated testing or security scanning in CI, and secrets were often plain YAML.</p><p>Sentinel is my attempt to close those gaps in one Kubernetes-native platform; GitOps, safe progressive releases, in-cluster policy enforcement, a security-gated CI pipeline, real secrets management, and observability that actually gates deployments.</p><p>This is Part 1 of a five-part series, covering the foundation with repo structure, a reference API, a local cluster, and GitOps with ArgoCD.</p><h3>What I’m building toward</h3><ul><li>A small API reporting its own version and deployment identity</li><li>Argo Rollouts managing canary and blue/green releases</li><li>Kyverno enforcing policy at admission time</li><li>CI that scans for vulnerabilities and leaked secrets before anything ships</li><li>Secrets pulled from AWS Secrets Manager, never committed to Git</li><li>Prometheus/Grafana gating whether a rollout is healthy enough to promote</li><li>Automated infrastructure drift detection</li><li>All deployed, for free, on k3s on an AWS free-tier instance</li></ul><h3>Setting up the repo</h3><p>Everything lives under one structure from day one:</p><pre>sentinel/<br>├── apps/api/<br>├── manifests/base/<br>├── manifests/overlays/{dev,prod}/<br>├── rollouts/<br>├── policies/{terraform,kyverno}/<br>├── infra/{kind,ec2-k3s}/<br>├── .github/workflows/<br>└── docs/medium/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcjSNNP7RJ32c_7FxDdCEA.png" /></figure><h3>The reference app</h3><p>A small API with one deliberate feature: an endpoint that reports its own deployment identity. Without that, “canary rollout” is just a claim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sB4VStg4c08G0PG1uLDpew.png" /></figure><p>In Part 2, hitting this endpoint during a rollout will show the traffic split in real time — some requests answering stable, some canary.</p><h3>A local cluster, for free</h3><p>Managed Kubernetes costs money the moment it exists — EKS runs about $70/month for the control plane alone, even idle. Kind (Kubernetes IN Docker) solves this. A real, spec-compliant cluster running in Docker on my own machine, free and disposable.</p><pre>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>name: sentinel-dev<br>nodes:<br>  - role: control-plane<br>  - role: worker</pre><p>Everything through Part 4 happens here. Only Part 5 moves to a real (still free-tier) cloud box.</p><h3>GitOps with ArgoCD</h3><p>Manual kubectl apply doesn&#39;t scale past one developer and leaves no audit trail. ArgoCD flips that. Git becomes the source of truth, and a controller continuously reconciles the live cluster against whatever is committed.</p><pre>kubectl create namespace argocd<br>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</pre><pre>argocd app create sentinel-dev \<br>  --repo https://github.com/amrufai/sentinel.git \<br>  --path manifests/overlays/dev \<br>  --dest-server https://kubernetes.default.svc \<br>  --dest-namespace sentinel</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kazLZTPkcIaCT2WMaMo0WA.png" /></figure><p>From here, deploying a change is a git push, not a kubectl apply — and any manual drift against the cluster gets flagged and reverted automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vcZAFkskgxNh2WvLuTgKg.png" /></figure><h3>What broke</h3><p><strong>ArgoCD wouldn’t respond over port-forward.</strong> curl -kvI showed the TCP connection succeeding but hanging right after the TLS handshake started. The cause: kubectl apply stores the full manifest as an annotation, capped at 256KiB, and the ApplicationSet CRD&#39;s schema exceeds that, so it silently failed to install. That left argocd-applicationset-controller crash-looping and the server unresponsive. Fixed with server-side apply, which skips that annotation:</p><pre>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml --server-side --force-conflicts</pre><p><strong>A stale port-forward squatted on 8080</strong> from an earlier attempt. Simplest fix was a forward on a different port (8888:443) instead of hunting the old process down.</p><p><strong>Kustomize refused to build the overlay</strong>, blocking my standard ../../base/namespace.yaml reference as a security restriction. Fixed by relaxing it cluster-wide:</p><pre>kubectl -n argocd patch configmap argocd-cm --type merge -p &#39;{&quot;data&quot;:{&quot;kustomize.buildOptions&quot;:&quot;--load-restrictor LoadRestrictionsNone&quot;}}&#39;<br>kubectl -n argocd rollout restart deployment argocd-repo-server</pre><p><strong>And the simplest failure:</strong> I built manifests/ and infra/ locally but never pushed them, so argocd app create failed — as far as GitHub was concerned, the files didn&#39;t exist. git status caught it instantly.</p><p>None of these were exotic, an annotation limit, a stale process, a default security setting, a forgotten push. But tracing each to its real cause, not just its symptom, is exactly what a tutorial skips and a real build doesn’t.</p><h3>What’s next</h3><p>By the end of this phase: a two-node Kind cluster, a containerized API with a working /version endpoint, and ArgoCD managing it declaratively — confirmed Synced and Healthy. Part 2 covers the part I&#39;m most looking forward to: replacing the plain Deployment with an Argo Rollout and watching traffic actually shift from stable to canary, with automatic rollback if things go wrong.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=795004e23710" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>