<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Syedusmanahmad on Medium]]></title>
        <description><![CDATA[Stories by Syedusmanahmad on Medium]]></description>
        <link>https://medium.com/@engr-syedusmanahmad?source=rss-543e77313b8d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7wcDyv6U84_kob2FCuU1eA.jpeg</url>
            <title>Stories by Syedusmanahmad on Medium</title>
            <link>https://medium.com/@engr-syedusmanahmad?source=rss-543e77313b8d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:17:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@engr-syedusmanahmad/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Pipelines to GitOps: Standing Up EKS and Argo CD]]></title>
            <link>https://engr-syedusmanahmad.medium.com/from-pipelines-to-gitops-standing-up-eks-and-argo-cd-b8169d32ca8a?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8169d32ca8a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-argocd]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 06:01:01 GMT</pubDate>
            <atom:updated>2026-10-01T06:01:01.727Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 5 of the CI/CD Pipeline Foundation series. Demos 1 through 4 built and compared two full CI pipelines against a single EC2 instance. This demo moves the deployment target to Kubernetes, standing up an EKS cluster with Terraform and bootstrapping Argo CD with an App-of-Apps pattern, so that adding a new service later means adding a file to Git rather than touching the cluster by hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hy-lhqSerTL4TXfy9vzjJQ.png" /></figure><h3>Why GitOps, and why now</h3><p>Everything through Demo 4 pushed an image somewhere and ran a script to update a running container. That works, but it means the cluster’s actual state lives partly in a deploy script and partly in whatever commands were run against it last. GitOps flips that: the cluster’s desired state lives entirely in Git, an agent inside the cluster continuously reconciles reality against it, and “what is currently deployed” becomes a question you answer by reading a repository instead of SSHing in or running kubectl get and hoping nothing drifted.</p><p>Argo CD is that agent. Getting it running, and proving it can actually deploy something end to end against a real, already-running application, took the rest of this demo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRNOlzZ3rLF0VNROY53e3A.png" /></figure><h3>Terraform first, and a lesson in not trusting stale knowledge</h3><p>The cluster was provisioned with the same Terraform module structure used in an earlier EKS series: a dedicated VPC, a managed node group, IAM access entries instead of the legacy aws-auth ConfigMap, S3-backed state with native locking instead of a DynamoDB lock table.</p><p>One number needed a hard correction along the way, and it is worth being honest about how that correction happened. A Kubernetes version was set based on what had been “current” during an earlier round of EKS work. That information was months stale. A quick check against the account’s own aws eks describe-addon-versions output showed newer versions already available and, more importantly, showed that the version being second-guessed as &quot;safe&quot; had already rolled out of EKS&#39;s standard support window and into extended support, which carries its own additional hourly charge per cluster. Correcting toward the version already in the account&#39;s supported list turned out to be both the technically current choice and the cheaper one. The lesson generalizes: for anything versioned as fast as Kubernetes, check the platform&#39;s own current state before trusting a number that &quot;worked before,&quot; especially when a wrong guess has a direct dollar cost attached.</p><p>A separate, smaller surprise showed up when scaling the node group after the fact. Increasing desired_capacity in terraform.tfvars and running apply only changed the node group&#39;s minimum size, and then failed outright, because EKS rejects a minimum size greater than the current desired size when they are set independently. The module, correctly, ignores lifecycle changes to desired_size so that the Cluster Autoscaler already running in the cluster can manage it without fighting Terraform. The fix was a single aws eks update-nodegroup-config call setting minSize, maxSize, and desiredSize together, which EKS accepts as one consistent state instead of two conflicting partial updates.</p><h3>Argo CD, deliberately without a load balancer</h3><pre>helm repo add argo https://argoproj.github.io/argo-helm<br>kubectl create namespace argocd<br>helm install argocd argo/argo-cd --namespace argocd --version 7.7.11</pre><p>Access to the UI went through kubectl port-forward rather than a LoadBalancer service. A LoadBalancer on EKS provisions a real AWS Network Load Balancer, at its own ongoing cost, for a cluster whose entire premise in this series is that it exists only while actively being worked on. Port-forwarding costs nothing and is more than adequate for a cluster that is not meant to run unattended.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0Xg1CITNh-hfWFivS5S2g.png" /></figure><h3>The App-of-Apps pattern, in three files</h3><p>The structure is a small chain: one root Argo CD Application watches a folder of other Application definitions, and each of those watches a folder of plain Kubernetes manifests.</p><pre># root-app.yaml, applied once by hand<br>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: root-app<br>  namespace: argocd<br>spec:<br>  project: default<br>  source:<br>    repoURL: &lt;your-manifests-repo&gt;.git<br>    targetRevision: containerize<br>    path: k8s/apps<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: argocd<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true</pre><pre># k8s/apps/psx-portfolio.yaml, discovered automatically by root-app<br>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: psx-portfolio<br>  namespace: argocd<br>spec:<br>  project: default<br>  source:<br>    repoURL: &lt;your-manifests-repo&gt;.git<br>    targetRevision: containerize<br>    path: k8s/base<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: default<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true</pre><p>The only file applied by hand, ever, is root-app.yaml. Everything under k8s/apps/ is picked up automatically the moment it exists in Git. Adding a second service later in this series means adding one more file in that folder, nothing more.</p><p>One detail worth remembering about this pattern specifically: the root Application lives outside the folder it manages, which means it is not self-managing. A git push that changes root-app.yaml itself does not take effect until kubectl apply -f root-app.yaml runs again by hand. Everything downstream of it is fully automatic; that one file is the deliberate exception.</p><h3>A private repository means Argo CD needs its own credentials</h3><p>The first sync attempt failed with a plain authentication required, because the manifests live in a private repository and Argo CD had no way to read it. Rather than introduce a new secrets mechanism just for this, the same pattern already used throughout the series, Secrets Manager plus the External Secrets Operator, extended cleanly to cover it: a fine-grained, read-only GitHub personal access token, scoped to Contents only on the one repository, stored as a secret, and synced into the cluster as a Kubernetes Secret carrying the specific label Argo CD looks for on repository credentials.</p><pre>apiVersion: external-secrets.io/v1<br>kind: ExternalSecret<br>metadata:<br>  name: github-repo-creds<br>  namespace: argocd<br>spec:<br>  refreshInterval: 1h<br>  secretStoreRef:<br>    name: aws-secrets-manager<br>    kind: ClusterSecretStore<br>  target:<br>    name: github-repo-creds<br>    creationPolicy: Owner<br>    template:<br>      metadata:<br>        labels:<br>          argocd.argoproj.io/secret-type: repository<br>      data:<br>        type: git<br>        url: &quot;{{ .url }}&quot;<br>        username: &quot;{{ .username }}&quot;<br>        password: &quot;{{ .password }}&quot;<br>  dataFrom:<br>    - extract:<br>        key: argocd/github-repo-creds</pre><p>That label, argocd.argoproj.io/secret-type: repository, is what tells Argo CD this Secret is a repository credential rather than an ordinary opaque Secret. Miss it, and the Secret exists correctly but is invisible to Argo CD.</p><p>A second, smaller gotcha surfaced applying this: the External Secrets Operator’s CRDs listed v1beta1 when queried, but that version was marked served: false, meaning only v1 was actually being served by the API server. Checking .spec.versions[*].served directly, rather than assuming a listed version is a usable one, was the difference between a working manifest and a confusing &quot;resource mapping not found&quot; error.</p><h3>Crossing a VPC boundary without setting up peering</h3><p>The Postgres database backing this application runs on the same EC2 instance used throughout the series, in a completely separate VPC from the new EKS cluster, with no peering connection between them. Rather than introduce one purely for this, the database’s existing configuration, opened up earlier in the series for practical, budget-driven reasons, already accepted connections from any source over its public IP. EKS pods reach it the same way any external client would: over the internet, through the node group’s outbound path, to the instance’s public address, with the connection string in the synced secret rewritten from localhost to that address by the same templating approach used for the standalone Docker deployment back in the earliest demos of this series.</p><p>It is not the design a production system would choose. It is the same conscious, documented tradeoff made earlier for the database’s network exposure, applied consistently rather than solved differently in every demo, and it meant zero additional network infrastructure to stand up, tear down, or pay for during the cluster’s short lifetime.</p><h3>Proof, not just a green checkmark</h3><p>The most convincing evidence this actually worked was not a Synced status in the Argo CD UI, though that appeared too. It was port-forwarding the resulting Service and loading the application in a browser: live KSE100 and KSE30 index values, a portfolio total, a &quot;last updated&quot; timestamp reading seconds old, all rendered by a pod that Argo CD had deployed from a Git commit, reading from a database across a VPC boundary, through credentials that had never been typed into the cluster by hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbVFhXyyVZUcQWD1cs4smg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKf-KZAAOdmJAAO2TyiA_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6QmivkFFT7K9k8sQZ7JPTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IfBlR8JzK-ToHmUU48dsiA.png" /></figure><h3>What this demo cost, and the discipline that keeps it that way</h3><p>Roughly the same per-hour math as earlier estimates in this series held: control plane, two small nodes for headroom once Argo CD’s own components were scheduled, and a single NAT Gateway. The actual dollar cost for a cluster run a handful of hours to prove all of this out was a few dollars, not a meaningful fraction of the series budget.</p><p>The discipline that keeps it that way is the same one repeated since the very first cost table in this series: nothing about EKS is expensive per experiment, everything about EKS is expensive if left running unattended. Scaling the node group to zero, or a full terraform destroy between working sessions, both cost nothing and both come back in minutes.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8169d32ca8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chat with Your Cluster: Building an AI Troubleshooting Agent for Kubernetes]]></title>
            <link>https://engr-syedusmanahmad.medium.com/chat-with-your-cluster-building-an-ai-troubleshooting-agent-for-kubernetes-54b5196b9fcc?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/54b5196b9fcc</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 06:01:02 GMT</pubDate>
            <atom:updated>2026-09-29T06:01:02.295Z</atom:updated>
            <content:encoded><![CDATA[<p>How I replaced the kubectl grind with a conversational agent that reasons over a live EKS cluster and what I learned building it for an AWS Community Talk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHkbeEYwHXO3EfB67zJWmA.png" /></figure><h3>The problem every DevOps engineer knows</h3><p>An alert fires. Something’s wrong in production. What happens next is almost always the same ritual:</p><pre>kubectl get pods -n my-namespace<br>kubectl describe pod broken-app-xxxx<br>kubectl logs broken-app-xxxx --previous<br>kubectl get events -n my-namespace</pre><p>Then you sit there mentally stitching together pod status, container state, log lines, and event timestamps until a root cause falls out. It works, but it depends entirely on how comfortable you are with Kubernetes. A senior engineer does this in ninety seconds. A junior engineer escalates, because the correlation itself is the hard part, not any single command.</p><p>I wanted to see what happens when you hand that correlation step to a model instead of a person not by scripting a fixed sequence of checks, but by giving an LLM the same tools an engineer has and letting it decide how to use them.</p><p>That became the subject of my AWS Community Talk, and this article walks through how it’s built.</p><h3>What “Chat with Your Cluster” actually does</h3><p>At its simplest: you ask a question in plain English, and an AI agent investigates your EKS cluster the way a human would, checking pod status, digging into logs, correlating events before explaining what’s wrong and how to fix it.</p><pre>You: Something seems wrong in the demo-app namespace, can you check?<br><br>Agent: [calls get_pods] → finds a CrashLoopBackOff pod<br>Agent: [calls describe_pod] → sees repeated restarts<br>Agent: [calls get_logs] → reads &quot;connection refused on port 5432&quot;<br>Agent: [calls get_events] → confirms the BackOff pattern<br><br>Agent: The pod is crash-looping because it can&#39;t reach a database<br>on port 5432. Likely causes: the database isn&#39;t running, or the<br>connection config is wrong. Suggested fix: verify the DB pod/service<br>exists and check the app&#39;s environment variables for host and port.</pre><p>Four tool calls, in that exact order, made without being told to make them, because the model decided, at each step, that it needed more information before it could answer confidently.</p><h3>Why this counts as “agentic” and not just a fancy script</h3><p>It would be easy to hardcode this: always run get_pods, then always run describe_pod, then always run get_logs. That&#39;s a script with extra steps, not an agent.</p><p>The difference is that the model is given the <em>tools</em>, not the <em>sequence</em>. On a different failure say, a missing container image the same agent, with the same code, chains a different path: it checks pods, sees ImagePullBackOff, and goes straight to events without ever calling get_logs, because logs are useless for a container that never started. Nobody told it to skip that step. It reasoned its way there.</p><p>That’s the actual value of tool-use over a monitoring script: the investigation adapts to what it finds, the way a person’s would.</p><h3>Architecture</h3><p>The system has four pieces:</p><pre>Browser (index.html)<br>      |  POST /chat<br>      v<br>FastAPI server (server.py) — streams progress via Server-Sent Events<br>      |<br>      v<br>Agent loop (bedrock_agent.py)<br>      |  \<br>      |   \--&gt; Amazon Bedrock (Claude Sonnet 4.6) — reasons, decides next tool call<br>      |<br>      \--&gt; k8s_tools.py --&gt; EKS cluster (read-only)</pre><p><strong>k8s_tools.py: </strong>the agent’s hands. Four read-only functions built on the Kubernetes Python client: get_pods, describe_pod, get_logs, and get_events. Nothing here can create, modify, or delete anything — which matters a lot once you&#39;re pointing this at a real cluster.</p><p><strong>bedrock_agent.py: </strong>the agent’s brain. This is where Amazon Bedrock’s Converse API and tool-use come in. The model is given a system prompt, a question, and a set of tool definitions. If it decides it needs data, it returns a tool-use request; the code executes the matching Python function and feeds the result back. This repeats sometimes once, sometimes four or five times until the model has enough signal to answer in plain English instead of asking for another tool.</p><p><strong>server.py: </strong>a thin FastAPI layer that turns the agent’s step-by-step reasoning into a live stream (Server-Sent Events) so a browser can render each tool call as it happens, not just the final answer.</p><p><strong>index.html: </strong>a chat interface that shows the investigation in progress: which tool is being called, with what arguments, and the formatted final diagnosis once the loop completes.</p><h3>The part that actually took the longest: model access</h3><p>If you’re trying this yourself, budget time for Bedrock model access. Listing an inference profile with list-inference-profiles doesn&#39;t mean your account can invoke it, several newer Claude models returned AccessDeniedException until I either requested access through the model card in the Bedrock console or fell back to a model I&#39;d already used successfully in an earlier project. If you hit this, don&#39;t burn your event-prep time on it: fall back to whatever model has already worked for you before, and revisit access requests separately.</p><h3>Read-only by design, on purpose</h3><p>Every tool in this project only reads from the cluster. No kubectl delete, no kubectl apply, nothing that mutates state. That wasn&#39;t a limitation I ran into it was a deliberate choice.</p><p>An agent that can <em>diagnose</em> production safely is immediately useful. An agent that can <em>act</em> on production needs a much higher bar of trust, guardrails, and probably a human in the loop before every action. Building the read-only version first meant I could point this at a real cluster, in front of a live audience, without worrying about what happens if the model reasons its way into something destructive.</p><p>Auto-remediation is the obvious next step and also exactly why it isn’t the first step.</p><h3>What I’d build next</h3><ul><li><strong>Auto-remediation for safe, pre-approved actions.</strong> Restarting a pod is low-risk; deleting a PVC is not. The interesting engineering problem is drawing that line correctly.</li><li><strong>Slack ChatOps.</strong> The same agent, reachable from the tool teams already live in during an incident.</li><li><strong>Multi-cluster support.</strong> Point the same reasoning loop at a fleet instead of a single kubeconfig context.</li></ul><h3>Build it yourself: step by step</h3><p>The full source is on GitHub. Here’s everything needed to get it running on your own EKS cluster.</p><p><strong>1. Prerequisites</strong></p><ul><li>An EKS cluster you already have kubectl access to</li><li>AWS credentials with Bedrock permissions (bedrock:InvokeModel, bedrock:Converse) and EKS read access</li><li>Python 3.9+</li><li>Model access enabled for an Anthropic model on Amazon Bedrock in your account and region (see the gotcha above check this first)</li></ul><p><strong>2. Clone the repo and set up a virtual environment</strong></p><pre>git clone https://github.com/engr-usman/aws-aiops-demo-repo.git<br>cd aws-aiops-demo-repo/demo-6-chat-with-your-cluster<br><br>python3 -m venv venv<br>source venv/bin/activate<br>pip install -r requirements.txt</pre><p><strong>3. Confirm your cluster access works</strong></p><pre>kubectl config current-context<br>kubectl get pods -A</pre><p>If this doesn’t return a pod list, fix cluster access before going further, the agent depends entirely on this working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*akuWsJ8xMfHRkTm8I-zuqQ.png" /></figure><p><strong>4. Deploy the demo namespace and a failure scenario</strong></p><p>The repo ships with three pre-built failure scenarios so you have something broken to diagnose:</p><pre>cd demo-manifests<br>kubectl apply -f namespace.yaml<br>chmod +x switch_scenario.sh<br><br>./switch_scenario.sh crashloop</pre><p>Check it deployed and is actually broken:</p><pre>kubectl get pods -n demo-app</pre><p>You should see a pod in CrashLoopBackOff. Swap crashloop for imagepull or oom to try the other two failure modes at any point, the script tears down and redeploys automatically.</p><p><strong>5. Run the agent from the command line first</strong></p><p>Before touching the web UI, confirm the core agent works end to end:</p><pre>cd ..<br>python bedrock_agent.py</pre><p>When prompted, ask something like:</p><pre>Something seems wrong in the demo-app namespace, can you check?</pre><p>You should see the tool calls print as they happen (get_pods, describe_pod, get_logs, get_events), followed by a plain-English diagnosis. If this works, the hard part, Bedrock access, cluster permissions, tool wiring, is confirmed working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UnFmlScmpPMlHgajfWyGuA.png" /></figure><p><strong>6. Launch the web UI</strong></p><pre>uvicorn server:app --reload --port 8000</pre><p>Open http://localhost:8000 in a browser. Ask the same question — this time you&#39;ll see each tool call stream in live as a card, with a checkmark once it resolves, followed by the formatted final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZZqBhuRfi3sN2cw7FyJ5pA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*WxJHWzLQPX_l--Cl_iM6hQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*bkz5901GLWsWHj8M-XWtEA.png" /></figure><p><strong>7. Try a different failure mode</strong></p><p>Switch scenarios and ask again without changing anything else:</p><pre>cd demo-manifests<br>./switch_scenario.sh imagepull<br>cd ..</pre><p>Ask the agent the same generic question in the UI. Watch which tools it calls this time, it should skip get_logs entirely, since a container that never pulled its image has nothing to log.</p><p><strong>If something breaks along the way</strong>, the most common failure point is Bedrock model access; see the gotcha section above before assuming it’s a code problem.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54b5196b9fcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitHub Actions vs Native AWS: Building the Same Pipeline Twice]]></title>
            <link>https://engr-syedusmanahmad.medium.com/github-actions-vs-native-aws-building-the-same-pipeline-twice-cf836249d102?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf836249d102</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[aws-developer-tools]]></category>
            <category><![CDATA[aws-codepipeline]]></category>
            <category><![CDATA[aws-codebuild]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Thu, 24 Sep 2026 06:01:02 GMT</pubDate>
            <atom:updated>2026-09-24T06:01:02.477Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 4 of the CI/CD Pipeline Foundation series. Demos 1 through 3 built a full GitHub Actions pipeline with a SonarQube quality gate. This demo rebuilds the build and push stage using AWS’s own native tooling, CodePipeline and CodeBuild, on a separate branch, purely to compare the two approaches side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJDrN8yCSrmTQg6FA547-A.png" /></figure><h3>Why build the same thing twice</h3><p>By this point in the series, GitHub Actions was already doing real work: build, test, a SonarQube quality gate, and a deploy to staging, all triggered by a push. Rebuilding that from scratch in a different tool might look redundant, but the two approaches solve the same problem with genuinely different tradeoffs, and the only honest way to compare them is to actually build both against the same application.</p><p>To avoid any chance of the two pipelines colliding, this demo runs on its own branch, codepipeline-demo, and stops at pushing an image to ECR rather than also deploying. The existing GitHub Actions pipeline on the main working branch was never touched.</p><h3>The one step that cannot be done by CLI alone</h3><p>Every other piece of AWS infrastructure in this series has been created with a CLI command. Connecting AWS to a GitHub repository is the exception. AWS CodeStar Connections require an actual OAuth handshake with GitHub, which means the connection starts in a PENDING state from the CLI and has to be finished in a browser.</p><pre>aws codestar-connections create-connection \<br>  --provider-type GitHub \<br>  --connection-name psx-portfolio-github \<br>  --region eu-central-1</pre><p>That command returns a connection ARN. In the AWS Console, under Developer Tools, Settings, Connections, the pending connection has an “Update pending connection” button, which walks through GitHub’s own authorization screen. Approve access for the specific repository, and the connection flips to Available. From that point forward, every other step is CLI or infrastructure as usual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OK7DpfyevoHo3s_GmFKlOw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s7ajxP5fNTIuW259bOyc-g.png" /></figure><h3>A buildspec is a workflow file with different syntax</h3><p>CodeBuild’s equivalent of a GitHub Actions workflow file is buildspec.yml, sitting at the repository root:</p><pre>version: 0.2<br>env:<br>  variables:<br>    AWS_REGION: &quot;eu-central-1&quot;<br>    ECR_REPOSITORY: &quot;psx-portfolio-repo&quot;<br>phases:<br>  install:<br>    runtime-versions:<br>      nodejs: 20<br>    commands:<br>      - npm ci<br>  pre_build:<br>    commands:<br>      - npm test --if-present<br>      - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com<br>      - IMAGE_TAG=codebuild-$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c1-12)<br>  build:<br>    commands:<br>      - docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$IMAGE_TAG .<br>  post_build:<br>    commands:<br>      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$IMAGE_TAG</pre><p>Two details worth calling out. The account ID is injected as an environment variable on the CodeBuild project rather than hardcoded here, so this file stays safe to commit even to a public repository. And the image tag uses a codebuild- prefix, deliberately different from the sha- prefix GitHub Actions uses, so a glance at the ECR repository immediately shows which pipeline produced which image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y6hkhKHRO4CCkUXqi6LH6A.png" /></figure><h3>Two IAM roles, matching the same least-privilege pattern as before</h3><p>CodeBuild and CodePipeline are separate AWS services with separate trust relationships, so each needs its own role. CodeBuild’s role covers exactly three things: writing its own logs, pushing and pulling from the one ECR repository it needs, and reading and writing the S3 bucket used to pass artifacts between pipeline stages.</p><pre>{<br>  &quot;Sid&quot;: &quot;ECRPushPull&quot;,<br>  &quot;Effect&quot;: &quot;Allow&quot;,<br>  &quot;Action&quot;: [<br>    &quot;ecr:BatchCheckLayerAvailability&quot;,<br>    &quot;ecr:GetDownloadUrlForLayer&quot;,<br>    &quot;ecr:BatchGetImage&quot;,<br>    &quot;ecr:PutImage&quot;,<br>    &quot;ecr:InitiateLayerUpload&quot;,<br>    &quot;ecr:UploadLayerPart&quot;,<br>    &quot;ecr:CompleteLayerUpload&quot;<br>  ],<br>  &quot;Resource&quot;: &quot;arn:aws:ecr:eu-central-1:ACCOUNT_ID:repository/psx-portfolio-repo&quot;<br>}</pre><p>CodePipeline’s role is narrower still: read and write the artifact bucket, start a build on the one named CodeBuild project, and use the one specific CodeStar connection. Neither role can touch anything outside its own job, the same discipline applied to the GitHub Actions roles back in Demo 2.</p><h3>The part that actually needs a flag most people forget</h3><pre>{<br>  &quot;environment&quot;: {<br>    &quot;type&quot;: &quot;LINUX_CONTAINER&quot;,<br>    &quot;image&quot;: &quot;aws/codebuild/amazonlinux2-x86_64-standard:5.0&quot;,<br>    &quot;computeType&quot;: &quot;BUILD_GENERAL1_SMALL&quot;,<br>    &quot;privilegedMode&quot;: true<br>  }<br>}</pre><p>privilegedMode: true is not optional here. Running docker build inside a CodeBuild container is Docker-in-Docker, and without privileged mode the build fails immediately trying to reach a Docker daemon that was never allowed to start. This is one of those settings that is obvious once you know to look for it and completely opaque from the error message alone if you do not.</p><h3>Wiring the pipeline itself</h3><pre>{<br>  &quot;pipeline&quot;: {<br>    &quot;name&quot;: &quot;psx-portfolio-pipeline&quot;,<br>    &quot;artifactStore&quot;: { &quot;type&quot;: &quot;S3&quot;, &quot;location&quot;: &quot;psx-portfolio-pipeline-artifacts&quot; },<br>    &quot;stages&quot;: [<br>      {<br>        &quot;name&quot;: &quot;Source&quot;,<br>        &quot;actions&quot;: [{<br>          &quot;name&quot;: &quot;GitHub_Source&quot;,<br>          &quot;actionTypeId&quot;: {<br>            &quot;category&quot;: &quot;Source&quot;, &quot;owner&quot;: &quot;AWS&quot;,<br>            &quot;provider&quot;: &quot;CodeStarSourceConnection&quot;, &quot;version&quot;: &quot;1&quot;<br>          },<br>          &quot;outputArtifacts&quot;: [{ &quot;name&quot;: &quot;SourceOutput&quot; }],<br>          &quot;configuration&quot;: {<br>            &quot;ConnectionArn&quot;: &quot;arn:aws:codestar-connections:eu-central-1:ACCOUNT_ID:connection/CONNECTION_ID&quot;,<br>            &quot;FullRepositoryId&quot;: &quot;engr-usman/Stock-Management-App&quot;,<br>            &quot;BranchName&quot;: &quot;codepipeline-demo&quot;<br>          }<br>        }]<br>      },<br>      {<br>        &quot;name&quot;: &quot;Build&quot;,<br>        &quot;actions&quot;: [{<br>          &quot;name&quot;: &quot;CodeBuild_Build&quot;,<br>          &quot;actionTypeId&quot;: { &quot;category&quot;: &quot;Build&quot;, &quot;owner&quot;: &quot;AWS&quot;, &quot;provider&quot;: &quot;CodeBuild&quot;, &quot;version&quot;: &quot;1&quot; },<br>          &quot;inputArtifacts&quot;: [{ &quot;name&quot;: &quot;SourceOutput&quot; }],<br>          &quot;outputArtifacts&quot;: [{ &quot;name&quot;: &quot;BuildOutput&quot; }],<br>          &quot;configuration&quot;: { &quot;ProjectName&quot;: &quot;psx-portfolio-build&quot; }<br>        }]<br>      }<br>    ]<br>  }<br>}</pre><p>Pinning BranchName to codepipeline-demo here is what kept this experiment fully isolated from the working GitHub Actions pipeline for the whole exercise.</p><h3>It worked on the first run, and one detail confirmed something unexpected</h3><p>Both the Source and Build stages passed on the very first execution. What was genuinely worth checking afterward: the resulting image size, 97.39MB, matched the GitHub Actions build almost exactly, despite CodeBuild running on Amazon Linux 2 and GitHub Actions running on an Ubuntu-based runner. Next.js’s output: &#39;standalone&#39; optimization from Demo 1 behaved identically across two different host operating systems, which is exactly what a build optimization should do, but is worth confirming rather than assuming.</p><p>Creating the pipeline also creates an EventBridge rule automatically, so a later push to the same branch triggered a new execution without touching any settings, no webhook secrets to manage on the GitHub side at all, the CodeStar connection handles that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*7s4VjgNLSNrzJYhvHtHRFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*96cRh_d3lB8-9XAJmDwSzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MszY8iM9LuOa7jRSShALog.png" /></figure><h3>GitHub Actions versus CodePipeline, having now built both</h3><p>GitHub Actions kept everything in one YAML file, in the same repository as the code, easy to read top to bottom. CodePipeline split configuration across a pipeline definition, a separate CodeBuild project, IAM roles for two different services, and an S3 bucket, more pieces, but each piece independently visible and auditable in the AWS console without leaving AWS at all.</p><p>Authentication tells a similar story from the opposite direction. GitHub Actions needed OIDC federation, set up by hand in Demo 2, to avoid static AWS keys. CodePipeline’s access to GitHub came from CodeStar Connections’ own OAuth flow, no equivalent setup required, because the trust relationship runs through AWS’s own connected-services model instead.</p><p>Neither is strictly better. A team already living in GitHub gets a tighter, single-file loop from Actions. A team standardizing on AWS-native tooling, wanting every pipeline stage visible in one console, or building on services that assume CodePipeline as the trigger, gets a more composable, more visible-to-AWS system from CodePipeline and CodeBuild. Having both running against the same real application, rather than reading about the tradeoffs secondhand, made the difference concrete instead of theoretical.</p><h3>Coming next</h3><p>Demo 5 moves toward Kubernetes: standing up an EKS cluster and installing Argo CD, the first step toward a full GitOps deployment flow, with the cluster’s lifetime deliberately kept short to control cost.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf836249d102" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Quality Gate That Actually Blocks Bad Code: Portfolio, Demo 3]]></title>
            <link>https://engr-syedusmanahmad.medium.com/a-quality-gate-that-actually-blocks-bad-code-portfolio-demo-3-f525933dbb40?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f525933dbb40</guid>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[sonarqube]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 06:01:01 GMT</pubDate>
            <atom:updated>2026-09-22T06:01:01.702Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 3 of the CI/CD Pipeline Foundation series. <a href="https://engr-syedusmanahmad.medium.com/812db5b63ebe?sharedUserId=engr-syedusmanahmad">Demo 2</a> automated a full build, push, and deploy pipeline. This demo adds a SonarQube quality gate in front of it and spends real effort proving the gate can actually say no.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jShIE7IGhjWCPUW8GoNe9A.png" /></figure><h3>The goal, and the honest problem with most “quality gate” tutorials</h3><p>The plan for this demo was simple on paper: add a SonarQube scan to the CI workflow, and stop the deployment if the scan fails. Most tutorials stop there, show a green checkmark, and call it done.</p><p>What they skip is the part that matters most: does the gate actually block anything, or does it just sit there looking authoritative while deployments sail through regardless? This demo spends more time answering that question than it does on the initial setup, and that time turned out to be the most useful part of the whole exercise.</p><h3>Step one: doing the memory arithmetic before touching Docker</h3><p>SonarQube Community Edition bundles Elasticsearch, and Elasticsearch wants at least 2GB of RAM just to run. The EC2 instance already hosting the containerized staging app also runs four pm2 processes and a local Postgres database. A quick check of available memory showed a little over 2GB free, with swap already seeing light use. That is not headroom, that is a rounding error away from a problem.</p><p>Rather than skip the check and find out the hard way, the right call was to run SonarQube with a hard memory cap and on demand only, started right before a scan and stopped right after, instead of as a permanent service. Docker’s --memory and --memory-swap flags create a cgroup boundary: if the process inside tries to exceed it, only that container gets killed. The host, and everything else running on it, is untouched.</p><pre>docker run -d \<br>  --name sonarqube \<br>  --memory=&quot;1800m&quot; \<br>  --memory-swap=&quot;1800m&quot; \<br>  -e SONAR_ES_JAVA_OPTS=&quot;-Xms512m -Xmx512m&quot; \<br>  -p 9000:9000 \<br>  --restart no \<br>  sonarqube:community</pre><p>--restart no was deliberate. The instinct with Docker is usually to add --restart unless-stopped so nothing needs babysitting. Here, that would have meant SonarQube silently coming back to life after every host reboot, permanently eating into memory that the live production app needs. On demand means on demand, not &quot;usually off.&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5T8NnrEVgk7e_BsaeHDPIg.png" /></figure><h3>Two OOM kills, and why they were actually good news</h3><p>The first attempt, capped at 1800MB, settled at 99% of its own limit just from starting up, before any scan had even run. Pushing a real analysis through it triggered a kernel out of memory kill. Raising the cap to 2200MB and giving Elasticsearch, the web process, and the Compute Engine process each their own explicit heap limit bought a little more room, but a second analysis run killed it again.</p><p>Both times, the kernel logs told a clear story:</p><pre>oom-kill:constraint=CONSTRAINT_MEMCG, ... task=java, ... oom_score_adj=0</pre><p>CONSTRAINT_MEMCG is the detail that matters. This was not a host level, &quot;the whole box ran out of memory&quot; kill. It was the kernel&#39;s cgroup controller enforcing the exact boundary that had been set, killing only the process inside that boundary. Both times, pm2 status showed the same restart counts before and after, and available host memory returned to normal within seconds. The safety mechanism did precisely what it was built to do.</p><p>That said, two consecutive crashes on the same host is also a signal, not something to push through by raising the cap a third time. Going higher would have meant leaving less and less headroom for the actual production processes, trading a container crash for a slower, harder to diagnose problem: a host under constant memory pressure. At that point the more honest fix was a dedicated instance, not a bigger number.</p><h3>Moving SonarQube to its own instance</h3><p>A small, SSM-managed EC2 instance, with no SSH key pair and no inbound port 22 at all, solved this cleanly. Its IAM role carries exactly one managed policy, AmazonSSMManagedInstanceCore, nothing else. Access happens entirely through aws ssm start-session, which also means every command run on the box is logged in AWS rather than living only in a terminal&#39;s shell history.</p><pre>aws ec2 run-instances \<br>  --image-id &lt;same-ami-as-main-instance&gt; \<br>  --instance-type t3.medium \<br>  --subnet-id &lt;same-subnet&gt; \<br>  --iam-instance-profile Name=SonarQube-EC2-Profile \<br>  --user-data file://sonarqube-userdata.sh \<br>  --block-device-mappings &#39;[{&quot;DeviceName&quot;:&quot;/dev/sda1&quot;,&quot;Ebs&quot;:{&quot;VolumeSize&quot;:20,&quot;VolumeType&quot;:&quot;gp3&quot;}}]&#39;</pre><p>The user data script installs Docker and sets the kernel’svm.max_map_count, the one prerequisite Elasticsearch insists on, at boot, so the instance is ready the moment it registers with SSM. With a full 4GB and nothing else competing for it, the same container, now given a genuinely generous cap, started cleanly and stayed stable through repeated scans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5T8NnrEVgk7e_BsaeHDPIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vTquv2oW8a9cXahdIv8gHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dF5zpit01YJvoieA6sb_aQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ia31PIEvnALj2e7S76rQng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hoeLNapCxck6q_5__Suo2Q.png" /></figure><h3>Proving the gate has teeth: three tests, two of them wrong</h3><p>This is the part most write-ups skip entirely, and it is where the real lessons were.</p><p><strong>Attempt one: a fake secret.</strong> The first instinct was to commit a string that looked like an AWS access key, on the reasoning that SonarQube’s secret scanner should flag it and fail the gate. It did not. The key used, AKIAIOSFODNN7EXAMPLE, is AWS&#39;s own publicly documented example credential, and essentially every secret scanner on the market explicitly allowlists it for exactly that reason. A useful reminder before writing a security test: check whether the &quot;vulnerable&quot; value is actually a known-safe placeholder first.</p><p><strong>Attempt two: comparing against </strong><strong>NaN.</strong> value === NaN is a classic, well-known JavaScript reliability bug; it is always false, regardless of the actual value. SonarQube&#39;s own rule catalogue includes a check for exactly this pattern. It still passed. Rather than assume something was broken, the dashboard&#39;s Issues view was checked directly with a filter for the specific file, and the rule&#39;s active status was checked directly in the Quality Profile. Both times the honest answer was: this did not need debugging, it needed verification, and the tool&#39;s own UI could give a definitive yes or no in under a minute instead of guessing.</p><p><strong>Attempt three, the one that actually worked, arrived in two parts.</strong> Unreachable code after a return statement (rule S1763) is about as unambiguous a bug as exists, and it did get correctly detected and listed as an issue in the dashboard. And the Quality Gate still passed. The dashboard&#39;s own issue detail showed why: the issue&#39;s &quot;introduced&quot; timestamp pointed to months earlier, not to the recent commit. The file had simply been renamed with git mv to test it, and Git&#39;s rename detection meant the line&#39;s blame history, which SonarQube uses to decide what counts as &quot;new&quot; code, still pointed at its original commit. A file that exists in a new location is not automatically new code; only content whose blame history is recent counts. Once the file&#39;s actual content was changed, not just its name, the same bug was correctly picked up as new, and the Quality Gate genuinely, provably failed:</p><pre>✗ Quality Gate has FAILED.<br>Error: Process completed with exit code 1.</pre><p>The workflow’s downstream steps, from the ECR push through the SSM deploy, showed as skipped rather than run, which is the actual proof the gate has teeth: not a green checkmark, but a chain of steps that visibly never got the chance to execute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRpXWxbHbD1k_2TENdQLng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UquWvWhe1FMhk4PKlhQ0Xw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azsD3tBKm08Zgi1g0jnUMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PsZgL8vA5fL-7X7OFzqqSw.png" /></figure><h3>The workflow, once the gate could be trusted</h3><pre>- name: SonarQube Scan<br>  uses: SonarSource/sonarqube-scan-action@7006c4492b2e0ee0f816d36501671557c97f5995 # v8.1.0<br>  env:<br>    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}<br>    SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}<br>- name: SonarQube Quality Gate<br>  uses: SonarSource/sonarqube-quality-gate-action@cf038b0e0cdecfa9e56c198bbb7d21d751d62c3b # v1.2.0<br>  timeout-minutes: 5<br>  env:<br>    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}</pre><p>Worth calling out: the official setup snippet ships with the Quality Gate step commented out by default, reporting results without ever blocking anything. Uncommenting it, and placing it before the AWS credential and deploy steps in the job, is what turns “we run a scan” into “a failing scan stops the deployment.” Left as-is, it is a dashboard. Enabled, it is a gate.</p><h3>What this demo actually cost, and what it proves</h3><p>In AWS terms: a second small EC2 instance, run only while actively scanning, and stopped the rest of the time. In debugging terms: two OOM kills correctly contained by cgroup limits, two false-negative security tests that turned out to be the tests being wrong rather than the tool, and one real bug that exposed exactly how SonarQube decides what “new code” means.</p><p>None of that shows up in a five-minute walkthrough. All of it is the difference between a pipeline that looks like it enforces quality and one that has actually been made to prove it does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Ia_5cj8o_IWuvZEJqREvg.png" /></figure><h3>Coming next</h3><p>Demo 4 brings AWS CodePipeline and CodeBuild into the picture as a native AWS alternative to GitHub Actions for the build stage, setting up a direct comparison between the two approaches before the series moves on to EKS and GitOps with Argo CD.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f525933dbb40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Manual Docker Commands to Push to Deploy: Portfolio, Demo 2]]></title>
            <link>https://engr-syedusmanahmad.medium.com/from-manual-docker-commands-to-push-to-deploy-portfolio-demo-2-812db5b63ebe?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/812db5b63ebe</guid>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nextjs]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 07:01:02 GMT</pubDate>
            <atom:updated>2026-09-16T07:01:02.936Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 2 of the CI/CD Pipeline Foundation series. <a href="https://engr-syedusmanahmad.medium.com/84f7bac57368?sharedUserId=engr-syedusmanahmad">Demo 1</a> containerized a live Next.js app by hand. This demo automates every step of that process with GitHub Actions, ECR, and AWS Systems Manager.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cWnwY6FVVfqy5Q46rctfOg.png" /></figure><h3>Where Demo 1 left off</h3><p>By the end of Demo 1, PSX Portfolio was running as a healthy Docker container on a staging port, alongside its untouched production process. Getting there required manually running docker buildmanually fetching secrets, and manually running docker run on the EC2 instance itself.</p><p>That is fine for proving a concept. It is not a pipeline. Demo 2 turns that sequence of manual commands into something that runs automatically on every push to the containerize branch: build, test, push to a private registry, and deploy, with no SSH session and no person watching a terminal.</p><h3>Design decision: OIDC over long-lived AWS keys</h3><p>The most common way to give GitHub Actions access to AWS is to generate an IAM user’s access key and secret key, then paste them into GitHub Secrets. It works, and it is also a standing liability: those credentials do not expire on their own, and if they ever leak, whoever has them can act as that IAM identity indefinitely.</p><p>GitHub Actions supports OpenID Connect federation instead. AWS trusts GitHub’s identity provider directly, and a workflow run requests short-lived, automatically expiring credentials scoped to a specific IAM role, no stored secret required beyond a role ARN, which is not sensitive on its own.</p><p>Setting this up has two parts: an OIDC identity provider registered once per AWS account, and an IAM role with a trust policy that only allows a specific GitHub repository, and in this case a specific branch, to assume it.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;token.actions.githubusercontent.com:aud&quot;: &quot;sts.amazonaws.com&quot;<br>        },<br>        &quot;StringLike&quot;: {<br>          &quot;token.actions.githubusercontent.com:sub&quot;: &quot;repo:&lt;github_account&gt;/&lt;repository_name&gt;:ref:refs/heads/containerize&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>That sub condition is doing real work. Without the ref:refs/heads/containerize suffix, any branch or pull request in the repository could assume this role. Scoping it down means a workflow running from an unrelated branch simply cannot authenticate as this role at all.</p><h3>Building the ECR repository</h3><pre>aws ecr create-repository \<br>  --repository-name psx-portfolio-repo \<br>  --region eu-central-1 \<br>  --image-scanning-configuration scanOnPush=true \<br>  --image-tag-mutability IMMUTABLE</pre><p>Immutable tags mean once an image is pushed under a given tag, that tag cannot be overwritten. Combined with tagging every build with its git commit SHA, this gives a hard guarantee: a given tag always points to exactly one build, forever. It also means a plain latest tag strategy will not work here, since latest would need to be reassigned on every push, so the workflow tags images as sha-&lt;commit sha&gt; only.</p><p>A lifecycle policy keeps storage costs predictable: untagged images expire after seven days, and only the ten most recent tagged images are retained.</p><h3>Least privilege, in two separate roles</h3><p>It would have been simpler to give one IAM role broad permissions and use it everywhere. Instead, this demo uses two roles with deliberately non-overlapping permissions.</p><p>The GitHub Actions role can push images to ECR and send commands through Systems Manager. It cannot pull images, and it cannot do anything else in the account.</p><p>The EC2 instance role can pull images from ECR and read one specific secret from Secrets Manager. It cannot push images, and it has no knowledge of the GitHub Actions role at all.</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;ECRPushPull&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;ecr:BatchCheckLayerAvailability&quot;,<br>        &quot;ecr:GetDownloadUrlForLayer&quot;,<br>        &quot;ecr:BatchGetImage&quot;,<br>        &quot;ecr:PutImage&quot;,<br>        &quot;ecr:InitiateLayerUpload&quot;,<br>        &quot;ecr:UploadLayerPart&quot;,<br>        &quot;ecr:CompleteLayerUpload&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;arn:aws:ecr:eu-central-1:ACCOUNT_ID:repository/psx-portfolio-repo&quot;<br>    }<br>  ]<br>}</pre><p>Neither role can do the other’s job. If the GitHub Actions role’s credentials were somehow exposed, an attacker could push a malicious image but could not deploy it. That separation is deliberate and costs nothing to maintain.</p><h3>Deploying without SSH: AWS Systems Manager</h3><p>Instead of SSH, the workflow uses ssm send-command to run a shell script on the EC2 instance directly through the AWS API. No inbound port 22 access is required for this to work, and every command executed is logged in AWS, which is a meaningfully better audit trail than shell history on a box.</p><p>Getting Systems Manager working required a debugging detour worth describing honestly, because the failure mode is confusing and the eventual cause was almost invisible from the error message alone.</p><p>The SSM agent was already running on the instance, but registration kept failing with an AccessDeniedException on ssm:UpdateInstanceInformation, even after attaching the standard AmazonSSMManagedInstanceCore managed policy to the instance role. The obvious suspects were checked and eliminated one at a time: no explicit Deny statement on any attached policy, no inline policies on the role, no permissions boundary, no AWS Organizations service control policy (the account is not part of an organization at all), and the instance profile was confirmed to wrap the exact role that had just been updated.</p><p>The actual cause was simpler than any of those: IAM policy attachments can take a short time to propagate, and the SSM agent had attempted registration and cached its failure before the new policy had fully taken effect. Restarting the agent after a longer wait resolved it immediately, with no configuration change required at all. The lesson generalizes well beyond this one bug: when an IAM permission was clearly just granted, and a service still reports it as missing, restart the thing consuming that permission before assuming the policy itself is wrong.</p><h3>The deploy script</h3><pre>#!/bin/bash<br>set -euo pipefail<br>AWS_REGION=&quot;eu-central-1&quot;<br>ECR_REPO=&quot;ACCOUNT_ID.dkr.ecr.eu-central-1.amazonaws.com/psx-portfolio-repo&quot;<br>IMAGE_TAG=&quot;${1:?Usage: deploy-staging.sh &lt;image-tag&gt;}&quot;<br>CONTAINER_NAME=&quot;psx-portfolio-staging&quot;<br>aws ecr get-login-password --region &quot;$AWS_REGION&quot; \<br>  | docker login --username AWS --password-stdin &quot;$ECR_REPO&quot;<br>docker pull &quot;$ECR_REPO:$IMAGE_TAG&quot;<br>./fetch-secrets.sh<br>docker stop &quot;$CONTAINER_NAME&quot; 2&gt;/dev/null || true<br>docker rm &quot;$CONTAINER_NAME&quot; 2&gt;/dev/null || true<br>docker run -d \<br>  --name &quot;$CONTAINER_NAME&quot; \<br>  --env-file .env.staging \<br>  --add-host=host.docker.internal:host-gateway \<br>  -p 3001:3000 \<br>  --restart unless-stopped \<br>  &quot;$ECR_REPO:$IMAGE_TAG&quot;<br>docker images &quot;$ECR_REPO&quot; --format &quot;{{.ID}} {{.CreatedAt}}&quot; \<br>  | sort -k2 -r | tail -n +4 | awk &#39;{print $1}&#39; \<br>  | xargs -r docker rmi 2&gt;/dev/null || true</pre><p>This is the exact sequence that was run by hand in Demo 1, turned into an idempotent script: log in, pull, refresh secrets, replace the running container, and clean up old images so disk usage does not grow unbounded on every deploy.</p><h3>Wiring it into the workflow</h3><pre>- name: Deploy to staging via SSM<br>  env:<br>    IMAGE_TAG: sha-${{ github.sha }}<br>    INSTANCE_ID: &lt;INSTANCE_ID&gt;<br>  run: |<br>    COMMAND_ID=$(aws ssm send-command \<br>      --instance-ids &quot;$INSTANCE_ID&quot; \<br>      --document-name &quot;AWS-RunShellScript&quot; \<br>      --parameters &quot;commands=[\&quot;cd /path/to/app &amp;&amp; ./deploy-staging.sh $IMAGE_TAG\&quot;]&quot; \<br>      --query &quot;Command.CommandId&quot; \<br>      --output text)<br><br>    aws ssm wait command-executed \<br>      --command-id &quot;$COMMAND_ID&quot; \<br>      --instance-id &quot;$INSTANCE_ID&quot; || true<br>    STATUS=$(aws ssm get-command-invocation \<br>      --command-id &quot;$COMMAND_ID&quot; \<br>      --instance-id &quot;$INSTANCE_ID&quot; \<br>      --query &quot;Status&quot; --output text)<br>    if [ &quot;$STATUS&quot; != &quot;Success&quot; ]; then<br>      aws ssm get-command-invocation \<br>        --command-id &quot;$COMMAND_ID&quot; \<br>        --instance-id &quot;$INSTANCE_ID&quot; \<br>        --query &quot;StandardErrorContent&quot; --output text<br>      exit 1<br>    fi</pre><p>The workflow does not just fire the command and assume success. It waits for the command to finish, checks the actual exit status, and surfaces the remote script’s output directly in the GitHub Actions log. A failed deploy shows up as a failed workflow run with the real error attached, not a green checkmark that quietly deployed nothing.</p><h3>The result</h3><p>A push to the containerize branch now triggers, without anyone touching a terminal on the EC2 instance: dependency install, test run, Docker build, an ECR push under an immutable, commit-scoped tag, and a Systems Manager deployment that pulls the new image, refreshes secrets from Secrets Manager, and replaces the running staging container. Production, still running through pm2 on port 3000, is never part of this flow.</p><h3>Coming next</h3><p>Demo 3 adds SonarQube as a quality gate ahead of the deploy step, so a build that fails static analysis never reaches staging in the first place. Later demos in this series move the deployment target from a single EC2 instance to an EKS cluster running Argo CD, at which point this same GitHub Actions workflow becomes the trigger for a GitOps sync rather than a direct SSM command.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=812db5b63ebe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containerizing a Live Production App Without Breaking It: Portfolio, Demo 1]]></title>
            <link>https://engr-syedusmanahmad.medium.com/containerizing-a-live-production-app-without-breaking-it-portfolio-demo-1-84f7bac57368?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/84f7bac57368</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[productionapp]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 07:01:03 GMT</pubDate>
            <atom:updated>2026-09-14T07:01:03.220Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1 of the CI/CD Pipeline Foundation series, a hands-on build of GitHub Actions, AWS, and GitOps around a real Next.js application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGmV85ggQx26cHvBz_hvTQ.png" /></figure><h3>Why a real app instead of a demo app</h3><p>Most CI/CD tutorials use a throwaway Node app with three files and a single API route. You follow along, it works, and then you try to apply the same steps to something real and half of it falls apart.</p><p>For this series, I decided to use PSX Portfolio, a Next.js 14 application I actually run and use, tracking my own equity trades on the Pakistan Stock Exchange. It has a Prisma schema with twenty plus models, NextAuth with Google OAuth, a Supabase client, the AWS Bedrock SDK for AI powered portfolio analysis, and a Content Security Policy that most tutorial apps never bother with.</p><p>It was already running in production on a single EC2 instance through pm2. That constraint shaped every decision in this demo: the live app could not go down, and the database (Postgres, running locally on the same EC2 instance) could not be touched.</p><h3>The plan: containerize without touching prod</h3><p>Rather than converting the existing pm2 process, I cloned the repository into a separate directory on the same EC2 instance, checked out a containerize branch, and built a parallel, independently running staging container on port 3001. The live app kept running on port 3000 through pm2, completely untouched, while I worked through every Docker issue in isolation.</p><p>This is a pattern worth stealing for any live migration: build the new thing next to the old thing, prove it works, then cut over deliberately instead of mid-flight.</p><h3>Step 1: Next.js standalone output</h3><p>The app has more than twenty dependencies, including the AWS SDK and Prisma. Copying the full node_modules folder into a Docker image would have produced something close to a gigabyte in size.</p><p>Next.js has a standalone output mode that traces the actual runtime dependency graph and copies only what is used:</p><pre>const nextConfig = {<br>  output: &#39;standalone&#39;,<br>  // ...rest of the existing config<br>};</pre><p>After npm run build, .next/standalone contains aserver.js, a trimmednode_modules, and a package.json. Static assets and the public folder are not included automatically, which becomes relevant later in the Dockerfile.</p><h3>Step 2: the Prisma binary target trap</h3><p>This is the single most common Docker plus Prisma bug, and it bit us immediately. Prisma generates a platform-specific query engine binary. Developing on a Mac and deploying to a Linux container means the engine built locally will not run in the container.</p><p>The fix is one line in schema.prisma:</p><pre>generator client {<br>  provider      = &quot;prisma-client-js&quot;<br>  binaryTargets = [&quot;native&quot;, &quot;debian-openssl-3.0.x&quot;]<br>}</pre><p>native keeps local development working on macOS, and debian-openssl-3.0.x produces the Linux binary the container will actually use. Skip this, and the app builds fine, then crashes at runtime with a query engine not found error, usually well after you have already moved on to the next problem.</p><h3>Step 3: a health endpoint, and a lesson about middleware</h3><p>Every container orchestration platform, from plain Docker healthchecks to an ALB target group to a Kubernetes liveness probe, needs a lightweight endpoint to poll. I added:</p><pre>// src/app/api/health/route.ts<br>import { NextResponse } from &#39;next/server&#39;;<br><br>export async function GET() {<br>  return NextResponse.json({ status: &#39;ok&#39; }, { status: 200 });<br>}</pre><p>Deliberately no database call here. This route answers one question only: is the process alive?</p><p>The first time I curled it, I got {&quot;error&quot;:&quot;Unauthorized&quot;}. Not a Docker problem, not a Prisma problem, a Next.js middleware problem. The app had a global middleware.ts intercepting every request before it reached a route handler, including this one, and its matcher pattern did not exclude /api/health. The fix was a one-line change to the matcher regex to explicitly carve out the health route.</p><p>It is a small bug, but it is the kind that costs real time if you do not know to check middleware first when a route behaves unexpectedly.</p><h3>Step 4: the Dockerfile</h3><pre>FROM node:20-slim AS deps<br>WORKDIR /app<br>COPY package.json package-lock.json ./<br>COPY prisma ./prisma<br>RUN npm ci<br><br>FROM node:20-slim AS builder<br>WORKDIR /app<br>COPY --from=deps /app/node_modules ./node_modules<br>COPY . .<br>RUN npm run build<br><br>FROM node:20-slim AS runner<br>WORKDIR /app<br>RUN apt-get update -y &amp;&amp; apt-get install -y openssl wget &amp;&amp; rm -rf /var/lib/apt/lists/*<br>ENV NODE_ENV=production<br>ENV HOSTNAME=&quot;0.0.0.0&quot;<br>RUN mkdir .next &amp;&amp; chown node:node .next<br>COPY --from=builder --chown=node:node /app/.next/standalone ./<br>COPY --from=builder --chown=node:node /app/.next/static ./.next/static<br>COPY --from=builder --chown=node:node /app/public ./public<br>USER node<br>EXPOSE 3000<br>HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \<br>  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1<br>CMD [&quot;node&quot;, &quot;server.js&quot;]</pre><p><strong>Two choices are worth explaining.</strong></p><p>node:20-slim (Debian) instead of Alpine. Alpine uses musl libc, and Prisma&#39;s musl binary target is a separate, less commonly used configuration than the Debian one we already set up. Debian slim avoided that extra complexity entirely.</p><p>Running as a non-root user (nodealready present in the base image). It costs nothing and closes off a class of container escape issues before they exist.</p><p>The healthcheck initially failed with an unhealthy status even though the app was clearly responding to curl from outside the container. The cause was almost funny: node:20-slim does not ship wgetso Docker&#39;s own healthcheck command was failing to run at all, unrelated to the application. Adding wget to the same apt-get install line as openssl fixed it.</p><h3>Step 5: .env never enters the image</h3><pre>node_modules<br>.next<br>.env<br>.env.*<br>.git<br>*.md<br>docs<br>scripts<br>logs</pre><p>This .dockerignore is a small file that prevents a large class of mistakes: local secrets accidentally baked into an image layer, where they would be recoverable by anyone who later pulls that image.</p><h3>Step 6: secrets from AWS Secrets Manager, not from files</h3><p>The app’s runtime configuration (database URL, NextAuth secret, Google OAuth credentials, Bedrock access) was already stored in AWS Secrets Manager under a single secret. The EC2 instance had an IAM role, so I extended that role with a tightly scoped, read-only policy limited to that one secret:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: &quot;secretsmanager:GetSecretValue&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:secretsmanager:eu-central-1:ACCOUNT_ID:secret:&lt;secret_name&gt;-*&quot;<br>    }<br>  ]<br>}</pre><p>A small shell script fetches the secret and writes it out as an env file at deploy time:</p><pre>aws secretsmanager get-secret-value \<br>  --secret-id &lt;secret_name&gt; \<br>  --region eu-central-1 \<br>  --query SecretString \<br>  --output text \<br>  | jq -r &#39;to_entries[] | &quot;\(.key)=\(.value)&quot;&#39; \<br>  | sed &#39;s/localhost:5432/host.docker.internal:5432/&#39; \<br>  &gt; .env.staging</pre><p>That sed substitution addresses one more networking detail: Postgres was running locally on the EC2 host, not in a container, and the connection string pointed at localhost. Inside a Docker container, localhost means the container itself, not the host machine. On Linux, reaching the host from inside a container requires an explicit --add-host=host.docker.internal:host-gateway flag on docker run, paired with rewriting the connection string to use that hostname.</p><h3>Running it</h3><pre>docker build -t psx-portfolio:staging .<br>docker run -d \<br>  --name psx-portfolio-staging \<br>  --env-file .env.staging \<br>  --add-host=host.docker.internal:host-gateway \<br>  -p 3001:3000 \<br>  --restart unless-stopped \<br>  psx-portfolio:staging</pre><p>docker ps eventually reported (healthy), and a request against a real data route, one that actually queries Postgres through Prisma, returned live KSE100 index data. The static health check proved the process was alive; a route with real Prisma traffic proved the database connection, the secrets, and the networking fix were all correct together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjQqoLLv4UmEPKWKsVjXXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*mIqveN4c6_J_kSjoW386fQ.png" /></figure><h3>What this demo actually proves</h3><p>Five real, unglamorous bugs got fixed in this one demo: a middleware matcher blocking a health route, a misplaced.dockerignore, a missing Prisma Linux binary target, container to host networking, and a missing wget binary causing a false negative healthcheck. None of them are exotic. All of them are the kind of thing that turns a &quot;simple&quot; containerization task into an afternoon of debugging if you do not know to expect them.</p><p>Production was never touched. The pm2 process kept serving real traffic on port 3000 the entire time, while a fully independent, properly containerized staging environment came up cleanly on port 3001.</p><h3>Coming next</h3><p>Demo 2 automates everything that was done by hand here: GitHub Actions builds the image on every push, pushes it to a private ECR repository using OIDC federation (no long-lived AWS keys stored in GitHub), and triggers a deployment on EC2 through AWS Systems Manager; no SSH required.</p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84f7bac57368" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking a Backend AI Service Off the Public Internet]]></title>
            <link>https://engr-syedusmanahmad.medium.com/taking-a-backend-ai-service-off-the-public-internet-d0a36ec04ff2?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a36ec04ff2</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[secure-ai-infrastructure]]></category>
            <category><![CDATA[bastion-host]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[secure-services]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 06:01:01 GMT</pubDate>
            <atom:updated>2026-09-04T06:01:01.254Z</atom:updated>
            <content:encoded><![CDATA[<p>Notes on private networking, service discovery, and giving engineers a safe way in</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOZ4PRMa0SGkM94ljRhZOw.png" /></figure><p>A few weeks ago we hit a problem that’s common enough to be almost boring, and dangerous enough that it’s worth writing down properly: an internal AI service, originally spun up for quick testing, had quietly become load-bearing. Another service depended on it in production. Developers needed to poke at it directly while debugging. And it was sitting on a public endpoint, reachable by anyone who found the URL.</p><p>Nothing had gone wrong yet. But “nothing has gone wrong yet” is not a security posture, and we didn’t want to find out the hard way what happens when it isn’t one.</p><p>This post walks through how we fixed it: how we took the service fully private, kept the service-to-service traffic that depended on it running without interruption, and gave our own developers a clean, auditable way to reach it when they genuinely needed to.</p><h4>The starting point</h4><p>The setup looked roughly like this:</p><ul><li><strong>Backend Core Service: </strong>a production service that calls the AI service as part of its normal request flow.</li><li><strong>AI Service:</strong> the service in question. Exposed on a public endpoint, reachable directly over the internet, with no network-level restriction on who could hit it.</li><li>Communication between the two internal services was handled through our container platform’s built-in service discovery, so <strong>Backend Core Service</strong> could resolve <strong>AI Service</strong> by name and talk to it directly inside the private network. That part was already working well.</li></ul><p>The public exposure of the AI service was the actual problem. It had never been the intended design, it was a leftover from an early testing phase that never got revisited once the service started handling real traffic. It’s an easy thing to miss, and an easy thing to normalize, because a URL that “has been fine so far” doesn’t feel urgent until it very suddenly is.</p><p>We set three requirements for the fix:</p><ol><li><strong>No public route to the AI service.</strong> Full stop.</li><li><strong>Zero disruption to Backend Core Service.</strong> The existing internal service-to-service traffic could not break, degrade, or even need a code change.</li><li><strong>Developers still need access, </strong>for debugging, for testing, for looking at interactive API docs, but that access has to be deliberate, authenticated, and limited to people who should have it.</li></ol><h4>Why we didn’t touch the internal service discovery</h4><p>It’s tempting, once you start pulling on this thread, to want to unify everything behind one access pattern. If we’re building a private entry point for developers, why not route the internal service-to-service traffic through it too?</p><p>We considered it and decided against it, for a few concrete reasons:</p><ul><li><strong>Latency.</strong> The existing internal service discovery lets one service talk to another directly inside the private network, with no extra hop. Routing that traffic through an additional network layer adds latency for no functional benefit.</li><li><strong>Resilience features we’d otherwise have to rebuild.</strong> Our service discovery layer already handles retries and endpoint health awareness for internal calls. Replacing it with a generic network layer means reimplementing behavior we already had for free.</li><li><strong>Blast radius.</strong> The internal service-to-service path was working and tested. Changing it to solve an unrelated access problem introduces risk into the one part of the system that wasn’t broken.</li></ul><p>So we left that path completely alone. The fix was scoped specifically to closing the public route and building a separate, narrow entry point for humans, not re-architecting how the two services talk to each other.</p><h4>Step 1: Confirm what’s actually reachable internally</h4><p>Before removing the public endpoint, we needed to be certain the AI service was already fully reachable from inside the private network by the service that depended on it. We verified this directly, rather than assuming it based on configuration:</p><pre># From inside the private network, resolve and call the service by its internal name<br>curl http://ai-service.internal:8000/health</pre><pre>{&quot;status&quot;: &quot;ok&quot;}</pre><p>Confirming this first meant that once we removed public access, we’d know immediately whether any resulting failure was a real regression or something we’d already ruled out.</p><h4>Step 2: Remove the public route</h4><p>This is the step that actually mattered. We removed the AI service’s public-facing listener entirely, no public IP, no public DNS record, no route in from the open internet. After this change, the only way to reach the service was from inside the private network.</p><p>We deliberately did <strong>not</strong> try to “restrict” the public endpoint with an allowlist or a bearer token bolted on top. A publicly routable endpoint with an access check in front of it is still a publicly routable endpoint, it’s one misconfiguration away from being open again. Removing the route entirely removes that entire category of risk.</p><p>At this point, <strong>Backend Core Service</strong> kept working exactly as before, since it had never talked to the AI service over the public route in the first place. This confirmed our assumption from Step 1 was correct.</p><h4>Step 3: Give the service a stable internal address</h4><p>Individual compute instances behind a service containers, tasks, whatever the unit is in your platform, come and go. They get replaced on deploys, on scaling events, on health-check failures. If developers tunnel in by targeting a specific instance’s address, that tunnel breaks the next time the service redeploys, and someone has to go rediscover the new address by hand.</p><p>To avoid that, we put a small <strong>internal load balancer</strong> in front of the AI service, scoped entirely to the private network, with no public exposure of its own. It does one job: give the service a single, stable internal address that doesn’t change no matter how many times the underlying instances are replaced.</p><pre># Example internal load balancer address (illustrative, not real)<br>internal-ai-service-lb.private.example.internal</pre><p>This address became the one thing developers would ever need to remember. Everything behind it can change new deploys, new instance counts, scaling up or down without a single developer-facing detail changing.</p><h4>Step 4: Build the developer access path: a bastion tunnel</h4><p>With the service now fully private, we needed a legitimate way in for engineers who needed it. We used a <strong>bastion host</strong>: a single, tightly controlled entry point inside the private network that developers reach over SSH, using individually issued key pairs, no shared credentials, no standing open ports on the AI service itself.</p><p>From there, developers open a local port-forward tunnel through the bastion straight to the internal load balancer:</p><pre>ssh -i my-key.pem \<br>  -N -L 8000:internal-ai-service-lb.private.example.internal:8000 \<br>  devuser@bastion.example.com</pre><p>What this does, in plain terms:</p><ul><li>-L 8000:...:8000 forwards port 8000 on the developer&#39;s own machine, through the encrypted SSH session, to port 8000 on the internal load balancer.</li><li>-N tells SSH to just hold the tunnel open, no remote shell is needed for this purpose.</li><li>Everything in between travels through the bastion, over an authenticated, encrypted connection. Nothing is exposed to the public internet at any point.</li></ul><p>Once the tunnel is running, the developer opens a browser on their own machine and points it at:</p><pre>http://localhost:8000/docs</pre><p>…and gets the AI service’s interactive API documentation, exactly as if they were sitting inside the private network because, for the duration of that tunnel, they effectively are.</p><h4>Step 5: Lock down who can even reach the door</h4><p>A bastion host is only as good as the fence around it. We made sure of a few things before calling this done:</p><ul><li><strong>Individual SSH keys per developer</strong>, not one shared key passed around in a chat thread. If someone leaves the team, you revoke one key, you don’t rotate a secret everyone had.</li><li><strong>Security group rules scoped tightly</strong>: the bastion only accepts inbound SSH from known IP ranges, and the AI service’s internal load balancer only accepts traffic from the bastion’s own security group, not from the wider private network at large.</li><li><strong>No inbound rule wider than it needs to be.</strong> It’s easy, mid-debugging, to open a security group rule “just to get this working” and forget to narrow it back down afterward. We treated every rule as something that had to justify its own existence.</li></ul><p>None of this is exotic. It’s the same pattern bastion host, private load balancer, scoped security groups, that’s been the standard way to reach private infrastructure for years. The value wasn’t in inventing something new; it was in actually doing it, instead of leaving a production-adjacent AI service sitting on the open internet because it was convenient during testing.</p><h4>The takeaway</h4><p>The uncomfortable part of this story isn’t the fix, the fix is genuinely simple, a few hours of careful work. The uncomfortable part is how ordinary the starting point was. A service that was never meant to be public-facing stayed public because removing that exposure wasn’t anyone’s job until it was.</p><p>If there’s one habit worth taking from this: whenever a “temporary” public endpoint outlives the reason it was made public, that’s the moment to go back and close it, not the moment something goes wrong because you didn’t.</p><p><em>If you’re setting up something similar, the shape is transferable regardless of what cloud platform or container orchestrator you’re on: keep internal service-to-service traffic on its native, already-optimized path; put a stable internal address in front of anything humans need to reach directly; and let a bastion host be the one deliberate door, instead of leaving several accidental ones open.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a36ec04ff2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes RBAC — Controlling Access Inside Your EKS Cluster]]></title>
            <link>https://engr-syedusmanahmad.medium.com/kubernetes-rbac-controlling-access-inside-your-eks-cluster-49fc5db55848?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/49fc5db55848</guid>
            <category><![CDATA[rbac]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 09:01:03 GMT</pubDate>
            <atom:updated>2026-08-15T09:01:03.417Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmz_lGD3-GbuatrPX7meVw.png" /></figure><blockquote>Who can do what — Role-Based Access Control from scratch</blockquote><p><em>This is Part 6 of the Amazon EKS on AWS series.</em> <em>Part 1 — EKS Cluster | Part 2 — Nginx + LB | Part 3 — HPA | Part 4 — Container Insights | Part 5 — Helm</em></p><blockquote><strong><em>Disclaimer:</em></strong><em> This is a learning demo intended for dev and pre-production environments. Always review and test RBAC policies carefully before applying to production clusters.</em></blockquote><p>A Kubernetes cluster with no access control is a security incident waiting to happen. Every developer has full access to everything. A CI/CD pipeline can delete production workloads. A misconfigured pod can read secrets it was never supposed to see.</p><p><strong>Kubernetes RBAC</strong> (Role-Based Access Control) solves this by giving every identity exactly the permissions it needs — and nothing more. This is the <strong>principle of least privilege</strong> applied to your cluster.</p><p>In this demo, I set up RBAC from scratch on Amazon EKS — service accounts, roles, bindings, and cluster-wide access control — then verified every permission with kubectl auth can-i.</p><h3>RBAC Core Concepts</h3><pre>ServiceAccount   → Identity (who is making the request?)<br>Role             → Permissions scoped to one namespace<br>ClusterRole      → Permissions scoped to the entire cluster<br>RoleBinding      → Links a ServiceAccount to a Role<br>ClusterRoleBinding → Links a ServiceAccount to a ClusterRole</pre><p>The key rule: <strong>if a permission is not explicitly granted, it is denied.</strong> Kubernetes RBAC is deny-by-default.</p><h3>Architecture — Three Personas</h3><pre>developer-sa  → dev-team namespace only<br>  ✅ pods, deployments, services (full)<br>  ✅ configmaps (read only)<br>  ❌ secrets (no access)<br>  ❌ prod-team namespace (no access)<br><br>readonly-sa   → dev-team namespace only<br>  ✅ everything: list/get/watch<br>  ❌ create, update, delete (no write)<br><br>monitoring-sa → entire cluster (ClusterRole)<br>  ✅ nodes, namespaces, pods across all namespaces<br>  ✅ metrics<br>  ❌ delete anything (read only cluster-wide)</pre><p>Step 1 — Namespaces and ServiceAccounts</p><pre>kubectl create namespace dev-team<br>kubectl create namespace prod-team<br><br>kubectl create serviceaccount developer-sa --namespace dev-team<br>kubectl create serviceaccount readonly-sa  --namespace dev-team<br>kubectl create serviceaccount monitoring-sa --namespace dev-team</pre><h3>Step 2 — Developer Role</h3><p>The developer role grants full pod and deployment access within dev-team — but explicitly excludes secrets:</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: developer-role<br>  namespace: dev-team<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;pods/log&quot;, &quot;pods/exec&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;delete&quot;]<br>  - apiGroups: [&quot;apps&quot;]<br>    resources: [&quot;deployments&quot;, &quot;replicasets&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;]<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;services&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;]<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>  # Secrets are NOT listed — denied by default</pre><p>Step 3 — Read-Only Role</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: readonly-role<br>  namespace: dev-team<br>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;, &quot;autoscaling&quot;]<br>    resources:<br>      - pods<br>      - deployments<br>      - replicasets<br>      - services<br>      - configmaps<br>      - horizontalpodautoscalers<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><p>Step 4 — RoleBindings</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: developer-binding<br>  namespace: dev-team<br>subjects:<br>  - kind: ServiceAccount<br>    name: developer-sa<br>    namespace: dev-team<br>roleRef:<br>  kind: Role<br>  apiGroup: rbac.authorization.k8s.io   # ← must be apiGroup, not apiRef<br>  name: developer-role</pre><p><strong>Real error encountered:</strong> The field is apiGroup — not apiRef. Using apiRef throws a strict decoding error at apply time. Always double-check the RoleBinding roleRef field name.</p><h3>Step 5 — Testing Permissions with kubectl auth can-i</h3><p>This command is one of the most useful debugging tools in Kubernetes — it lets you check what any service account can or cannot do without actually performing the action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1veQTm8hqa-Ril3kdVZ3fw.png" /></figure><pre># Developer — can list pods in dev-team?<br>kubectl auth can-i list pods \<br>  --namespace dev-team \<br>  --as system:serviceaccount:dev-team:developer-sa<br># yes ✅<br><br># Developer — can access secrets?<br>kubectl auth can-i get secrets \<br>  --namespace dev-team \<br>  --as system:serviceaccount:dev-team:developer-sa<br># no ✅ (correctly denied)<br><br># Developer — can access prod-team?<br>kubectl auth can-i list pods \<br>  --namespace prod-team \<br>  --as system:serviceaccount:dev-team:developer-sa<br># no ✅ (namespace isolation works)<br><br># ReadOnly — can delete pods?<br>kubectl auth can-i delete pods \<br>  --namespace dev-team \<br>  --as system:serviceaccount:dev-team:readonly-sa<br># no ✅</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8lqgV0BxrFLZLmIJVoKow.png" /></figure><h3>Step 6 — ClusterRole for Monitoring</h3><p>A ClusterRole provides read access across the entire cluster — useful for monitoring tools that need to see all namespaces:</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: cluster-viewer<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;nodes&quot;, &quot;namespaces&quot;, &quot;pods&quot;, &quot;services&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>  - apiGroups: [&quot;apps&quot;]<br>    resources: [&quot;deployments&quot;, &quot;daemonsets&quot;, &quot;replicasets&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>  - apiGroups: [&quot;metrics.k8s.io&quot;]<br>    resources: [&quot;nodes&quot;, &quot;pods&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;]</pre><pre># Monitoring — can list nodes cluster-wide?<br>kubectl auth can-i list nodes \<br>  --as system:serviceaccount:dev-team:monitoring-sa<br># yes ✅<br><br># Monitoring — can list pods in prod-team?<br>kubectl auth can-i list pods \<br>  --namespace prod-team \<br>  --as system:serviceaccount:dev-team:monitoring-sa<br># yes ✅ (cluster-wide access)<br><br># Monitoring — can delete nodes?<br>kubectl auth can-i delete nodes \<br>  --as system:serviceaccount:dev-team:monitoring-sa<br># no ✅ (read-only)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HHgJQEoMiL3oY3POwH9uUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSDOUyqFVEcrQP0hXQ-bxw.png" /></figure><p><strong>Role vs ClusterRole</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*txVv1C89J7Nwd0Po-KOx_Q.png" /></figure><p><strong>Permission Test Results Summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLs73ntUr0UruUq8qrZJ9g.png" /></figure><p>Every result is exactly what was intended — this is what proper RBAC looks like.</p><h3>Key Lessons</h3><p><strong>RBAC is deny-by-default.</strong> You do not need to explicitly deny permissions — anything not listed in a Role is automatically denied. This makes RBAC easier to reason about than allowlist/blocklist systems.</p><p><strong>kubectl auth can-i is your best friend.</strong> Before deploying anything to production, verify every service account&#39;s permissions using this command. It is fast, safe, and gives you exact yes/no answers without performing any actual operations.</p><p><strong>The field is </strong><strong>apiGroup — not </strong><strong>apiRef.</strong> This is a simple but easy-to-make typo in RoleBinding and ClusterRoleBinding manifests. Kubernetes will reject the resource with a strict decoding error. Always check roleRef.apiGroup.</p><p><strong>Never grant </strong><strong>cluster-admin to application service accounts.</strong> The built-in cluster-admin ClusterRole has unrestricted access to everything. It should only be used by human administrators during initial setup — never by pods, CI/CD pipelines, or operators.</p><p><strong>Namespace isolation is one of the most valuable RBAC patterns.</strong> By scoping a Role to a single namespace and verifying that the service account cannot access other namespaces, you create hard boundaries between teams and environments within the same cluster.</p><h3>Full Source Code</h3><p>All RBAC manifests from this demo are available on GitHub:</p><p>🔗 <a href="https://github.com/engr-usman/aws-aiops-demo-repo"><strong>github.com/engr-usman/aws-aiops-demo-repo</strong></a></p><p><em>This completes the Amazon EKS on AWS series — 6 demos from cluster setup to production-grade access control.</em></p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a>, if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49fc5db55848" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Package Management with Helm on Amazon EKS]]></title>
            <link>https://engr-syedusmanahmad.medium.com/kubernetes-package-management-with-helm-on-amazon-eks-d73ee4af9fae?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d73ee4af9fae</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 06:31:01 GMT</pubDate>
            <atom:updated>2026-08-13T06:31:01.349Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F78N1y35x3FcSKPGnEhnzA.png" /></figure><p>Deploy, upgrade, rollback — managing applications the right way</p><p><em>This is Part 5 of the Amazon EKS on AWS series.</em> <em>Part 1 — EKS Cluster | Part 2 — Nginx + LoadBalancer | Part 3 — HPA | Part 4 — Container Insights</em></p><blockquote><strong><em>Disclaimer:</em></strong><em> This is a learning demo intended for dev and pre-production environments. Always test Helm charts thoroughly before deploying to production workloads.</em></blockquote><p>Managing Kubernetes applications with raw YAML files works — until it does not. When you have five YAML files per application, multiple environments, and a team making changes, things break quickly. Values get hardcoded, files go out of sync, and rollbacks mean manually reverting each file one by one.</p><p><strong>Helm</strong> solves all of this. It is the package manager for Kubernetes — the same way npm manages Node packages or apt manages Ubuntu packages, Helm manages Kubernetes applications as versioned, configurable packages called charts.</p><p>In this demo, I created a custom Helm chart for Nginx, deployed it on EKS, upgraded it with new configuration, and rolled it back — all through clean, repeatable Helm commands.</p><h3>What Helm Provides</h3><pre>Without Helm:<br>  kubectl apply -f deployment.yaml<br>  kubectl apply -f service.yaml<br>  kubectl apply -f hpa.yaml<br>  kubectl apply -f configmap.yaml<br>  (every file managed separately)<br><br>With Helm:<br>  helm install nginx-release ./nginx-chart<br>  (everything deployed in one command)<br><br>Upgrade:<br>  helm upgrade nginx-release ./nginx-chart --set replicaCount=4<br><br>Rollback:<br>  helm rollback nginx-release 1</pre><h3>Helm Chart Structure</h3><p>Running helm create nginx-chart generates a standard chart skeleton:</p><pre>nginx-chart/<br>  ├── Chart.yaml          ← chart metadata (name, version, description)<br>  ├── values.yaml         ← default configuration values<br>  └── templates/<br>        ├── deployment.yaml   ← uses {{ .Values.x }} for dynamic config<br>        ├── service.yaml<br>        ├── hpa.yaml<br>        └── _helpers.tpl      ← reusable template functions</pre><p>The power is in the templates — they reference values from values.yaml using Go template syntax. This means the same chart can deploy to dev with 1 replica and production with 10, just by changing a single value.</p><h3>Chart.yaml</h3><pre>apiVersion: v2<br>name: nginx-chart<br>description: AIOps Demo Nginx Helm Chart<br>type: application<br>version: 1.0.0        # chart version — bump on chart changes<br>appVersion: &quot;1.25.0&quot;  # application version being packaged</pre><p>values.yaml — The Configuration Layer</p><pre>replicaCount: 2<br><br>image:<br>  repository: nginx<br>  tag: &quot;1.25-alpine&quot;<br>  pullPolicy: IfNotPresent<br><br>service:<br>  type: LoadBalancer<br>  port: 80<br>  annotations:<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;external&quot;<br>    service.beta.kubernetes.io/aws-load-balancer-scheme: &quot;internet-facing&quot;<br><br>resources:<br>  requests:<br>    cpu: &quot;100m&quot;<br>    memory: &quot;128Mi&quot;<br>  limits:<br>    cpu: &quot;200m&quot;<br>    memory: &quot;256Mi&quot;<br><br>autoscaling:<br>  enabled: true<br>  minReplicas: 2<br>  maxReplicas: 8<br>  targetCPUUtilizationPercentage: 30</pre><p>Every value here can be overridden at install or upgrade time — no YAML editing required.</p><h3>Template Example — Deployment</h3><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: {{ include &quot;nginx-chart.fullname&quot; . }}<br>  namespace: {{ .Values.namespace }}<br>spec:<br>  {{- if not .Values.autoscaling.enabled }}<br>  replicas: {{ .Values.replicaCount }}<br>  {{- end }}<br>  template:<br>    spec:<br>      containers:<br>        - name: {{ .Chart.Name }}<br>          image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag }}&quot;<br>          resources:<br>            {{- toYaml .Values.resources | nindent 12 }}</pre><p>{{ .Values.image.tag }} pulls from values.yaml. Override it at deploy time and the template renders with the new value — no file editing needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7oSQYEsrP_EJxoC5Q05w-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYM6sLiZXlySQoIy757fIg.png" /></figure><h3>Step 1 — Lint Before Deploy</h3><pre>helm lint nginx-chart/<br># ==&gt; Linting nginx-chart/<br># 1 chart(s) linted, 0 chart(s) failed ✅</pre><p>Always lint before deploying — catches template errors before they reach the cluster.</p><p><strong>Common lint issue encountered:</strong> helm create generates templates for serviceaccount.yaml, ingress.yaml, and httproute.yaml by default, but these reference values.serviceAccount, values.ingress, and values.httpRoute which do not exist in the custom values.yaml. This causes a nil pointer error during lint.</p><p>Fix: delete the unused templates:</p><pre>rm nginx-chart/templates/serviceaccount.yaml<br>rm nginx-chart/templates/ingress.yaml<br>rm nginx-chart/templates/httproute.yaml</pre><p>Step 2 — Install</p><pre>kubectl create namespace helm-demo<br><br>helm install nginx-release nginx-chart/ \<br>  --namespace helm-demo<br><br># Verify<br>helm list -n helm-demo<br># NAME           NAMESPACE  REVISION  STATUS    CHART<br># nginx-release  helm-demo  1         deployed  nginx-chart-1.0.0 ✅<br><br>kubectl get all -n helm-demo<br># Pods, Service, HPA all running ✅</pre><p>Step 3 — Upgrade (Override Values)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6rbGmu-5mzfr4LwTSxAqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6rbGmu-5mzfr4LwTSxAqQ.png" /></figure><pre># Bump replicas to 4 — no YAML editing<br>helm upgrade nginx-release nginx-chart/ \<br>  --namespace helm-demo \<br>  --set replicaCount=4<br><br># Upgrade to newer Nginx image<br>helm upgrade nginx-release nginx-chart/ \<br>  --namespace helm-demo \<br>  --set image.tag=1.26-alpine</pre><p>Each upgrade creates a new revision tracked by Helm automatically.</p><h3>Step 4 — History</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKa_ZgeUs9P0zK9oek8AYQ.png" /></figure><pre>helm history nginx-release -n helm-demo<br><br># REVISION  STATUS      CHART              DESCRIPTION<br># 1         superseded  nginx-chart-1.0.0  Install complete<br># 2         superseded  nginx-chart-1.0.0  Upgrade complete<br># 3         deployed    nginx-chart-1.0.0  Upgrade complete</pre><p>Every change is recorded. This is what makes rollbacks reliable.</p><h3>Step 5 — Rollback</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TvmW-SjGY5KTtrPyMSsoKg.png" /></figure><pre># Roll back to revision 1<br>helm rollback nginx-release 1 -n helm-demo<br><br># Verify<br>helm history nginx-release -n helm-demo<br># REVISION  STATUS      DESCRIPTION<br># 1         superseded  Install complete<br># 2         superseded  Upgrade complete<br># 3         superseded  Upgrade complete<br># 4         deployed    Rollback to 1 ✅</pre><p>One command. Previous state restored. No manual YAML reverting needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIHqZH_7EaCiI0UpuqYKcg.png" /></figure><h3>Step 6 — Uninstall</h3><pre>helm uninstall nginx-release -n helm-demo<br># All Kubernetes resources removed ✅<br># Helm release record removed ✅</pre><p>Helm vs Raw kubectl</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCSzpJZ32ZarZ0NvssOveg.png" /></figure><h3>Key Lessons</h3><p><strong>Always lint before install.</strong> helm lint catches template errors locally before they reach your cluster. A failed lint is far less painful than a failed production deployment.</p><p><strong>helm create generates more than you need.</strong> The default skeleton includes ServiceAccount, Ingress, and HTTPRoute templates that reference values not present in the default values.yaml. Delete unused templates or add the corresponding values — otherwise lint will fail with nil pointer errors.</p><p><strong>Helm tracks every change as a revision.</strong> Every install and upgrade is versioned. This is what makes helm rollback reliable — you are not guessing what the previous state was, Helm knows exactly.</p><p><strong>--set overrides are powerful but </strong><strong>values.yaml files are better for teams.</strong> For quick demos, --set image.tag=1.26 works well. In production, maintain separate values-dev.yaml and values-prod.yaml files and deploy with helm install -f values-prod.yaml — this keeps configuration in version control.</p><h3>Full Source Code</h3><p>All Helm chart files are available on GitHub:</p><p>🔗 <a href="https://github.com/engr-usman/aws-aiops-demo-repo"><strong>github.com/engr-usman/aws-aiops-demo-repo</strong></a></p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a>, if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d73ee4af9fae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Kubernetes with CloudWatch Container Insights on Amazon EKS]]></title>
            <link>https://engr-syedusmanahmad.medium.com/monitoring-kubernetes-with-cloudwatch-container-insights-on-amazon-eks-44fd649ba79e?source=rss-543e77313b8d------2</link>
            <guid isPermaLink="false">https://medium.com/p/44fd649ba79e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Syedusmanahmad]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 06:31:01 GMT</pubDate>
            <atom:updated>2026-08-11T06:31:01.934Z</atom:updated>
            <content:encoded><![CDATA[<p>Pod-level metrics, container logs, and a live EKS operations dashboard</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qO7c-CF3DeLIcDheYtK36A.png" /></figure><p><em>This is Part 4 of the Amazon EKS on AWS series.</em> <a href="https://engr-syedusmanahmad.medium.com/12f7b612beae?sharedUserId=engr-syedusmanahmad"><em>Part 1 </em></a><em>— EKS Cluster Setup | </em><a href="https://engr-syedusmanahmad.medium.com/59eacf9e9e59?sharedUserId=engr-syedusmanahmad"><em>Part 2</em></a><em> — Nginx Deployment | </em><a href="https://engr-syedusmanahmad.medium.com/8bb559076eb4?sharedUserId=engr-syedusmanahmad"><em>Part 3</em></a><em> — Horizontal Pod Autoscaler</em></p><blockquote><strong><em>Disclaimer:</em></strong><em> This is a learning demo intended for dev and pre-production environments. Always validate monitoring configurations before applying to production workloads.</em></blockquote><p>Running a Kubernetes cluster without proper observability is like driving with your eyes closed. You know pods are running, but you have no idea which ones are consuming the most memory, which nodes are under pressure, or where that intermittent crash is coming from.</p><p><strong>CloudWatch Container Insights</strong> solves this for Amazon EKS — it gives you pod-level metrics, container logs, and a live performance dashboard, all integrated into the same CloudWatch you already use for your other AWS resources.</p><p>In this demo, I set up Container Insights on the EKS cluster from the previous demos and explored the data it collects.</p><h3>What Container Insights Provides</h3><pre>Standard CloudWatch:<br>  EC2 metrics → CPU, Memory at instance level<br><br>Container Insights adds:<br>  Cluster level  → Node count, overall CPU/Memory<br>  Node level     → Per-node resource utilization<br>  Pod level      → CPU and Memory per pod<br>  Container      → Individual container metrics<br>  Logs           → Pod application logs<br>                   Node host logs<br>                   Kubernetes dataplane logs</pre><h3>Architecture</h3><pre>EKS Cluster (aiops-eks)<br>        │<br>        ├── CloudWatch Agent (DaemonSet)<br>        │     Collects: CPU, Memory, Network, Disk metrics<br>        │     Destination: /aws/containerinsights/aiops-eks/performance<br>        │<br>        └── FluentBit (DaemonSet)<br>              Collects: Application logs, Host logs, Dataplane logs<br>              Destinations:<br>                /aws/containerinsights/aiops-eks/application<br>                /aws/containerinsights/aiops-eks/host<br>                /aws/containerinsights/aiops-eks/dataplane</pre><p>Two DaemonSets run on every node — CloudWatch Agent for metrics, FluentBit for logs. A DaemonSet ensures exactly one pod per node regardless of how many nodes exist.</p><h3>Step 1 — IAM Permissions</h3><p>The node group IAM role needs permission to write metrics and logs to CloudWatch:</p><pre>IAM → Roles → (node group role) → Attach policy:<br>  CloudWatchAgentServerPolicy</pre><p>Without this, the CloudWatch Agent pods will run but silently fail to publish any data.</p><h3>Step 2 — CloudWatch Agent (Metrics)</h3><p>The CloudWatch Agent was deployed as a DaemonSet using AWS’s official manifest. A ConfigMap defines what to collect:</p><pre>{<br>  &quot;logs&quot;: {<br>    &quot;metrics_collected&quot;: {<br>      &quot;kubernetes&quot;: {<br>        &quot;cluster_name&quot;: &quot;aiops-eks&quot;,<br>        &quot;metrics_collection_interval&quot;: 60<br>      }<br>    },<br>    &quot;force_flush_interval&quot;: 5<br>  }<br>}</pre><pre>kubectl get daemonset -n amazon-cloudwatch<br># NAME               DESIRED   CURRENT   READY<br># cloudwatch-agent   1         1         1     ✅</pre><h3>Step 3 — FluentBit (Logs)</h3><p>FluentBit requires a ConfigMap named fluent-bit-cluster-info with cluster details before it will start. This is a common setup mistake — deploying the DaemonSet before creating the ConfigMap causes CrashLoopBackOff:</p><pre>Error: configmap &quot;fluent-bit-cluster-info&quot; not found</pre><p>The fix is straightforward — create the ConfigMap first:</p><pre>kubectl create configmap fluent-bit-cluster-info \<br>  --from-literal=cluster.name=aiops-eks \<br>  --from-literal=http.server=On \<br>  --from-literal=http.port=2020 \<br>  --from-literal=read.head=Off \<br>  --from-literal=read.tail=On \<br>  --from-literal=logs.region=eu-central-1 \<br>  -n amazon-cloudwatch</pre><p>After the ConfigMap was created and the DaemonSet restarted, both pods came up healthy:</p><pre>kubectl get pods -n amazon-cloudwatch<br># NAME                     READY   STATUS<br># cloudwatch-agent-xxxxx   1/1     Running  ✅<br># fluent-bit-xxxxx         1/1     Running  ✅</pre><h3>Step 4 — Container Insights Dashboard</h3><p>Once both DaemonSets were running, metrics started flowing into CloudWatch within 2–3 minutes.</p><p><strong>CloudWatch → Container Insights → Performance monitoring → aiops-eks</strong></p><p>The dashboard provides multiple views:</p><p><strong>Cluster view:</strong></p><pre>Nodes status        : 2 Ready<br>Node CPU util       : 4.84%<br>Node Memory util    : 27.8%<br>Container restarts  : 0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZe0iESs3BStXe3o5e3s7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wTQwd9wgsv_D0tpNoP6dg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyKLfet-6a5XyEktQkhIYQ.png" /></figure><p><strong>Pod CPU and Memory over pod limit:</strong> Both charts show live utilization relative to the resource limits defined in the pod spec — this is what makes Container Insights more useful than raw CloudWatch metrics. You see utilization in context of what the pod is actually allowed to use.</p><p><strong>Additional views available:</strong></p><ul><li>Namespaces — resource breakdown per namespace</li><li>Nodes — per-node CPU and memory</li><li>Services — network metrics per service</li><li>Workloads — deployment-level view</li><li>Pods — individual pod metrics</li><li>Containers — container-level granularity</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzOd1GxCk0tLwGaRc50lhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1y58RDPjPlpAUNicOIFgJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sk7yFjVeLsbKNQnUzV-KBw.png" /></figure><h3>Step 5 — Log Groups Created</h3><p>After FluentBit started, four log groups appeared automatically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9G0lsMYcLbNsAzvUtgCD6A.png" /></figure><pre>/aws/containerinsights/aiops-eks/application  ← Nginx and app container logs<br>/aws/containerinsights/aiops-eks/host          ← Node OS logs<br>/aws/containerinsights/aiops-eks/performance   ← CloudWatch Agent metrics<br>/aws/containerinsights/aiops-eks/dataplane     ← Kubernetes control plane logs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jBC3J-wWdGJUp5rc7g3DYA.png" /></figure><p>Step 6 — Querying Logs with CloudWatch Logs Insights</p><pre>CloudWatch → Logs Insights → /aws/containerinsights/aiops-eks/application<br><br>Query:<br>fields @timestamp, kubernetes.pod_name, log<br>| filter kubernetes.namespace_name = &quot;aiops-demo&quot;<br>| sort @timestamp desc<br>| limit 20</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-i0TQl8dT76U-z6fD6tW9g.png" /></figure><p>This returns the last 20 log lines from every container running in the aiops-demo namespace — no SSH, no kubectl exec required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dyuzwtd6G29CRnMeO74qPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*99Ppq1ZUDw-J3AIm6-kIAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P7QlPxLzWrQSTWIq7_FetQ.png" /></figure><h3>Key Lessons</h3><p><strong>The FluentBit ConfigMap must exist before the DaemonSet is deployed.</strong> If you deploy the DaemonSet first, it enters CrashLoopBackOff and keeps pulling the image every 30 seconds. Create the ConfigMap first, then apply the DaemonSet — or restart it after the ConfigMap is created.</p><p><strong>IAM permissions are always the first thing to verify.</strong> The CloudWatch Agent ran without errors but published no data until CloudWatchAgentServerPolicy was attached to the node role. If your metrics are missing, check IAM before debugging the agent configuration.</p><p><strong>Container Insights gives you context that raw metrics cannot.</strong> Knowing a pod uses 27.8% memory is useful — knowing it uses 27.8% of its 256Mi limit is actionable. Container Insights shows utilization relative to resource limits, which is what actually matters for capacity planning.</p><p><strong>DaemonSets automatically scale with your cluster.</strong> When a new node is added by Cluster Autoscaler, the CloudWatch Agent and FluentBit pods are automatically scheduled on it. You do not need to do anything — monitoring scales with your cluster.</p><h3>Container Insights vs Standard CloudWatch</h3><p><strong>Capability</strong> Standard CloudWatch <strong>Container Insights</strong></p><p>EC2 instance metrics ✅ ✅</p><p>Pod-level metrics ❌ ✅</p><p>Container-level metrics ❌ ✅</p><p>Application logs❌ ✅ (FluentBit)</p><p>K8s-aware dashboard ❌ ✅</p><p>Auto-scales with cluster N/A ✅ (DaemonSet)</p><h3>Full Source Code</h3><p>All manifests and setup scripts used in this demo are available on GitHub:</p><p>🔗 <a href="https://github.com/engr-usman/aws-aiops-demo-repo"><strong>github.com/engr-usman/aws-aiops-demo-repo</strong></a></p><p>I hope you enjoyed reading this article. Please feel free to contact me, <a href="https://medium.com/u/543e77313b8d">Syedusmanahmad</a> if you have any questions.</p><p>Please feel free to write to @<strong><em>engr.syedusmanahmad@gmail.com</em></strong> | <strong>LinkedIn </strong><a href="https://www.linkedin.com/in/engrusman-ahmad/">https://www.linkedin.com/in/engrusman-ahmad</a> for any queries on AWS/DevOps &amp; stay tuned for the next write-up.</p><p>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44fd649ba79e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>