<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by The kube guy on Medium]]></title>
        <description><![CDATA[Stories by The kube guy on Medium]]></description>
        <link>https://medium.com/@thekubeguy?source=rss-54b070394829------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*H4apdPEHhS1tPqvuxXz4Bg.png</url>
            <title>Stories by The kube guy on Medium</title>
            <link>https://medium.com/@thekubeguy?source=rss-54b070394829------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:49:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thekubeguy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Volume Group Snapshots Hit GA — Crash-Consistent Backups Across All Your Volumes]]></title>
            <link>https://medium.com/@thekubeguy/volume-group-snapshots-hit-ga-crash-consistent-backups-across-all-your-volumes-852fedd36f24?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/852fedd36f24</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Thu, 02 Jul 2026 05:21:00 GMT</pubDate>
            <atom:updated>2026-07-02T05:21:00.842Z</atom:updated>
            <content:encoded><![CDATA[<p><em>You’ve been snapshotting your database volumes one at a time and praying the data lines up. Kubernetes v1.36 finally lets you snapshot them all at once — and the storage industry has been waiting three years for this.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*p6BQzKmTTWvug8Uki77p5g.png" /><figcaption>Image by Author</figcaption></figure><p>Let me describe a disaster scenario that’s probably already happened to you.</p><p>You’re running a distributed database on Kubernetes — maybe PostgreSQL with a WAL volume and a data volume. Or Cassandra with three PVCs across three pods. Or an application that splits its state between a data directory and a transaction log on separate volumes. Backup time comes around, so you take a snapshot of volume A. Three seconds later, you snapshot volume B. And then volume C.</p><p>Everything looks fine. Your snapshots completed. Your monitoring is green. You sleep well.</p><p>Six months later, disaster strikes and you need to restore. You bring up your database from those three snapshots and… it won’t start. Or worse, it starts but the data is corrupted. Because in those three seconds between snapshot A and snapshot C, your application wrote a transaction that committed to one volume but not the others. Your “backup” captured an impossible state — one that never existed in reality. It’s like taking a group photo where everyone blinked at a different time and Photoshopping them together. Technically a photo. Practically useless.</p><p>This is the crash consistency problem, and it has haunted stateful workloads on Kubernetes since day one. Until now.</p><p>Kubernetes v1.36, released April 22, 2026, graduates <strong>Volume Group Snapshots</strong> to General Availability. This feature lets you snapshot multiple volumes together as a single atomic operation — every volume captured at exactly the same point in time, guaranteed by your storage backend. No more sequential snapshots. No more prayer-driven backup strategies. Just one API call, one consistent recovery point, across all your volumes.</p><p>Let’s break this down.</p><h4>Why Individual Snapshots Aren’t Enough?</h4><p>Kubernetes has supported VolumeSnapshots since v1.20 — the ability to take a point-in-time copy of a PersistentVolume through the CSI interface. And for single-volume workloads, they work great. Snapshot your PostgreSQL data directory, restore it later, and the database uses its transaction log to recover to a consistent state.</p><p>But the moment your application spreads state across multiple volumes, individual snapshots become a liability.</p><p>Think of it like a bank transaction. You’re transferring $1,000 from Account A to Account B. The operation has two steps: debit A, credit B. If you take a “snapshot” of Account A after the debit but snapshot Account B <em>before</em> the credit, your backup shows $1,000 missing from both accounts. The money vanished. In reality, the transfer was in-flight — but your inconsistent snapshots captured a state that never actually existed.</p><p>Databases deal with exactly this problem across volumes. A write-ahead log (WAL) on volume A records the intent. The actual data files on volume B get updated afterward. If you snapshot volume B mid-write but capture volume A a moment later, the WAL says “transaction completed” but the data files don’t reflect it. Or vice versa — the data has been modified but the WAL doesn’t know about it. Either way, your database’s recovery logic can’t make sense of it.</p><p>The same problem hits distributed systems. Cassandra, MongoDB with replica sets, etcd clusters, Kafka with separate log and data volumes — any workload where state spans multiple disks is vulnerable to this consistency gap.</p><p>Before Volume Group Snapshots, teams worked around this with application-level quiescing. Freeze the database, flush buffers to disk, take all your snapshots as fast as possible, then unfreeze. It works, but it requires intimate knowledge of each application’s freeze/thaw mechanism, introduces downtime during the snapshot window, and still doesn’t guarantee true atomicity because you’re still taking snapshots sequentially — just with the application paused.</p><h4>What Volume Group Snapshots Actually Do?</h4><p>Volume Group Snapshots solve this by pushing the atomicity requirement down to the storage backend — where it belongs.</p><p>When you create a VolumeGroupSnapshot, Kubernetes doesn’t just take individual snapshots in quick succession. It sends a single CreateVolumeGroupSnapshot RPC to your CSI driver, which tells the storage system: &quot;Snapshot these five volumes together, right now, at the same instant.&quot; The storage backend — whether that&#39;s a SAN, a cloud disk service, or a software-defined storage system — uses its native capabilities to freeze all the volumes at the same I/O point and capture them atomically.</p><p>The result is a set of snapshots that represent a single, crash-consistent recovery point across all your volumes. “Crash-consistent” means the state is equivalent to what would exist if someone had yanked the power cord at that exact instant — all completed writes are captured, no partial writes are included, and the application can recover using its normal crash-recovery mechanisms (journal replay, WAL recovery, fsck, whatever).</p><p>It’s the difference between taking individual photos of each musician in an orchestra and recording a single video frame of the entire performance. Both capture the musicians, but only the video frame shows them all at the same moment in time.</p><h4>The Three API Resources</h4><p>Volume Group Snapshots introduce three new Custom Resource Definitions that follow the same pattern as existing Kubernetes snapshot APIs:</p><p><strong>VolumeGroupSnapshotClass</strong> — The cluster-scoped template. Defines which CSI driver handles the group snapshot and any driver-specific parameters. Think of it like a StorageClass, but for group snapshots.</p><pre>apiVersion: groupsnapshot.storage.k8s.io/v1<br>kind: VolumeGroupSnapshotClass<br>metadata:<br>  name: csi-groupsnap-class<br>driver: pd.csi.storage.gke.io<br>deletionPolicy: Delete</pre><p><strong>VolumeGroupSnapshot</strong> — The user-facing, namespaced resource. This is what you create to trigger a group snapshot. It uses a label selector to identify which PVCs should be captured together.</p><pre>apiVersion: groupsnapshot.storage.k8s.io/v1<br>kind: VolumeGroupSnapshot<br>metadata:<br>  name: my-database-snapshot<br>  namespace: production<br>spec:<br>  volumeGroupSnapshotClassName: csi-groupsnap-class<br>  source:<br>    selector:<br>      matchLabels:<br>        app: my-database</pre><p><strong>VolumeGroupSnapshotContent</strong> — The cluster-scoped resource representing the actual snapshot data on the storage backend. Created automatically when a VolumeGroupSnapshot is provisioned. You generally don’t interact with this directly, just like most users never touch PersistentVolumes directly.</p><p>That’s it. Three resources, and you’ll mostly interact with just two of them — the class (set up once by your platform team) and the snapshot itself (triggered by your backup automation).</p><h4>A Practical Example — Backing Up a Database</h4><p>Let’s walk through a real scenario. You’re running a PostgreSQL instance with two PVCs — one for the data directory and one for the WAL:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: postgres-data<br>  namespace: production<br>  labels:<br>    app: postgres<br>    component: database<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: fast-ssd<br>  resources:<br>    requests:<br>      storage: 100Gi<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: postgres-wal<br>  namespace: production<br>  labels:<br>    app: postgres<br>    component: database<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: fast-ssd<br>  resources:<br>    requests:<br>      storage: 20Gi</pre><p>Notice the shared label app: postgres, component: database on both PVCs. That label is the glue. When you create a VolumeGroupSnapshot with a selector matching those labels, both volumes get captured atomically:</p><pre>apiVersion: groupsnapshot.storage.k8s.io/v1<br>kind: VolumeGroupSnapshot<br>metadata:<br>  name: postgres-backup-2026-05-12<br>  namespace: production<br>spec:<br>  volumeGroupSnapshotClassName: csi-groupsnap-class<br>  source:<br>    selector:<br>      matchLabels:<br>        app: postgres<br>        component: database</pre><p>When this resource is created, the snapshot controller finds all PVCs in the production namespace matching those labels, sends a single CreateVolumeGroupSnapshot RPC to the CSI driver, and the storage backend captures both volumes at the same instant. The result is a VolumeGroupSnapshotContent on the cluster side, plus individual VolumeSnapshot and VolumeSnapshotContent objects for each volume in the group — all linked back to the parent group snapshot.</p><h3>Restoring — It’s Just Individual Snapshots</h3><p>Here’s an elegant design decision: restoration doesn’t use any new APIs. When a VolumeGroupSnapshot is created, Kubernetes automatically creates individual VolumeSnapshot objects for each volume in the group. To restore, you simply create new PVCs that reference those individual snapshots as their data source:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: postgres-data-restored<br>  namespace: production<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: fast-ssd<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  dataSource:<br>    name: snapshot-abc123-postgres-data<br>    kind: VolumeSnapshot<br>    apiGroup: snapshot.storage.k8s.io<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: postgres-wal-restored<br>  namespace: production<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: fast-ssd<br>  resources:<br>    requests:<br>      storage: 20Gi<br>  dataSource:<br>    name: snapshot-abc123-postgres-wal<br>    kind: VolumeSnapshot<br>    apiGroup: snapshot.storage.k8s.io</pre><p>Because both snapshots were taken at the same point in time, the restored volumes represent a consistent state. Your PostgreSQL instance can start up, replay whatever was in the WAL, and reach a clean state — exactly like recovering from a crash. No data corruption. No phantom transactions. No missing money.</p><h3>The Old Way vs. The New Way</h3><p>Let’s compare. Here’s what a “group backup” looked like before this feature, using a CronJob with application quiescing:</p><pre># THE OLD WAY — sequential snapshots with manual quiescing<br>apiVersion: batch/v1<br>kind: CronJob<br>metadata:<br>  name: postgres-backup<br>spec:<br>  schedule: &quot;0 2 * * *&quot;<br>  jobTemplate:<br>    spec:<br>      template:<br>        spec:<br>          containers:<br>          - name: backup<br>            image: backup-tools:latest<br>            command:<br>            - /bin/sh<br>            - -c<br>            - |<br>              # Step 1: Freeze the database<br>              psql -c &quot;SELECT pg_start_backup(&#39;snapshot&#39;);&quot;<br>              # Step 2: Take snapshot of data volume<br>              kubectl apply -f snapshot-data.yaml<br>              # Step 3: Wait for it to be ready<br>              kubectl wait --for=jsonpath=&#39;{.status.readyToUse}&#39;=true \<br>                volumesnapshot/data-snap --timeout=120s<br>              # Step 4: Take snapshot of WAL volume<br>              kubectl apply -f snapshot-wal.yaml<br>              # Step 5: Wait again<br>              kubectl wait --for=jsonpath=&#39;{.status.readyToUse}&#39;=true \<br>                volumesnapshot/wal-snap --timeout=120s<br>              # Step 6: Unfreeze the database<br>              psql -c &quot;SELECT pg_stop_backup();&quot;</pre><p>Count the failure modes. The freeze could fail. The first snapshot could timeout. The second snapshot could fail after the first succeeded, leaving you with a partial backup. The unfreeze could fail, leaving your database frozen. The CronJob needs RBAC to create VolumeSnapshots AND database credentials to run psql. It’s a fragile, credential-heavy, multi-step process held together with shell script and hope.</p><p>Now the new way:</p><pre># THE NEW WAY — one resource, atomic guarantee<br>apiVersion: groupsnapshot.storage.k8s.io/v1<br>kind: VolumeGroupSnapshot<br>metadata:<br>  name: postgres-backup-2026-05-12<br>  namespace: production<br>spec:<br>  volumeGroupSnapshotClassName: csi-groupsnap-class<br>  source:<br>    selector:<br>      matchLabels:<br>        app: postgres<br>        component: database</pre><p>Eight lines of YAML. No database credentials. No freeze/thaw choreography. No sequential waiting. No shell scripts. The storage backend handles the atomicity, the CSI driver orchestrates it, and Kubernetes gives you a clean API for triggering and managing it. Wrap that in a CronJob and you’ve got automated, crash-consistent, multi-volume backups in under 20 lines of total YAML.</p><h3>Real-World Use Cases</h3><p><strong>Databases with separate data and log volumes.</strong> PostgreSQL, MySQL, MongoDB — most production database deployments split data files and transaction logs onto separate volumes for performance. Group snapshots ensure you never capture one without the other.</p><p><strong>Distributed stateful applications.</strong> Cassandra, Kafka, etcd clusters — workloads where data is sharded across multiple volumes on the same or different pods. Label all the PVCs with a common selector and snapshot the entire dataset atomically.</p><p><strong>Multi-tier applications.</strong> An application that stores user data in one PVC, session state in another, and uploaded files in a third. A group snapshot gives you a consistent recovery point across all three, so you can restore the entire application state to a single moment in time.</p><p><strong>Compliance and audit requirements.</strong> Many regulatory frameworks require point-in-time recovery capabilities for data stores. Group snapshots provide auditable, atomic recovery points that satisfy these requirements without custom tooling.</p><p><strong>Disaster recovery automation.</strong> Pair group snapshots with cross-region replication (if your CSI driver supports it) for automated DR that captures your entire application’s persistent state consistently.</p><h3>Things to Keep in Mind</h3><p><strong>CSI drivers only.</strong> Volume Group Snapshots work exclusively with CSI volume drivers. In-tree volume plugins don’t support this feature. Your CSI driver must implement the CreateVolumeGroupSnapshot and DeleteVolumeGroupSnapshot RPCs, and the underlying storage must support atomic multi-volume snapshots.</p><p><strong>Crash-consistent, not application-consistent.</strong> The snapshots capture disk state, not application state. Your database won’t be gracefully quiesced — the snapshot is equivalent to a power failure at that instant. This is fine for most databases (they’re designed to recover from crashes), but if your application requires a clean shutdown to produce consistent data, you’ll still need to handle quiescing separately.</p><p><strong>Same CSI driver required.</strong> All PVCs in a group snapshot must use the same CSI driver. You can’t group a GCE PD volume with an EBS volume into a single group snapshot. This makes sense — the atomicity guarantee comes from a single storage backend.</p><p><strong>Extension APIs via CRDs.</strong> The group snapshot resources are delivered as CRDs, not built into the core Kubernetes API. You’ll need to install the CRDs and deploy the external-snapshotter controller and sidecar to use this feature.</p><p><strong>Label selector semantics.</strong> The PVC grouping is based on label selectors, which means you need disciplined labeling. If you add the app: postgres label to an unrelated PVC, it&#39;ll get pulled into your group snapshot. Be precise with your labels.</p><p><strong>No direct restore from group snapshot.</strong> You restore from the individual VolumeSnapshot objects that are created as part of the group, not from the VolumeGroupSnapshot directly. This keeps the restore path simple and consistent with existing snapshot workflows.</p><h4>Wrapping Up</h4><p>Volume Group Snapshots graduating to GA in Kubernetes v1.36 closes one of the last major gaps in Kubernetes storage management. For the first time, you can take atomic, crash-consistent snapshots across multiple volumes using a native Kubernetes API — no shell scripts, no application-level freeze/thaw, no prayer.</p><p>Here’s what to take away:</p><ul><li>Volume Group Snapshots let you capture multiple PVCs atomically using a label selector — one API call, one consistent recovery point.</li><li>The API introduces three CRDs: VolumeGroupSnapshotClass, VolumeGroupSnapshot, and VolumeGroupSnapshotContent, all now at v1.</li><li>Restoration uses existing VolumeSnapshot objects — no new restore APIs to learn.</li><li>Crash consistency is guaranteed by the storage backend, not by Kubernetes — your CSI driver and storage system do the heavy lifting.</li><li>All PVCs in a group must use the same CSI driver.</li><li>The feature went through alpha, two betas, and GA over three years — the API is battle-tested.</li><li>Available right now in Kubernetes v1.36 with the API group groupsnapshot.storage.k8s.io/v1.</li></ul><p><strong>Your volumes don’t exist in isolation. Your backups shouldn’t either. Volume Group Snapshots finally make multi-volume consistency a first-class Kubernetes primitive — and your 3 AM self is going to thank you.</strong></p><p>If this post helped you rethink your backup strategy, smash that clap button — it helps other K8s engineers find articles like this. Running stateful workloads in production? I’d love to hear how you’ve been handling multi-volume backups in the comments.</p><p>And remember: a backup that isn’t crash-consistent isn’t a backup. It’s a liability.</p><p><strong>Follow The Kube Guy</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=852fedd36f24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pod Security Standards (PSS) in Kubernetes Explained: The Successor to PodSecurityPolicy]]></title>
            <link>https://medium.com/@thekubeguy/pod-security-standards-pss-in-kubernetes-explained-the-successor-to-podsecuritypolicy-2a47145eb063?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a47145eb063</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 05:56:00 GMT</pubDate>
            <atom:updated>2026-06-29T05:56:00.781Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine you own an apartment building.</p><p>Some tenants are trusted maintenance staff who need access to every room. Others are regular residents who should only access their own apartments. You wouldn’t hand everyone a master key, right?</p><p>Kubernetes works the same way.</p><p>Every Pod that runs inside your cluster can either have limited permissions or almost unlimited access to the worker node. If an attacker manages to compromise a highly privileged Pod, they could potentially gain control over the entire node.</p><p>That’s exactly why <strong>Pod Security Standards (PSS)</strong> exist.</p><p>They provide a simple, built-in way to define what Pods are allowed to do inside a Kubernetes cluster. They replaced the old <strong>PodSecurityPolicy (PSP)</strong> mechanism, which was powerful but difficult to configure and maintain. Starting from <strong>Kubernetes v1.25</strong>, PodSecurityPolicy was completely removed and replaced by <strong>Pod Security Admission (PSA)</strong>, which enforces Pod Security Standards using namespace labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*gA-VkP8qxUhjXS8dXm0ujg.png" /><figcaption>Image by Author</figcaption></figure><h4>Why Do Pods Need Security Restrictions?</h4><p>By default, a container should behave like an isolated process.</p><p>But Kubernetes allows containers to request additional permissions such as:</p><ul><li>Running as root</li><li>Accessing the host filesystem</li><li>Using the host network</li><li>Running privileged containers</li><li>Mounting sensitive directories from the node</li></ul><p>These features are useful for infrastructure components but dangerous for normal applications.</p><h4>Real-life example to make things simple</h4><p>Think of a hotel.</p><p>Guests should only have access to their own rooms.</p><p>Now imagine every guest also gets:</p><ul><li>The master key</li><li>CCTV access</li><li>Electrical room access</li><li>Kitchen access</li></ul><p>One compromised guest could take over the entire hotel.</p><p>A Pod with too many privileges creates the same problem inside Kubernetes.</p><h4>The Problem with PodSecurityPolicy (PSP)</h4><p>Before Kubernetes v1.25, clusters used <strong>PodSecurityPolicy (PSP)</strong>.</p><p>PSP allowed administrators to define exactly what Pods could or couldn’t do.</p><p>Although it was powerful, it had several issues:</p><ul><li>Very difficult to configure</li><li>Required RBAC permissions alongside the policy</li><li>Hard to debug when Pods failed</li><li>Different clusters implemented it differently</li><li>Too many configuration options</li></ul><p>Many Kubernetes administrators either disabled PSP completely or configured it incorrectly.</p><p>To simplify security, Kubernetes introduced <strong>Pod Security Admission (PSA)</strong> with predefined security levels instead of fully customizable policies.</p><h4>What are Pod Security Standards (PSS)?</h4><p>Pod Security Standards are predefined security profiles.</p><p>Instead of creating dozens of custom rules, Kubernetes provides three ready-made security levels.</p><p>Think of them like security clearance levels.</p><p>ProfileSecurity LevelTypical UsePrivilegedAlmost no restrictionsSystem workloadsBaselineModerate securityMost applicationsRestrictedStrong securityProduction workloads</p><p>Every namespace can choose which level it wants.</p><h4>How Pod Security Admission Works?</h4><p>Whenever someone creates a Pod:</p><pre>kubectl apply -f deployment.yaml</pre><p>The request reaches the Kubernetes API Server.</p><p>Before creating the Pod, the Pod Security Admission controller checks:</p><p>“Does this Pod satisfy the namespace’s security policy?”</p><p>If yes:</p><pre>Pod Created</pre><p>If no:</p><pre>Request Rejected</pre><p>No extra controller is required because this functionality is built into Kubernetes.</p><h4>The Three Pod Security Profiles</h4><p>Let’s understand each one.</p><p><strong>1. Privileged Profile</strong></p><p>This profile is basically:</p><p><em>“Do whatever you want.”</em></p><p>Almost everything is allowed.</p><p>Examples include:</p><ul><li>Privileged containers</li><li>Host networking</li><li>Host PID</li><li>Host IPC</li><li>HostPath volumes</li><li>Running as root</li><li>Extra Linux capabilities</li></ul><p>This profile should only be used by trusted infrastructure components.</p><p>Examples:</p><ul><li>CNI plugin</li><li>CSI storage drivers</li><li>Monitoring agents</li><li>Node maintenance tools</li></ul><p>Normal business applications should almost never use this profile.</p><p><strong>2. Baseline Profile</strong></p><p>Baseline is the middle ground.</p><p>It blocks the most dangerous privilege escalation techniques while still allowing most applications to run without modification.</p><p>It prevents things like:</p><ul><li>Privileged containers</li><li>Host PID</li><li>Host IPC</li><li>Host Network</li><li>Dangerous capabilities</li><li>HostProcess containers on Windows</li></ul><p>Most web applications, REST APIs, and backend services can run successfully under Baseline.</p><p><strong>3. Restricted Profile</strong></p><p>Restricted is the strongest built-in policy.</p><p>It follows Kubernetes hardening best practices.</p><p>Typical requirements include:</p><ul><li>Must not run as root</li><li>allowPrivilegeEscalation: false</li><li>Drop unnecessary Linux capabilities</li><li>Use the default seccomp profile</li><li>Restrict unsafe volume types</li><li>Disallow privileged containers</li></ul><p>This is the recommended target for most production application namespaces, although some workloads may need changes before they comply.</p><h4>How Kubernetes Knows Which Profile to Use?</h4><p>Kubernetes uses namespace labels.</p><p>Example:</p><pre>kubectl label namespace production \<br>pod-security.kubernetes.io/enforce=restricted</pre><p>Now every new Pod created inside that namespace must satisfy the Restricted profile.</p><p>You can also use:</p><pre>pod-security.kubernetes.io/enforce=baseline</pre><p>or</p><pre>pod-security.kubernetes.io/enforce=privileged</pre><p>The policy applies only to new Pods and updates that are validated by admission; existing running Pods are not automatically changed.</p><h4>The Three Enforcement Modes</h4><p>Pod Security Admission supports three modes.</p><p><strong>1. Enforce</strong></p><p>Violating Pods are rejected.</p><pre>Developer<br>    ↓<br>Create Pod<br>    ↓<br>Fails Validation<br>    ↓<br>Pod Rejected</pre><p>This is the strongest mode.</p><p><strong>2. Warn</strong></p><p>The Pod is created.</p><p>But Kubernetes prints warnings.</p><p>Useful during migration.</p><p>Example:</p><pre>Warning:<br>Pod violates Restricted policy</pre><p>Developers can fix issues before enforcement begins.</p><p><strong>3. Audit</strong></p><p>The Pod is created.</p><p>No warning is shown to users.</p><p>Instead, violations are recorded in audit logs for administrators to review.</p><p>This is useful for understanding the impact before enforcing stricter policies.</p><h4>Enforcing Multiple Modes Together</h4><p>A common production setup is:</p><pre>pod-security.kubernetes.io/enforce=baseline</pre><pre>pod-security.kubernetes.io/warn=restricted</pre><pre>pod-security.kubernetes.io/audit=restricted</pre><p>This means:</p><ul><li>Baseline violations are blocked.</li><li>Restricted violations only generate warnings and audit records.</li></ul><p>It gives teams time to prepare before moving to Restricted.</p><h4>Version Pinning</h4><p>You can pin policies to a Kubernetes version.</p><p>Example:</p><pre>pod-security.kubernetes.io/enforce-version=v1.36</pre><p>Why?</p><p>Because security rules can evolve between Kubernetes releases.</p><p>Version pinning ensures consistent behavior during upgrades.</p><h4>What Breaks When You Turn On Restricted?</h4><p>This is where many administrators get surprised.</p><p>Existing workloads may suddenly fail validation because they request permissions that Restricted doesn’t allow.</p><p>Common problems include:</p><p><strong>Running as Root</strong></p><p>Old images often assume they run as the root user.</p><p>Restricted expects:</p><pre>runAsNonRoot: true</pre><p><strong>Privileged Containers</strong></p><p>This will fail:</p><pre>securityContext:<br>  privileged: true</pre><p><strong>Privilege Escalation</strong></p><p>Restricted expects:</p><pre>allowPrivilegeEscalation: false</pre><p><strong>Missing Seccomp Profile</strong></p><p>Many older manifests don’t specify one.</p><p>Restricted expects something like:</p><pre>seccompProfile:<br>  type: RuntimeDefault</pre><p><strong>Linux Capabilities</strong></p><p>Instead of adding many capabilities, Restricted expects them to be dropped unless absolutely required.</p><p>Example:</p><pre>capabilities:<br>  drop:<br>    - ALL</pre><p><strong>HostPath Volumes</strong></p><p>Many monitoring or backup applications mount the host filesystem.</p><p>Restricted blocks many of these patterns.</p><p><strong>Host Network</strong></p><p>This will fail:</p><pre>hostNetwork: true</pre><p>unless you’re using a more permissive profile.</p><h3>Some practical recommendations:</h3><ul><li>Use <strong>Restricted</strong> for production application namespaces whenever possible.</li><li>Reserve <strong>Privileged</strong> only for trusted infrastructure components.</li><li>Start with <strong>Warn</strong> and <strong>Audit</strong> before enabling <strong>Enforce</strong>.</li><li>Pin the policy version to match your Kubernetes version.</li><li>Test workloads in a staging environment before enforcing stricter policies.</li><li>Combine Pod Security Standards with RBAC, NetworkPolicies, image scanning, and admission tools such as Kyverno or OPA Gatekeeper if you need more granular rules.</li></ul><h3>Conclusion</h3><p>Pod Security Standards made Kubernetes security much simpler than the old PodSecurityPolicy model.</p><p>Instead of writing complex security rules, you now choose one of three built-in profiles and apply it to a namespace. Start with <strong>Warn</strong> and <strong>Audit</strong>, fix your workloads, and then move to <strong>Enforce</strong>. It’s a simple path that greatly improves your cluster’s security without adding extra components.</p><p>And if your Pods are still asking for privileged: true just because &quot;it works,&quot; don&#39;t be surprised when the API server starts treating them like an unschedulable Pod—ignored until fixed.</p><p>If this blog helped you understand Kubernetes a little better, follow <strong>TheKubeGuy</strong>. Unlike CrashLoopBackOff, the content keeps restarting with useful updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a47145eb063" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ConfigMaps and Secrets — How They Actually Work?]]></title>
            <link>https://medium.com/@thekubeguy/configmaps-and-secrets-how-they-actually-work-81dddde46e6d?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/81dddde46e6d</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 09:58:21 GMT</pubDate>
            <atom:updated>2026-06-22T09:58:21.022Z</atom:updated>
            <content:encoded><![CDATA[<p><em>You’ve been stuffing configuration into ConfigMaps and Secrets since your first Deployment. But do you know why env vars don’t update, why subPath mounts are secretly broken, why your “encrypted” Secrets are sitting in etcd as plain base64, and why that 1 MiB limit exists? Let’s talk about what’s really going on under the hood.</em></p><p>It’s 2 AM. Your on-call phone buzzes. The payment service is returning 500s. You check the logs — it’s connecting to the staging database. Someone updated the ConfigMap with the production database URL three hours ago, did a kubectl apply, saw the ConfigMap update, and went to bed. The pods never picked up the change. They’re still running with the old value because the database URL was injected as an environment variable.</p><p>This exact scenario happens more often than anyone wants to admit. ConfigMaps and Secrets are the most used — and most misunderstood — configuration primitives in Kubernetes. Everyone learns the basics in their first week: create a ConfigMap, reference it in your pod spec, done. But the <em>behavior</em> of these objects — how they propagate, when they don’t, what’s actually happening in etcd, and where the security model falls apart — that’s where production incidents live.</p><p>Let’s go deep.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rV1et0hcbWBLwp-U4GDYdw.png" /></figure><h4><strong>What ConfigMaps and Secrets Actually Are</strong></h4><p>At the API level, a ConfigMap is just a Kubernetes object that holds key-value pairs. That’s it. No magic, no special runtime. It’s a map of strings stored in etcd, accessible through the Kubernetes API.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>name: app-config<br>namespace: production<br>data:<br>DATABASE_HOST: &quot;postgres.production.svc.cluster.local&quot;<br>DATABASE_PORT: &quot;5432&quot;<br>LOG_LEVEL: &quot;info&quot;<br>app.properties: |<br>server.port=8080<br>server.timeout=30s<br>cache.ttl=300</pre><p>A Secret is structurally almost identical. The only differences are:</p><ol><li>Values in data are base64-encoded (or you use stringData for plain text that gets encoded on creation).</li><li>Kubernetes applies some minimal access controls — kubectl get secrets requires explicit RBAC, and Secrets aren’t printed in kubectl describe pod output.</li><li>Secrets have a type field that provides some validation for specific use cases.</li></ol><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>name: db-credentials<br>namespace: production<br>type: Opaque<br>stringData:<br>DB_USERNAME: &quot;app_service&quot;<br>DB_PASSWORD: &quot;s3cret-but-not-really&quot;</pre><p>That stringData field is a write-only convenience. Kubernetes accepts it, base64-encodes the values, and stores them in the data field. When you kubectl get secret db-credentials -o yaml, you’ll see data with base64 values, never stringData.</p><h4><strong>The Base64 Elephant in the Room</strong></h4><p>Let’s get this out of the way: <strong>base64 is not encryption</strong>. It’s encoding. It’s the same as ROT13 for security purposes — which is to say, none. Anyone who can read the Secret object from the API can decode the values instantly:</p><pre>kubectl get secret db-credentials -o jsonpath=&#39;{.data.DB_PASSWORD}&#39; | base64 -d<br># Output: s3cret-but-not-really</pre><p>By default, Secrets are stored in etcd as plain text (well, base64-encoded plain text, which is the same thing). If someone gets access to your etcd data directory, they can read every Secret in your cluster. If someone gets read access to the Kubernetes API with the right RBAC, same deal.</p><p>This is not a flaw — it’s a design choice. Kubernetes Secrets were never meant to be a vault. They’re a <em>configuration delivery mechanism</em> that happens to have slightly more access controls than ConfigMaps. For actual secret management, you need encryption at rest and/or an external secrets manager. We’ll cover both later.</p><h4><strong>The Two Ways to Consume: Env Vars vs. Volume Mounts</strong></h4><p>You have two options for getting ConfigMap and Secret data into your containers: environment variables and volume mounts. They behave very differently, and choosing the wrong one will bite you.</p><p><strong>Environment Variables</strong></p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: my-app<br>spec:<br>containers:<br>- name: app<br>image: my-app:latest<br>env:<br>- name: DATABASE_HOST<br>valueFrom:<br>configMapKeyRef:<br>name: app-config<br>key: DATABASE_HOST<br>- name: DB_PASSWORD<br>valueFrom:<br>secretKeyRef:<br>name: db-credentials<br>key: DB_PASSWORD<br>envFrom:<br>- configMapRef:<br>name: app-config<br>prefix: CFG_</pre><p>The env approach lets you pick individual keys. The envFrom approach dumps the entire ConfigMap or Secret into the container’s environment, optionally with a prefix.</p><p>Here’s the critical thing: <strong>environment variables are set once, at container startup, and never change</strong>. If you update the ConfigMap after the pod is running, the environment variables inside the container keep their original values. The kubelet doesn’t re-inject them. The container runtime doesn’t restart the process. Nothing happens. Your pod is running with stale configuration until it’s restarted.</p><p>This isn’t a bug. Environment variables are a process-level construct in Linux. Once a process starts with its environment, that environment is fixed for the life of the process. Kubernetes can’t reach into a running container and modify the environment of PID 1. The only way to pick up the new values is to restart the pod — either by deleting it, rolling the Deployment, or using some external mechanism.</p><p><strong>Volume Mounts</strong></p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: my-app<br>spec:<br>containers:<br>- name: app<br>image: my-app:latest<br>volumeMounts:<br>- name: config-volume<br>mountPath: /etc/config<br>readOnly: true<br>- name: secret-volume<br>mountPath: /etc/secrets<br>readOnly: true<br>volumes:<br>- name: config-volume<br>configMap:<br>name: app-config<br>- name: secret-volume<br>secret:<br>secretName: db-credentials</pre><p>This mounts each key as a file inside the container. DATABASE_HOST becomes /etc/config/DATABASE_HOST containing postgres.production.svc.cluster.local. DB_PASSWORD becomes /etc/secrets/DB_PASSWORD containing the decoded secret value.</p><p>Here’s where it gets interesting: <strong>volume-mounted ConfigMaps and Secrets do update automatically</strong>. When you change the ConfigMap, the kubelet detects the change and updates the files inside the container. No pod restart needed. The update happens within the kubelet’s sync period — typically 60 seconds, plus the configured ConfigMapAndSecretChangeDetectionStrategy cache TTL (default is another 60 seconds). So in the worst case, you’re looking at about two minutes for the new values to appear in the mounted files.</p><p>But — and this is a big but — <strong>your application has to be watching for file changes</strong>. If your app reads the config file once at startup and caches the values in memory, the file updating on disk doesn’t help you. You need inotify watches, a file-polling loop, or a framework that handles config reloading (like Spring Cloud’s @RefreshScope or Viper’s WatchConfig() in Go).</p><h4><strong>The subPath Trap</strong></h4><p>Here’s a gotcha that has burned countless teams. Sometimes you want to mount a single key from a ConfigMap as a specific file, without replacing the entire directory:</p><pre>volumeMounts:<br>- name: config-volume<br>mountPath: /etc/app/config.yaml<br>subPath: config.yaml</pre><p>This mounts just the config.yaml key as /etc/app/config.yaml without blowing away whatever else is in /etc/app/. Sounds perfect, right?</p><p><strong>subPath mounts never update.</strong> Ever. Even when the ConfigMap changes, the file stays frozen at the value it had when the pod started. This is a fundamental limitation of how subPath works — it uses a bind mount tied to the original inode, and the kubelet’s atomic symlink swap mechanism (which is how regular volume updates work) doesn’t apply to bind mounts.</p><p>This has been a known issue since 2017 (<a href="https://github.com/kubernetes/kubernetes/issues/50345">kubernetes/kubernetes#50345</a>), and it’s not going to be fixed because it’s architecturally inherent to how bind mounts work in Linux.</p><p>Your options if you need a single file without clobbering the directory:</p><ol><li>Mount the full ConfigMap to a different path and symlink from your app.</li><li>Use a projected volume with items to control which keys are mounted.</li><li>Use an init container to copy the file into an emptyDir, then mount the emptyDir at the target path.</li><li>Just accept the pod restart and use a rolling update strategy.</li></ol><h4><strong>Secret Types: More Than Just Opaque</strong></h4><p>Most people only use Opaque Secrets. But Kubernetes has several built-in types that provide validation and structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*lN_WBabY27Lr2lplPGraVA.png" /></figure><p>The typed Secrets aren’t just labels — Kubernetes validates the required keys exist when you create them. Try creating a kubernetes.io/tls Secret without a tls.key field and the API server will reject it:</p><pre>kubectl create secret tls my-cert - cert=server.crt<br># error: - key is required for tls secret</pre><p>A note on kubernetes.io/service-account-token: this type was how ServiceAccount tokens used to work — Kubernetes created a long-lived, non-expiring Secret containing a JWT. Since Kubernetes 1.24, the LegacyServiceAccountTokenNoAutoGeneration feature gate (now permanent) stopped this auto-generation. Modern Kubernetes uses the TokenRequest API to issue short-lived, audience-bound tokens that are mounted via projected volumes. If you’re still creating service-account-token Secrets manually, stop. Use the TokenRequest API or projected service account tokens instead.</p><h4><strong>Immutable ConfigMaps and Secrets</strong></h4><p>Since Kubernetes 1.21 (GA), you can mark any ConfigMap or Secret as immutable:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>name: app-config-v3<br>data:<br>DATABASE_HOST: &quot;postgres.production.svc.cluster.local&quot;<br>LOG_LEVEL: &quot;warn&quot;<br>immutable: true</pre><p>Once you set immutable: true, you cannot change the data or binaryData fields. You also cannot flip immutable back to false. The only way to “update” an immutable ConfigMap is to delete it and create a new one — which is usually done by versioning the name (e.g., app-config-v3, app-config-v4) and updating the pod spec to reference the new version.</p><h4><strong>Why You Want This</strong></h4><p><strong>Performance.</strong> The kubelet maintains watches on every ConfigMap and Secret that’s mounted into pods on its node. In a cluster with thousands of pods, each referencing multiple ConfigMaps, that’s a lot of watch connections to the API server. When a ConfigMap is immutable, the kubelet closes its watch — it knows the data will never change, so there’s nothing to watch for. On large clusters, this measurably reduces API server load and kubelet memory consumption.</p><p><strong>Safety.</strong> A typo in a mutable ConfigMap gets propagated to every pod mounting it. With immutable ConfigMaps, changes require a deliberate create-new-version-and-update-pod-spec cycle, which naturally goes through your Deployment’s rolling update process. This means you get rollback for free — if the new config is bad, you roll back the Deployment, which rolls back to the previous ConfigMap version.</p><p><strong>Predictability.</strong> When a ConfigMap is immutable, you know that every pod referencing app-config-v3 has the exact same configuration. There’s no window where some pods have the old data and others have the new data.</p><p>The pattern that works well: treat ConfigMaps like container images. Version them, reference them explicitly, and roll forward. It’s more work than mutating in place, but it’s also more reliable.</p><h4><strong>Encryption at Rest: What It Means and How to Actually Do It</strong></h4><p>By default, everything in etcd is stored as plain text. ConfigMaps, Secrets, all of it. The base64 encoding on Secrets is purely for binary safety (so you can store TLS certs without escaping issues), not security.</p><p>To actually encrypt Secrets at rest, you need to configure the API server’s EncryptionConfiguration:</p><pre>apiVersion: apiserver.config.k8s.io/v1<br>kind: EncryptionConfiguration<br>resources:<br>- resources:<br>- secrets<br>providers:<br>- aescbc:<br>keys:<br>- name: key1<br>secret: &lt;base64-encoded-32-byte-key&gt;<br>- identity: {}</pre><p>The providers list is ordered. The first provider is used for <strong>writing</strong> (encrypting new Secrets). All providers are tried for <strong>reading</strong> (decrypting existing Secrets). The identity: {} provider means “no encryption” and acts as a fallback for reading Secrets that were stored before encryption was enabled.</p><p>Available encryption providers, from least to most secure:</p><ul><li><strong>identity</strong>: No encryption. This is the default.</li><li><strong>secretbox</strong>: Uses XSalsa20 + Poly1305. Fast, modern, good for most use cases. The key is stored in the EncryptionConfiguration file on the API server node.</li><li><strong>aescbc</strong>: AES-256 in CBC mode with PKCS#7 padding. Widely used, but the key is still local.</li><li><strong>aesgcm</strong>: AES-256 in GCM mode. Faster than CBC with built-in authentication, but you must rotate keys frequently because GCM nonce reuse is catastrophic.</li><li><strong>kms v2</strong>: Integrates with an external KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault, HashiCorp Vault). The encryption key is managed by the KMS, and the API server only holds a data encryption key (DEK) that’s wrapped by the KMS key encryption key (KEK). This is the production recommendation.</li></ul><p>With aescbc or secretbox, the encryption key sits in a file on your control plane node. If someone compromises the node, they get the key. With KMS v2, the key never leaves the KMS — the API server sends the DEK to the KMS for wrapping/unwrapping, and the KMS handles the actual cryptographic operations. This is a meaningful security improvement for anyone running in a cloud environment.</p><p>After enabling encryption, existing Secrets are still stored unencrypted. You need to re-write them:</p><pre># Re-encrypt all existing Secrets<br>kubectl get secrets - all-namespaces -o json | kubectl replace -f -</pre><p>This reads each Secret (decrypting with the old provider) and writes it back (encrypting with the new provider).</p><h4><strong>Managed Kubernetes Caveat</strong></h4><p>If you’re on GKE, EKS, or AKS, encryption at rest works differently. GKE encrypts etcd by default using Google-managed keys, with an option to use customer-managed encryption keys (CMEK) via Cloud KMS. EKS encrypts etcd by default. AKS supports bringing your own key. In all cases, you don’t manage the EncryptionConfiguration directly — the managed service handles it. Check your provider’s documentation for the specifics.</p><h4><strong>The 1 MiB Size Limit</strong></h4><p>Both ConfigMaps and Secrets are capped at 1 MiB (1,048,576 bytes). This isn’t arbitrary — it comes from etcd’s value size limit. etcd is designed for small key-value pairs (cluster metadata, not file storage), and large values degrade performance for every other read and write in the cluster.</p><p>The limit includes the entire object — metadata, labels, annotations, and the data itself. In practice, you have slightly less than 1 MiB for your actual configuration data.</p><p>If you’re hitting this limit, it’s usually a sign you’re using ConfigMaps for something they weren’t designed for:</p><ul><li><strong>Large config files</strong>: Break them into multiple ConfigMaps, or reconsider whether a ConfigMap is the right delivery mechanism. An init container that downloads config from S3 or a config service might be better.</li><li><strong>Bundled assets</strong>: Use a container image layer or a persistent volume.</li><li><strong>ML models</strong>: Definitely not a ConfigMap. Use a volume, an init container, or a model-serving framework.</li></ul><p>One non-obvious consequence: if you have many pods on a single node, each mounting different ConfigMaps, the kubelet has to watch all of them. A node with 50 pods each mounting 3 ConfigMaps means 150 watch connections. The API server and etcd are doing work for every one of those, even when nothing changes. This is another reason immutable ConfigMaps are valuable at scale.</p><h4><strong>Practical Patterns and Anti-Patterns</strong></h4><p><strong>Pattern: ConfigMap Per Environment, Sealed Secrets for Credentials</strong></p><pre># Structure<br>config/<br>base/<br>configmap.yaml # shared config<br>overlays/<br>staging/<br>configmap.yaml # staging overrides<br>sealed-secret.yaml # staging credentials (encrypted in git)<br>production/<br>configmap.yaml # production overrides<br>sealed-secret.yaml # production credentials (encrypted in git)</pre><p>Use Kustomize overlays to compose environment-specific ConfigMaps. Use Sealed Secrets (or External Secrets Operator) for credentials, so encrypted values can live in Git safely.</p><h4><strong>Pattern: Hash-Based ConfigMap Rolling Updates</strong></h4><p>Deployments don’t automatically roll when a referenced ConfigMap changes. The pod spec references the ConfigMap by name, and the name didn’t change, so the Deployment sees no diff. Helm solves this with the checksum annotation trick:</p><pre># In your Deployment template (Helm)<br>spec:<br>template:<br>metadata:<br>annotations:<br>checksum/config: {{ include (print $.Template.BasePath &quot;/configmap.yaml&quot;) . | sha256sum }}</pre><p>When the ConfigMap content changes, the annotation changes, which changes the pod template, which triggers a rolling update. Kustomize has a similar mechanism with configMapGenerator and its behavior: create default, which appends a content hash to the ConfigMap name.</p><h4><strong>Anti-Pattern: Storing Secrets in ConfigMaps</strong></h4><p>Don’t. Even if your Secret “isn’t that sensitive.” ConfigMaps have weaker RBAC defaults, they appear in kubectl describe pod output, they show up in events, and they’re more likely to end up in logs or debugging output. Use a Secret. The overhead is zero.</p><h4><strong>Anti-Pattern: Giant Monolithic ConfigMaps</strong></h4><p>One ConfigMap with every config value for every microservice. When you update one value, every pod that mounts it gets updated (or needs a restart). Break ConfigMaps along service boundaries. Each service gets its own ConfigMap with just the keys it needs.</p><h4><strong>Anti-Pattern: Using ConfigMaps as a Database</strong></h4><p>ConfigMaps are not a key-value store for application data. They’re for configuration. They go through the API server, get stored in etcd, and trigger watch events on every change. Using them for frequently-changing application state hammers the control plane. Use Redis, a database, or literally anything else.</p><h4><strong>The External Secrets Ecosystem</strong></h4><p>For production workloads, most teams eventually outgrow native Kubernetes Secrets. The common pattern is to store secrets in an external vault and sync them into Kubernetes Secrets:</p><ul><li><strong>External Secrets Operator (ESO)</strong>: Creates Kubernetes Secrets from external providers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, and many more). You define an ExternalSecret resource, and ESO creates and maintains the corresponding Kubernetes Secret.</li><li><strong>Sealed Secrets (Bitnami)</strong>: Encrypts Secrets with a cluster-side certificate so you can store them in Git. The controller in the cluster decrypts them into regular Secrets.</li><li><strong>HashiCorp Vault with CSI driver</strong>: Mounts Vault secrets directly as files using the Secrets Store CSI Driver, bypassing Kubernetes Secrets entirely.</li></ul><p>Each approach has tradeoffs. ESO is the most flexible and widely adopted. Sealed Secrets are simplest for GitOps workflows. Vault with CSI is the most secure (secrets never exist as Kubernetes objects) but the most complex to operate.</p><h4><strong>What’s New: Recent Kubernetes Changes</strong></h4><p>A couple of recent additions worth knowing about:</p><p><strong>Relaxed Environment Variable Validation (Kubernetes 1.34)</strong>: The RelaxedEnvironmentVariableValidation feature gate loosens the restrictions on environment variable names. Previously, names had to match the POSIX regex [A-Za-z_][A-Za-z0–9_]*. With this feature enabled, any printable ASCII character except = is allowed. This also means ConfigMap and Secret keys that would previously fail as env var names can now be used with envFrom. If you’ve ever hit “invalid environment variable name” errors when mounting a ConfigMap with keys like my.config.key, this feature is for you.</p><p><strong>File-based Environment Variables (Kubernetes 1.34)</strong>: A new fileKeyRef in the pod spec’s env.valueFrom lets an init container write environment variables to a file in a shared emptyDir, which the main container then reads as env vars. This opens up dynamic configuration patterns that were previously awkward to implement.</p><p><strong>Image Pull Credential Re-validation (Kubernetes 1.35)</strong>: The KubeletEnsureSecretPulledImages feature, now enabled by default, means the kubelet re-validates image pull credentials for every pod, even if the image is already cached locally. This tightens the security model for kubernetes.io/dockerconfigjson Secrets — a cached image can no longer be used by a pod that doesn’t have valid pull credentials.</p><p><strong>Debugging ConfigMaps and Secrets</strong></p><p>When things go wrong, here’s your toolkit:</p><p><strong>Check if the ConfigMap exists and has the expected data</strong></p><pre>kubectl get configmap app-config -o yaml</pre><p><strong>Check if the Secret has the expected keys (don’t print values in shared terminals)</strong></p><pre>kubectl get secret db-credentials -o jsonpath=’{.data}’ | jq ‘keys’</pre><p><strong>Check what a pod actually sees</strong></p><pre>kubectl exec my-app - env | grep DATABASE<br>kubectl exec my-app - cat /etc/config/DATABASE_HOST</pre><p><strong>Check if volume mounts are correct</strong></p><pre>kubectl describe pod my-app | grep -A 5 “Mounts:”</pre><p><strong>Check events for mount failures</strong></p><pre>kubectl get events — field-selector involvedObject.name=my-app</pre><p><strong>Verify a Secret value without exposing it in shell history</strong></p><pre>kubectl get secret db-credentials -o go-template=’{{index .data “DB_PASSWORD” | base64decode}}’</pre><h4><strong>The Summary</strong></h4><p>Here’s what you should take away:</p><ul><li><strong>ConfigMaps and Secrets are structurally the same</strong> — the difference is access controls and base64 encoding, not encryption.</li><li><strong>Env vars never update. Volume mounts do</strong> (except with subPath). Choose your consumption method accordingly.</li><li><strong>subPath mounts are frozen at pod start.</strong> This is a permanent architectural limitation, not a bug.</li><li><strong>Immutable ConfigMaps and Secrets save API server resources</strong> and give you safer rollout semantics. Use them by default.</li><li><strong>Base64 is not encryption.</strong> Enable encryption at rest, preferably with a KMS provider.</li><li><strong>The 1 MiB limit comes from etcd.</strong> If you’re hitting it, rethink your approach.</li><li><strong>Version your ConfigMaps</strong> (via Kustomize hash suffixes, Helm checksums, or manual naming) to get proper rolling updates and rollback.</li><li><strong>Use an external secrets manager</strong> (ESO, Sealed Secrets, or Vault) for anything resembling a real credential.</li><li><strong>Secrets are a configuration delivery mechanism, not a vault.</strong> Treat them accordingly.</li></ul><p>Configuration management in Kubernetes isn’t glamorous. Nobody’s giving conference talks about their ConfigMap strategy. But getting it wrong causes outages, security incidents, and the kind of 2 AM pages that make you reconsider your career choices. Get it right, and it just works — quietly, reliably, boringly. And boring infrastructure is the best kind.</p><p>⸻</p><p><em>If this helped you understand what’s actually happening inside your cluster’s configuration layer, clap, comment, or share it with someone who’s still injecting database passwords as env vars in a mutable ConfigMap. They need this more than they know.</em></p><p><em>Follow @thekubeguy for more deep dives into Kubernetes internals — the stuff that actually matters when things break at 2 AM.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81dddde46e6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OCI Volumes Hit GA — Your Registry Is Now a Volume Backend]]></title>
            <link>https://medium.com/@thekubeguy/oci-volumes-hit-ga-your-registry-is-now-a-volume-backend-5863b91b3c73?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/5863b91b3c73</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Thu, 28 May 2026 10:11:00 GMT</pubDate>
            <atom:updated>2026-05-28T10:11:00.520Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Kubernetes v1.36 just turned your container registry into a storage layer. Mount ML models, config bundles, and static assets directly into pods — no PVCs, no initContainers, no hacks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qH6K7CSPWRJVcrytKH3kSQ.png" /><figcaption>Image by author</figcaption></figure><p>Here’s a scenario that’ll sound painfully familiar. You’ve got a machine learning model that’s 8 GB. You need to serve it in a pod. So what do you do? You bake it into the container image. Now your image takes 12 minutes to pull, your CI pipeline wants to cry, and every time you update the model, you’re rebuilding an image that’s 95% unchanged data. Or maybe you went the initContainer route — write a little script that downloads the model from S3, waits for it to finish, then signals the main container to start. It works… until it doesn’t. Authentication tokens expire, downloads fail silently, and you’ve got race conditions that only show up at 3 AM.</p><p>There had to be a better way. And now there is.</p><p>Kubernetes v1.36, released April 22, 2026, graduates the <strong>OCI Image Volume Source</strong> to General Availability. This feature lets you mount any OCI artifact — container images, model weights, config files, binary tools — directly as a read-only volume in your pod. No PersistentVolumeClaims. No initContainers downloading things. No baking data into your application image. Just declare it in your pod spec and the kubelet handles the rest.</p><p>Let’s break this down from scratch.</p><h4><strong>What Is an OCI Volume, Exactly?</strong></h4><p>You already know what an OCI image is — it’s the standard format that container images use. When you do docker push myapp:v1, you’re pushing an OCI image to a registry. That image has layers, a manifest, and metadata. Registries like Docker Hub, GitHub Container Registry, Amazon ECR, and Google Artifact Registry all speak this protocol.</p><p>Here’s the key insight: <strong>an OCI image doesn’t have to contain a runnable application.</strong> It can contain anything. A directory of config files. A trained ML model. A dataset. A collection of static assets. A set of binary plugins. As long as it’s packaged as an OCI artifact and pushed to a registry, Kubernetes can now mount it directly into your pod as a volume.</p><p>Think of your container registry like a USB drive factory. Previously, Kubernetes could only <em>run</em> the programs on those USB drives. Now it can also <em>read the files</em> on them. Same infrastructure, new superpower.</p><h4><strong>The Old Way…</strong></h4><p>Before OCI volumes, teams had to get creative (read: hacky) to get data into pods. Let’s look at the common patterns and why they all have problems.</p><p><strong>Baking data into the container image.</strong> This is the simplest approach — just add your model weights or config files to your Dockerfile. But it couples your data to your application lifecycle. Update a 50 MB config file? Rebuild and redeploy a 10 GB image. Your CI pipeline is now doing heavy lifting for what should be a data change. And image pull times go through the roof.</p><p><strong>InitContainers that download from object storage.</strong> Write an initContainer that pulls files from S3, GCS, or Azure Blob Storage before the main container starts. This works, but now you need to manage credentials, handle download failures, deal with retry logic, and debug a multi-container startup sequence. Oh, and you need a network path from your pod to the storage service, which gets fun in air-gapped environments.</p><p><strong>ConfigMaps and Secrets.</strong> Great for small files, terrible for anything over 1 MB. ConfigMaps have a hard size limit, and base64-encoding a 5 GB model weight is… not going to work.</p><p><strong>PersistentVolumes with pre-loaded data.</strong> Create a PV, mount it, populate it with data somehow, then reference it from your pods. This works, but now you’re managing storage provisioning, access modes, and lifecycle. And good luck versioning that data with the same rigor you version your container images.</p><p>Every one of these approaches is a workaround for a missing primitive. OCI volumes fill that gap.</p><h4><strong>The New Way — Declare and Mount</strong></h4><p>Here’s what an OCI volume looks like in practice. Say you have an ML model packaged as an OCI artifact at registry.example.com/models/llama-3:v2.1. Here’s how you serve it:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: model-server<br>spec:<br>containers:<br>- name: inference<br>image: registry.example.com/serving/vllm:latest<br>command: [&quot;python&quot;, &quot;-m&quot;, &quot;vllm.entrypoints.api_server&quot;,<br>&quot; - model&quot;, &quot;/models/llama-3&quot;]<br>volumeMounts:<br>- name: model-weights<br>mountPath: /models/llama-3<br>volumes:<br>- name: model-weights<br>image:<br>reference: registry.example.com/models/llama-3:v2.1<br>pullPolicy: IfNotPresent</pre><p>That’s it. No initContainers. No download scripts. No PVC claims. The kubelet sees the image volume source, pulls the OCI artifact from the registry (using the same pull infrastructure it uses for container images), and mounts the contents at /models/llama-3. The volume is read-only and non-executable by default — which is exactly what you want for data volumes.</p><p>Let’s compare this side by side with the old initContainer approach:</p><p><strong>THE OLD WAY — initContainer downloading from S3</strong></p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: model-server<br>spec:<br>initContainers:<br>- name: download-model<br>image: amazon/aws-cli:latest<br>command: [&quot;aws&quot;, &quot;s3&quot;, &quot;cp&quot;,<br>&quot;s3://my-models/llama-3-v2.1/&quot;, &quot;/models/&quot;,<br>&quot; - recursive&quot;]<br>env:<br>- name: AWS_ACCESS_KEY_ID<br>valueFrom:<br>secretKeyRef:<br>name: aws-creds<br>key: access-key<br>- name: AWS_SECRET_ACCESS_KEY<br>valueFrom:<br>secretKeyRef:<br>name: aws-creds<br>key: secret-key<br>volumeMounts:<br>- name: model-data<br>mountPath: /models<br>containers:<br>- name: inference<br>image: registry.example.com/serving/vllm:latest<br>command: [&quot;python&quot;, &quot;-m&quot;, &quot;vllm.entrypoints.api_server&quot;,<br>&quot; - model&quot;, &quot;/models/llama-3&quot;]<br>volumeMounts:<br>- name: model-data<br>mountPath: /models<br>volumes:<br>- name: model-data<br>emptyDir: {}</pre><p>Count the differences. The old way needs an initContainer, AWS credentials in a Secret, an emptyDir volume to shuttle data between containers, and a network path to S3. The new way needs… a reference to an OCI artifact. That’s a significant reduction in moving parts, credential surface area, and things that can break at 3 AM.</p><h4><strong>How to Package Your Data as an OCI Artifact</strong></h4><p>“Okay cool,” you’re thinking, “but how do I actually get my model weights or config files into an OCI registry?” Great question. You have two main options.</p><p><strong>Option 1: Use a Dockerfile (yes, seriously).</strong> You can create a minimal image that just contains your data:</p><pre>FROM scratch<br>COPY ./model-weights/ /<br>Build and push it like any other container image:<br>docker build -t registry.example.com/models/llama-3:v2.1 -f Modelfile .<br>docker push registry.example.com/models/llama-3:v2.1</pre><p>The FROM scratch base means there’s no operating system, no runtime, no extra layers — just your raw data. When Kubernetes mounts this as a volume, you get exactly the files you packed in.</p><p><strong>Option 2: Use ORAS (OCI Registry As Storage).</strong> ORAS is a purpose-built CLI tool for pushing arbitrary files to OCI registries. It’s designed exactly for this use case:</p><pre># Install ORAS<br>brew install oras<br># Push your model weights as an OCI artifact<br>oras push registry.example.com/models/llama-3:v2.1 \<br>./model-weights/:application/octet-stream<br># Push config files<br>oras push registry.example.com/configs/my-app:v1.0 \<br>./config.yaml:application/yaml \<br>./rules.json:application/json</pre><p>ORAS treats your registry as a generic storage backend. No Dockerfiles needed. No build step. Just push files directly. This is especially nice when you’re dealing with data pipelines that produce artifacts outside of a traditional CI/CD workflow — like a training pipeline that spits out new model weights every night.</p><h4><strong>Real-World Use Cases That Make This Shine</strong></h4><p>Let me walk through some practical scenarios where OCI volumes genuinely change the game.</p><p><strong>ML Model Serving.</strong> This is the poster child use case. Package your model weights as an OCI artifact, version them with tags (:v1.0, :v2.1, :latest), and mount them into inference pods. When you retrain your model, push new weights to the registry and roll out new pods. The serving container image stays the same — only the model volume changes. This also means your model and your serving code can have completely independent release cycles.</p><p><strong>Shared Configuration Bundles.</strong> Got a set of config files that multiple microservices need? Package them as an OCI artifact and mount them across pods. Update the config, push a new version, and roll it out. No more giant ConfigMaps, no more copying the same files into 15 different container images.</p><p><strong>Plugin Systems.</strong> Running a platform that supports plugins? Package each plugin as an OCI artifact and mount them into the base application container. Enable or disable plugins by adding or removing volumes from the pod spec. Version each plugin independently.</p><p><strong>Static Assets for Web Apps.</strong> Package your frontend build artifacts (HTML, CSS, JS) as an OCI artifact and mount them into an NGINX container. Your web server image stays tiny and generic, while your static assets have their own versioning and release pipeline.</p><p><strong>Air-Gapped and Edge Environments.</strong> This is a big one. In disconnected environments, you often can’t reach external object storage. But you probably have a local container registry (for pulling images into the cluster). Now that same registry serves double duty — it distributes both your application images and your data volumes. One artifact distribution mechanism to rule them all.</p><h4><strong>The Pull Policy Matters</strong></h4><p>Just like container images, OCI volumes support three pull policies:</p><p><strong>IfNotPresent</strong> — The kubelet only pulls the artifact if it’s not already cached on the node. This is what you want for most production workloads. Pin your reference to a specific tag or digest, and the kubelet will cache it after the first pull.</p><p><strong>Always</strong> — The kubelet always checks the registry and pulls fresh content. Use this when your tag is mutable (like :latest) and you want pods to always get the newest version. Be aware this means a registry round-trip on every pod startup.</p><p><strong>Never</strong> — The kubelet only uses locally available images. Useful when you’ve pre-loaded artifacts onto your nodes and don’t want any registry traffic.</p><pre>volumes:<br>- name: config<br>image:<br>reference: registry.example.com/configs/app:v3.2<br>pullPolicy: IfNotPresent</pre><p>Pro tip: for production, always use a specific tag or, even better, a <strong>digest reference</strong> like registry.example.com/models/llama-3@sha256:abc123…. This guarantees you get exactly the artifact you expect, regardless of whether someone has retagged things in the registry.</p><h4><strong>Sharing Volumes Across Containers in a Pod</strong></h4><p>One pattern that’s immediately useful is mounting the same OCI volume into multiple containers within a pod. Since the volume is read-only and pulled once by the kubelet, multiple containers can read from it simultaneously without any coordination.</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: multi-container-app<br>spec:<br>containers:<br>- name: api-server<br>image: myapp/api:v2.0<br>volumeMounts:<br>- name: shared-config<br>mountPath: /etc/app-config<br>- name: worker<br>image: myapp/worker:v2.0<br>volumeMounts:<br>- name: shared-config<br>mountPath: /etc/app-config<br>- name: metrics-exporter<br>image: myapp/exporter:v1.3<br>volumeMounts:<br>- name: shared-config<br>mountPath: /etc/app-config<br>volumes:<br>- name: shared-config<br>image:<br>reference: registry.example.com/configs/shared:v3.0<br>pullPolicy: IfNotPresent</pre><p>All three containers get the same config files, pulled once, cached on the node. Compare this to the old approach where you’d either copy the config into each container image (tripling the duplication) or use a ConfigMap (which falls over for large datasets).</p><p><strong>SubPath — Mount Just What You Need</strong></p><p>Starting from Kubernetes v1.33, you can use subPath with OCI volumes to mount specific subdirectories or files from the artifact, rather than the entire thing. This is useful when your OCI artifact contains multiple datasets or configs but a container only needs a subset:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: selective-mount<br>spec:<br>containers:<br>- name: app<br>image: myapp:latest<br>volumeMounts:<br>- name: data-bundle<br>mountPath: /app/config.yaml<br>subPath: configs/production/config.yaml<br>volumes:<br>- name: data-bundle<br>image:<br>reference: registry.example.com/data/bundle:v1.0<br>pullPolicy: IfNotPresent</pre><p><strong>Things to Keep in Mind</strong></p><p>OCI volumes are powerful, but they come with a few constraints worth knowing about.</p><p><strong>Read-only, always.</strong> OCI volumes are mounted as read-only. Your application can’t write to them. This is by design — they’re for distributing immutable data, not for scratch space. If you need to modify data from an OCI volume, copy it to an emptyDir first.</p><p><strong>Non-executable files.</strong> Files mounted from OCI volumes have the noexec flag set. You can’t execute binaries directly from the mount point. Again, this is a deliberate security decision. If you need to run executables, copy them to a writable volume first.</p><p><strong>Node-level caching.</strong> The kubelet caches pulled artifacts just like it caches container images. This means the first pod on a node that uses a given OCI volume will pull it, but subsequent pods on the same node won’t. This also means you inherit the same garbage collection behavior — if the kubelet needs disk space, it may evict cached artifacts.</p><p><strong>Registry authentication.</strong> OCI volumes use the same image pull credentials as container images. Your imagePullSecrets on the pod spec or service account cover both. No separate credential management needed.</p><p><strong>Size considerations.</strong> There’s no hard size limit on OCI volumes, but remember that the kubelet has to pull the entire artifact before the pod can start. A 50 GB model will take time to pull. Plan your node storage and image pull timeouts accordingly. For very large artifacts, pre-pulling to nodes using a DaemonSet or node image cache can help.</p><p><strong>Wrapping Up</strong></p><p>OCI Image Volumes going GA in Kubernetes v1.36 is one of those features that seems simple on the surface but fundamentally changes how you think about data distribution in a cluster. Your container registry — the thing you already operate, secure, and monitor — is now a first-class volume backend. No new infrastructure. No new protocols. No new credential systems.</p><p>Here’s what to take away:</p><ul><li>OCI volumes let you mount images and artifacts directly as read-only pod volumes — no initContainers or PVCs needed.</li><li>Package data with FROM scratch Dockerfiles or use ORAS to push files directly to your registry.</li><li>Three pull policies — IfNotPresent, Always, and Never — give you full control over freshness vs. performance.</li><li>Perfect for ML models, shared configs, plugin systems, static assets, and air-gapped environments.</li><li>Volumes are read-only and noexec by default — secure by design.</li><li>Uses your existing imagePullSecrets — no new credential management.</li><li>Available right now in Kubernetes v1.36 with no feature gates required.</li></ul><p><strong>Your registry already stores your code. Now it stores your data too. That’s not just convenient — it’s the kind of infrastructure convergence that makes platform teams smile.</strong></p><p>If this post helped you rethink how you distribute data to your pods, smash that clap button — it helps other K8s engineers find articles like this. Got an interesting use case for OCI volumes? Drop it in the comments — I’m especially curious to hear how ML teams are using this in production. Don’t forget to follow <a href="https://medium.com/u/54b070394829">The kube guy</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5863b91b3c73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Resource Allocation — GPUs Story]]></title>
            <link>https://medium.com/@thekubeguy/dynamic-resource-allocation-gpus-story-43195bc568ba?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/43195bc568ba</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Fri, 22 May 2026 07:26:00 GMT</pubDate>
            <atom:updated>2026-05-22T07:26:00.495Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Your AI workloads need GPUs, but Kubernetes has been treating them like second-class citizens. DRA changes everything — and v1.36 is about to make it real.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*yepY8fJB-K4Tana0yk30Sg.png" /><figcaption>Image by author</figcaption></figure><p>If you’ve ever tried to run a GPU workload on Kubernetes, you know the pain. You slap <em>nvidia.com/gpu: “1”</em> into your resource requests, cross your fingers, and hope the scheduler puts your pod on a node that actually has a free GPU. No preference for which GPU. No way to say “I need the one with 80GB of VRAM, not the 16GB one.” No ability to share a GPU between two lightweight inference containers. Just… “give me a GPU. Any GPU. Please.”</p><p>It’s like walking into a car rental place and being told you can ask for “one car” — but you can’t specify the make, model, engine type, or even whether it has air conditioning. You get what you get. And if the only car available is a massive truck when all you needed was a compact? Tough luck.</p><p>This has been the reality of hardware management in Kubernetes for years. The <strong>device plugin framework</strong> that handles GPUs, FPGAs, network adapters, and other specialized hardware has served us well, but it was designed in a simpler era. An era before every startup wanted to run LLM inference on Kubernetes. Before ML training jobs needed to request “8 GPUs on the same node with NVLink interconnect.” Before platform teams needed to take a single A100 and slice it into seven isolated partitions.</p><p>Enter <strong>Dynamic Resource Allocation (DRA)</strong> — the feature that finally teaches Kubernetes how to properly manage hardware. And with Kubernetes v1.36 dropping on April 22, 2026, DRA is getting some serious upgrades that make it production-ready for the GPU-hungry masses. Let’s break this down.</p><h4>What’s Wrong with the Device Plugin Framework?</h4><p>Before we talk about DRA, let’s understand why we need it. The current device plugin framework (introduced way back in Kubernetes v1.8) works through a simple model: a plugin runs on each node, advertises a count of available devices as extended resources, and the scheduler treats them like any other resource — CPU, memory, GPUs — just numbers.</p><p>The problem? That’s <strong><em>all</em></strong> they are. Numbers. When you request `nvidia.com/gpu: “2”`, the scheduler knows it needs a node with at least 2 of those things available. But it has no idea what “those things” actually are. Are they A100s or T4s? Do they have 16GB or 80GB of memory? Are they connected via NVLink or PCIe? The scheduler doesn’t know, doesn’t care, and can’t be told.</p><p>Think of it like a warehouse inventory system that only tracks “boxes.” You have 50 boxes. Someone needs 3 boxes. Great, you have enough! But the person needed boxes that can hold 50kg each, and you just gave them three shoe boxes. The system worked perfectly according to its own logic — and completely failed the user.</p><p>Here’s a concrete example of the old way:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: ml-training<br>spec:<br>containers:<br>- name: trainer<br>image: my-training-job:latest<br>resources:<br>requests:<br>nvidia.com/gpu: &quot;2&quot;<br>limits:<br>nvidia.com/gpu: &quot;2&quot;</pre><p>That’s it. That’s all the expressiveness you get. Two GPUs, please. Which ones? Whatever. The device plugin picks them and you find out at runtime whether they’re what you needed.</p><p>And forget about sharing. If you have one GPU but two containers that each only need a fraction of it, the device plugin framework says no — one device, one container, no exceptions. In a world where a single H100 costs more per hour than most engineers’ daily rate, that’s not just inconvenient. It’s expensive.</p><h4>How DRA Fixes Everything</h4><p>Dynamic Resource Allocation, which hit <strong>General Availability in Kubernetes v1.34</strong>, takes a fundamentally different approach. Instead of treating devices as anonymous, countable things, DRA introduces a <strong>structured model</strong> where every device has attributes, capacities, and identity.</p><p>The core idea is beautifully simple: instead of saying “I need 2 GPUs,” you describe <strong><em>what kind</em></strong> of device you need, and Kubernetes finds devices that match. It’s the difference between telling a recruiter “hire me two people” versus “hire me two engineers with Go experience, Kubernetes knowledge, and at least 5 years of backend development.”</p><p>DRA introduces three key resources that work together:</p><p><strong>DeviceClass</strong> — Defines a category of devices. Think of it as a filter that says “these are the kinds of devices I’m interested in.” A cluster might have a `gpu.nvidia.com` DeviceClass that selects all NVIDIA GPUs, or a more specific `gpu-a100` class that only selects A100s.</p><pre>apiVersion: resource.k8s.io/v1<br>kind: DeviceClass<br>metadata:<br>name: gpu-a100<br>spec:<br>selectors:<br>- cel:<br>expression: &gt;<br>device.driver == &quot;gpu.nvidia.com&quot; &amp;&amp;<br>device.attributes[&quot;gpu.nvidia.com&quot;].model == &quot;A100&quot;</pre><p>See that? CEL expressions. You’re writing real selection logic, not hoping annotations work. You can filter on any attribute the driver exposes — model, memory size, interconnect type, compute capability, you name it.</p><p><strong>ResourceClaim</strong> — This is where you declare what you actually need. It references a DeviceClass and can add additional constraints. Think of it like a purchase order — “I need 2 devices from the A100 class, and they must each have at least 80GB of memory.”</p><pre>apiVersion: resource.k8s.io/v1<br>kind: ResourceClaim<br>metadata:<br>name: training-gpus<br>spec:<br>devices:<br>requests:<br>- name: gpu<br>exactly:<br>deviceClassName: gpu-a100<br>allocationMode: ExactCount<br>count: 2<br>selectors:<br>- cel:<br>expression: &gt;<br>device.capacity[&quot;gpu.nvidia.com&quot;].memory &gt;= quantity(&quot;80Gi&quot;)</pre><p><strong>ResourceClaimTemplate</strong>— Same idea as a ResourceClaim, but the claim is created dynamically for each pod that uses it. Use ResourceClaim when multiple pods can share the same device (like an inference endpoint), and ResourceClaimTemplate when each pod needs its own exclusive allocation (like distributed training).</p><p>And here’s how a pod actually uses all of this:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>name: ml-training-job<br>spec:<br>resourceClaims:<br>- name: gpu-claim<br>resourceClaimTemplateName: training-gpu-template<br>containers:<br>- name: trainer<br>image: my-training-job:latest<br>resources:<br>claims:<br>- name: gpu-claim</pre><p>Compare this to the old `nvidia.com/gpu: “2”` approach. It’s night and day. You’re not requesting anonymous numbers anymore — you’re making structured, attributable, verifiable claims about exactly what hardware your workload needs.</p><h4>What’s New in v1.36 — The Features That Make DRA Production-Ready</h4><p>DRA has been around since v1.26 in alpha, and the core went GA in v1.34. But Kubernetes v1.36, landing on April 22, 2026, brings three features that take DRA from “works in theory” to “ready for your production GPU fleet.”</p><p><strong>Device Taints and Tolerations (Beta — Enabled by Default)</strong></p><p>If you’ve used node taints and tolerations, this will feel instantly familiar. Devices can now be **<strong>tainted</strong>**, which prevents workloads from being scheduled on them unless the workload explicitly tolerates that taint.</p><p>Why does this matter? Picture this: you have a fleet of 64 GPUs across 8 nodes. One GPU starts throwing ECC memory errors. In the old world, you’d have to either take the entire node offline or play scheduling roulette hoping no pod lands on the bad GPU. With device taints, the DRA driver can mark that specific GPU as tainted — say, `hardware-error:NoSchedule` — and the scheduler will simply skip it. Every other GPU on that node keeps serving workloads normally.</p><p>Cluster admins can also create <strong>DeviceTaintRules</strong> to apply taints based on device attributes. Want to prevent any non-production workload from being scheduled on your expensive H100s? Taint them and only let production workloads tolerate it:</p><pre>apiVersion: resource.k8s.io/v1beta1<br>kind: DeviceTaintRule<br>metadata:<br>name: reserve-h100s<br>spec:<br>deviceSelector:<br>cel:<br>expression: &gt;<br>device.driver == &quot;gpu.nvidia.com&quot; &amp;&amp;<br>device.attributes[&quot;gpu.nvidia.com&quot;].model == &quot;H100&quot;<br>taint:<br>key: gpu-tier<br>value: premium<br>effect: NoSchedule</pre><p>Then only pods with the matching toleration get scheduled on those devices. Clean, declarative, Kubernetes-native. No more duct tape.</p><p><strong>Prioritized Lists (Beta — Enabled by Default)</strong></p><p>Here’s one that ML engineers are going to love. Say your training job can run on A100s, H100s, or even L40S GPUs — but you’d <em>*prefer*</em> H100s if they’re available, fall back to A100s if not, and use L40S as a last resort. Before v1.36, you’d have to either hard-code one choice or build your own retry logic.</p><p>With prioritized lists, you give the scheduler an ordered preference, and it picks the best available option:</p><pre>apiVersion: resource.k8s.io/v1<br>kind: ResourceClaim<br>metadata:<br>name: flexible-gpu<br>spec:<br>devices:<br>requests:<br>- name: gpu<br>firstAvailable:<br>- deviceClassName: gpu-h100<br>allocationMode: ExactCount<br>count: 1<br>- deviceClassName: gpu-a100<br>allocationMode: ExactCount<br>count: 1<br>- deviceClassName: gpu-l40s<br>allocationMode: ExactCount<br>count: 1</pre><p>The scheduler walks the list top to bottom, tries to allocate H100s first, falls back to A100s, then L40S. Your job gets the best hardware available without manual intervention. This is huge for multi-tenant clusters where GPU types are mixed and availability fluctuates.</p><p><strong>Admin Access Controls (Stable)</strong></p><p>In multi-tenant clusters, you probably don’t want every developer to be able to claim any GPU they want. The admin access feature gates DRA access so that only users in namespaces with the `resource.k8s.io/admin-access: “true”` label can create ResourceClaims with admin-level access. This keeps your expensive hardware allocations under control without building custom admission webhooks.</p><h4>GPU Partitioning with MIG — DRA’s Killer Feature</h4><p>One of the most practical applications of DRA is **Multi-Instance GPU (MIG) support**. NVIDIA’s A100 and H100 GPUs can be partitioned into up to seven isolated instances, each with dedicated compute cores and memory. Before DRA, using MIG with Kubernetes was… messy. You’d need specialized device plugins, custom configurations, and a lot of hope.</p><p>With DRA, a single physical GPU can be exposed as multiple devices, each with its own attributes. A DRA driver can advertise MIG partitions as individual devices with specific memory and compute capabilities. Your inference workload that only needs 10GB of GPU memory? It gets a MIG slice. Your training job that needs the full 80GB? It gets the whole thing. All managed through the same API.</p><p>This means you can run seven lightweight inference containers on a single A100, each with guaranteed isolation. At current cloud GPU pricing, that’s the difference between spending $30/hour on seven full GPUs and $4.30/hour on one partitioned GPU. Do that math across a fleet and you’ll see why FinOps teams are very excited about DRA.</p><h4>Things That Might Bite You</h4><p>A few gotchas to watch out for:</p><p><strong>Driver maturity varies.</strong> NVIDIA’s DRA driver is the most mature, but drivers for other hardware (FPGAs, custom accelerators, SmartNICs) are at different stages. Check your vendor’s support status before going all-in.</p><p><strong>Scheduler complexity </strong>DRA adds more work for the scheduler. In clusters with hundreds of devices and complex selection criteria, you might see slightly longer scheduling times. Profile your cluster before and after migration.</p><p><strong>ResourceClaim lifecycle</strong> Unlike regular resource requests that are tied to pod lifecycle, ResourceClaims can exist independently. This is powerful (you can pre-allocate devices) but also means you need to clean them up. Watch for orphaned claims eating up your hardware budget.</p><p><strong>CEL expression gotchas </strong>The attribute filtering uses CEL (Common Expression Language), which is powerful but has a learning curve. A typo in your expression won’t give you an error at apply time — it’ll just silently fail to match any device. Test your expressions with `kubectl get resourceslices -o yaml` to see what attributes your driver actually exposes.</p><h4>Wrapping Up</h4><p>Dynamic Resource Allocation represents the biggest shift in how Kubernetes handles hardware since the original device plugin framework. We’ve gone from “give me N things” to “give me exactly what I need, with fallback options, protected by access controls, and the ability to take devices offline without downtime.”</p><p>If this post helped you wrap your head around DRA, smash that clap button — it helps other K8s engineers find articles like this. Running GPU workloads on Kubernetes and have war stories? Drop them in the comments. I want to hear about your wildest scheduling nightmares.</p><p>And hey, at least now when your ML engineer asks for “the good GPUs,” you can actually give them a YAML file instead of a Slack message that says “try again in 10 minutes.” Follow <a href="https://medium.com/u/54b070394829">The kube guy</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43195bc568ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In-Place Pod Resize — Explained]]></title>
            <link>https://medium.com/@thekubeguy/in-place-pod-resize-explained-f5ada5211a36?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5ada5211a36</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 05:22:36 GMT</pubDate>
            <atom:updated>2026-04-01T05:22:36.843Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Your pod is running out of memory, but you don’t want to kill it and start over. What if you could just… give it more? Now you can.</em></p><p>Have you ever been in a situation where your application pod is chugging along, doing its job perfectly well, but it’s slowly eating up all the memory you gave it? Maybe it’s a Java app whose heap is growing, or an ML inference service handling a traffic spike. Your instinct says “just bump the resources” — but in Kubernetes, that used to mean deleting and recreating the entire pod. Connections dropped. State lost. Users unhappy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWCxlJpb1iHHuYvcIfQ9YA.png" /><figcaption>Image by Author</figcaption></figure><p>Well, not anymore. <strong>In-Place Pod Resize</strong> has officially graduated to stable (GA) in Kubernetes v1.35, and it changes the game for how we think about resource management. Let’s break this down from scratch.</p><h4>The Old Way — Delete, Recreate, Pray</h4><p>Before in-place resize, if you wanted to change the CPU or memory allocated to a running pod, here’s what happened. You’d update your Deployment spec with new resource requests and limits. Kubernetes would see the change, terminate the old pod, and spin up a new one with the updated resources. The rolling update strategy would help, sure, but your specific pod — with its warm caches, its established connections, its in-progress work — was gone.</p><p>Think of it like this: you’re at a restaurant, your plate is too small for the amount of food you ordered, and instead of the waiter bringing you a bigger plate, they take your food away, throw it out, and bring you a completely new meal on a bigger plate. You’d be… frustrated.</p><p>For stateless web servers behind a load balancer, this was annoying but manageable. For **stateful workloads** — databases, ML training jobs, long-running batch processes — it was painful. And for workloads where startup time is measured in minutes (looking at you, JVM applications), it was downright expensive.</p><h4>What Is In-Place Pod Resize?</h4><p>**In-Place Pod Resize** (tracked as KEP-1287) makes CPU and memory requests and limits **mutable** on a running pod. That’s it. That’s the magic. You can now change how much CPU and memory a container is allowed to use without restarting it.</p><p>Under the hood, when you update a pod’s resource spec, the kubelet on the node receives the change and adjusts the container’s **cgroups** — the Linux kernel mechanism that controls resource allocation. No new container. No new pod. No downtime. The container keeps running with its new resource boundaries.</p><p>It’s like upgrading your internet plan and having the speed change take effect immediately, without the technician making you unplug your router.</p><p>Here’s what a simple pod with resize policies looks like:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: my-app<br>spec:<br>  containers:<br>  - name: app<br>    image: my-app:latest<br>    resources:<br>      requests:<br>        cpu: &quot;250m&quot;<br>        memory: &quot;256Mi&quot;<br>      limits:<br>        cpu: &quot;500m&quot;<br>        memory: &quot;512Mi&quot;<br>    resizePolicy:<br>    - resourceName: cpu<br>      restartPolicy: NotRequired<br>    - resourceName: memory<br>      restartPolicy: RestartContainer</pre><p>Notice the <strong>resizePolicy</strong> field — this is new and important. It tells Kubernetes how to handle each resource type when a resize happens. We’ll dig into this more in a moment.</p><h4>How It Actually Works — The Flow</h4><p>Let’s walk through what happens when you resize a pod, step by step. Understanding the flow helps you debug when things don’t go as expected… and trust me, knowing this will save you one day.</p><p><strong>Step 1</strong>: You issue a patch command to update the pod’s resources. You use kubectl with the ` — subresource resize` flag:</p><pre>kubectl patch pod my-app - subresource resize \<br>- patch &#39;{&quot;spec&quot;:{&quot;containers&quot;:[{&quot;name&quot;:&quot;app&quot;,&quot;resources&quot;:{&quot;requests&quot;:{&quot;cpu&quot;:&quot;500m&quot;,&quot;memory&quot;:&quot;512Mi&quot;},&quot;limits&quot;:{&quot;cpu&quot;:&quot;1&quot;,&quot;memory&quot;:&quot;1Gi&quot;}}}]}}&#39;</pre><p><strong>Step 2</strong>: The API server validates the change and updates the pod spec. It records your desired resources in `spec.containers[].resources` and keeps the currently allocated resources in `status.containerStatuses[].allocatedResources`.</p><p><strong>Step 3</strong>: The kubelet on the node picks up the change. It checks: “Do I have enough resources on this node to satisfy this request?” If yes, it proceeds. If not, the pod enters a <strong>PodResizePending</strong> state.</p><p><strong>Step 4:</strong> If the resources are available, kubelet talks to the container runtime (containerd, CRI-O) via the CRI interface and asks it to update the container’s cgroups. The CPU shares and memory limits get adjusted in real time.</p><p><strong>Step 5: </strong>The pod’s status is updated to reflect the new allocated resources. Done. Your container never stopped running.</p><p>The whole process typically takes just a few seconds. Your application doesn’t even notice — it just suddenly has more (or less) room to work with.</p><h4>Resize Policies — NotRequired vs. RestartContainer</h4><p>Here’s a nuance that trips people up. Not all resource changes can be applied seamlessly. This is where the <strong>resizePolicy</strong> comes in.</p><p><strong>NotRequired</strong> means “just change the cgroup limits, don’t restart the container.” This works beautifully for <strong>CPU</strong>— the Linux kernel can adjust CPU shares and quotas on the fly without any process being aware of it. Your application keeps running and suddenly has more (or fewer) CPU cycles available.</p><p><strong>RestartContainer</strong> means “restart the container to apply this change.” This is often necessary for <strong>memory</strong> — and here’s why. When you increase a container’s memory limit, the kernel can adjust the cgroup limit without issues. But some applications allocate memory pools at startup (think JVM heap, database buffer pools, Go’s memory arena). They won’t automatically use the new memory unless they’re restarted. And when you <strong><em>decrease</em></strong> memory, it gets even trickier — the kernel can’t easily reclaim memory that a process is actively using.</p><p>Here’s the key insight: even when <strong>restartPolicy</strong> is set to <strong>RestartContainer,</strong> Kubernetes restarts <strong><em>just that one container</em> </strong>inside the pod. The pod itself stays alive. Other containers in the pod (your sidecars, your log collectors) keep running undisturbed. It’s like rebooting one appliance in your kitchen, not the whole house.</p><p>If you don’t specify a resize policy at all, the default is `NotRequired` for both CPU and memory. That’s fine for many workloads, but be aware that your app might not actually <em>*use*</em> newly available memory until it’s restarted.</p><h4>The Resize Status — What to Watch For</h4><p>When you resize a pod, the status tells you exactly what’s happening. Here’s what to look for:</p><pre>kubectl get pod my-app -o yaml</pre><p>Check the `status.resize` field. It can be one of several values:</p><p><strong>Proposed </strong>— the resize has been requested but kubelet hasn’t acted on it yet.</p><p><strong>InProgress</strong> — kubelet has accepted the request and is applying the changes. Resources have been allocated but the container runtime hasn’t finished adjusting everything.</p><p><strong>Deferred</strong> — kubelet can’t satisfy the request right now (maybe the node is too busy), but it might be possible later. Think of it as “I’ll get to it when resources free up.”</p><p><strong>Infeasible</strong>— the resize is impossible on this node. Maybe you’re asking for 64GB of memory on a node that only has 32GB total. This won’t resolve itself, and you’ll need to either reduce your ask or move the pod to a bigger node.</p><p>No status field at all means the resize completed successfully and everything is in its desired state.</p><h4>The VPA Connection — Automatic In-Place Scaling</h4><p>Now here’s where things get really exciting. Manual resizing is great for one-off situations, but the real power comes when you combine in-place resize with the <strong>Vertical Pod Autoscaler (VPA).</strong></p><p>VPA has a new update mode called <strong>InPlaceOrRecreate</strong> (graduated to beta alongside the in-place resize feature). When you set your VPA to this mode, it monitors your pod’s actual resource usage, calculates optimal requests and limits, and applies those changes <strong><em>in-place</em></strong> whenever possible. If in-place update isn’t possible (say, the node doesn’t have enough resources), it falls back to the traditional recreate approach.</p><p>Here’s what a VPA configuration looks like with in-place resize:</p><pre>apiVersion: autoscaling.k8s.io/v1<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: my-app-vpa<br>spec:<br>  targetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: my-app<br>  updatePolicy:<br>    updateMode: &quot;InPlaceOrRecreate&quot;<br>  resourcePolicy:<br>    containerPolicies:<br>    - containerName: app<br>      minAllowed:<br>        cpu: &quot;100m&quot;<br>        memory: &quot;128Mi&quot;<br>      maxAllowed:<br>        cpu: &quot;2&quot;<br>        memory: &quot;4Gi&quot;</pre><h4>Things to Know</h4><p>Before you go resizing everything in sight, there are a few things to keep in mind.</p><p>First, <strong>you can only resize up to what the node can provide</strong>. If your node has 4 CPUs and other pods are using 3.5, you can’t resize your pod to request 2 CPUs. The scheduler won’t move your pod to a bigger node — that’s still the delete-and-recreate flow.</p><p>Second, <strong>LimitRange and ResourceQuota still apply</strong>. If your namespace has a ResourceQuota that caps total memory at 8Gi and you’re already using 7Gi, your resize request to add another 2Gi will be rejected by the API server.</p><p>Third, <strong>not all container runtimes handle memory decreases gracefully</strong>. Reducing memory limits on a running container can cause OOMKills if the container is already using more memory than the new limit. Be very careful when scaling down memory — it’s much safer to scale up than down.</p><p>Finally, <strong>pod-level resources</strong> (a newer feature graduating alongside in-place resize) follow the same pattern. If you’re using pod-level resource specifications instead of container-level, the resize mechanism works the same way but applies at the pod boundary.</p><h4>Wrapping Up</h4><p>In-Place Pod Resize might not be the flashiest feature Kubernetes has shipped, but it’s one of the most practical. It addresses a pain point that every Kubernetes operator has felt: the frustration of needing to restart pods just to change resource allocations.</p><p>- In-Place Pod Resize is <strong>GA in Kubernetes v1.35</strong> — it’s production-ready and here to stay.</p><p>- CPU and memory requests and limits are now <strong>mutable</strong> on running pods.</p><p>- The **resizePolicy** field controls whether a container needs to restart for each resource type.</p><p>- Resize status fields (Proposed, InProgress, Deferred, Infeasible) tell you exactly what’s happening.</p><p>- <strong>VPA’s InPlaceOrRecreate mode</strong> enables automatic, disruption-free vertical scaling.</p><p>- This is a game-changer for stateful workloads, JVM apps, ML inference, and batch jobs.</p><p>If you found this useful, drop a clap (or fifty!) and share it with your team. Got a war story about pod restarts ruining your day? Drop it in the comments — I’d love to hear it. And if you want more Kubernetes deep-dives that don’t read like documentation, follow <a href="https://medium.com/u/54b070394829">The kube guy</a> for weekly breakdowns that actually make sense.</p><p>May your pods always have the resources they need… without ever having to restart for them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5ada5211a36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How etcd Works in Kubernetes?]]></title>
            <link>https://medium.com/@thekubeguy/how-etcd-works-in-kubernetes-4db19f544b82?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/4db19f544b82</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 05:41:00 GMT</pubDate>
            <atom:updated>2026-04-02T05:57:39.115Z</atom:updated>
            <content:encoded><![CDATA[<p>You’ve probably heard the word “etcd” thrown around in Kubernetes conversations. Maybe you’ve seen it listed as a control plane component. Maybe you’ve accidentally broken a cluster and someone pointed at etcd and said “that’s the problem.” But if you’ve never really understood <em>what</em> it is or <em>how</em> it works… you’re not alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gjYreXwuIiJbgge2pcr2VA.png" /><figcaption>Image by Author</figcaption></figure><p>Here’s the thing — etcd is the most important component in your entire Kubernetes cluster. Not the API server. Not the scheduler. Not the controller manager. etcd. Because without it, none of those other components have anything to work with. It is the single source of truth for everything in your cluster — every Pod, every Deployment, every Secret, every ConfigMap… all of it lives in etcd.</p><p>So let’s break this down from scratch. By the end of this post, you’ll understand what etcd actually is, how it stores and protects your data using a clever consensus algorithm, how it talks to the rest of Kubernetes, and — critically — why you absolutely must be backing it up. Let’s dive in.</p><h3>What is etcd?</h3><p>Before we get into how etcd works in Kubernetes, let’s understand what etcd <em>is</em> at its core.</p><p>etcd is an open-source, distributed key-value store. Think of it like a highly reliable dictionary — you put something in with a key, and you can get it back with that same key. Simple concept, but the magic is in what happens when you distribute that dictionary across multiple servers and ensure they all stay perfectly in sync, even when nodes crash or networks hiccup.</p><p>The name itself gives you a clue about its purpose. In Linux, the /etc directory is where all system configuration files live — it&#39;s the &quot;phone book&quot; of the OS. The &quot;d&quot; at the end stands for <strong>distributed</strong>. So etcd is essentially a distributed /etc — a shared configuration store for a cluster of machines, not just one.</p><p>Kubernetes has used etcd as its backing store since day one. Every time you create a resource — say a Deployment with 3 replicas — that specification gets written into etcd. Every time a Pod gets scheduled to a node, that state gets written into etcd. Every time you create a Secret, every time you update a ConfigMap, every time a node joins the cluster… etcd is the thing that remembers all of it.</p><p>Here’s what you can actually find in there. If you exec into the etcd pod and poke around with etcdctl, you&#39;ll see keys that look like this:</p><pre>ETCDCTL_API=3 etcdctl get / — prefix — keys-only</pre><p>The output looks something like:</p><pre>/registry/deployments/default/my-app<br>/registry/pods/kube-system/coredns-abc123<br>/registry/secrets/default/my-secret<br>/registry/services/specs/default/my-service<br>/registry/configmaps/default/app-config</pre><p>Every Kubernetes object has its own key under /registry. The value stored at each key is the full YAML/JSON spec of that object, serialized into binary using Protocol Buffers. So when you run kubectl get pod my-pod -o yaml, Kubernetes is reading from this exact store and deserializing it back for you.</p><figure><a href="https://fourninecloud.com/services/kubernetes/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysttX6noRF8OIDt7w9KaHg.png" /></a></figure><h3>How etcd Stays in Sync</h3><p>Now here’s where things get really interesting. You can’t just run a single etcd instance in production and call it a day. What happens if that node crashes? Your entire cluster state is gone. That’s why etcd runs as a <strong>cluster</strong> — typically 3 or 5 nodes — and uses a consensus algorithm called <strong>Raft</strong> to keep all of them perfectly in sync.</p><p>Think of Raft like a democracy with very strict rules. Every etcd cluster elects one node as the <strong>Leader</strong>. The other nodes are called <strong>Followers</strong>. The Leader is the one in charge — all write requests go through it. Followers just replicate whatever the Leader says and are ready to step up if the Leader goes down.</p><p>Here’s how a write operation actually flows through the Raft algorithm:</p><ol><li>A write request arrives at the Leader (say, “create a new Deployment”)</li><li>The Leader appends this to its own log and sends it to all Followers</li><li>Followers write it to their logs and send back an acknowledgment</li><li>Once the <strong>majority</strong> of nodes (a quorum) have acknowledged it, the Leader commits the entry</li><li>The Leader then notifies all Followers that the entry is committed and they apply it too</li><li>Finally, the response is sent back to the client — success!</li></ol><p>The key word there is <strong>majority</strong>. A 3-node cluster needs 2 nodes to agree. A 5-node cluster needs 3. This is why etcd clusters always run with an <strong>odd number of members</strong> — not because even numbers are unlucky, but because odd numbers always guarantee a clear majority during a split. A 4-node cluster loses the same number of nodes as a 3-node cluster (one), but with an extra node adding complexity. An odd-sized cluster always gets you the most fault tolerance for the fewest nodes.</p><p><strong>What happens when the Leader fails?</strong> This is where Raft’s heartbeat mechanism kicks in. The Leader constantly sends heartbeats to its Followers to say “hey, I’m still alive.” If a Follower doesn’t hear a heartbeat for a certain period (the election timeout, usually 150–300ms), it assumes the Leader is dead, promotes itself to a <strong>Candidate</strong>, and starts an election. It votes for itself and asks the other nodes to vote for it. First Candidate to get a majority vote becomes the new Leader. Simple, fast, and automatic.</p><p>This whole dance happens without any human intervention. Your cluster might be in the middle of scheduling a Pod when the etcd leader crashes — and within milliseconds, a new leader is elected and operations continue. This is distributed systems engineering at its finest.</p><h4>How etcd Talks to Kubernetes?</h4><p>Here’s a key design choice in Kubernetes that’s easy to miss: <strong>only the API server talks to etcd</strong>. Not the scheduler. Not the controller manager. Not kubelet. Only kube-apiserver.</p><p>Every component in the Kubernetes control plane goes through the API server to read or write cluster state. The API server is the bouncer — it handles authentication, authorization (RBAC), admission webhooks, and validation. Only after all of that does it persist the data to etcd or return what’s stored there.</p><p>This architecture means etcd doesn’t need to worry about who’s asking or whether they’re allowed to — that’s the API server’s problem. etcd just stores and retrieves data as instructed.</p><p>The API server connects to etcd over gRPC on port <strong>2379</strong> (client traffic). etcd nodes talk to each other on port <strong>2380</strong> (peer traffic). The entire connection is secured with mutual TLS, so certificates are everywhere — and you’ll need them when you run etcdctl commands manually.</p><p>Here’s another superpower etcd gives to Kubernetes: the <strong>Watch API</strong>. The Kubernetes control loop works something like this — “what is the desired state? what is the actual state? make them match.” But how do components <em>know</em> when something has changed without constantly polling?</p><p>etcd’s watch mechanism. Controllers and the scheduler can “watch” specific keys or key prefixes in etcd and get notified the instant something changes. When you kubectl apply a new Deployment, the API server writes it to etcd, and etcd instantly notifies the controller manager via a watch event. The deployment controller wakes up, sees the new spec, and starts creating ReplicaSets and Pods. No polling. Just events. That&#39;s the magic of the reconciliation loop.</p><h4>What’s Actually Stored in etcd (And What’s Not)?</h4><p>Let’s be clear about what lives in etcd and what doesn’t.</p><p><strong>In etcd:</strong></p><ul><li>All Kubernetes API objects: Pods, Deployments, Services, ConfigMaps, Secrets, Namespaces, RBAC roles, ServiceAccounts, PersistentVolumes, CustomResourceDefinitions — everything you create via kubectl or the API</li><li>Cluster configuration and metadata</li><li>Leader election locks used by components like the scheduler and controller manager (yes, they use etcd for distributed locking too!)</li></ul><p><strong>Not in etcd:</strong></p><ul><li>Container logs (those live on the node)</li><li>Metrics (those go to your monitoring stack)</li><li>The actual container images</li><li>Any data your application writes to disk</li></ul><p>This distinction matters enormously. etcd is not a general-purpose database for your applications. It’s specifically designed for storing small amounts of configuration data with high consistency requirements. The recommended etcd database size is to keep it under 8 GB — and you can check your current size with</p><pre>ETCDCTL_API=3 etcdctl endpoint status \<br> — endpoints=https://127.0.0.1:2379 \<br> — cacert=/etc/kubernetes/pki/etcd/ca.crt \<br> — cert=/etc/kubernetes/pki/etcd/server.crt \<br> — key=/etc/kubernetes/pki/etcd/server.key \<br> — write-out=table</pre><p>If your etcd is growing large, you might need to run defragmentation — that’s what etcdctl defrag is for. Think of it like vacuuming your database. Over time, as keys get updated and deleted, etcd accumulates fragmented space. Defrag reclaims it. Schedule it during low-traffic windows.</p><h4>Backup and Restore — Please Don’t Skip This Section</h4><p>If etcd is the brain of your cluster… you need to back it up. I can’t stress this enough. etcd is the <strong>only stateful component</strong> in the Kubernetes control plane. If you back up etcd, you can restore your entire cluster state — every Deployment, every Service, every Secret. If you lose etcd without a backup, those resources are gone. Your nodes are still there but they won’t know what to run anymore.</p><p><strong>Taking a snapshot</strong> is simple and non-disruptive:</p><pre>ETCDCTL_API=3 etcdctl snapshot save /opt/backup/etcd-$(date +%Y%m%d).db \<br> — endpoints=https://127.0.0.1:2379 \<br> — cacert=/etc/kubernetes/pki/etcd/ca.crt \<br> — cert=/etc/kubernetes/pki/etcd/server.crt \<br> — key=/etc/kubernetes/pki/etcd/server.key</pre><p>A quick note on tools: there’s etcdctl (for live cluster operations over the network) and etcdutl (for direct file operations, introduced as etcdctl replacements for certain commands in etcd v3.5+). For restores specifically, etcdutl is now the recommended approach since etcdctl snapshot restore is deprecated in v3.5.x:</p><pre># Modern restore using etcdutl (recommended for etcd 3.5+)<br>etcdutl — data-dir /var/lib/etcd-restored snapshot restore /opt/backup/etcd-20260323.db</pre><h4>etcd in Production — A Few Things to Know</h4><p>A few hard-won truths if you’re running etcd in production:</p><p><strong>Disk I/O is everything.</strong> etcd is extremely sensitive to disk latency. It uses a write-ahead log (WAL) and has to fsync to disk on every committed write. If your disk is slow (looking at you, shared virtualized disks), you&#39;ll start seeing election timeouts and cluster instability. Use SSDs. Dedicated SSDs if you can. etcd on a spinning disk in prod is a bad time.</p><p><strong>CPU and memory matter less than you think.</strong> etcd is not CPU-intensive. It’s I/O-intensive. Throwing more cores at it won’t help if your disk is the bottleneck.</p><p><strong>5 nodes for critical production.</strong> A 3-node cluster can tolerate 1 node failure. A 5-node cluster can tolerate 2 node failures simultaneously. For production clusters you can’t afford to lose, run 5 etcd members.</p><p><strong>Defrag regularly.</strong> Run etcdctl defrag on your etcd members, but be careful — defragging the leader causes a brief leadership transfer. Do followers first, leader last.</p><p><strong>Monitor these metrics:</strong></p><ul><li>etcd_server_leader_changes_seen_total — frequent leadership changes mean instability</li><li>etcd_disk_wal_fsync_duration_seconds — if p99 is above 10ms, your disk is struggling</li><li>etcd_mvcc_db_total_size_in_bytes — watch for database size creep</li></ul><h4>Wrapping Up</h4><p>etcd might not be the component you interact with every day, but it’s the one quietly holding everything together. Every kubectl command ultimately reads from or writes to etcd. Every controller loop depends on it. Every leader election goes through it.</p><p>Here’s what to take away from this:</p><ul><li>etcd is a distributed key-value store that acts as the single source of truth for all Kubernetes cluster state</li><li>It uses the <strong>Raft consensus algorithm</strong> to stay consistent across its cluster members — always deploy it with an odd number of nodes (3 or 5)</li><li>Only the <strong>API server</strong> communicates with etcd; all other components go through the API server</li><li>etcd’s <strong>Watch API</strong> powers the Kubernetes reconciliation loop without polling</li><li>Disk I/O is etcd’s biggest performance bottleneck — SSDs are not optional in production</li><li><strong>Back it up. Regularly. And test your restores.</strong></li></ul><p><strong>Remember: In Kubernetes, if the API server is the front desk, etcd is the filing cabinet — lose the cabinet, and nothing in the office makes sense anymore.</strong></p><p><em>And may your etcd clusters always have quorum. It’s always DNS… unless it’s etcd.</em></p><p><strong>Follow </strong><a href="https://medium.com/u/54b070394829"><strong>The kube guy</strong></a> for more deep dives into the components that actually make Kubernetes tick.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4db19f544b82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How DNS Actually Works in Kubernetes? — CoreDNS Explained]]></title>
            <link>https://medium.com/@thekubeguy/how-dns-actually-works-in-kubernetes-coredns-explained-f3fb14885f77?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3fb14885f77</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 12:19:20 GMT</pubDate>
            <atom:updated>2026-03-16T12:19:20.560Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever wondered how your pods find each other inside a Kubernetes cluster? Let’s break down DNS from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*191UbPopvTlPoEmf0ZkC7g.png" /><figcaption>Image by Author</figcaption></figure><p>Kubernetes has a lot of moving parts… and somehow they all manage to find each other. Your frontend pod talks to the backend service, the backend talks to the database, and everything just works. But have you ever stopped and wondered — how does a pod actually know where to send traffic?</p><p>The answer is DNS — specifically, CoreDNS. It’s the unsung hero running quietly in your kube-system namespace, making sure every service name resolves to the right IP. And trust me, once you understand how it works under the hood, debugging those random “name resolution failed” errors becomes a whole lot easier.</p><p>Let’s break this down from scratch.</p><h3>What is CoreDNS?</h3><p>CoreDNS is a flexible, plugin-based DNS server that became the default cluster DNS in Kubernetes starting from version 1.13. Before CoreDNS, Kubernetes used kube-dns — but CoreDNS replaced it because it’s lighter, more configurable, and easier to extend.</p><p>Think of CoreDNS as the phone book of your Kubernetes cluster. When a pod wants to talk to a service, it doesn’t memorize IP addresses — it just asks CoreDNS, “Hey, where is my-service?” and CoreDNS responds with the right IP. Simple as that.</p><p>CoreDNS runs as a Deployment in the kube-system namespace with usually two replicas for high availability. It also has a Service (typically called kube-dns) with a stable ClusterIP that all pods in the cluster use as their DNS resolver.</p><h4>How DNS Resolution Works Inside a Pod?</h4><p>Here’s what happens step by step when a pod tries to resolve a service name:</p><ol><li>Your pod makes a DNS query — say it wants to reach my-service in the same namespace. It sends the query to the DNS resolver configured in <strong><em>/etc/resolv.conf </em></strong>inside the pod.</li></ol><p>2. The resolver points to the CoreDNS Service IP (the kube-dns ClusterIP). This IP is automatically injected into every pod’s /etc/resolv.conf by the kubelet when the pod starts.</p><p>3. CoreDNS receives the query and checks — is this an internal Kubernetes name or an external domain? If it’s internal (like my-service.default.svc.cluster.local), CoreDNS uses the Kubernetes API to look up the Service and return its ClusterIP.</p><p>4. If the query is for an external domain (like google.com), CoreDNS forwards it to an upstream DNS server — usually whatever DNS your cluster nodes are configured to use.</p><p>5. The resolved IP is returned to the pod, and the connection is made. All of this happens in milliseconds.</p><h4>The Kubernetes DNS Naming Convention</h4><p>This is important. Kubernetes follows a very specific DNS naming pattern for services:</p><blockquote><strong><em>service-name.namespace.svc.cluster.local</em></strong></blockquote><p>Let’s break that down:</p><p><strong>service-name</strong> — the name of the Kubernetes Service you created.</p><p><strong>namespace</strong> — the namespace the Service lives in (like default, kube-system, etc.).</p><p><strong>svc</strong> — indicates this is a Service resource.</p><p><strong>cluster.local</strong> — the default cluster domain (configurable, but rarely changed).</p><p>So if you have a service called backend-api in the production namespace, its full DNS name would be backend-api.production.svc.cluster.local. But here’s the cool part — if your pod is in the same namespace, you can just use backend-api and Kubernetes will figure out the rest. More on that in a bit.</p><h4>What’s Inside /etc/resolv.conf?</h4><p>This is where the magic starts. If you exec into any pod and check /etc/resolv.conf, you’ll see something like this:</p><pre>nameserver 10.96.0.10<br>search default.svc.cluster.local svc.cluster.local cluster.local<br>options ndots:5</pre><p>Let’s decode each line:</p><p><strong>nameserver 10.96.0.10</strong> — This is the ClusterIP of the kube-dns Service, which points to your CoreDNS pods. Every DNS query from this pod goes here first.</p><p><strong>search domains</strong> — These are the suffixes Kubernetes appends to short names. So when you just say my-service, Kubernetes will try my-service.default.svc.cluster.local first, then my-service.svc.cluster.local, and so on until it finds a match. This is why you can use short names inside the same namespace!</p><p><strong>ndots:5 </strong>— This is the sneaky one. It tells the resolver: “If the name has fewer than 5 dots in it, try appending search domains before treating it as an absolute name.” And this is where things can get interesting (and sometimes slow). More on this next.</p><h4>The ndots Problem — Why Your DNS Might Be Slow</h4><p>This is one of the most common Kubernetes DNS gotchas, and most people don’t even know it’s happening.</p><p>With ndots:5, when your pod tries to resolve an external domain like api.stripe.com (which has only 2 dots), the resolver thinks: “Hmm, that’s less than 5 dots, let me try the search domains first.” So before it even attempts the real domain, it fires off queries like:</p><pre>api.stripe.com.default.svc.cluster.local → NXDOMAIN<br>api.stripe.com.svc.cluster.local → NXDOMAIN<br>api.stripe.com.cluster.local → NXDOMAIN<br>api.stripe.com → Finally resolves!</pre><p>That’s 4 unnecessary DNS queries for every single external connection. Multiply that by hundreds of pods making thousands of requests, and you’ve got yourself a DNS bottleneck.</p><p><strong>How to fix it?</strong></p><p><strong>Option 1</strong>: Add a trailing dot to external domains in your app config (api.stripe.com.) — the dot tells the resolver it’s already a fully qualified domain name, skip the search domains.</p><p><strong>Option 2: </strong>Set ndots to a lower value (like 2) in your pod spec’s dnsConfig. This reduces unnecessary lookups but still works for internal service discovery.</p><h4><strong>The Corefile — CoreDNS Configuration</strong></h4><p>CoreDNS is configured through a file called the Corefile, which lives inside a ConfigMap named coredns in the kube-system namespace. You can check it out by running:</p><pre>kubectl get configmap coredns -n kube-system -o yaml</pre><p>A typical Corefile looks something like this:</p><pre>.:53 {<br>    errors<br>    health<br>    ready<br>    kubernetes cluster.local in-addr.arpa ip6.arpa {<br>        pods insecure<br>        fallthrough in-addr.arpa ip6.arpa<br>    }<br>    prometheus :9153<br>    forward . /etc/resolv.conf<br>    cache 30<br>    loop<br>    reload<br>    loadbalance<br>}</pre><p>DNS issues in Kubernetes are more common than you’d think. Here are some quick commands to help you troubleshoot:</p><p>Check if CoreDNS pods are running:</p><p>Here’s what the key plugins do:</p><p><strong>kubernetes </strong>— This is the main plugin that watches the Kubernetes API and resolves service/pod names within the cluster domain (cluster.local).</p><p><strong>forward </strong>— Anything that doesn’t match the cluster domain gets forwarded to upstream DNS servers (usually from /etc/resolv.conf on the node).</p><p><strong>cache</strong> — Caches DNS responses for 30 seconds to reduce load on CoreDNS and upstream servers.</p><p><strong>health and ready </strong>— Expose health check endpoints so Kubernetes knows CoreDNS is alive and ready.</p><p><strong>loop</strong> — Detects infinite DNS forwarding loops and kills CoreDNS if one is found. This is a lifesaver.</p><p><strong>prometheus</strong> — Exposes DNS metrics on port 9153 for monitoring with Prometheus.</p><p><strong>Debugging DNS Issues — Quick Commands</strong></p><pre>kubectl get pods -n kube-system -l k8s-app=kube-dns</pre><p><strong>Test DNS from inside a pod:</strong></p><pre>kubectl run dnsutils — image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 — restart=Never — sleep 3600<br>kubectl exec -it dnsutils - nslookup kubernetes.default</pre><p>If <strong>nslookup</strong> returns the <strong>ClusterIP</strong> of the kubernetes service, your DNS is working fine. If it times out or returns <strong>SERVFAIL</strong>, something’s off — check CoreDNS pod logs next.</p><p><strong>Check CoreDNS logs:</strong></p><pre>kubectl logs -n kube-system -l k8s-app=kube-dns</pre><p>Look for errors like REFUSED, SERVFAIL, or i/o timeout — these usually point to upstream DNS issues or network policy problems blocking DNS traffic on port 53.</p><h4>Wrapping Up</h4><p><strong>Remember: In Kubernetes, if networking is the highway, DNS is the GPS. Without it, nothing knows where to go.</strong></p><p>If you found this helpful, give it a clap and follow <a href="https://medium.com/u/54b070394829">The kube guy</a> for more Kubernetes deep dives. Got questions about CoreDNS or DNS issues you’ve faced? Drop them in the comments — I’d love to hear your war stories!</p><p>Until next time, happy resolving! And remember — if your pods can’t find each other, it’s probably DNS. It’s always DNS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3fb14885f77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes v1.36 — Sneak Peek]]></title>
            <link>https://medium.com/google-cloud/kubernetes-v1-36-sneak-peek-7c5422ffd841?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c5422ffd841</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 16:11:34 GMT</pubDate>
            <atom:updated>2026-03-17T01:47:24.002Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes v1.36 is around the corner, and here’s what’s cooking behind the scenes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*YmpcvTuT0EEMVsehSQsfiA.png" /><figcaption>v1.36 — Sneak Peek</figcaption></figure><p>After covering Timbernetes (v1.35) in my last blog, I’ve been keeping a close eye on what the Kubernetes community is brewing for the next release. Kubernetes v1.36 is officially scheduled for April 22, 2026 — and while the final feature list isn’t locked in yet, we already have enough signal from the enhancements freeze to know this release is going to be a meaningful one.</p><p>Let’s break down the highlights you should be watching.</p><h4>HPA Scale-to-Zero — Finally Enabled by Default</h4><p>This one’s been a long time coming. The HPAScaleToZero feature gate has been sitting in alpha since Kubernetes v1.16 — yes, you read that right. In v1.36, this feature is expected to be enabled by default, which means the Horizontal Pod Autoscaler can finally scale your workloads down to zero replicas.</p><p><strong>Why This Matters</strong></p><p>If you’ve ever operated staging or test environments, you know the pain of paying for idle pods that sit around doing nothing. With minReplicas: 0 now viable in HPA, you can let workloads scale down completely when there’s no traffic — and scale back up when demand returns. Think of it like turning off the lights when nobody’s in the room.</p><p>This is particularly useful for intermittent workloads, batch processing pipelines, and cost-optimized clusters where every unused pod is wasted money.</p><h4>2. Ephemeral Service Account Tokens for Image Pulls</h4><p>Security-conscious teams will love this one. Traditionally, pulling images from private registries meant storing long-lived credentials as Kubernetes Secrets. The problem? Those secrets can be leaked, shared across namespaces accidentally, or sit around forever without rotation.</p><p>Kubernetes v1.36 is pushing forward the ability to use ephemeral Kubernetes Service Account (KSA) tokens for authenticating image pulls. Instead of relying on static secrets, the kubelet can now use short-lived, auto-rotating tokens that are scoped to the pod’s identity. Think of it like replacing your house key with a smart lock that generates a new code every time you need to enter — much safer.</p><p>This builds on KEP-2535 that started gaining traction in v1.33 and went beta in v1.34. By v1.36, expect this to be a mature, production-ready approach to image pull authentication.</p><h4>3. Smarter HPA Pod Selection</h4><p>Here’s a subtle but impactful one. The HPA currently gathers metrics from all pods matching a selector — including pods that might be owned by a different controller or in an unexpected state. This can throw off scaling decisions badly.</p><p>A new alpha feature in v1.36 refines this logic to ensure the HPA only considers metrics from pods directly managed by the target workload. If you’ve ever debugged a situation where your HPA was scaling based on metrics from stale or orphaned pods, you’ll appreciate this fix. It’s like a GPS that finally stops recalculating because of cars on a different highway.</p><h4>4. IPVS Mode in kube-proxy — Deprecated and on Its Way Out</h4><p>If you’re running kube-proxy in IPVS mode, heads up — IPVS mode was deprecated in v1.35 and is expected to be removed in v1.36. The Kubernetes community has been gradually pushing towards nftables and eBPF-based alternatives for service proxying.</p><p>If your clusters rely on IPVS, now is the time to start planning your migration to iptables (nftables backend) or explore Cilium/eBPF-based proxying. Don’t wait until the removal hits you in a patch upgrade.</p><h4>5. Gateway API and Ingress NGINX Retirement</h4><p>This is a big ecosystem shift happening around the v1.36 timeframe. The upstream Kubernetes project is retiring Ingress NGINX — the controller that practically everyone learned Kubernetes ingress with. If you’re still relying on Ingress NGINX, it’s time to evaluate alternatives like Gateway API or third-party ingress controllers.</p><p><strong>Gateway API is the Future</strong></p><p>Gateway API offers richer routing capabilities, cross-namespace references, and a more standardized approach to traffic management. Think of it as Kubernetes finally getting a proper highway system instead of the patchwork of country roads we’ve been navigating with Ingress resources. Start experimenting with Gateway API now if you haven’t already.</p><h4>6. Improved Windows Container Support and Containerd Changes</h4><p>Kubernetes v1.36 continues to improve Windows container support, making it easier to run mixed Linux/Windows workloads in the same cluster. If your organization runs .NET applications alongside microservices, this is great news.</p><p><strong>Containerd Heads-Up</strong></p><p>v1.36 is expected to be the last Kubernetes release that supports older versions of containerd. If you’re still running containerd 1.6.x, it’s time to upgrade. The Kubernetes project is aligning with containerd 2.x, so make sure your container runtime is up to date before the next release catches you off guard.</p><h4>Wrapping Up</h4><p>Kubernetes v1.36 might not have dropped yet, but the roadmap is packed with meaningful changes. From HPA finally learning to scale to zero, to ephemeral tokens tightening image pull security, to the retirement of Ingress NGINX — this release is all about maturing the platform and cleaning up technical debt that’s been lingering for years.</p><p>The bottom line: Start testing these features in your staging clusters now. When v1.36 officially lands on April 22, 2026, you’ll be ready to upgrade without breaking a sweat.</p><p>Until next time, may your pods never enter CrashLoopBackOff and may your YAML indentation always be correct on the first try. (Let’s be honest, it never is.)</p><p>Follow <a href="https://medium.com/u/54b070394829">The kube guy</a> for more Kubernetes content, tips, and the occasional bad container joke.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c5422ffd841" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-v1-36-sneak-peek-7c5422ffd841">Kubernetes v1.36 — Sneak Peek</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are leases in Kubernetes?]]></title>
            <link>https://faun.pub/what-are-leases-in-kubernetes-caf60be752bf?source=rss-54b070394829------2</link>
            <guid isPermaLink="false">https://medium.com/p/caf60be752bf</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[The kube guy]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 07:37:00 GMT</pubDate>
            <atom:updated>2025-12-31T16:58:30.474Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes has a lot of moving parts… and somehow they all manage to stay in sync. Ever wondered <strong>how Kubernetes components coordinate without stepping on each other’s toes?</strong> One of the answers is a simple but powerful resource: <strong>Leases</strong>.<br>Let’s break this down from scratch, with real-world examples, so you can understand how Leases keep your cluster stable and reliable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ybXPVA_fZ8jYfbuwS_Pxg.png" /><figcaption>What are leases in Kubernetes?</figcaption></figure><h4>What is a Lease in Kubernetes?</h4><p>A <strong>Lease</strong> is a special Kubernetes object that represents <strong>temporary ownership of a resource</strong>. It’s like saying, “Hey, I’m using this thing right now — don’t touch it until I’m done.”</p><p><strong>Technical definition:</strong><br> A Lease is an API object in the coordination.k8s.io/v1 API group. It stores a record with fields like:</p><ul><li><strong>holderIdentity</strong> (Who holds the lease)</li><li><strong>leaseDurationSeconds</strong> (How long the lease is valid)</li><li><strong>renewTime</strong> (When it was last renewed)</li></ul><p>Here’s what a Lease object looks like in YAML:</p><pre>apiVersion: coordination.k8s.io/v1<br>kind: Lease<br>metadata:<br>  name: my-lease<br>  namespace: default<br>spec:<br>  holderIdentity: &quot;worker-node-2&quot;<br>  leaseDurationSeconds: 15</pre><p>This tells Kubernetes:<br> <strong>“worker-node-2 holds the lease for 15 seconds. If it doesn’t renew by then, it’s up for grabs.”</strong></p><h4>A Simple Real-Life Example</h4><p>Imagine you run a <strong>busy car workshop</strong>. You have <strong>one diagnostic machine</strong>, but multiple mechanics need it.<br> So you keep a <strong>sign-up board</strong> where a mechanic writes their name and the time they start using it.</p><ul><li>Mechanic writes name → <strong>Lease acquired</strong></li><li>They keep updating the time → <strong>Lease renewed</strong></li><li>They leave without updating → <strong>Lease expires</strong>, and someone else takes over</li></ul><p>If this board didn’t exist?<br> Two mechanics might plug in the machine at the same time → chaos and burnt wiring.<br> <strong>This is Kubernetes without Leases.</strong></p><h4>Why Leases Matter?</h4><p>Leases play a big role in preventing conflicts and downtime. They ensure only <strong>one component controls something at a time</strong>. Without Leases, Kubernetes would be full of race conditions and conflicts.</p><h4>Why you should care as a Kubernetes engineer</h4><p>Leases help with:</p><ul><li>High availability</li><li>Failover and leader election</li><li>Node health tracking</li><li>Avoiding duplicate work/processing</li><li>Better reliability in distributed systems</li></ul><p>If you want to understand how Kubernetes autopilots itself, you need to understand Leases.</p><h4>Where Are Leases Used in Kubernetes?</h4><h4>1. Node Heartbeats</h4><p>Every node needs to prove it’s alive. Previously, this was done via NodeStatus updates, but that caused load on the API server.</p><p><strong>Leases changed that.</strong><br> Now nodes update a Lease object under the hood to show they’re alive.</p><ul><li>Less API load</li><li>Faster node failure detection</li><li>Better scalability</li></ul><h4>2. Leader Election</h4><p>When multiple replicas of the same controller run, <strong>only one should act as leader</strong>.</p><p>Used in:</p><ul><li>Kube-controller-manager</li><li>Kube-scheduler</li><li>Custom controllers &amp; Operators</li></ul><p>Example:<br> Three controllers try to manage jobs. Only one gets the Lease and becomes leader. If it crashes, the Lease expires, and another takes over.<br> No humans needed. No drama.</p><h4>3. Custom Applications</h4><p>You can use Leases too.</p><p>If you build a microservice where only one instance should perform a job (like sending emails or processing payments), use Leases.</p><h4>Creating and Checking Leases</h4><h4>Create a Lease</h4><pre>kubectl apply -f my-lease.yaml</pre><h4>Check if a Lease is still active</h4><pre>kubectl get lease my-lease -n default -o yaml</pre><h4>Force Remove a Lease</h4><p>Useful in stuck leader situations:</p><pre>kubectl delete lease my-lease -n default</pre><p>⚠️⚠️ Do this only if you are sure the owner is dead. ⚠️⚠️<br> Deleting active leases is like pulling the plug on someone still using the diagnostic machine. Expect swearing.</p><h4>Practical Real-Life Use Case</h4><p>Let’s say you have a <strong>batch job</strong> that sends invoices every night. You run <strong>3 instances</strong> for redundancy.</p><p>Use a Lease so only ONE instance sends invoices, while others wait.<br> If the active one goes down, another takes the Lease and continues.<br> No duplicate invoices. No angry clients.</p><p>If you know Leases, you’re one step closer to understanding Kubernetes internals — and that’s where the real magic is.</p><p>Leases silently manage coordination, failover, and heartbeat tracking across Kubernetes.<br> If you understand Leases, you understand <strong>why Kubernetes doesn’t explode under pressure</strong>.</p><p>If you liked this and want more Kubernetes wisdom that doesn’t sound like it was written by a YAML-powered robot…<br> <strong>Follow </strong><a href="https://medium.com/u/54b070394829"><strong>The kube guy</strong></a><strong> </strong>before your cluster decides to failover <em>you</em> using a Lease and elect someone else as the leader. 🚀</p><p>Because let’s be real — even your Pods probably follow me by now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*QDkVCfJNmECTBq8g.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN.dev() &amp; get similar stories in your inbox each week for free!</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caf60be752bf" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/what-are-leases-in-kubernetes-caf60be752bf">What are leases in Kubernetes?</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>