<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by muntashir islam on Medium]]></title>
        <description><![CDATA[Stories by muntashir islam on Medium]]></description>
        <link>https://medium.com/@islam-muntashir?source=rss-16debb4ed02b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*WiO-ygsOVIGRMHQsui4VJQ.jpeg</url>
            <title>Stories by muntashir islam on Medium</title>
            <link>https://medium.com/@islam-muntashir?source=rss-16debb4ed02b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:54:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@islam-muntashir/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Every Port Was Taken]]></title>
            <link>https://islam-muntashir.medium.com/every-port-was-taken-c1770af567cf?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1770af567cf</guid>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[agones]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 03:18:29 GMT</pubDate>
            <atom:updated>2026-09-13T03:19:57.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPjfPNokZHN8-ObSNTjSBQ.png" /></figure><p><strong><em>Cluster Field Reports series. Case 02/10</em></strong></p><p><em>Rolling out an autoscaler should be routine. This one collided with a game-server API, a load balancer controller, and the cluster’s own network overlay for three times, on three different ports, before it would run.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*m-UJJQmPMEKn3pvMBGorSw.png" /></figure><p>Three collisions, three environments, one moving target</p><p>KEDA is, in the abstract, a small thing: an operator, an admission webhook, a metrics adapter. We packaged it as a Helm chart, pointed ArgoCD at dev, stage, and production, and watched the admission webhook crash-loop identically on all three.</p><pre>kubectl logs deploy/keda-admission-webhooks --previous <br>ERROR setup unable to start admission webhooks {&quot;error&quot;: &quot;error listening on :8081: bind: address already in use&quot;}</pre><p>Nothing in KEDA’s own chart was wrong. The problem was where it landed. This cluster already had opinions about which ports meant what. we just didn’t know it until KEDA asked for the same ones.</p><p>Agones runs an extension API server for game-server allocation, and it reserves two host ports outright 8081 for its own webhook, 8080 for plain HTTP. KEDA&#39;s chart defaults its webhook health probe to 8081 and, less obviously, always opens a metrics listener on 8080 whether or not metrics are enabled - the flag is unconditional in the chart&#39;s own template. Any node running both pods at once refused the second bind outright.</p><p>We moved both: 8091 and 8092. Simple enough - except the same unconditional-flag pattern showed up twice more before the day was out.</p><p>KEDA’s webhook itself listens on 9443 by default. So does aws-load-balancer-controller&#39;s own admission webhook - a completely unrelated component, already running on two of stage&#39;s handful of untainted nodes. The third was occupied by KEDA&#39;s own previous, still-crash-looping pod, waiting on the exact same port.</p><blockquote><em>Three nodes could take the new pod. All three already had something holding the port it needed including the old version of itself.</em></blockquote><pre>kubectl describe pod keda-admission-webhooks-... <br>0/15 nodes are available: 12 node(s) had untolerated taint(s), 3 node(s) didn&#39;t have free ports for the requested pod ports.</pre><p>This wasn’t a crash loop anymore, it was a genuine scheduling deadlock, and it would have stayed that way indefinitely without a real fix. We moved KEDA’s webhook to 9446, letting its own Service follow automatically since the chart derives the target port from the same value.</p><p>The last failure mode didn’t look like a port collision at all as every pod came up healthy, and the Application still refused to report Healthy. The external metrics APIService was failing its own discovery check:</p><pre>kubectl get apiservice v1beta1.external.metrics.k8s.io <br>message: failing or missing response from https://10.2.5.38:6443/...: Address is not allowed</pre><p>The control plane simply can’t route to a pod IP sitting on Cilium’s own overlay CIDR which is the same structural gap that had already forced hostNetwork onto cert-manager&#39;s and the load balancer controller&#39;s webhooks, long before KEDA ever showed up. The metrics adapter needed the same treatment, and its own unconditional metrics flag needed the same port shift as the webhook before it.</p><p>Nothing about KEDA’s own code changed across any of this. Every fix lived in nine lines of Helm values and all three ports moved, one flag flipped to true - because the actual constraint was never KEDA. It was everything already living on these nodes, quietly, that KEDA had no way of knowing about until it tried to stand next to it.</p><p><em>All three environments, zero port conflicts remaining.Case 02 of 10 Cluster Field Reports series</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1770af567cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cluster Field Reports series — Case 01/10]]></title>
            <link>https://islam-muntashir.medium.com/the-job-nobody-inherited-895a153ef3cf?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/895a153ef3cf</guid>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cilium]]></category>
            <category><![CDATA[falco]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 21:03:18 GMT</pubDate>
            <atom:updated>2026-09-13T03:03:19.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cZOxXRXJmF58yT09wJkMJg.png" /></figure><p><em>Cluster Field Reports series — Case 01/10</em></p><p><em>One Falco pod stuck for thirty-one hours turned out to be the first symptom of a switch flipped a week earlier and a job that quietly stopped being anyone’s responsibility the moment it was flipped.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*QuDqHlt8rMX6CF7lyFJw3g.png" /></figure><blockquote><em>Nobody removed the plugins. The thing that used to install them just stopped running.</em></blockquote><p>The ticket was almost apologetic in how small it looked: a single Falco pod, falco-hvjkb, stuck in a non-terminal state for over a day. Falco pods restart, get stuck, resolve themselves - this looked like exactly one of those, until the first two obvious fixes didn&#39;t touch it at all.</p><p>The first theory was the boring one: bad instance. Cordon it, drain it, terminate it, let the ASG launch a replacement. The replacement came up on fresh hardware and hit the identical wall within minutes and which meant this wasn’t a hardware fluke at all.</p><p>The second theory was almost as boring: some first-boot race condition, resolved by giving the node a second try. A soft reboot can’t re-run whatever only happens once at first boot, and it didn’t — the node came back exactly as broken as it left.</p><pre>kubectl describe pod falco-hvjkb<br><br>Warning FailedCreatePodSandBox failed to find plugin &quot;portmap&quot; in path [/opt/cni/bin]</pre><p>Not a crash. Not a timeout. A file that was supposed to exist on every node, and didn’t on a fresh instance, after a reboot, every single time.</p><p>Debugging a missing file usually means looking forward to find what should have written it, and why didn’t it run. The break here came from asking the opposite question: what used to write this file, before anyone thought about it at all?</p><blockquote><em>What if AWS says to use the VPC CNI instead?</em></blockquote><p>The VPC CNI addon, when it’s enabled, is what actually populates /opt/cni/bin with the standard plugin binaries - bridge, host-local, portmap, all of it - on every node, as a side effect nobody thinks about because it&#39;s always just been there. Cilium&#39;s own installer only ever writes one file: its own cilium-cni binary. It was never responsible for the rest, because it never needed to be, while something else was quietly doing that job underneath it.</p><pre>git log - follow -p - eks.tf<br>- enable_vpc_cni_addon = true <br>+ enable_vpc_cni_addon = false # Cilium is now the cluster&#39;s exclusive CNI ...</pre><p>The flag had been flipped about a week earlier, as the last step of a Cilium migration that had every reason to look complete. Every already-running node still had its old plugin binaries sitting untouched on disk. The gap only ever showed up on a node that hadn’t existed yet when the flag changed — which is exactly why it took a week to notice, and why it looked random until it very much wasn’t.</p><h3>Why Nothing Else Noticed</h3><p>The fleet’s own health checks stayed green the entire time, for a reason that made sense the moment we saw it: kube-proxy runs with hostNetwork: true, which skips the CNI entirely and shares the node&#39;s own network namespace outright. Every hostNetwork pod on every broken node kept running perfectly. Only a pod that actually needed the CNI to build it a sandbox - like Falco - would ever hit the missing plugin at all.</p><p>The honest fix wasn’t reverting to the VPC CNI the Cilium migration was real, deliberate work, already finished everywhere else. It was making sure the one job VPC CNI used to do silently had an explicit owner again: a small DaemonSet, running with hostNetwork: true so it can repair a node before the CNI is even working, that downloads the standard plugin binaries and installs them itself. Shipped through ArgoCD to all three environments, it turned an invisible, week-old assumption into a piece of infrastructure someone could actually point to.</p><p>That DaemonSet did its job well for weeks. It’s also, as it happens, exactly where the next incident in this series begins.</p><p><strong>Durable fix shipped:</strong> <a href="https://github.com/containernetworking/plugins">cni-plugins-installer</a>, running on every node, all three environments.</p><p><em>Case 01 of 10— Cluster Field Reports series</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=895a153ef3cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hard Way: MIG GPU Slicing on AWS EKS with Karpenter and the p5.4xlarge instance]]></title>
            <link>https://islam-muntashir.medium.com/the-hard-way-mig-gpu-slicing-on-aws-eks-with-karpenter-and-the-p5-4xlarge-instance-d0f906cb455e?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f906cb455e</guid>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[mig]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Sun, 05 Jul 2026 02:05:57 GMT</pubDate>
            <atom:updated>2026-07-05T02:05:57.646Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OpnmbapNOmusNXiJl1Codg.jpeg" /></figure><p><em>A real-world troubleshooting journey through NVIDIA MIG, GPU Operator, node taints, disk pressure, and everything in between.</em></p><h3>Introduction</h3><p>GPU instances are expensive. A single p5.4xlarge on AWS packing one NVIDIA H100 80GB runs on EC2 Capacity Blocks that aren&#39;t cheap. If your workload doesn&#39;t saturate a full H100, you&#39;re leaving a lot of money on the table. That&#39;s where <strong>MIG (Multi-Instance GPU)</strong> comes in.</p><p>Introduced by NVIDIA with the Ampere architecture and refined in Hopper (H100), MIG allows a single physical GPU to be partitioned into multiple isolated instances, each with its own dedicated streaming multiprocessors (SMs), L2 cache, memory controllers, and DRAM. Unlike GPU time-slicing which just context-switches between workloads, MIG creates true hardware-level isolation. Each slice gets a guaranteed, predictable chunk of compute and memory. No noisy neighbors.</p><p>This post documents a real end-to-end setup of MIG on AWS EKS using Karpenter for node provisioning and the NVIDIA GPU Operator to manage the GPU stack. We hit every possible obstacle along the way wrong tolerations, missing labels, Bottlerocket configs on AL2023, disk pressure, CNI failures and this is the honest account of how we solved each one.</p><h3>The Target Setup</h3><ul><li><strong>Instance</strong>: p5.4xlarge (1× NVIDIA H100 80GB, 16 vCPUs, 256GB RAM)</li><li><strong>OS</strong>: Amazon Linux 2023 (AL2023)</li><li><strong>Node provisioner</strong>: Karpenter v1.x</li><li><strong>GPU stack</strong>: NVIDIA GPU Operator v25.10.1</li><li><strong>MIG profile</strong>: all-1g.10gbwhich is 7 slices of ~10GB each per GPU</li><li><strong>Goal</strong>: Run 7 independent GPU workloads on a single H100</li></ul><h3>H100 80GB MIG Profiles</h3><p>Profile Memory Max Instances 1g.10gb ~10 GB 7 1g.20gb ~20 GB 4 2g.20gb ~20 GB 3 3g.40gb ~40 GB 2 4g.40gb ~40 GB 1 7g.80gb ~80 GB 1</p><h3>Step 1: Setting Up the Karpenter NodePool and EC2NodeClass</h3><p>The p5.4xlarge was launched by AWS in August 2025 as a single-GPU variant of the P5 family, available via EC2 Capacity Blocks for ML. You cannot get it directly but to use this instance you must have capacity block configured. We booked this 1 week early and it was commissioned in us-wast-1f region. Our NodePool targets it explicitly:</p><pre>apiVersion: karpenter.sh/v1<br>kind: NodePool<br>metadata:<br>  name: mig-gpu-workers<br>spec:<br>  template:<br>    metadata:<br>      labels:<br>        node-type: mig-gpu<br>        nvidia.com/mig.config: all-1g.10gb   # Auto-labels for MIG Manager<br>    spec:<br>      nodeClassRef:<br>        apiVersion: karpenter.k8s.aws/v1<br>        kind: EC2NodeClass<br>        name: mig-gpu-nodeclass<br>      taints:<br>        - effect: NoSchedule<br>          key: nvidia.com/gpu<br>          value: &quot;true&quot;<br>        - effect: NoSchedule<br>          key: nvidia.com/mig<br>          value: &quot;true&quot;<br>        - effect: NoSchedule<br>          key: workload<br>          value: general<br>      requirements:<br>        - key: node.kubernetes.io/instance-type<br>          operator: In<br>          values: [&quot;p5.4xlarge&quot;]<br>        - key: karpenter.sh/capacity-type<br>          operator: In<br>          values: [&quot;reserved&quot;, &quot;on-demand&quot;] # Must mentioned Reserved as we did pre reserved<br>        - key: kubernetes.io/arch<br>          operator: In<br>          values: [&quot;amd64&quot;]<br>        - key: topology.kubernetes.io/zone<br>          operator: In<br>          values: [&quot;us-east-1f&quot;]      # We have capacity only in this AZ<br><br>  limits:<br>    cpu: &quot;64&quot;<br>    memory: 512Gi<br>  disruption:<br>    consolidationPolicy: WhenEmpty<br>    consolidateAfter: 30s</pre><p>And the EC2NodeClass — <strong>AL2023, single root volume, no userData</strong>:</p><pre>apiVersion: karpenter.k8s.aws/v1<br>kind: EC2NodeClass<br>metadata:<br>  name: mig-gpu-nodeclass<br>spec:<br>  amiSelectorTerms:<br>    - alias: al2023@latest<br>  role: k8s-poc-x<br>  securityGroupSelectorTerms:<br>    - tags:<br>        karpenter.sh/discovery: k8s-poc-x<br>  subnetSelectorTerms:<br>    - tags:<br>        karpenter.sh/discovery: k8s-poc-x<br>  capacityReservationSelectorTerms:<br>    - id: cr-0f1ebcbxxxxxx   # this is capacity reservation ID. <br>  tags:<br>    karpenter.sh/discovery: k8s-poc-x<br>  blockDeviceMappings:<br>    - deviceName: /dev/xvda<br>      ebs:<br>        volumeSize: 300Gi      # Large enough for GPU container images<br>        volumeType: gp3<br>        encrypted: true<br>        deleteOnTermination: true</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nd_19DYufVPcPpNiX1aR0g.png" /><figcaption>p5.4xlarge node pop up in aws</figcaption></figure><blockquote><strong><em>Gotcha #1</em></strong><em>: Do NOT add a second </em><em>/dev/xvdb volume for AL2023. Unlike Bottlerocket (which uses two volumes one for OS, one for containers), AL2023 uses a single root volume </em><em>/dev/xvda. Adding </em><em>/dev/xvdb doesn&#39;t give you extra container storage.</em></blockquote><blockquote><strong><em>Gotcha #2</em></strong><em>: Do NOT put Bottlerocket TOML userData in an AL2023 nodeclass. The </em><em>[settings.kubelet-device-plugins.nvidia.mig.profile] config block is Bottlerocket-specific and is silently ignored on AL2023.</em></blockquote><h3>Step 2: Don’t Install nvidia-device-plugin Separately</h3><p>Our first instinct was to install the NVIDIA device plugin via Helm:</p><pre>helm install nvidia-device-plugin nvdp/nvidia-device-plugin \<br>  --namespace gpu-operator \<br>  --set migStrategy=single</pre><p>This immediately failed:</p><pre>Error: INSTALLATION FAILED: ServiceAccount &quot;node-feature-discovery&quot; in namespace<br>&quot;gpu-operator&quot; exists and cannot be imported into the current release: invalid<br>ownership metadata; annotation validation error: key &quot;meta.helm.sh/release-name&quot;<br>must equal &quot;nvidia-device-plugin&quot;: current value is &quot;gpu-operator-1770960549&quot;</pre><p>The GPU Operator was already installed and owns that namespace. We are working with other GPU instances so it was obvious to use gpu operator. The GPU Operator includes the device plugin internally you should never install them both. Configure MIG through the GPU Operator instead.</p><h3>Step 3: GPU Operator — The Taint Problem</h3><p>With GPU Operator v25.10.1 installed, we checked the DaemonSet status:</p><pre>kubectl get daemonsets -n gpu-operator</pre><p>Everything showed 0 desired pods. The culprit: our GPU node had three NoSchedule taints, and the GPU Operator&#39;s DaemonSets (including Node Feature Discovery worker) couldn&#39;t schedule on it.</p><p>The key insight was finding the correct Helm values key. We initially tried per-component tolerations later we fix this with correct value:</p><pre># ✅ This is what actually controls all DaemonSet tolerations<br>daemonsets:<br>  tolerations:<br>  - key: nvidia.com/gpu<br>    operator: Exists<br>    effect: NoSchedule</pre><p>Here’s the complete working gpu-operator-values.yaml:</p><pre>daemonsets:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br><br>operator:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br><br>node-feature-discovery:<br>  worker:<br>    tolerations:<br>      - key: mig-only<br>        operator: Exists<br>        effect: NoSchedule<br>      - key: nvidia.com/gpu<br>        operator: Exists<br>        effect: NoSchedule<br>      - key: nvidia.com/mig<br>        operator: Exists<br>        effect: NoSchedule<br>      - key: workload<br>        operator: Exists<br>        effect: NoSchedule<br><br>mig:<br>  strategy: single<br><br>migManager:<br>  enabled: true<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br><br>driver:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br><br>devicePlugin:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br><br>dcgmExporter:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule<br>  serviceMonitor:<br>    enabled: true<br><br>gfd:<br>  tolerations:<br>    - key: mig-only<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/gpu<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: nvidia.com/mig<br>      operator: Exists<br>      effect: NoSchedule<br>    - key: workload<br>      operator: Exists<br>      effect: NoSchedule</pre><p>Install with:</p><pre>helm install gpu-operator-1770960549 nvidia/gpu-operator \<br>  -n gpu-operator \<br>  --create-namespace \<br>  -f gpu-operator-values.yaml \<br>  --timeout 10m \<br>  --wait</pre><blockquote><strong><em>Gotcha #3</em></strong><em>: The </em><em>node-feature-discovery subchart requires tolerations set at </em><strong><em>both</em></strong><em> the top-level </em><em>node-feature-discovery.tolerations AND </em><em>node-feature-discovery.worker.tolerations. Setting only one of them isn&#39;t enough.</em></blockquote><h3>Step 4: Verifying the GPU Operator Chain</h3><p>The GPU Operator deploys components in a strict dependency chain:</p><pre>NFD Worker → GPU Labels Applied → Driver → Container Toolkit → Device Plugin → Validator → MIG Manager</pre><p>Verify NFD worker landed on your GPU node:</p><pre><br>➜  storageclass git:(mig-work) kubectl get pods -n gpu-operator -o wide | grep node-feature-discovery-worker<br>gpu-operator-1770960549-node-feature-discovery-worker-22j9w       1/1     Running     0          19h   10.0.35.176   ip-10-0-44-38.ec2.internal    &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-4xhn2       1/1     Running     0          26h   10.0.77.22    ip-10-0-76-230.ec2.internal   &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-7ntt8       1/1     Running     0          24h   10.0.67.149   ip-10-0-72-129.ec2.internal   &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-8nztk       1/1     Running     0          26h   10.0.36.104   ip-10-0-34-17.ec2.internal    &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-cxmf8       1/1     Running     0          19h   10.0.10.74    ip-10-0-12-176.ec2.internal   &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-jvhgj       1/1     Running     0          26h   10.0.28.106   ip-10-0-19-237.ec2.internal   &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-kwv27       1/1     Running     0          26h   10.0.67.13    ip-10-0-67-223.ec2.internal   &lt;none&gt;           &lt;none&gt;<br>gpu-operator-1770960549-node-feature-discovery-worker-tm4wt       1/1     Running     0          19h   10.0.77.232   ip-10-0-70-165.ec2.internal   &lt;none&gt;           &lt;none&gt;</pre><p>Once NFD runs on the node, it applies nvidia.com/gpu.deploy.* labels that trigger the rest of the chain:</p><pre>➜ kubectl get node ip-10-0-72-129.ec2.internal --show-labels | tr &#39;,&#39; &#39;\n&#39; | grep &quot;nvidia.com/gpu.deploy&quot;<br>nvidia.com/gpu.deploy.container-toolkit=true<br>nvidia.com/gpu.deploy.dcgm-exporter=true<br>nvidia.com/gpu.deploy.dcgm=true<br>nvidia.com/gpu.deploy.device-plugin=true<br>nvidia.com/gpu.deploy.driver=pre-installed<br>nvidia.com/gpu.deploy.gpu-feature-discovery=true<br>nvidia.com/gpu.deploy.mig-manager=true<br>nvidia.com/gpu.deploy.node-status-exporter=true<br>nvidia.com/gpu.deploy.nvsm=true<br>nvidia.com/gpu.deploy.operator-validator=true</pre><h3>Step 5: The aws-node CNI Problem</h3><p>After the GPU Operator pods started coming up, some hit FailedCreatePodSandBox errors:</p><pre>plugin type=&quot;aws-cni&quot; failed (add): rpc error: code = Unavailable desc =<br>connection error: dial tcp 127.0.0.1:50051: connect: connection refused</pre><p>The aws-node VPC CNI DaemonSet also didn&#39;t tolerate the GPU node taints, so it never scheduled on the node — meaning networking wasn&#39;t initialized when other pods tried to start. This was a <strong>timing issue</strong>, not a toleration issue. EKS managed addons (aws-node, kube-proxy) automatically tolerate all taints by default — they were scheduled on the node correctly. However, on a freshly provisioned node the CNI plugin takes a moment to fully initialize. If the GPU Operator pods start too quickly before the CNI is ready, they hit this transient networking error.</p><p><strong>This resolves itself automatically </strong>as<strong> </strong>the pods will retry and succeed once the CNI is ready. No patching required.</p><p>If you see this error persisting beyond a few minutes, check the aws-node pod status on the node:</p><pre>kubectl get pods -n kube-system -o wide | grep aws-node | grep &lt;node-name&gt;<br>kubectl logs -n kube-system &lt;aws-node-pod&gt; --tail=30</pre><blockquote><strong><em>Gotcha #4</em></strong><em>: </em><em>FailedCreatePodSandBox with CNI errors on a new node is almost always a </em><strong><em>race condition</em></strong><em> during node bootstrap, not a toleration problem. Wait 2-3 minutes before troubleshooting further — EKS core addons tolerate all taints automatically and don&#39;t need manual patching.</em></blockquote><h3>Step 6: Disk Pressure — The Hidden Killer</h3><p>After finally getting the GPU Operator running, our workload pod was evicted immediately:</p><pre>Warning  Evicted  kubelet  The node was low on resource: ephemeral-storage.<br>Threshold quantity: 2139512454, available: 1822456Ki.</pre><p>And then:</p><pre>write /var/lib/containerd/.../libcuda-injection.so: no space left on device</pre><p>The node only had ~20Gi of ephemeral storage. GPU container images are massive — the CUDA runtime alone can be 10–15Gi, and the full application image pushed us over the limit.</p><p>The fix was to increase /dev/xvda to 300Gi in the EC2NodeClass (and remove the incorrect /dev/xvdb entry):</p><pre>blockDeviceMappings:<br>  - deviceName: /dev/xvda<br>    ebs:<br>      volumeSize: 300Gi<br>      volumeType: gp3<br>      encrypted: true<br>      deleteOnTermination: true</pre><p>After recycling the node, allocatable ephemeral storage jumped to 288764809146 bytes (~288Gi).</p><h3>Step 7: Activating MIG</h3><p>With the GPU Operator fully running and the node healthy, activating MIG is a single label Although we did it from karpenter nodepool but we can easily change profile :</p><pre>kubectl label node &lt;gpu-node&gt; nvidia.com/mig.config=all-1g.10gb --overwrite</pre><p>The MIG Manager pod watches for this label, enables MIG mode on the H100 (which requires a GPU reset), destroys any existing MIG instances, and creates 7 new 1g.10gb instances. The node reboots automatically.</p><p>Watch it happen:</p><pre>kubectl logs -n gpu-operator -l app=nvidia-mig-manager -f<br><br>time=&quot;2026-03-13T21:14:39Z&quot; level=info msg=&quot;Skipping: /host/dev/nvidia-uvm-tools already exists&quot;<br>time=&quot;2026-03-13T21:14:39Z&quot; level=info msg=&quot;Creating management CDI spec&quot;<br>time=&quot;2026-03-13T21:14:39Z&quot; level=info msg=&quot;Creating management CDI spec (simplified implementation)&quot;<br>Starting kubelet.service<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Restarting all GPU clients previously shutdown on the host by restarting their systemd services&quot;<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Restarting validator pod to re-run all validations&quot;<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Restarting all GPU clients previously shutdown in Kubernetes by reenabling their component-specific nodeSelector labels&quot;<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Changing the &#39;nvidia.com/mig.config.state&#39; node label to &#39;success&#39;\n&quot;<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Successfully updated to MIG config: all-1g.10gb&quot;<br>time=&quot;2026-03-13T21:14:41Z&quot; level=info msg=&quot;Waiting for change to &#39;nvidia.com/mig.config&#39; label&quot;</pre><p>Verify the result:</p><pre>kubectl get node &lt;gpu-node&gt; -o json \<br>  | jq &#39;.status.capacity | with_entries(select(.key | contains(&quot;nvidia&quot;)))&#39;</pre><pre>{<br>  &quot;nvidia.com/gpu&quot;: &quot;7&quot;<br>}</pre><p>One H100, seven GPU slices, ready to schedule.</p><h3>Step 8: Automating MIG for Future Nodes</h3><p>To avoid manually labeling every new node Karpenter provisions, add the MIG label to the NodePool template:</p><pre>spec:<br>  template:<br>    metadata:<br>      labels:<br>        node-type: mig-gpu<br>        nvidia.com/mig.config: all-1g.10gb   # 👈 automatic MIG on every new node</pre><p>Now every p5.4xlarge node that Karpenter provisions will have MIG configured from the start.</p><h3>Step 9: Deploying Workloads</h3><p>With 7 GPU slices available, here’s a corrected deployment that targets the MIG node:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: workload-mig<br>  namespace: default<br>spec:<br>  replicas: 7   # One pod per MIG slice<br>  selector:<br>    matchLabels:<br>      app: workload-mig<br>  template:<br>    spec:<br>      nodeSelector:<br>        node-type: mig-gpu          # Must match NodePool label<br>      tolerations:<br>      - key: mig-only<br>        operator: Exists<br>        effect: NoSchedule<br>      - effect: NoSchedule<br>        key: nvidia.com/gpu<br>        operator: Exists   <br>      - effect: NoSchedule <br>        key: nvidia.com/mig<br>        operator: Exists<br>      - effect: NoSchedule <br>        key: workload<br>        operator: Exists<br>      containers:<br>        - name: workload-mig<br>          image: your-gpu-image:tag<br>          resources:<br>            limits:<br>              nvidia.com/gpu: &quot;1&quot;   # Gets one 10GB MIG slice<br>          volumeMounts:<br>            - mountPath: /opt/profiler<br>              name: profiler<br>              readOnly: true<br>      volumes:<br>        - name: profiler<br>          hostPath:<br>            path: /var/lib/profiler<br>            type: DirectoryOrCreate  # Use DirectoryOrCreate, not Directory</pre><h3>Changing MIG Profiles</h3><p>To switch to a different profile (e.g., fewer but larger slices), just update the node label:</p><pre># Scale down first<br>kubectl scale deployment example-deployment-mig --replicas=0</pre><pre># Switch to 4 slices of 20GB each<br>kubectl label node &lt;gpu-node&gt; nvidia.com/mig.config=all-1g.20gb --overwrite</pre><pre># Scale back up<br>kubectl scale deployment example-deployment-mig --replicas=4</pre><p>The MIG Manager handles everything — no GPU Operator reinstall, no node draining.</p><h3>Final Architecture</h3><pre>p5.4xlarge node (1× H100 80GB)<br>├── MIG slice 1: 1g.10gb (~10GB) → Pod 1<br>├── MIG slice 2: 1g.10gb (~10GB) → Pod 2<br>├── MIG slice 3: 1g.10gb (~10GB) → Pod 3<br>├── MIG slice 4: 1g.10gb (~10GB) → Pod 4<br>├── MIG slice 5: 1g.10gb (~10GB) → Pod 5<br>├── MIG slice 6: 1g.10gb (~10GB) → Pod 6<br>└── MIG slice 7: 1g.10gb (~10GB) → Pod 7</pre><p>7 isolated GPU workloads from one $10/hr instance. That’s the power of MIG done right.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f906cb455e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Cost Optimization at Scale: Lessons from Running Global Kubernetes Platforms]]></title>
            <link>https://islam-muntashir.medium.com/cloud-cost-optimization-at-scale-lessons-from-running-global-kubernetes-platforms-89af3f7c3f90?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/89af3f7c3f90</guid>
            <category><![CDATA[kubernetes-optimization]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[cloud-finops]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 00:32:02 GMT</pubDate>
            <atom:updated>2026-01-16T11:28:13.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vfapBE6wAITKLP4IZks_SA.png" /></figure><p>Cloud cost optimization is rarely about a single clever trick. It’s about discipline, visibility, and hundreds of small decisions made consistently over time.</p><p>A few years ago, I was part of a DevOps team managing a high-pace, globally distributed Kubernetes environment. Multiple teams across different time zones were deploying workloads daily. Alongside development and staging clusters, we were also responsible for production clusters owned by our clients but fully managed by our team.</p><p>At peak scale, we were running several hundred nodes across Azure and AWS, spanning multiple regions, environments, and workload profiles. Cloud spend was not theoretical. It was painfully real.</p><p>Our monthly compute bill alone crossed minimum limit of USD 100,000 even with a mix of Spot and On-Demand instances.</p><p>That number became the starting point of a long journey toward cloud cost optimization.</p><h3>When Cost Becomes a First-Class Problem</h3><p>Initially, cloud cost felt like “someone else’s concern.” Teams focused on reliability, delivery speed, and uptime and rightly so. But as usage grew, every new node, every idle database, and every unnecessary egress byte began to matter.</p><p>Clients started asking questions:</p><ul><li><em>Why is the bill increasing when traffic is flat?</em></li><li><em>Why do dev clusters cost almost as much as production?</em></li><li><em>Why are we paying for capacity no one is using at night?</em></li></ul><p>That’s when cost optimization stopped being a finance problem and became a DevOps responsibility.</p><h3>Strategy #1: Reserved Instances for What We Knew Would Stay</h3><p>Production workloads were predictable. These clusters ran 24/7, year-round, with stable baseline capacity.</p><p>For these environments, 3-year Reserved Instances became an obvious win.</p><ul><li>We carefully analyzed historical usage</li><li>Identified the non-negotiable baseline</li><li>Committed to long-term reservations</li></ul><p>💡 <strong>Result:</strong><br> We reduced compute costs for production by more than 50% without changing a single workload.</p><p>The key lesson here was simple:</p><blockquote>If something is guaranteed to run, don’t pay on-demand prices for it.</blockquote><h3>Strategy #2: Treat Databases as Workloads, Not Sacred Snowflakes</h3><p>Databases are often the most expensive and most overprotected resources in the cloud.</p><p>In our case, many relational databases had:</p><ul><li>Low CPU utilization</li><li>Predictable workloads</li><li>Long idle periods</li></ul><p>Fortunately, most of them were PostgreSQL.</p><p>We decided to move these databases into Kubernetes using the CloudNativePG (CNPG) operator.</p><h4>What changed?</h4><ul><li>Databases became schedulable workloads</li><li>Scaling was trivial</li><li>Operational tasks became simpler</li></ul><p>Need to troubleshoot an any issue on db level? Fix is as easy as</p><ul><li>Scale down</li><li>Scale back up</li></ul><p>No VM resizing. No waiting. No over provisioned instances “just in case.”</p><p>💡 <strong>Result:</strong><br> Lower infrastructure cost, simpler operations, and better resource utilization. Obviously we set priority on these pods to be schedule in the cluster seamlessly.</p><h3>Strategy #3: Turning Dev and Staging Clusters Off (Yes, Completely)</h3><p>One uncomfortable truth we uncovered:</p><blockquote>Most development clusters were idle most of the time.</blockquote><p>Developers worked in business hours. Clusters ran <strong>24/7</strong>.</p><p>So we changed the model.</p><ul><li>Development clusters scale to zero when no work is happening</li><li>The node pools of staging clusters scale to zero outside business hours</li><li>Everything automated via schedules and CI/CD signals</li></ul><p>No human intervention. No manual toggles.</p><p>💡 <strong>Result:</strong><br> Massive savings with zero impact on developer productivity.</p><p>If a developer needed the environment, it would come back automatically. We had a separate control plane that could communicate with the API server of these clusters and instruct all workloads to be brought back with a single click and a cup of coffee.</p><h3>Strategy #4: Spot Instances Everywhere (With Guardrails)</h3><p>Spot instances are powerful but dangerous if misused.</p><p>We leaned heavily on Spot instances for:</p><ul><li>Development</li><li>Staging batch workloads</li><li>Non-critical batch workloads</li></ul><p>But with rules:</p><ul><li>Multi-AZ node pools</li><li>Pod disruption budgets</li><li>Graceful termination handling</li></ul><p>At times, 90% of non-production workloads ran on Spot.</p><p>💡 <strong>Result:</strong><br> We paid a fraction of on-demand pricing while maintaining acceptable reliability.</p><h3>Strategy #5: Killing the NAT Gateway Bill</h3><p>One month, we noticed something alarming:<br><em> NAT Gateway costs were silently exploding.</em></p><p>In AWS especially, NAT Gateways are:</p><ul><li>Convenient</li><li>Reliable</li><li>Shockingly expensive at scale</li></ul><p>So we made a bold move.</p><p>We replaced AWS NAT Gateways for dev and staging with custom NAT instances, built using FreeBSD as we have very limited trafic went outbound on that time.</p><p>Even more importantly:</p><ul><li>We enforced policies ensuring no unnecessary outbound traffic</li><li>Ensured traffic stayed within AWS and Azure networks whenever possible</li><li>Blocked accidental internet egress at the network level</li></ul><p>💡 <strong>Result:</strong><br> Significant savings and far better control over outbound traffic.</p><h3>Strategy #6: Visibility Before Optimization</h3><p>None of these changes would have worked without clear visibility.</p><p>We invested heavily in:</p><ul><li>Cost allocation tags</li><li>Per-namespace cost tracking</li><li>Team-level cost dashboards</li></ul><p>Once teams could see their own cloud bill, behavior changed naturally.</p><ul><li>Idle resources disappeared</li><li>Requests became intentional</li><li>Architecture discussions included cost by default.</li></ul><h3>Strategy #7: Right-Sizing Batch Jobs From Kubernetes to Lambda</h3><p>As our platform matured, we started questioning a fundamental assumption:</p><blockquote>Does every workload really need to run on Kubernetes?</blockquote><p>The answer was clearly no.</p><p>We had several batch jobs that:</p><ul><li>Ran once a day or on specific events</li><li>Had very low execution time</li><li>Didn’t need persistent compute</li><li>Were not latency-sensitive</li></ul><p>Running these jobs on Kubernetes meant:</p><ul><li>Nodes stayed alive longer than needed</li><li>Spot interruptions became unnecessary risks</li><li>Compute was reserved for tasks that didn’t justify it</li></ul><p>So we moved these workloads to AWS Lambda.</p><p>Using event-driven triggers, the functions spun up only when required, completed the task, and disappeared no idle cost, no node pressure.</p><p>💡 <strong>Result:</strong><br> We eliminated entire classes of always-on workloads while improving reliability for short-lived jobs.</p><h3>Strategy #8: Data-Driven Workloads That Wait for Cheap Compute</h3><p>Not all workloads are urgent.</p><p>We had data-heavy analytical jobs that:</p><ul><li>Were compute-intensive</li><li>Had no strict execution deadlines</li><li>Could tolerate delays</li></ul><p>Instead of forcing them onto on-demand nodes, we made cost a scheduling constraint.</p><p>These workloads were:</p><ul><li>Paused intentionally</li><li>Triggered only when Spot capacity was available</li><li>Rescheduled automatically if interrupted</li></ul><p>This approach required a mindset shift:</p><blockquote>Time flexibility is a powerful cost-optimization lever.</blockquote><p>💡 <strong>Result:</strong><br> Heavy analytics ran at a fraction of the cost, with no impact on business outcomes.</p><h3>Strategy #9: Bin-Packing Compute with a Custom Scheduler</h3><p>One of the biggest hidden costs in Kubernetes is fragmentation.</p><p>Plenty of clusters look optimal, yet still spin up new nodes because:</p><ul><li>Pods don’t fit efficiently</li><li>Default scheduling favours availability over density</li></ul><p>To fix this, we introduced a custom scheduler focused on bin-packing.</p><p>Its goals were simple:</p><ul><li>Pack workloads tightly onto existing nodes</li><li>Maximize node CPU and memory utilization</li><li>Avoid unnecessary node scale-ups</li></ul><p>This reduced:</p><ul><li>Excess Spot instance requests</li><li>Unplanned On-Demand fallbacks</li><li>Idle resources across node pools</li></ul><p>💡 <strong>Result:</strong><br> Higher utilization, fewer nodes, and better predictability. Previously We need 5 nodes to fit workload Now we optimized it to two nodes only.</p><h3>Strategy #10: Event-Driven Compute Shutdown with EventBridge</h3><p>Humans forget. Automation doesn’t.</p><p>Despite policies and best practices, developers occasionally:</p><ul><li>Left clusters running overnight</li><li>Spun up large node pools for testing</li><li>Forgot to clean up temporary environments</li></ul><p>Instead of relying on reminders, we used EventBridge extensively.</p><h3>What we automated:</h3><ul><li>Detecting idle node pools</li><li>Identifying unused compute resources</li><li>Automatically shutting them down</li><li>Notifying teams after the fact</li></ul><p>This created a safety net, not a punishment system.</p><p>💡 <strong>Result:</strong><br> Cloud hygiene became automatic, and accidental overspending disappeared.</p><h3>The Bigger Picture: Cost as a Scheduling Dimension</h3><p>Over time, cost stopped being an afterthought and became part of our platform design philosophy.</p><p>We started asking different questions:</p><ul><li><em>Does this workload need to run now?</em></li><li><em>Does it need guaranteed capacity?</em></li><li><em>Can it wait for cheaper compute?</em></li><li><em>Does it belong on Kubernetes at all?</em></li></ul><p>By treating cost as a first-class scheduling signal, we unlocked optimizations that tooling alone could never achieve.</p><h3>What This Journey Taught Me</h3><p>Cloud cost optimization is not about cutting corners it’s about engineering responsibly at scale.</p><p>Key takeaways:</p><ul><li>Cost must be treated as a non-functional requirement</li><li>Kubernetes gives incredible flexibility but only if you use it intentionally</li><li>Dev and staging environments deserve as much optimization as production</li><li>Small optimizations compound into massive savings</li></ul><h3>Final Reflection</h3><p>There was no single “big win.” Instead, there were dozens of practical, sometimes boring decisions each shaving a little cost, each improving efficiency.</p><p>Together, we transformed a six-figure monthly bill into something predictable, explainable, and optimized.</p><p>And the best part?<br>We didn’t just save money, we built a more disciplined, more resilient platform. This journey wasn’t about chasing the cheapest cloud bill.</p><p>It was about:</p><ul><li>Building intentional platforms</li><li>Matching workloads to the right execution model</li><li>Letting automation enforce discipline</li><li>Designing systems that are cost-aware by default</li></ul><p>If you’re running Kubernetes at scale and not actively optimizing for cost, you’re almost certainly paying more than you should.</p><p>From Reserved Instances and Spot fleets to Lambda, EventBridge, and custom schedulers, every decision was guided by one principle:</p><blockquote>Pay only for what you truly need and exactly when you need it.</blockquote><p>At scale, that principle makes all the difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89af3f7c3f90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time-Based Autoscaling in Kubernetes: From KEDA to a Custom Operator]]></title>
            <link>https://islam-muntashir.medium.com/time-based-autoscaling-in-kubernetes-from-keda-to-a-custom-operator-7c75306e56f8?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c75306e56f8</guid>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Thu, 15 Jan 2026 04:56:00 GMT</pubDate>
            <atom:updated>2026-01-15T04:56:00.143Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>In one of my recent projects, I worked with an e-commerce–focused company where traffic patterns were highly time-dependent. Unlike traditional 24/7 workloads, this platform experienced predictable traffic spikes during specific hours of the day, aligned with business campaigns, flash sales, and regional shopping behavior.</p><p>Several microservices in the system had heavy application initialization times:</p><ul><li>Large JVM-based services</li><li>Cold cache warming</li><li>External dependency handshakes (databases, payment gateways)</li></ul><p>Because of this, reactive autoscaling often lagged behind real demand.</p><h3>Initial Scaling Approach: CPU &amp; Memory-Based HPA</h3><p>The first approach was the standard Kubernetes Horizontal Pod Autoscaler (HPA) using CPU and memory metrics.</p><h4>Problems encountered:</h4><ul><li>Pods scaled after traffic arrived</li><li>Cold starts caused timeouts and degraded user experience</li><li>CPU spikes happened too late to prevent failures</li></ul><p>This approach worked well for steady workloads, but not for predictable, time-based spikes.</p><h3>Migration to Event-Based Scaling</h3><p>Next, the team migrated to <strong>event-driven autoscaling</strong> using a queue-based system:</p><ul><li>Incoming requests were buffered in queues</li><li>Scaling decisions were driven by queue depth</li></ul><pre>                ┌──────────────┐<br>                │   Customers  │<br>                └──────┬───────┘<br>                       │<br>                ┌──────▼───────┐<br>                │  API Gateway │<br>                └──────┬───────┘<br>                       │<br>                ┌──────▼───────┐<br>                │ Message Queue│<br>                │ (Kafka/SQS)  │<br>                └──────┬───────┘<br>                       │<br>        ┌──────────────▼──────────────┐<br>        │ KEDA                        │<br>        │ (Event-based Autoscaler)    │<br>        └──────────────┬──────────────┘<br>                       │<br>               ┌───────▼────────┐<br>               │ Microservice   │<br>               │ Deployment     │<br>               │ (Pods)         │<br>               └────────────────┘</pre><h4>Improvements:</h4><ul><li>Better alignment with workload demand</li><li>Reduced CPU metric lag</li></ul><blockquote>But the core issue remained:<em>The microservices still took too long to start serving.</em></blockquote><p>Even with event-based scaling, pods were not ready when traffic arrived. What we really needed was proactive scaling.</p><h3>Introducing Time-Based Scaling with KEDA</h3><p>To solve this, we introduced KEDA (Kubernetes Event-Driven Autoscaling) with cron-based triggers.</p><h4>Business Requirement</h4><p>Scale a service:</p><ul><li>Up to 5 replicas from 9 AM — 6 PM</li><li>Down to 0 replicas outside business hours</li></ul><h4>KEDA ScaledObject Example</h4><pre>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: app-cron-scale<br>  namespace: default<br>spec:<br>  scaleTargetRef:<br>    name: my-app<br>  minReplicaCount: 0<br>  maxReplicaCount: 5<br>  triggers:<br>  - type: cron<br>    metadata:<br>      timezone: America/Toronto<br>      start: &quot;0 9 * * *&quot;<br>      end: &quot;0 18 * * *&quot;<br>      desiredReplicas: &quot;5&quot;</pre><pre>                 ┌───────────────────┐<br>                 │  Business Hours   │<br>                 │  (9 AM – 6 PM)    │<br>                 └─────────┬─────────┘<br>                           │<br>                   ┌───────▼────────┐<br>                   │ KEDA Cron      │<br>                   │ Trigger        │<br>                   └───────┬────────┘<br>                           │<br>                   ┌───────▼────────┐<br>                   │ Deployment     │<br>                   │ (Pre-scaled)   │<br>                   │ 5 Pods Ready   │<br>                   └────────────────┘</pre><p>Example microservice was like this one. It scales the deployment during certain time using KEDA.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>spec:<br>  replicas: 0   # Important: start at 0<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>      - name: app<br>        image: nginx</pre><h4>Outcome</h4><p>This approach worked extremely well:</p><ul><li>Pods were already running before traffic arrived</li><li>No cold-start latency during peak hours</li><li>Infrastructure cost reduced outside business hours</li></ul><h3>The Organizational Constraint</h3><p>Despite the technical success, we faced a platform-level challenge.</p><blockquote><em>The internal platform team was not fully comfortable introducing an external dependency like KEDA into the core infrastructure stack.</em></blockquote><p>Concerns included:</p><ul><li>Long-term maintenance ownership</li><li>Security reviews</li><li>Dependency management policies</li></ul><p>So we needed a solution that was:</p><ul><li>Kubernetes-native</li><li>Simple</li><li>Fully in-house maintained</li></ul><p>This led me to the development of:</p><p><strong><em>Timedependent Autoscaling Operator</em></strong></p><h3>Building an In-House Time-Based Autoscaling Operator</h3><p>A lightweight Kubernetes operator that scales deployments purely based on time, without CPU, memory, or external event dependencies.</p><h3>Operator Design</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xe6ZAaGPCbXWRbLYlXixtg.png" /></figure><h4><strong>Key Idea</strong></h4><p>Kubernetes does not natively support time-based scaling, but its reconciliation loop makes this pattern easy to implement.</p><h4>Custom Resource Definition (CRD)</h4><pre>apiVersion: api.muntashirislam.com/v1alpha1<br>kind: Scaler<br>metadata:<br>  name: scaler-sample<br>spec:<br>  start: 23<br>  end: 1<br>  replicas: 2<br>  deployments:<br>    - name: nginx<br>      namespace: default</pre><h3>What This Means</h3><ul><li>Between the defined time window, deployments scale to the desired replica count</li><li>Outside the window, they can scale back down (including scale-to-zero)</li></ul><h3>How this work at high level</h3><p>At a high level, the controller:</p><ol><li>Watches Scaler custom resources</li><li>Checks the current UTC hour</li><li>Compares it with the configured time window</li><li>Scales the target deployments accordingly</li><li>Requeues reconciliation every 30 seconds</li></ol><pre>                ┌────────────────────┐<br>                │   Scaler CRD       │<br>                │ (Start / End Time) │<br>                └─────────┬──────────┘<br>                          │<br>                ┌─────────▼──────────┐<br>                │ Timed Autoscaling  │<br>                │ Operator           │<br>                │ (Controller Loop)  │<br>                └─────────┬──────────┘<br>                          │<br>        ┌─────────────────▼─────────────────┐<br>        │ Kubernetes API Server             │<br>        └─────────────────┬─────────────────┘<br>                          │<br>            ┌─────────────▼─────────────┐<br>            │ Deployment(s)             │<br>            │ - Scale Up (Business Hrs) │<br>            │ - Scale Down (Off Hours)  │<br>            └───────────────────────────┘</pre><h3>Core Reconciliation Logic</h3><pre>currentHour := time.Now().UTC().Hour()<br>if currentHour &lt;= startTime {<br>    if err = scaleDeployment(scaler, r, ctx, int32(scaler.Spec.Replicas)); err != nil {<br>        return ctrl.Result{}, err<br>    }<br>}<br>return ctrl.Result{RequeueAfter: time.Duration(30 * time.Second)}, nil</pre><pre>┌─────────────────────────┐<br>│ Reconcile Loop          │<br>└───────────┬─────────────┘<br>            │<br>     ┌──────▼──────┐<br>     │ Read Scaler │<br>     │ CRD Spec    │<br>     └──────┬──────┘<br>            │<br>     ┌──────▼──────┐<br>     │ Get Current │<br>     │ Time (UTC)  │<br>     └──────┬──────┘<br>            │<br>     ┌──────▼────────────┐<br>     │ Is time in window │<br>     └──────┬────────────┘<br>            │<br>     ┌──────▼──────┐<br>     │ Scale Target│<br>     │ Deployment  │<br>     └──────┬──────┘<br>            │<br>     ┌──────▼──────┐<br>     │ Requeue     │<br>     │ (30s)       │<br>     └─────────────┘</pre><p>The operator uses:</p><ul><li>controller-runtime</li><li>Standard Kubernetes reconciliation patterns</li><li>Native RBAC and CRDs</li></ul><h3>Deployment Workflow</h3><h4>Prerequisites</h4><ul><li>Go v1.22+</li><li>Docker 19.03+</li><li>Kubernetes v1.24+</li></ul><h4>Build and Deploy</h4><pre>make docker-build docker-push IMG=&lt;registry&gt;/timely-autoscaling-operator:tag<br>make install<br>make deploy IMG=&lt;registry&gt;/timely-autoscaling-operator:tag<br>kubectl apply -k config/samples/</pre><h3>Why This Worked for Us</h3><h4>Advantages</h4><ul><li>No external autoscaling dependencies</li><li>Fully owned and customizable</li><li>Easy to reason about and debug</li><li>Perfect for predictable traffic patterns</li></ul><h4>Trade-offs</h4><ul><li>Not event-driven</li><li>Best suited for stateless workloads</li><li>Time windows must be carefully managed (DST, timezone)</li></ul><h3>Final Thoughts</h3><p>Autoscaling is not a one-size-fits-all problem.</p><ul><li>CPU/memory-based HPA works for reactive scaling</li><li>Event-based scaling works for asynchronous workloads</li><li>Time-based scaling is ideal when traffic is predictable</li></ul><p>In our case, building a simple, time-based autoscaling operator gave us the right balance between reliability, simplicity, and organizational constraints.</p><p>Sometimes, the best solution isn’t the most feature-rich tool — it’s the one that fits your platform and your team.</p><p>Github Link:<br><a href="https://github.com/muntashir-islam/timebased-autoscaling-operator">https://github.com/muntashir-islam/timebased-autoscaling-operator</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c75306e56f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Touch Safety: Automated Canary Rollbacks with Argo Rollouts & Istio]]></title>
            <link>https://islam-muntashir.medium.com/zero-touch-safety-automated-canary-rollbacks-with-argo-rollouts-istio-090cc165bc96?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/090cc165bc96</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[argo-rollouts]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 05:28:20 GMT</pubDate>
            <atom:updated>2025-12-22T06:00:11.735Z</atom:updated>
            <content:encoded><![CDATA[<p>In a modern microservices architecture, “deploying” is easy, but “deploying safely” is hard. If you push a new version that increases p95 latency from 150ms to 850ms, you want your infrastructure to realize it’s broken and roll back before your customers notice.</p><p>In this guide, we’ll configure a Kubernetes Rollout that uses Istio for traffic splitting and Prometheus metrics to trigger an automatic rollback.</p><h3>The Deployment That Stopped Itself</h3><p>In my previous SaaS company, I implemented SLO-driven automated deployments, which worked roughly like this – during a Friday afternoon release:</p><ol><li><strong>Deployment started</strong> via ArgoCD (GitOps Approach)</li><li><strong>Service mesh detected</strong> latency increasing (example p95)</li><li><strong>SLO evaluation triggered</strong>: Error budget burning faster than acceptable</li><li><strong>Automated decision</strong>: Deployment paused, rollback initiated automatically</li><li><strong>Alert sent to Slack to internal SRE channel</strong>: “Deployment stopped due to SLO violation — latency spike to on Service name demo-app”</li></ol><p>The deployment was automatically rolled back before customers even software team or on-call find out and filed tickets. The root cause? Acquiring a old api due to wrong patch applied to one of the microservice.</p><p>Total customer impact: Near zero.<br>Time to resolution: 10 minutes (automated).</p><p>Without automation, this would have been a major incident with hours of downtime.</p><p>Here a am showing this scenario using a demo application. There are lots of tools can be associated in real life setup but for simplicity I am using simple approch</p><h3>The Architecture and Initial Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWVxSBQOUck3LEgaQdYGuA.png" /></figure><p>We use three main components:</p><ol><li><strong>Argo Rollouts:</strong> Manages the deployment strategy.</li><li><strong>Istio Service Mesh:</strong> Handles the precise traffic splitting (e.g., 20% to Canary).</li><li><strong>Prometheus:</strong> Provides the “ground truth” metrics used to decide if a rollout should continue or abort.</li></ol><p>Git Repo → Argo CD → Argo Rollouts → Istio → Prometheus → Auto Rollback</p><h3>Step 1: Prepare the Demo Service</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybErZVt9E4RGoreKQ8X9Fw.png" /></figure><p>We’ll use <strong>Nicholas Jackson’s fake-service</strong> because it supports:</p><ul><li>Artificial latency</li><li>Error simulation</li><li>Simple HTTP responses</li></ul><p>This makes it perfect for testing canary deployments.</p><pre>containers:<br>- name: app<br>  image: nicholasjackson/fake-service:v0.26.2<br>  env:<br>  - name: LISTEN_ADDR<br>    value: &quot;0.0.0.0:8080&quot;<br>  - name: NAME<br>    value: &quot;demo-app&quot;<br>  - name: DELAY<br>    value: &quot;2000ms&quot;  # Introduces latency</pre><h3>Step 2: Configure Stable &amp; Canary Services</h3><p>Argo Rollouts with Istio requires <strong>two services</strong>:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo-app-stable<br>spec:<br>  selector:<br>    app: demo-app<br>  ports:<br>  - port: 80<br>    targetPort: 8080<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: demo-app-canary<br>spec:<br>  selector:<br>    app: demo-app<br>  ports:<br>  - port: 80<br>    targetPort: 8080</pre><p>Istio will route traffic between them dynamically during the canary rollout.</p><h3>Step 3: Istio VirtualService</h3><pre>apiVersion: networking.istio.io/v1beta1<br>kind: VirtualService<br>metadata:<br>  name: demo-app<br>spec:<br>  hosts:<br>  - demo-app<br>  http:<br>  - route:<br>    - destination:<br>        host: demo-app-stable<br>      weight: 100<br>    - destination:<br>        host: demo-app-canary<br>      weight: 0</pre><p>During the rollout, Argo Rollouts updates the weight field to gradually send traffic to the canary.</p><h3>Step 4: AnalysisTemplate for p95 Latency</h3><pre>apiVersion: argoproj.io/v1alpha1<br>kind: AnalysisTemplate<br>metadata:<br>  name: p95-latency<br>spec:<br>  metrics:<br>  - name: p95<br>    interval: 30s<br>    count: 5<br>    # Since metric is milliseconds, 500 = 0.5s. <br>    # Use len(result) &gt; 0 to avoid errors if Prometheus is slow to update.<br>    successCondition: len(result) &gt; 0 &amp;&amp; result[0] &lt; 500<br>    failureCondition: len(result) &gt; 0 &amp;&amp; result[0] &gt;= 500<br>    provider:<br>      prometheus:<br>        address: http://prometheus.istio-system:9090<br>        # FIXED: &quot;by (le)&quot; is now inside the sum(...)<br>        query: &gt;-<br>          histogram_quantile(0.95, <br>            sum(<br>              rate(istio_request_duration_milliseconds_bucket{<br>                reporter=&quot;destination&quot;,<br>                destination_service_name=&quot;demo-app-canary&quot;<br>              }[2m])<br>            ) by (le)<br>          ) or on() vector(0)</pre><p><strong>Key points:</strong></p><ul><li>len(result) &gt; 0 &amp;&amp; result[0] &lt; 500 ensures a scalar comparison</li><li>count: 5 measures 5 times per step</li><li>Prometheus query returns the <strong>max p95</strong> across all requests</li></ul><h3>Step 5: Canary Rollout</h3><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Rollout<br>metadata:<br>  name: demo-app<br>spec:<br>  replicas: 2<br>  progressDeadlineSeconds: 90<br>  selector:<br>    matchLabels:<br>      app: demo-app<br>  strategy:<br>    canary:<br>      stableService: demo-app-stable<br>      canaryService: demo-app-canary<br>      trafficRouting:<br>        istio:<br>          virtualService:<br>            name: demo-app<br>      steps:<br>      - setWeight: 20<br>      - pause: { duration: 1m }<br>      - analysis:<br>          templates:<br>          - templateName: p95-latency<br>      - setWeight: 100<br>  template:<br>    metadata:<br>      labels:<br>        app: demo-app<br>    spec:<br>      containers:<br>      - name: app<br>        image: nicholasjackson/fake-service:v0.26.2<br>        env:<br>        - name: LISTEN_ADDR<br>          value: &quot;0.0.0.0:8080&quot;<br>        - name: NAME<br>          value: &quot;demo-app&quot;<br>        - name: DELAY<br>          value: &quot;2000ms&quot;<br>        ports:<br>        - containerPort: 8080</pre><p>Checking for traffic patern as rollout has steps to forward 20% traffic first</p><h3>Step 6: Generate Traffic</h3><p>To see it in action, follow these steps:</p><ol><li><strong>Generate Traffic:</strong> Use a tool like fortio to hit your <strong>stable</strong> service. fortio load -qps 50 -t 5m <a href="http://demo-app-stable/">http://demo-app-stable</a></li></ol><pre>kubectl run -i --tty load-generator --rm --image=curlimages/curl --restart=Never -- \<br>  sh -c &quot;while true; do curl -s http://demo-app-canary; sleep 0.1; done&quot;</pre><p>We are using simple load generator</p><ol><li><strong>Deploy the “Broken” Version:</strong> Update your container image or set an environment variable like DELAY=2000ms.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSADp0Q23rjVebDaTJUF0Q.png" /><figcaption>Initiate Canary Release</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*a-xd_sKDnkZ5AFSnFQrZlA.png" /><figcaption>kubectl argo rollouts get rollout demo-app — watch</figcaption></figure><ol><li><strong>Watch the Magic:</strong> Argo will shift 20% of traffic to the broken pod.</li></ol><ul><li>Prometheus will record the 2000ms latency.</li><li>The Analysis will fail.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u8hdpuqbda3yfW-BfFZAJQ.png" /><figcaption>Analysis Failed</figcaption></figure><ul><li>Argo will instantly flip the Istio weight back to <strong>100% Stable</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*urQLecRSLOUCg95P2KtfQg.png" /><figcaption>ArgoCD Flip 100% traffic to stable release</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*dfZze5ujD3Mv_y0Ci_6aww.png" /><figcaption>Scaled Down the Latest release as it facing error</figcaption></figure><h4>The Challenge: Why p95 Metrics are Tricky here</h4><p>When you run a Canary, you are only sending a fraction of traffic to the new pods. A standard Prometheus query often fails because:</p><ul><li><strong>Empty Results (NaN):</strong> If no traffic hits the canary in a 30-second window, the query returns “No Data.”</li><li><strong>Syntax Errors:</strong> Aggregating histograms requires a specific by (le) grouping.</li><li><strong>Metric Units:</strong> Istio often reports in milliseconds, while your threshold might be in seconds.</li></ul><h3><strong>Key Takeways</strong></h3><p>When your infrastructure is designed to self-heal during critical production deployments, the fear of hitting ‘deploy’ disappears. By automating the response to performance regressions, you effectively neutralize 50% of potential production incidents before they escalate. For the modern SRE, this isn’t just about technical safety — it’s about reclaiming your time and peace of mind</p><p>By moving the responsibility of “checking for errors” from a human on-call to a Prometheus-backed AnalysisTemplate, we achieve more than just a faster pipeline—we achieve <strong>true peace of mind.</strong></p><p>In this setup, we’ve bridged the gap between raw infrastructure and business value:</p><ul><li><strong>The infrastructure</strong> (Istio) provides the precision.</li><li><strong>The data</strong> (Prometheus) provides the truth.</li><li><strong>The controller</strong> (Argo Rollouts) provides the judgment.</li></ul><p>Implementing zero-touch safety means that your SRE team stops fighting fires and starts building fire-proof systems. While we used a simple latency check here, the same pattern can be extended to 5xx error rates, throughput drops, or even custom business metrics.</p><p>As you move toward a fully automated production environment, remember: <strong>the best deployment is the one you didn’t have to worry about.</strong></p><p>You can findout Demo project <a href="https://github.com/muntashir-islam/kubernetes-sample-projects/tree/main/k8s-argo-istio-canary-demo"><strong><em>here</em></strong></a></p><h3>Final Pro-Tips for Production</h3><ul><li><strong>Initial Delay:</strong> Always give your canary pods 30–60 seconds to “warm up” before starting the analysis to avoid false positives.</li><li><strong>Metric Overlap:</strong> Use a [2m] window in your PromQL even if your interval is 30s. This &quot;overlap&quot; ensures the calculation always has enough data points to be accurate.</li><li><strong>Slack Integration:</strong> Connect Argo Rollouts to your Slack or MS Teams. Getting a notification that says <em>“Rollout aborted: Latency spike detected”</em> is much better than getting a paged call at 3 AM.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=090cc165bc96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bin Packing Strategies for Cost-Optimized Kubernetes Clusters]]></title>
            <link>https://islam-muntashir.medium.com/bin-packing-strategies-for-cost-optimized-kubernetes-clusters-2647c4917da9?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2647c4917da9</guid>
            <category><![CDATA[kubernetes-development]]></category>
            <category><![CDATA[kubenetes]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 23:32:47 GMT</pubDate>
            <atom:updated>2025-12-08T23:41:13.871Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltu1UKT7AFbePQT7jQHKmw.png" /></figure><p>Building a Custom Scheduler for High-Density, Low-Cost Dev/Test Environments</p><p>Running Kubernetes clusters — especially <strong>development or CI clusters</strong> — often leads to a common problem:<br>Nodes sit mostly empty, workloads are spread out, and you end up paying for far more compute than you actually use.</p><p>The default Kubernetes scheduler intentionally balances pods across nodes for reliability — but this behavior wastes capacity in environments where cost and node efficiency matter more than perfect distribution.</p><p>There are many approches in kubernetes cluster to optimized placement of pods into nodes. For example:</p><h4>Configure Resource Requests Thoughtfully</h4><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: testpod<br>spec:<br>  containers:<br>  - name: testcontainer<br>    image: image<br>    resources:<br>      requests:<br>        cpu: 100m<br>        memory: 128Mi<br>      limits:<br>        cpu: 200m<br>        memory: 256Mi</pre><p>In this configuration, the testpod defines clear CPU and memory requests and limits for its container. Setting these values ensures the workload receives the necessary resources to operate consistently while preventing excess allocation that could lead to wasted capacity.</p><h4>Ensuring Stable Workloads in Cost-Optimized Clusters</h4><p>Maintaining stability while managing costs is a critical part of cluster design. Workloads that rely heavily on BestEffort Pods or Burstable Pods with low memory requests can introduce performance variability and, in some cases, reliability problems. By understanding how these QoS classes behave, cluster operators can make informed decisions that reduce risk and improve overall cluster health.</p><h4>Example: Configuring Workloads for Consistent Performance</h4><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: testpod<br>spec:<br>  containers:<br>  - name: testcontainer<br>    image: testimage<br>    resources:<br>      requests:<br>        cpu: 200m<br>        memory: 256Mi<br>      limits:<br>        cpu: 400m<br>        memory: 512Mi<br>  qosClass: Burstable</pre><p>In this configuration, the testpod defines clear CPU and memory requests and limits for its container. Assigning the Pod to the Burstable QoS class provides flexibility for handling variable workloads while still maintaining predictable performance and reducing the likelihood of resource-related disruptions.</p><h4>Keeping User Experience at the Center</h4><p>Cost optimization should never come at the expense of application reliability. When clusters rely heavily on BestEffort or lightly provisioned Burstable Pods, bin packing efficiency may drop, increasing the risk of performance issues. Striking the right balance between saving costs and delivering a smooth, uninterrupted user experience is essential.</p><h4>Example: Designing for a Strong User Experience</h4><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: testpod<br>spec:<br>  containers:<br>  - name: testcontainer<br>    image: testimage<br>    resources:<br>      requests:<br>        cpu: 100m<br>        memory: 128Mi<br>      limits:<br>        cpu: 200m<br>        memory: 256Mi<br>  qosClass: Guaranteed</pre><p>Here, the testpod sets both CPU and memory requests and limits, placing it in the Guaranteed QoS class. This approach ensures consistent performance and helps maintain a reliable experience for end users.</p><h4>Using Demand-Based Downscaling for Smarter Cost Control</h4><p>High-performing teams achieve significant savings by scaling their clusters down during low-traffic periods. This is made possible through tools like the Cluster Autoscaler, HPA, VPA, and cost-visibility tools such as Kubecost. These solutions allow clusters to automatically reduce or increase resources based on real usage.</p><h4>Example: Adaptive Scaling With HPA</h4><pre>apiVersion: autoscaling/v2beta2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: testhpa<br>spec:<br>  selector:<br>    matchLabels:<br>      app: testapp<br>  minReplicas: 1<br>  maxReplicas: 10<br>  metrics:<br>  - type: Resource<br>    resource:<br>      name: cpu<br>      target:<br>        type: Utilization<br>        averageUtilization: 70</pre><p>In this setup, the testhpa automatically adjusts the number of running replicas based on CPU utilization. This ensures resources scale up when demand increases and scale down when traffic slows, helping maintain efficiency without compromising performance.</p><p>Even after applying these optimizations in a test or development environment, you may still encounter situations where the cluster reports no available nodes for scheduling new Pods. However, checking the node pressure metrics often reveals that there <em>is</em> sufficient capacity. In many cases, the default scheduler simply selects other nodes based on its standard logic, leaving some Pods unscheduled even though certain nodes still have room. With smarter scheduling, some of these Pods could be placed on nodes that actually do have adequate resources.</p><h3><strong>Custom Kubernetes scheduler using a bin-packing strategy</strong></h3><p>To solve this, I built a custom Kubernetes scheduler using a bin-packing strategy, written entirely in Python. It tightly packs workloads onto the fewest nodes possible, improving utilization and lowering cloud spend.</p><p>This article walks you through the approach, architecture, code, and real results.</p><h4>🚀 Why Bin Packing for Kubernetes?</h4><p><strong>Bin packing</strong> is a classic optimization technique:</p><ul><li>Pods = items</li><li>Nodes = bins</li><li>Goal = use the smallest number of bins</li></ul><p>Kubernetes’ default scheduler spreads pods, but bin packing does the opposite:</p><p>✔Packs workloads tightly</p><p>✔ Frees unnecessary nodes</p><p>✔ Reduces cloud bills</p><p>✔ Ideal for dev/test clusters</p><p>This approach is <strong>not</strong> meant to replace the <strong>production</strong> scheduler; it’s built for:</p><ul><li>Developer clusters</li><li>Ephemeral environments</li><li>Low-cost home labs</li><li>CI/automation clusters</li><li>Resource-constrained setups</li></ul><p>If you want more density and lower compute cost → bin packing is perfect.</p><h4>🧠 Architecture Overview</h4><p>Here’s the simplified architecture of the scheduler:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bevW0bDZnhF660T3YbY2HA.png" /></figure><p>It works by:</p><ol><li>Watching for pending pods that request schedulerName: binpack-scheduler</li><li>Calculating usage on every node</li><li>Finding the “best fit” node (minimal leftover CPU + memory)</li><li>Binding the pod directly using the Kubernetes API</li></ol><h4>🛠 Full Scheduler Code (Python)</h4><p>Below is the exact scheduler implementation, explained in later sections:</p><pre>import time<br>import logging<br>from kubernetes import client, config, watch<br><br>logging.basicConfig(level=logging.INFO, format=&#39;%(asctime)s %(levelname)s: %(message)s&#39;)<br>config.load_incluster_config()<br>v1 = client.CoreV1Api()<br><br>scheduler_name = &quot;binpack-scheduler&quot;<br><br>def parse_cpu(cpu_str):<br>    try:<br>        if cpu_str.endswith(&quot;m&quot;):<br>            return int(cpu_str[:-1]) / 1000<br>        return float(cpu_str)<br>    except Exception:<br>        logging.warning(f&quot;Unable to parse CPU: {cpu_str}, defaulting to 1 core&quot;)<br>        return 1<br><br>def parse_mem(mem_str):<br>    mem_str = mem_str.lower()<br>    try:<br>        if mem_str.endswith(&quot;mi&quot;):<br>            return int(mem_str[:-2])<br>        elif mem_str.endswith(&quot;gi&quot;):<br>            return int(mem_str[:-2]) * 1024<br>        elif mem_str.endswith(&quot;ki&quot;):<br>            return int(mem_str[:-2]) // 1024<br>        return int(mem_str)<br>    except Exception:<br>        logging.warning(f&quot;Unable to parse Memory: {mem_str}, defaulting to 128Mi&quot;)<br>        return 128<br><br>def get_pod_requests(pod):<br>    total_cpu = 0<br>    total_mem = 0<br>    for container in pod.spec.containers:<br>        res = container.resources.requests or {}<br>        cpu = parse_cpu(res.get(&quot;cpu&quot;, &quot;100m&quot;))<br>        mem = parse_mem(res.get(&quot;memory&quot;, &quot;128Mi&quot;))<br>        total_cpu += cpu<br>        total_mem += mem<br>    return total_cpu, total_mem<br><br>def get_node_allocations():<br>    allocations = {}<br>    pods = v1.list_pod_for_all_namespaces().items<br>    for node in v1.list_node().items:<br>        allocations[node.metadata.name] = {&quot;cpu&quot;: 0, &quot;memory&quot;: 0}<br>    <br>    for pod in pods:<br>        if pod.spec.node_name:<br>            cpu, mem = get_pod_requests(pod)<br>            allocations[pod.spec.node_name][&quot;cpu&quot;] += cpu<br>            allocations[pod.spec.node_name][&quot;memory&quot;] += mem<br>    return allocations<br><br>def schedule():<br>    w = watch.Watch()<br>    for event in w.stream(v1.list_pod_for_all_namespaces):<br>        pod = event[&#39;object&#39;]<br>        if pod.spec.scheduler_name != scheduler_name:<br>            continue<br>        if pod.status.phase != &quot;Pending&quot;:<br>            continue<br>        cpu_req, mem_req = get_pod_requests(pod)<br>        node_allocations = get_node_allocations()<br>        nodes = v1.list_node().items<br>        <br>        best_node = None<br>        min_free = None<br>        for node in nodes:<br>            name = node.metadata.name<br>            alloc = node_allocations.get(name, {&quot;cpu&quot;: 0, &quot;memory&quot;: 0})<br>            cpu_cap = parse_cpu(node.status.allocatable.get(&quot;cpu&quot;, &quot;1&quot;))<br>            mem_cap = parse_mem(node.status.allocatable.get(&quot;memory&quot;, &quot;1Gi&quot;))<br>            cpu_free = cpu_cap - alloc[&quot;cpu&quot;]<br>            mem_free = mem_cap - alloc[&quot;memory&quot;]<br>            if cpu_free &gt;= cpu_req and mem_free &gt;= mem_req:<br>                if min_free is None or (cpu_free + mem_free) &lt; min_free:<br>                    best_node = name<br>                    min_free = cpu_free + mem_free<br>        if best_node:<br>            target = client.V1ObjectReference(api_version=&quot;v1&quot;, kind=&quot;Node&quot;, name=best_node)<br>            meta = client.V1ObjectMeta(name=pod.metadata.name, namespace=pod.metadata.namespace)<br>            body = client.V1Binding(target=target, metadata=meta)<br>            v1.create_namespaced_binding(namespace=pod.metadata.namespace, body=body)<br>        else:<br>            logging.warning(f&quot;No suitable node found for pod {pod.metadata.name}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    while True:<br>        try:<br>            schedule()<br>        except Exception as e:<br>            logging.error(f&quot;Scheduler crashed: {e}&quot;)<br>            time.sleep(5)</pre><h3>📦 How this too simple Bin Packing Algorithm works</h3><p>The bin packing logic is simple but effective:</p><p><strong>Step 1 — Gather node usage</strong></p><p>Compute how much CPU/memory each node is currently using.</p><p><strong>Step 2 — Parse pod resource requests</strong></p><p>Defaults are applied if not defined:</p><ul><li><strong>CPU:</strong> 100m</li><li><strong>Memory:</strong> 128Mi</li></ul><p><strong>Step 3 — Evaluate candidate nodes</strong></p><p>A node is valid if:</p><pre>cpu_free &gt;= cpu_req AND mem_free &gt;= mem_req</pre><p><strong>Step 4 — Choose the “best fit”</strong></p><p>Select the node where (cpu_free + mem_free) is minimum.</p><p>This packs workloads tightly instead of spreading them.</p><h3>🔐 Required RBAC</h3><p>The scheduler needs permission to:</p><ul><li>list/watch pods</li><li>list/watch nodes</li><li>create bindings</li></ul><p>Here is the RBAC and Deployment YAML:</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: binpack-scheduler<br>  namespace: kube-system<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: binpack-scheduler-role<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;nodes&quot;, &quot;bindings&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: binpack-scheduler-rolebinding<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: binpack-scheduler-role<br>subjects:<br>  - kind: ServiceAccount<br>    name: binpack-scheduler<br>    namespace: kube-system<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: binpack-scheduler<br>  namespace: kube-system<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: binpack-scheduler<br>  template:<br>    metadata:<br>      labels:<br>        app: binpack-scheduler<br>    spec:<br>      serviceAccountName: binpack-scheduler<br>      containers:<br>        - name: scheduler<br>          image: muntashir/binpacked-scheduler:v9<br>          imagePullPolicy: Always</pre><h3>🧪 Example: Scheduling an Nginx Deployment</h3><pre>Simple Nginx Running with default schedular<br><br>NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE                         NOMINATED NODE   READINESS GATES<br>nginx-binpack-7748c79896-nfk7t   1/1     Running   0          62s   10.244.1.7   multi-node-cluster-worker    &lt;none&gt;           &lt;none&gt;<br>nginx-binpack-7748c79896-zgz4t   1/1     Running   0          54s   10.244.2.8   multi-node-cluster-worker2   &lt;none&gt;           &lt;none&gt;</pre><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx-binpack<br>  namespace: default<br>spec:<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx-binpack<br>    spec:<br>      schedulerName: binpack-scheduler<br>      containers:<br>        - name: nginx<br>          image: nginx:stable<br>          resources:<br>            requests:<br>              cpu: &quot;10m&quot;<br>              memory: &quot;18Mi&quot;</pre><pre><br>After applyed this scheduler<br><br>NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE                               NOMINATED NODE   READINESS GATES<br>nginx-binpack-67996d955-hqpd2   1/1     Running   0          31s   10.244.0.29   multi-node-cluster-worker          &lt;none&gt;           &lt;none&gt;<br>nginx-binpack-67996d955-zgchq   1/1     Running   0          31s   10.244.0.30   multi-node-cluster-worker          &lt;none&gt;           &lt;none&gt;</pre><p>Look at the worker node here.</p><p>This results in:</p><p>✔ fewer active nodes</p><p>✔ lower cloud cost</p><p>✔ higher resource utilization</p><h3>🔎 Observed Benefits</h3><p>From actual cluster results:</p><p>✔ Both pods packed onto a single worker node</p><p>✔ Unused nodes become scale-down candidates</p><p>✔ Test clusters require fewer VMs</p><p>✔ Perfect for local kind/minikube high-density testing</p><h3>📘 Feature Summary</h3><ul><li>Custom scheduler using Kubernetes Python client</li><li>Lightweight and safe (targets only pods with schedulerName)</li><li>CPU and memory parsing supports m, Mi, Gi, Ki</li><li>Implements best-fit bin packing</li><li>Real-time event-driven scheduling</li><li>Cluster-safe with proper RBAC</li></ul><h3>🏁 Final Thoughts</h3><p>Bin packing is a powerful strategy when your goal is density and cost optimization, not redundancy or spreading. With just a few hundred lines of Python, Kubernetes becomes a much more predictable, efficient platform for dev/test workloads.</p><p>This scheduler has already helped reduce compute usage in my dev clusters — sometimes by more than 40%.</p><p>Here is the github repo <a href="https://github.com/muntashir-islam/k8s-binpacked-scheduler">https://github.com/muntashir-islam/k8s-binpacked-scheduler</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2647c4917da9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Lightweight Tool to Create Kubeconfig Files for Kubernetes Service Accounts(Advanced K8s Series…]]></title>
            <link>https://islam-muntashir.medium.com/a-lightweight-tool-to-create-kubeconfig-files-for-kubernetes-service-accounts-advanced-k8s-series-396358372e33?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/396358372e33</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Fri, 28 Nov 2025 15:40:12 GMT</pubDate>
            <atom:updated>2025-11-28T15:40:12.310Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Lightweight Tool to Create Kubeconfig Files for Kubernetes Service Accounts(Advanced K8s Series: Day 2)</h3><p>There are many situations where you need a kubeconfig file that uses a service account with restricted permissions. Sometimes you may need to share kubeconfig access with teammates or cross-functional teams — especially when the cluster doesn’t have OIDC or any centralized authentication set up.</p><p>However, the ideal approach is usually to create a kubeconfig file for a service account so it can only view or deploy to specific namespaces, for example from a CI/CD pipeline.</p><p>Generating this kubeconfig for a service account normally requires several manual steps inside the cluster. There are some tools on GitHub, but I often find it difficult to locate the right one or use it quickly. So I decided to write a simple tool that meets my needs.</p><p>It’s not intended for production use if you’re not comfortable reviewing the code, but it works well for my day to day ops work.</p><p>Here is the Github link for the tool</p><p><a href="https://github.com/muntashir-islam/kubeconfig-generator">https://github.com/muntashir-islam/kubeconfig-generator</a></p><h4><strong>Overview</strong></h4><p>kubeconfig-generator allows you to easily create kubeconfig files for ServiceAccounts in your Kubernetes cluster. This enables applications or users to authenticate to Kubernetes using a ServiceAccount identity and its associated RBAC permissions.</p><p>The tool works with both modern Kubernetes clusters (1.24+) that use the TokenRequest API and older clusters that use ServiceAccount secrets for tokens.</p><h4><strong>Features:</strong></h4><ul><li>Generate kubeconfig files for any existing ServiceAccount</li><li>Works with both modern and legacy Kubernetes authentication mechanisms</li><li>Automatically extracts cluster information from your current context</li><li>Preserves CA certificate data for secure communication</li><li>Supports configurable token expiration</li><li>Handles proper file permissions for secure kubeconfig files</li></ul><h3>Building from source</h3><ol><li>Clone the repository or create the source file:</li></ol><pre># If using git<br>git clone https://github.com/muntashir-islam/kubeconfig-generator.git<br>cd kubeconfig-generator</pre><pre># Or if starting with just the main.go file<br>mkdir -p kubeconfig-generator<br>cd kubeconfig-generator<br># Copy the main.go file here</pre><p>2. Initialize the Go module and get dependencies:</p><pre>go mod init kubeconfig-generator<br>go get k8s.io/client-go@latest<br>go get k8s.io/apimachinery@latest</pre><p>3. Build the binary:</p><pre>go build -o kubeconfig-generator .</pre><p>4. (Optional) Move the binary to your PATH:</p><pre>sudo mv kubeconfig-generator /usr/local/bin/</pre><h3>Usage</h3><p><strong>Basic usage</strong></p><pre>./kubeconfig-generator -sa SERVICE_ACCOUNT_NAME -namespace NAMESPACE -output KUBECONFIG_PATH</pre><p><strong>All available options</strong></p><pre>Usage:<br>  kubeconfig-generator [flags]<br><br>Flags:<br>  -sa string            Name of the ServiceAccount (required)<br>  -namespace string     Namespace of the ServiceAccount (default &quot;default&quot;)<br>  -output string        Output path for the kubeconfig file (default &quot;./sa-kubeconfig&quot;)<br>  -context string       Context name to use in kubeconfig (defaults to &lt;sa-name&gt;-context)<br>  -cluster string       Cluster name to use in kubeconfig (defaults from current context)<br>  -api-server string    API server URL (defaults from current context)<br>  -kubeconfig string    Path to the kubeconfig file (default &quot;~/.kube/config&quot;)<br>  -expiry int           Token expiry in hours (default 8760 - 1 year)</pre><h3>Example: Creating a ServiceAccount for Pod Viewing</h3><p>This example demonstrates creating a ServiceAccount with permissions to list pods in the default namespace, then generating a kubeconfig for it.</p><p>Create the ServiceAccount and simple RBAC resources:</p><pre># Create a namespace for our ServiceAccount<br><br>kubectl create namespace sa-namespace# Create the ServiceAccount<br><br>kubectl create serviceaccount pod-viewer --namespace sa-namespace# Create a Role with permissions to list pods in default namespace<br><br>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  namespace: default<br>  name: pod-viewer-role<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;pods&quot;]<br>  verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]<br>EOF# Create a RoleBinding<br><br>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: pod-viewer-rolebinding<br>  namespace: default<br>subjects:<br>- kind: ServiceAccount<br>  name: pod-viewer<br>  namespace: sa-namespace<br>roleRef:<br>  kind: Role<br>  name: pod-viewer-role<br>  apiGroup: rbac.authorization.k8s.io<br>EOF</pre><p>Generate a kubeconfig for the ServiceAccount:</p><pre>./kubeconfig-generator -sa pod-viewer -namespace sa-namespace -output ./pod-viewer-kubeconfig</pre><p>Test the kubeconfig:</p><pre># Should succeed - list pods in default namespace<br>KUBECONFIG=./pod-viewer-kubeconfig kubectl get pods -n default# Should fail - no permission to list pods in other namespaces<br>KUBECONFIG=./pod-viewer-kubeconfig kubectl get pods -n kube-system# Should fail - no permission to delete pods<br>KUBECONFIG=./pod-viewer-kubeconfig kubectl delete pod some-pod-name -n default</pre><h4>How It Works</h4><ol><li>The tool first loads your current kubeconfig to get cluster information (API server URL, CA certificate).</li><li>It verifies that the ServiceAccount exists in the specified namespace.</li><li>For Kubernetes 1.24+, it attempts to create a token using the kubectl create token command.</li><li>For older Kubernetes versions, it falls back to retrieving the token from the ServiceAccount’s secret.</li><li>It constructs a new kubeconfig file with the cluster information, token, and appropriate context.</li><li>The file permissions are set to 0600 (read/write for owner only) for security.</li></ol><h4>Security Considerations</h4><ul><li>The generated kubeconfig contains a token with the permissions of the ServiceAccount</li><li>By default, tokens are generated with a 1-year expiry (configurable with -expiry)</li><li>The kubeconfig file permissions are set to be readable only by the owner</li><li>For production use, consider setting shorter expiry times and securely distributing the kubeconfig</li></ul><h3>How this ServiceAccount Kubeconfig Generator Works</h3><p>This Go program automates the process of creating a kubeconfig file for a Kubernetes ServiceAccount. Normally, doing this manually requires several steps:</p><ul><li>Verify the ServiceAccount exists</li><li>Generate a token for it</li><li>Fetch cluster certificate data</li><li>Assemble a kubeconfig YAML correctly</li><li>Save it with the right permissions</li></ul><p>This tool does it all in one command.</p><h4>1. CLI Flags &amp; Configuration Handling</h4><p>The program starts by defining and parsing command-line flags. These flags let you customize the ServiceAccount name, namespace, output file, token expiry, and more.</p><pre>flag.StringVar(&amp;config.ServiceAccountName, &quot;sa&quot;, &quot;&quot;, &quot;Name of the ServiceAccount (required)&quot;)<br>flag.StringVar(&amp;config.Namespace, &quot;namespace&quot;, &quot;default&quot;, &quot;Namespace of the ServiceAccount&quot;)<br>flag.StringVar(&amp;config.OutputPath, &quot;output&quot;, &quot;sa-kubeconfig&quot;, &quot;Output path for the kubeconfig file&quot;)<br>flag.IntVar(&amp;config.TokenExpiryHours, &quot;expiry&quot;, 8760, &quot;Token expiry in hours (default 1 year)&quot;)</pre><p>If the user doesn’t specify a context name, the tool automatically sets one:</p><pre>config.ContextName = fmt.Sprintf(&quot;%s-context&quot;, config.ServiceAccountName)</pre><h3>2. Loading the Current Kubeconfig</h3><p>Before generating a new kubeconfig, the tool loads the user’s current kubeconfig file:</p><pre>currentConfig, err := clientcmd.LoadFromFile(config.KubeconfigPath)</pre><p>From the current context, it extracts the default:</p><ul><li>cluster name</li><li>API server URL</li><li>CA certificate data</li></ul><p>This ensures the new kubeconfig matches your cluster’s correct settings.</p><h3>3. Checking the ServiceAccount Exists</h3><p>Next, it verifies that the ServiceAccount actually exists in the given namespace:</p><pre>clientset.CoreV1().ServiceAccounts(config.Namespace).Get(<br>    context.TODO(),<br>    config.ServiceAccountName,<br>    metav1.GetOptions{},<br>)</pre><p>If not found, the tool stops with an error.</p><h3>4. Retrieving a Token for the ServiceAccount</h3><p>The tool supports <strong>both new and old Kubernetes versions</strong>.</p><h4>Path A: New K8s Versions (TokenRequest API)</h4><p>It first tries:</p><pre>kubectl create token &lt;serviceaccount&gt;</pre><p>Snippet:</p><pre>cmd := exec.Command(&quot;kubectl&quot;, &quot;create&quot;, &quot;token&quot;, config.ServiceAccountName)<br>out, err := cmd.Output()</pre><h3>Path B: Older K8s Versions (Secrets-based tokens)</h3><p>If the above fails, it falls back to fetching the token from the legacy secret:</p><pre>secretName := sa.Secrets[0].Name<br>secret, _ := clientset.CoreV1().Secrets(config.Namespace).Get(...)<br>token := string(secret.Data[&quot;token&quot;])</pre><p>This hybrid method ensures compatibility with almost any cluster.</p><h3>5. Building the New Kubeconfig</h3><p>Once we have the token, the tool builds an in-memory kubeconfig:</p><h3>Cluster Section</h3><pre>newConfig.Clusters[config.ClusterName] = &amp;api.Cluster{<br>    Server: config.APIServer,<br>    CertificateAuthorityData: currentCluster.CertificateAuthorityData,<br>}</pre><p>If no CA exists, it safely falls back to:</p><pre>InsecureSkipTLSVerify = true</pre><h3>User Section</h3><p>The ServiceAccount token is injected:</p><pre>newConfig.AuthInfos[config.ServiceAccountName] = &amp;api.AuthInfo{<br>    Token: token,<br>}</pre><h3>Context Section</h3><pre>newConfig.Contexts[config.ContextName] = &amp;api.Context{<br>    Cluster:   config.ClusterName,<br>    AuthInfo:  config.ServiceAccountName,<br>    Namespace: config.Namespace,<br>}</pre><p>Finally, the context is set as current:</p><pre>newConfig.CurrentContext = config.ContextName</pre><h3>6. Writing the Kubeconfig File</h3><p>The completed kubeconfig is written to disk:</p><pre>clientcmd.WriteToFile(*newConfig, config.OutputPath)</pre><p>Then permissions are hardened to read-only for the user:</p><pre>os.Chmod(config.OutputPath, 0600)</pre><p>This ensures CI/CD agents or other users cannot leak credentials.</p><h3>7. Output</h3><p>When completed, the tool prints:</p><pre>Kubeconfig file created at: sa-kubeconfig<br>Use with: export KUBECONFIG=sa-kubeconfig</pre><h3>Summery</h3><p>Building tools like this is actually quite simple, and I create similar automation regularly as part of my day-to-day Kubernetes work. They save a lot of time, especially when existing GitHub utilities aren’t updated or compatible with the latest Kubernetes releases. When you own the code yourself, you can quickly adapt it to new API changes and ensure it always works with whatever Kubernetes version you’re running.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=396358372e33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes API Aggregation Layer — A Deep Dive (Advanced K8s Series: Day 1)]]></title>
            <link>https://islam-muntashir.medium.com/kubernetes-api-aggregation-layer-a-deep-dive-advanced-k8s-series-day-1-430dc7748388?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/430dc7748388</guid>
            <category><![CDATA[advanced-kubernetes]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kube-api-server]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 19:23:58 GMT</pubDate>
            <atom:updated>2025-11-23T19:53:30.164Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes API Aggregation Layer — A Deep Dive (Advanced K8s Series: Day 1)</h3><p>The Kubernetes API is one of the most powerful and extensible components of the entire ecosystem. Most engineers know how to interact with kubectl or create CRDs, but very few understand the API Aggregation Layer — a mechanism that allows you to extend the Kubernetes API with completely new API servers.</p><p>In this first post of the <em>Advanced Kubernetes Series</em>, we’ll explore the architecture, the reasoning behind aggregation, and how to build a small aggregated API service.</p><h3>What Is the API Aggregation Layer?</h3><p>The API Aggregation Layer allows you to add new API endpoints to Kubernetes without modifying the core API server. Think of it as plugging in another API server that Kubernetes routes traffic to.</p><pre>kubectl -&gt; kube-apiserver -&gt; (Core API) -&gt; (Extension API via Aggregation)</pre><p>Where CRDs extend resource types <em>inside</em> the existing API server, aggregation adds entire APIs using a separate backend server.</p><h3>Why Do We Need the Aggregation Layer?</h3><p>It’s used when:</p><ol><li>You need custom APIs that behave like first-class Kubernetes APIs. For<strong> </strong>Examples</li></ol><ul><li>Metrics API (/apis/metrics.k8s.io)</li><li>Service Catalog API</li><li>Custom controllers requiring complex logic</li></ul><p>2. You need advanced validation, admission, or storage logic</p><p>CRDs are limited compared to a full API server:</p><ul><li>No custom storage strategies</li><li>No complex validation logic beyond webhooks</li><li>No custom etcd schemas</li><li>Limited subresource capabilities</li></ul><p>3. Or you need performance and scale beyond CRDs</p><p>CRDs share storage and API handling with the core API server.<br> Aggregated APIs can have their own storage engine and performance tuning.</p><h3>How the Aggregation Layer Works (Architecture)</h3><p>A simplified flow(Design using chatgpt):</p><pre><strong>kubectl get gadgets</strong><br>↓<br><strong>kube-apiserver</strong><br>↓ (If CRD exists, handle internally)<br>↓ (Else check APIService)<br><strong>APIService object → points to a Service</strong><br>↓<br><strong>Extension API Server (your custom API server)</strong><br>↓<br><strong>Response returned via kube-apiserver</strong></pre><p>The magic resource is <strong>APIService</strong> (not to be confused with CRDs).</p><h3><strong>Hands-On Lab: Build a Simple Aggregated API Server</strong></h3><p>🧪 <strong>Step 1: Start a local cluster</strong></p><p>Use Kind or Minikube:</p><pre>kind create cluster --name agg-demo</pre><p>🧪<strong> Step 2: Deploy a simple extension API server</strong></p><p>Here I have written a small spi server using Golang. We’ll use this small Go API server based on Kubernetes to demonstrate how this integration work</p><p>Clone the sample:</p><pre>git clone https://github.com/muntashir-islam/simple-agreegated-api-server.git<br>cd simple-agreegated-api-server</pre><p>deploy with the provided manifests:</p><pre>make deploy</pre><p>This installs:</p><ul><li>A Deployment running the extension API server</li><li>A Service exposing it</li><li>A matching APIService that tells Kubernetes to route requests</li></ul><p><strong>🧪 Step 3: Verify the new API group</strong></p><pre>➜  simple-agreegated-api-server git:(main) ✗ kubectl get apiservices | grep gadget<br>v1.gadgets.muntashirislam.com     gadgets/gadget-server   True        69m</pre><p><strong>🧪 Step 4: Create a resource from the new API</strong></p><pre>apiVersion: muntashirislam.com/v1<br>kind: Gadget<br>metadata:<br>  name: my-first-gadget<br>  namespace: gadgets   # optional, must match your namespace<br>spec:<br>  size: large<br>  color: red</pre><p>Apply this</p><pre>-&gt; kubectl apply -f deploy/example-resource.yaml<br>-&gt; kubectl-n gadgets get gadgets.muntashirislam.com <br><br>NAME              AGE<br>my-first-gadget   40s<br><br>-&gt; kubectl describe -n gadgets gadgets.muntashirislam.com<br><br>Name:         my-first-gadget<br>Namespace:    gadgets<br>Labels:       &lt;none&gt;<br>Annotations:  &lt;none&gt;<br>API Version:  muntashirislam.com/v1<br>Kind:         Gadget<br>Metadata:<br>  Creation Timestamp:  2025-11-23T17:59:09Z<br>  Generation:          1<br>  Resource Version:    7304<br>  UID:                 323f1c0f-233f-4352-8b38-3e0059e90bbb<br>Spec:<br>  Color:  red<br>  Size:   large<br>Events:   &lt;none&gt;</pre><h3>How Is This Better Than CRDs?</h3><p>CRDs are great for:</p><ul><li>Most controllers</li><li>Simple schemas</li><li>Non-performance-critical resources</li></ul><p>But aggregated APIs win when you need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*y7hoQ2XFdLA0DZlv5YaHRQ.png" /><figcaption>CRD vs AA server</figcaption></figure><p>This is why Kubernetes’ <strong>metrics.k8s.io</strong> and <strong>servicecatalog.k8s.io</strong> are aggregated APIs — CRDs aren’t enough.<br>🎯 When Should You Use It?</p><p>Use the API Aggregation Layer when:</p><ul><li>CRDs hit performance limits</li><li>You need custom storage semantics</li><li>You’re implementing custom subresources</li><li>You’re building a platform-level API</li></ul><p>If your logic is simple → use CRDs.<br>If you need full control → use API aggregation.</p><h3>How this small Aggregated api (AA) server works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7J3R7VVH0rXaowPpdCuOA.png" /></figure><p>The provided Go code implements a minimal Aggregated API (AA) Server that extends the Kubernetes API by serving a custom resource, Gadget, under the API group gadgets.muntashirislam.com/v1.</p><p>Unlike Custom Resource Definitions (CRDs), which rely on the core Kubernetes API server for storage, this AA server runs completely independently and handles its own logic, routing, and data storage.</p><p>Here is a detailed breakdown of how the components work together to register and serve the custom API.</p><h3>1. Server Initialization and Secure Endpoint (main.go, server.go)</h3><p>The core function of the AA server is to be an independent, secure web server that the Kubernetes control plane can trust and communicate with.</p><p><strong>Server Structure (</strong><strong>APIServer struct):</strong></p><ul><li>The APIServer struct holds an *http.ServeMux, the standard Go HTTP request router.</li></ul><p><strong>Initialization (</strong><strong>New()):</strong></p><ul><li>New() creates the server instance and immediately calls s.registerRoutes(), setting up the API endpoints.</li></ul><p><strong>Secure Execution (</strong><strong>RunTLS()):</strong></p><ul><li>The server must use HTTPS for communication with the Kubernetes API server (for security).</li><li>RunTLS(&quot;:8443&quot;) starts the server, listening on port 8443 (a common port for AA servers), and provides paths to the required TLS certificate (/certs/tls.crt) and key (/certs/tls.key). This certificate is crucial for the $kube-apiserver$ to verify the identity of the aggregated server.</li></ul><h3>2. API Registration and Routing (registerRoutes.go, handlers.go)</h3><p>The kube-apiserver needs two things to properly delegate requests: the API group structure and the actual resource handlers.</p><p><strong>API Root Endpoint (</strong><strong>APIRoot handler):</strong></p><ul><li>The server registers the route /apis/gadgets.muntashirislam.com/v1.</li><li>When the kube-apiserver is initially registering and validating the APIService, it queries this root path.</li><li>The handler responds with an APIResourceList (a standard Kubernetes object type), explicitly listing the available resource: &quot;name&quot;: &quot;gadgets&quot;, &quot;kind&quot;: &quot;Gadget&quot;. This tells Kubernetes exactly what resources are served by this endpoint.</li></ul><p><strong>Resource Endpoint (</strong><strong>ListOrCreate handler):</strong></p><ul><li>The server registers the main resource route: /apis/gadgets.muntashirislam.com/v1/gadgets.</li><li>This handler implements the <strong>CRUD (Create, Read, Update, Delete)</strong> operations.</li><li>It uses a switch r.Method block to differentiate between HTTP verbs:</li><li><strong>GET:</strong> Calls storage.List() and returns a GadgetList.</li><li><strong>POST:</strong> Decodes the request body into a new Gadget object, calls storage.Create(), and returns the newly created object.</li></ul><h3>3. Data Structure and Storage (types.go, store.go)</h3><p>This AA server defines its own internal data structures and storage mechanism, entirely separate from Kubernetes’ etcd.</p><p><strong>Data Types (</strong><strong>types.go):</strong></p><ul><li>The Gadget struct defines the custom resource schema, including the mandatory Kubernetes fields (Kind, APIVersion, Metadata).</li><li>The GadgetList struct is used to package multiple Gadget objects when responding to a LIST request (e.g., kubectl get gadgets).</li></ul><p><strong>In-Memory Storage (</strong><strong>MemoryStore):</strong></p><ul><li>store.go implements a simple, non-persistent, <strong>in-memory storage</strong> solution using a Go slice ([]Gadget).</li><li>This demonstrates the AA server’s ability to use any storage backend (SQL, MongoDB, custom cloud APIs) without being restricted to etcd.</li></ul><h3>4. Integration with Kubernetes (External APIService and CRD)</h3><p>The AA server’s Go code is useless to Kubernetes until it’s configured in the cluster.</p><p><strong>APIService Resource (Not shown, but required):</strong></p><ul><li>An APIService object is deployed to the cluster, pointing the kube-apiserver to the AA server’s Kubernetes Service (gadget-api.default.svc:8443).</li><li>It contains the AA server’s group (gadgets.muntashirislam.com) and version (v1).</li><li>Crucially, it includes the <strong>CA Bundle</strong> of the AA server’s TLS certificate, allowing the kube-apiserver to trust the connection.</li></ul><p><strong>CustomResourceDefinition (CRD, shown):</strong></p><ul><li>The accompanying CRD is deployed to define the group structure (muntashirislam.com) and schema properties (size, color) for the client-side (kubectl) to validate YAML definitions before sending them.</li><li>Although the AA server handles the actual data serving, the CRD helps ensure tooling and client compatibility.</li></ul><p>When a user runs kubectl get gadgets.muntashirislam.com, the Aggregation Layer in the $kube-apiserver$ sees the $APIService$, checks the request, and proxies the HTTP request directly to the running Go server on port 8443.<br>Github: <a href="https://github.com/muntashir-islam/simple-agreegated-api-server.git">https://github.com/muntashir-islam/simple-agreegated-api-server.git</a></p><p>#Kubernetes #Api Server</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=430dc7748388" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to upgrade EKS cluster using Terraform]]></title>
            <link>https://islam-muntashir.medium.com/how-to-upgrade-eks-cluster-using-terraform-ec0a93c54afb?source=rss-16debb4ed02b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec0a93c54afb</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[muntashir islam]]></dc:creator>
            <pubDate>Mon, 27 Sep 2021 00:47:22 GMT</pubDate>
            <atom:updated>2021-09-27T00:47:22.419Z</atom:updated>
            <content:encoded><![CDATA[<p>To upgrade EKS using terraform I have prepared 3 node clusters and deployed some workload over there. I am going to install the 1.20 version of EKS cluster. Then I will upgrade this cluster into 1.21 and then safely upgrade the worker node of the cluster. We have 2 worker group name worker-group-1 with 2 nodes and worker-group-2 with 1 worker node.</p><p>For your information I have used this <a href="https://github.com/hashicorp/learn-terraform-provision-eks-cluster">repo</a> for this demonstration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*QX98xLQODNWA2dFs" /><figcaption>Current node status of cluster</figcaption></figure><p>We have an EKS cluster with version UP and running. I am going to deploy some workload for realisting cluster upgradation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*co4e0E8cC_GrzNWT" /><figcaption>Deployed Nginx and Redis workload using helm chart</figcaption></figure><p>Next, update the cluster version in your eks_cluster module to the next desired version of EKS. Remember clusters managed by EKS can only be upgraded one minor version at a time, so if you are currently at 1.20, you can upgrade to 1.21.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/0*LRmrsxttQqRdTuNI" /><figcaption>Change cluster version from 1.20 -&gt; 1.21</figcaption></figure><p>When you run the terraform plan command you see some changes that seem unrelated to upgrading the EKS cluster version. The EKS terraform module is updated often which may cause some items to be renamed, which will show up as an update or a destroy and recreate. I highly recommend storing the kubeconfig and aws_auth_config files that are generated by this module in version control so you can always access them in case something goes wrong with the cluster update. Once you are satisfied with the changes, run terraform apply and confirm the changes again.</p><p>I have found the following resource changes during plan command.</p><p><em>Plan: 2 to add, 3 to change, 2 to destroy.</em></p><p>For me it will take almost 33 minutes. If you check into EKS cluster dashboard then you will see that your cluster version is already updated to 1.21.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*o4tYafxISHwiCGrT" /><figcaption>Current status of EKS cluster in EKS dashboard</figcaption></figure><p>But you will see the worker node version is like previous version 1.20</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/0*ay-wYB6-ryUfV1uD" /><figcaption>Worker node did not upgrade as a part of cluster upgrade</figcaption></figure><p>Now it is important to upgrade the kube proxy also to support 1.21 properly. For doing that First check which exact kubernetes version is actually installed in the cluster. From this <a href="https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html">link</a> we found that only 1.21.2 is supported for 1.21 release. We can ensure this by running <strong>kubectl version</strong> command</p><p><em>Server Version: version.Info{Major:”1&quot;, Minor:”21+”, GitVersion:”v1.21.2-eks-0389ca3&quot;, GitCommit:”8a4e27b9d88142bbdd21b997b532eb6d493df6d2&quot;, GitTreeState:”clean”, BuildDate:”2021–07–31T01:34:46Z”, GoVersion:”go1.16.5&quot;, Compiler:”gc”, Platform:”linux/amd64&quot;}</em></p><p>So for upgrading kube-proxy we need to upgrade the existing daemonset’s image. From this <a href="https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html">link</a> you will find which image version is actually supported by 1.21 eks cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*d1TdRU8thtn1iWro" /><figcaption>check kubeproxy version for particular kubernetes version</figcaption></figure><p>You will find that image tag 1.21.2-eksbuild.2 is supported as a kube proxy image for 1.21 eks cluster version. Lets upgrade the image in kube-proxy daemonset.</p><blockquote>kubectl edit daemonsets.apps kube-proxy -n kube-system</blockquote><p>And set the image version to 1.21.2-eksbuild.2. And save the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*9K2RpzCpRF-QUVNj" /><figcaption>Changing the image version in daemonsets</figcaption></figure><p>You will see new deployment triggered and the kube proxy in rolling update state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*hATNYqlV2WfFc3Lu" /><figcaption>Applying the change will replace existing kubeproxy daemonsets</figcaption></figure><p>Now to upgrade our worker nodes to the new version, we will create a new worker group of nodes at the new version, and then move our pods over to them. The first step is to add a new configuration block to your worker groups configuration in terraform. Your configuration will be different depending on what features you are using of the eks module, and it is important that you copy your worker groups so that they match your old configuration. You can also point your new ami if any and point that here. You will find your AMI in this <a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html">link</a>.</p><p>Here is my new worker group:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*gQkenizXsoBHHcz6" /><figcaption>New worker group for migrating existing workload</figcaption></figure><p>Now apply the Terraform Plan and apply command. It will show you following including volume and other stuffs</p><p><em>Plan: 3 to add, 0 to change, 0 to destroy.</em></p><p>After running successfully this command you will find that two new node come up with version 1.21.2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*MbUsNqxQaXucf6t1" /><figcaption>Node status after adding new worker node with v1.21.2 version</figcaption></figure><p>Now it’s time to remove previous worker-group-2 from my nodepool. For doing that I need to make the node cordon so that all workload will migrate into the new node. But before that if you have any running cluster autoscaler then you need to set replica to zero otherwise it will hamper your cluster scaling.</p><p><em>kubectl scale deployments/cluster-autoscaler — replicas=0 -n kube-system</em></p><p>Then find out your nodes list from your AWS dashboard associated with worker-group-2. In my case <strong>ip-10–0–3–179.ec2.internal</strong> is the candidate.</p><p>So make the node unschedulable first by cordon the node</p><p><em>kubectl cordon ip-10–0–3–179.ec2.internal</em></p><p>Now check the status of the node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*tU-rQEwlF7SRbgTJ" /><figcaption>Initiate migrating workload from existing node to new node. The targeted node is unscheduable now</figcaption></figure><p>Now it’s time to drain the old nodes and force the pods to move to new nodes. I recommend doing this one node at a time to ensure that everything goes smoothly, especially in a production cluster:</p><p><em>kubectl drain ip-10–0–3–179.ec2.internal — ignore-daemonsets — delete-local-data</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C_0Kz0vNOrguvHsX" /><figcaption>Evicting workload from legacy node</figcaption></figure><p>If you get all pods then you will see all workload running as it is without any interruption. And no pods are running in our legacy node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zX3uARoR2wuHYDHq" /><figcaption>all pods now running on other nodes then our targeted one</figcaption></figure><p>If you have multiple nodes in your node group then you need to repeat this task for each and every node. Remember to create a new nodegroup with at least an exact number of nodes that you have in your legacy nodegroup.</p><p>Now Once you have confirmed that all non-DaemonSet pods are running on the new nodes, we can terminate your old worker group. Since the eks module uses an ordered array of worker group config objects in the <strong>worker groups</strong> key, you cannot just delete the old config. Terraform will see this change and assume that the order must have changed and try to recreate the AutoScaling groups. Instead, we should recognize that this will not be the last time we will do an upgrade, and that empty AutoScaling groups are free. So we will keep the old worker group configuration and just run 0 capacity in it like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*oNK_kXg3EZ0IEKdk" /><figcaption>Now terminate the old worker group by setting desire size to zero</figcaption></figure><p>Run <strong>terraform plan</strong> and <strong>terraform apply. </strong>After successful rollout you will find the following. Find that there is no worker-group-2 available here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*KYOraGt06ynNFd5-" /><figcaption>there is no Worker group 2 is available now</figcaption></figure><p>Repeat the same steps for Worker-group-1 also. This will help you to upgrade your worker node without downtime. Next time when we upgrade, we can put the new config in this worker group and easily spin up workers without worrying about the terraform state. Finally, if you scaled down your cluster-autoscaler, you can revert those changes so that auto scaling works properly again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec0a93c54afb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>