<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Johnson Isaac Joseph on Medium]]></title>
        <description><![CDATA[Stories by Johnson Isaac Joseph on Medium]]></description>
        <link>https://medium.com/@js.johni?source=rss-337114d59f1c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Johnson Isaac Joseph on Medium</title>
            <link>https://medium.com/@js.johni?source=rss-337114d59f1c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:30:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@js.johni/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ready, but not reachable: Kubernetes pod readiness on secondary NICs]]></title>
            <link>https://medium.com/@js.johni/ready-but-not-reachable-kubernetes-pod-readiness-on-secondary-nics-fca9dc4e8637?source=rss-337114d59f1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fca9dc4e8637</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Johnson Isaac Joseph]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 05:48:06 GMT</pubDate>
            <atom:updated>2026-10-25T05:48:06.418Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>I have watched a pod go Ready and immediately start making outbound API calls, and every one of them failed. The pod was Ready, but the network programming for its secondary interface had not finished, so the rest of the network did not know how to reach that NIC yet. Nothing in the pod’s status said otherwise.</p><p>“Ready” in Kubernetes carries more weight than it should. We read a Ready pod as a pod that is on the network and can serve traffic. For a pod’s primary interface that is usually true (the exception is the first time the control plane wires up networking on a fresh node, or during a dynamic scale up). For a second interface it often is not, and at scale that gap stops being a curiosity. In performance critical workloads, it increases the latency to start.</p><h3>Who this is for</h3><p>This assumes you are comfortable with Pods, readiness probes and conditions, and the basics of CNI. If “readiness gate” or “CNI ADD” are new terms, read those first and come back.</p><h3>Why Ready usually does mean on-network (for the primary NIC)</h3><p>When a pod starts, the kubelet calls the CNI plugin’s ADD operation, and for most default setups that call programs the dataplane for the pod’s primary interface before the pod is considered started. By the time the pod can go Ready, eth0 is wired. That is why the shortcut “Ready means on the network” holds up most of the time, and why almost nobody questions it.</p><h3>Why it breaks for secondary interfaces</h3><p>Secondary interfaces do not follow that timeline. You can attach one a few ways: <a href="https://github.com/k8snetworkplumbingwg/multus-cni">Multus</a> with a NetworkAttachmentDefinition, Dynamic Resource Allocation (DRA, which <a href="https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/">went GA in Kubernetes v1.34</a>) for GPU and RDMA fabrics, or a controller that provisions the interface during pod creation. The attach itself can be quick. The catch is what counts as done. Moving the device into the pod’s network namespace and assigning an address is local setup. That secondary IP is only usable once the rest of the network knows how to reach that IP, and that catching up can lag well behind the local setup.</p><p>During that window the pod can be Running, and even Ready by the kubelet’s own checks, while its second NIC is not yet reachable. For an expensive GPU or RDMA pod that is wasted capacity: the scheduler thinks the pod is working, traffic lands on the fabric NIC, and it fails in a way that is hard to attribute. “Ready but not routable” here is not a bug in any one component. It is structural, because nothing in the readiness path is looking at the second interface.</p><h3>What Kubernetes already does (so we are clear this is not reinvention)</h3><p>Kubernetes already cares about how long network programming takes. There are two groups of existing tools.</p><p>Things that enforce readiness:</p><ul><li><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate">Pod readiness gates</a> let a controller hold a pod out of Ready until a custom condition flips. They enforce a wait. They do not tell you how long it took, and nothing checks that the condition reflects real routability.</li><li>The <a href="https://github.com/kubernetes-sigs/node-readiness-controller">Node Readiness Controller</a> (KEP-5233) applies the same idea at the node level, tainting a node until conditions like a network-ready signal are met. Node scoped, condition based, not a probe.</li></ul><p>Things that measure how long it takes:</p><ul><li><a href="https://github.com/kubernetes/community/blob/main/sig-scalability/slos/network_programming_latency.md">NetworkProgrammingLatency</a> is a SIG-Scalability SLO, but it measures Service and Endpoints programming through kube-proxy, not per-pod interface readiness.</li><li>Cilium measures endpoint datapath programming time (<a href="https://docs.cilium.io/en/stable/observability/metrics/">cilium_endpoint_regeneration_time_stats_seconds</a>). That is the closest in spirit, a per-endpoint “the pod’s datapath is ready” timing, but it is the default interface, single NIC.</li><li><a href="https://kubernetes.io/blog/2025/09/17/kubernetes-v1-34-pods-report-dra-resource-health/">DRA resource health</a> (KEP-4680, alpha in v1.34) surfaces device health in pod status, but it is health reported by the driver as Healthy, Unhealthy or Unknown, not a connectivity check, and it targets accelerators.</li><li><a href="https://github.com/kubernetes/perf-tests/tree/master/clusterloader2">clusterloader2</a> can measure DRA ResourceClaim allocation latency, but that clock stops at allocation, before the datapath is ready.</li></ul><h3>The gap</h3><p>Every signal above is scoped to a Service, a policy, a node, or an allocation step. The one thing none of them measure is, per pod and per secondary interface, how long until that specific NIC is actually routable.</p><p>The newer building blocks confirm the gap rather than close it. <a href="https://github.com/kubernetes-sigs/dranet">DRANET</a> and the <a href="https://arxiv.org/abs/2506.23628">Kubernetes Network Driver model</a> move a network interface into the pod namespace, but they do not verify the secondary IP is reachable from off host, and the work to even standardize what gets reported in ResourceClaim status (<a href="https://www.kubernetes.dev/resources/keps/4817/">KEP-4817</a>) is still in flight. <a href="https://www.kubernetes.dev/resources/keps/6313/">KEP-6313</a>, the current direction for network-isolated pod sandboxes, explicitly puts secondary interface attachment and its lifecycle out of scope. For now, if you want this number, you measure it yourself.</p><h3>The obvious fix, and why I did not take it</h3><p>There is a tempting counter argument here. If a secondary interface is not routable yet, why mark the pod usable at all? You could probe the interface during CNI ADD, for example a DHCP or reachability check bound to that NIC, and if it fails, move the interface back out of the pod namespace and fail the ADD. Fail closed, so no pod ever starts believing a NIC works when it does not.</p><p>It is a clean idea, and for some workloads it is the right one. I did not make it the default, for two reasons.</p><p>First, failing the ADD couples pod startup to the rest of the network catching up. Under churn or a large scale up, that is exactly when the rest of the network is slowest to catch up, so you turn a slow secondary NIC into failed and retried pod starts right when the system is already under pressure. The blocking behavior bites hardest in the case it is meant to protect.</p><p>Second, a probe at ADD time only proves the NIC could reach one target at one moment. It is a point check, not a promise the path stays routable afterward.</p><p>And eventual consistency is often acceptable. A lot of what a pod does at startup does not need the secondary NIC in the first millisecond. Init containers and other non blocking setup can run while the rest of the network catches up. So my default is the opposite of failing closed: let the pod start, let the work that does not need the secondary NIC proceed, and measure how long that NIC actually takes to become routable. Block only when the workload genuinely needs the NIC at startup.</p><h3>How I measured it</h3><p>The approach that worked for me was simple: let each pod check itself.</p><p>Give the pod an init container (or a sidecar that runs early) that:</p><ol><li>waits for the secondary interface to appear and get an address,</li><li>tries to reach a known target bound specifically to that NIC,</li><li>records the time from pod start to first success, and</li><li>reports that number out.</li></ol><p>An aggregator collects the per-pod numbers and gives you p50, p90 and p99 per interface, plus the top N slowest pods. The distribution is the point. In my own testing the median stayed low while the p99 blew up under churn, and the tail is where the surprise lives.</p><p>Why a self-probe instead of probing from outside? Because external exec or tunnel based probing gets unreliable exactly when the cluster is under the load that makes this problem show up. A pod checking its own NIC, on its own clock, scales with the workload and degrades more gracefully.</p><h3>Example</h3><p>A NetworkAttachmentDefinition for the second network:</p><pre>apiVersion: k8s.cni.cncf.io/v1<br>kind: NetworkAttachmentDefinition<br>metadata:<br>  name: fabric-net<br>spec:<br>  config: |<br>    {<br>      &quot;cniVersion&quot;: &quot;1.0.0&quot;,<br>      &quot;type&quot;: &quot;macvlan&quot;,<br>      &quot;master&quot;: &quot;eth1&quot;,<br>      &quot;mode&quot;: &quot;bridge&quot;,<br>      &quot;ipam&quot;: {<br>        &quot;type&quot;: &quot;host-local&quot;,<br>        &quot;subnet&quot;: &quot;10.10.0.0/16&quot;,<br>        &quot;rangeStart&quot;: &quot;10.10.1.10&quot;,<br>        &quot;rangeEnd&quot;: &quot;10.10.1.250&quot;<br>      }<br>    }</pre><p>A pod that attaches the second NIC and times how long until it is routable:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: nic-ready-demo<br>  annotations:<br>    k8s.v1.cni.cncf.io/networks: fabric-net<br>spec:<br>  initContainers:<br>  - name: nic-ready-probe<br>    image: nicolaka/netshoot        # has iproute2 + iputils<br>    command: [&quot;/bin/sh&quot;, &quot;-c&quot;]<br>    args:<br>      - |<br>        set -eu<br>        IFACE=net1                  # Multus names the 2nd iface net1<br>        TARGET=&quot;${PROBE_TARGET}&quot;<br>        START=$(date +%s%3N)<br>        # 1) wait for the secondary iface to exist and get an IPv4 address<br>        until ip -o addr show dev &quot;$IFACE&quot; 2&gt;/dev/null | grep -q &#39;inet &#39;; do<br>          sleep 0.1<br>        done<br>        # 2) wait until the target is reachable bound to that NIC<br>        until ping -I &quot;$IFACE&quot; -c1 -W1 &quot;$TARGET&quot; &gt;/dev/null 2&gt;&amp;1; do<br>          sleep 0.1<br>        done<br>        END=$(date +%s%3N)<br>        # emit the number; a real setup pushes this to a pushgateway/aggregator<br>        echo &quot;nic_ready_ms{iface=\&quot;$IFACE\&quot;} $((END - START))&quot;<br>    env:<br>      - name: PROBE_TARGET<br>        value: &quot;10.10.0.1&quot;<br>  containers:<br>  - name: app<br>    image: registry.k8s.io/pause:3.9</pre><p>The init container also blocks the app from starting until net1 is routable, so it doubles as a crude readiness gate. But the value is the number it emits: collect nic_ready_ms across every pod and you have the per-NIC distribution the built-in signals never gave you.</p><h3>Close</h3><p>I’m moving on to different work, so I wanted to get this down while it’s still fresh. The big lesson for me was simple: on a secondary interface, Ready tells you the kubelet is happy, not that the NIC can carry traffic. If you run Multus, DRA, or GPU fabrics, it’s worth timing that gap yourself even once. The init container above is enough to see your own numbers, and the tail is usually where the surprise is.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fca9dc4e8637" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Cost of Cloud-Dependent CI Pipelines in the Age of AI Agents]]></title>
            <link>https://medium.com/@js.johni/the-hidden-cost-of-cloud-dependent-ci-pipelines-in-the-age-of-ai-agents-5969706ccc31?source=rss-337114d59f1c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5969706ccc31</guid>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Johnson Isaac Joseph]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 16:51:23 GMT</pubDate>
            <atom:updated>2026-04-15T01:46:33.338Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s a bug that only shows up in CI. You push, wait for the pipeline, get an error, copy the logs into your terminal or IDE, try a fix, push again, wait again. Most engineers have been through this loop. With AI coding tools and agents, it gets a lot more painful, because you’re iterating much faster now.</p><p>The root cause usually isn’t the code. It’s how the pipeline is set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14owx7iIz8SKemWZiNbyVA.png" /></figure><h4>When your pipeline can only run in the cloud</h4><p>CI platforms like Azure DevOps and GitHub Actions come with a lot of built-in steps — cloud login tasks, resource provisioning helpers, managed credential providers. These are useful, but it’s easy to end up just reaching for them out of habit, and then every stage in your pipeline needs cloud access just to start. A typical pipeline in this pattern looks something like:</p><ul><li>Stage 1: Cloud authentication</li><li>Stage 2: Resource provisioning</li><li>Stage 3: Network setup</li><li>Stage 4: Checkout and build</li><li>Stage 5: Run tests</li></ul><p>Each stage makes sense on its own. But the whole thing becomes impossible to run locally. If Stage 4 is failing, you can’t run just that step on your laptop — you have to push a commit and run the full chain remotely. During peak CI hours, each stage waits in a queue before it even starts, so a single iteration can easily take 10–15 minutes.</p><h4>Shell scripts vs compiled binaries</h4><p>A related problem is using CLI tools directly in shell scripts — things like az cli or kubectl. It seems like the natural choice. They&#39;re familiar, well-documented, and quick to write.</p><p>But there are real costs. Every time you need retry logic, you’re writing it from scratch or copy-pasting a pattern. With a proper Go client like <a href="https://github.com/Azure/azure-sdk-for-go">azure-sdk-for-go Client</a> or <a href="https://github.com/kubernetes/client-go">Kubernetes go-Client</a>, retry and error handling are already built in. If you need the same behavior across multiple stages, you end up either duplicating shell code or writing wrapper functions that don&#39;t travel well between projects.</p><p>More importantly, shell scripts don’t have a build step. There’s no compiler telling you about a typo or a wrong flag until it actually blows up in CI. With Go, go vet and go build catch those things before anything runs. You can also write real unit tests for your pipeline logic — something that&#39;s not really possible with shell scripts in any clean way.</p><p>The better approach is to build a binary that handles all the API calls. Compile it once, publish it to an artifact registry, and each CI stage just pulls it and calls it with different parameters. No git checkout in the stage. No install steps. No cloud login just to run a build. The binary already has everything baked in.</p><p>This is the pattern that actually makes pipelines fast to work with, especially when you’re using AI agents. The agent can run the binary locally, see real errors from real clients, and fix them. It doesn’t need CI at all for most of the iteration cycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkbftfudyYjnSfF3Ge-ClQ.png" /><figcaption>Diagram 1— binary built once, stages just call it (the solution)</figcaption></figure><h4>Why this matters more now</h4><p>When you’re working with an AI coding assistant or agent, the feedback loop is the whole point. A pipeline that requires pushing a commit, waiting for a queue, and copying logs back into your editor completely breaks that loop. If the pipeline runs locally, the feedback is immediate — the agent sees the failure, proposes a fix, and you validate it in seconds. That difference adds up fast over a day of active development.</p><h4>Not all stages are bad</h4><p>This isn’t an argument against multi-stage pipelines. Running tests across multiple deployment regions, enforcing approval gates before production, or parallel test suites that are genuinely independent — these are all valid reasons to use stages.</p><p>For parallelism, a DAG approach (Argo Workflows, Tekton, or DAG mode in various CI platforms) is cleaner than just stacking stages. It makes dependencies explicit so you can see exactly what can run in parallel and what has to wait.</p><p>The goal isn’t fewer stages — it’s making sure each stage is portable and doesn’t pull in cloud dependencies it doesn’t actually need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*4qMIvZHio7FjP9jnBhfJ6w.png" /><figcaption>Diagram 2 — iteration loop: old way vs new way</figcaption></figure><h4>The bigger picture</h4><p>AI tools are raising the bar for what good developer experience means. A pipeline that was fine when you iterated once an hour becomes a real bottleneck when you’re iterating ten times faster. Local reproducibility isn’t a nice-to-have anymore — it’s what lets you actually take advantage of AI-assisted development.</p><p>Before adding a platform-specific step to your pipeline, ask: could I debug a failure in this step without pushing a commit? If the answer is no, that step is worth rethinking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5969706ccc31" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>