<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DAMAN on Medium]]></title>
        <description><![CDATA[Stories by DAMAN on Medium]]></description>
        <link>https://medium.com/@sdaman830?source=rss-5e90c25b156a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DGymskcDz9Z7OnzVdrdajQ.jpeg</url>
            <title>Stories by DAMAN on Medium</title>
            <link>https://medium.com/@sdaman830?source=rss-5e90c25b156a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:13:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sdaman830/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pods Could Talk to Each Other. Kubernetes Services Couldn’t. Here’s the Kernel Bug Behind It.]]></title>
            <link>https://medium.com/@sdaman830/pods-could-talk-to-each-other-kubernetes-services-couldnt-here-s-the-kernel-bug-behind-it-31da3d443377?source=rss-5e90c25b156a------2</link>
            <guid isPermaLink="false">https://medium.com/p/31da3d443377</guid>
            <category><![CDATA[on-prem]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[DAMAN]]></dc:creator>
            <pubDate>Sat, 25 Jul 2026 13:29:41 GMT</pubDate>
            <atom:updated>2026-07-25T13:29:41.963Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A field guide to a failure mode that has nothing to do with your code — and everything to do with a handful of missing lines in a kernel config.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsKmyNYnna9VKYuXzFAgdA.png" /></figure><h3>The symptom that started it all</h3><p>It began with one crashing pod. A backend service kept restarting, and the logs showed something every backend engineer has seen a hundred times:</p><pre>panic: runtime error: invalid memory address or nil pointer dereference</pre><p>The stack trace pointed at a database client. Standard stuff — probably a bad connection string, a missing secret, a typo in a hostname. I pulled the logs from the step before the panic and found this instead:</p><pre>{&quot;level&quot;:&quot;error&quot;,&quot;error&quot;:&quot;server selection error: server selection timeout,<br>current topology: { Servers: [{ Addr: mongo.mongo.svc.cluster.local:27017,<br>Last error: dial tcp 10.43.x.x:27017: connect: connection refused }] }&quot;}</pre><p>The app was panicking because it couldn’t reach its database, and it wasn’t handling that failure gracefully. Fair enough — that’s a real bug, but it’s a <em>symptom</em>. The real question was: why couldn’t it reach the database?</p><p>What followed was a multi-day descent through label mismatches, a broken kernel, three failed Kubernetes networking backends, a failed attempt at a <em>fourth</em>, and a set of workarounds that got everything running again. I’m writing this up not because the specific hardware matters, but because <strong>the failure mode is generic</strong>, and I suspect a lot of people deploying Kubernetes on embedded boards, minimal cloud images, or hardened Linux builds are going to run into exactly this — often without realizing what they’re actually looking at.</p><p>If you’re running Kubernetes anywhere that <em>isn’t</em> a standard cloud VM or a full desktop Linux distro — an SBC, an appliance, a Yocto or Buildroot image, a security-hardened kernel, a stripped-down container host — this article is for you.</p><h3>Setting the scene</h3><p>Here’s the environment, for anyone who wants the specifics before the story:</p><p>Hardware Embedded ARM64 single-board computer (vendor dev kit) Kernel Linux 6.6, custom-compiled as part of a board support package (BSP) — not a general-purpose distro kernel Orchestrator k3s, a lightweight Kubernetes distribution Container runtime containerd Application stack ~15 microservices, MongoDB, Redis, a vector database, an ingress controller — an entirely ordinary setup What turned out to be missing Four separate kernel subsystems, one for each possible Kubernetes Service-routing backend (details below)</p><p>Nothing about the <em>application layer</em> here was unusual — a dozen or so services talking to a database and to each other over gRPC, fronted by an ingress controller. If you’ve built a microservices app on Kubernetes, you’ve built this one.</p><p>What <em>was</em> different from a typical cloud deployment was the kernel underneath it. It wasn’t Ubuntu, wasn’t a standard cloud image, wasn’t anything from a general-purpose distro’s kernel team. It was a kernel built by hardware engineers, for one specific device, optimized for boot time and image size — the kind of build where every kernel config option gets evaluated against “does this device need it,” never against “does Kubernetes need it.” Nobody involved in assembling that kernel had Kubernetes in mind, because the hardware wasn’t originally being spec’d for it.</p><p>By the end of this investigation, it turned out that <em>every single Service-routing mechanism Kubernetes offers</em> was missing its corresponding kernel piece — not one unlucky gap, but four for four. That’s not a coincidence; it’s what happens when a kernel gets trimmed down without anyone specifically checking whether Kubernetes’ networking layer still has what it needs.</p><p>That single fact turned out to explain everything that follows.</p><h3>Step one: separate the endpoint problem from the routing problem</h3><p>Before getting to the deep issue, there was a shallower one worth mentioning, because it’s a common first stop on this kind of journey and it’s worth ruling out early.</p><p>The database’s Kubernetes Service had <strong>no endpoints</strong> — kubectl get endpoints mongo returned &lt;none&gt;. A Service with no endpoints is a dead end by design: nothing is listening behind it. The cause was mundane — the Service&#39;s label selector required four labels, the pod only carried one. Classic drift between a hand-edited manifest and a Helm-templated one.</p><p>Fixing the selector so it matched the pod’s actual labels resolved <em>that</em> specific error. But the next test run produced a different failure, and the shift between the two error messages turned out to be the important clue.</p><p><strong>Before the fix:</strong> connection refused <strong>After the fix:</strong> i/o timeout</p><p>These are not interchangeable, and the distinction is worth internalizing:</p><ul><li><strong>connection refused</strong> means something answered instantly and said &quot;nothing is here.&quot; This is what you get from a Service with no endpoints, or a port nothing is listening on.</li><li><strong>i/o timeout</strong> means the packet was sent and <em>nothing ever responded</em> — not even a rejection. This is the signature of a routing black hole: the destination address exists as a concept, but there&#39;s no path to it.</li></ul><p>Once the endpoint existed but the timeout persisted, the problem had moved from “is anything listening” to “can traffic get there at all.” That’s a fundamentally different layer of the stack, and it’s where things got interesting.</p><h3>Step two: the test that actually isolates the layer</h3><p>If you take away one practical technique from this article, make it this one. When a Kubernetes Service seems unreachable, don’t guess — run this comparison:</p><pre># Get the pod&#39;s real IP (bypasses the Service entirely)<br>kubectl get pod &lt;pod-name&gt; -o jsonpath=&#39;{.status.podIP}&#39;</pre><pre># From another pod, test BOTH paths<br>kubectl run netcheck --rm -it --restart=Never --image=busybox -- sh -c &#39;<br>  echo &quot;via Service (ClusterIP/DNS):&quot;;<br>  nc -zv -w5 my-service.my-namespace.svc.cluster.local 5432;<br>  echo &quot;via pod IP directly:&quot;;<br>  nc -zv -w5 &lt;pod-ip&gt; 5432&#39;</pre><p>The result splits the problem cleanly in two:</p><ul><li><strong>Both work</strong> → networking is fine, look elsewhere (app config, auth, etc.)</li><li><strong>Both fail</strong> → this is a pod-networking / CNI problem, a different investigation entirely</li><li><strong>Pod IP works, Service fails</strong> → this is the case in this article. Pod-to-pod networking (your CNI — Flannel, Calico, Cilium’s networking layer, whatever you’re running) is completely healthy. The <strong>Service abstraction</strong> specifically is broken. And Services are implemented by exactly one component: <strong>kube-proxy</strong>.</li></ul><p>That third outcome is the one to know cold. It immediately tells you the problem isn’t your application, isn’t DNS, isn’t your CNI, and isn’t your database — it’s the layer that translates a stable Service address into a real, current pod address. Every time.</p><h3>Step three: watch kube-proxy fail in real time</h3><p>kube-proxy’s job, in its default (iptables) mode, is unglamorous but critical: watch the API server for Services and Endpoints, and continuously rewrite a set of kernel packet-filtering rules so that traffic to a Service’s virtual IP gets redirected to a real pod. It does this by shelling out to iptables-restore with a freshly generated rule set, on a loop, forever.</p><p>On this board, that command was failing. Every single time:</p><pre>journalctl -u k3s | grep -A5 &#39;iptables-restore&#39;</pre><pre>Warning: Extension statistic revision 0 not supported, missing kernel module?<br>iptables-restore: line 590 failed</pre><p>There it was, in plain English: the kernel doesn’t have a module that iptables-restore needs, so the <em>entire</em> rule-update fails, every cycle, forever.</p><p>Two details about <em>how</em> it fails matter a lot for understanding the downstream symptoms:</p><p><strong>First — kube-proxy runs its restores with </strong><strong>--noflush, and that&#39;s the key to the confusing part.</strong> --noflush exists so that kube-proxy&#39;s sync doesn&#39;t wipe out rules it doesn&#39;t own. The side effect: when a sync fails partway through — which, on this kernel, was <em>every</em> sync — nothing from that failed attempt gets applied, but nothing from the <em>last successful</em> sync gets removed either. iptables-restore&#39;s atomicity is scoped per table (bounded by that table&#39;s COMMIT line), so a rule that fails to parse aborts the table&#39;s update before it&#39;s ever replaced in the kernel. The practical effect is that the cluster gets stuck at whatever state existed the last time a full sync actually succeeded — before whatever change (a new Service, a Service gaining a second endpoint, anything that produces a rule needing the missing extension) started tripping the failure on every cycle after. Services that already had working rules from that last good sync kept working, untouched. Services added or changed afterward never got their rules installed at all, because every attempt to update anything failed before it could commit. That&#39;s why it looked like arbitrary luck which services worked — it wasn&#39;t luck at the moment you checked it, it was a frozen snapshot from however long ago the kernel gap started actually mattering.</p><p><strong>Second — the missing piece is genuinely small.</strong> xt_statistic is a single netfilter match extension. Its entire job is enabling probabilistic load-balancing across multiple pod IPs behind one Service. It&#39;s not exotic — it&#39;s just one line item among hundreds of possible kernel config options, and on a general-purpose Linux distro it&#39;s virtually always compiled in. On a purpose-built, size-optimized kernel, there&#39;s a real chance nobody ever had a reason to turn it on.</p><h3>Step four: confirm it, don’t assume it</h3><p>At this point it would be easy to declare victory — “found the missing module, done.” Resist that. Confirm it directly, because the fix depends on <em>exactly</em> what’s missing, and there’s often more than one thing.</p><pre># Try to load the module the log is complaining about<br>modprobe xt_statistic</pre><pre>modprobe: FATAL: Module xt_statistic not found in directory /lib/modules/&lt;kernel-version&gt;</pre><p>That’s a different failure than “module exists but isn’t loaded” (which modprobe would have fixed on the spot). &quot;Not found&quot; means it was never built for this kernel at all — there&#39;s no file to load. No amount of modprobe-ing gets you out of that; the kernel image itself has to change.</p><p>The next question is whether the <em>kernel config</em> even asked for it, which you can check directly if /proc/config.gz is exposed (many minimal builds keep this on for exactly this kind of debugging):</p><pre>zcat /proc/config.gz | grep -E &#39;STATISTIC|NFT_CT|IP_VS|DEBUG_INFO_BTF&#39;</pre><pre># CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set<br># CONFIG_NFT_CT is not set<br>CONFIG_IP_VS=m<br># CONFIG_IP_VS_RR is not set<br># CONFIG_IP_VS_SH is not set<br># CONFIG_DEBUG_INFO_BTF is not set</pre><p>This is worth reading carefully, because it explains something that would otherwise look like bad luck: <strong>every mode of Service routing Kubernetes offers depends on this same missing category of kernel functionality</strong>, just in different specific modules.</p><p>kube-proxy mode What it needs Present here? iptables (default) xt_statistic for load-balancing across endpoints No nftables nft_ct (connection tracking in the nftables framework) No IPVS Scheduler modules like ip_vs_rr, ip_vs_sh No — only the core ip_vs module was present, none of the schedulers eBPF (e.g. Cilium&#39;s kube-proxy replacement) CONFIG_DEBUG_INFO_BTF for its CO-RE compiler No</p><p>I tried switching kube-proxy to nftables mode, expecting it to sidestep the iptables-specific gap. (Worth checking your Kubernetes version before you try this yourself — nftables mode is relatively new, GA only in fairly recent releases, so it may not be available at all depending on what you’re running.) It failed with its own version of the same story:</p><pre>Error: Could not process rule: No such file or directory<br>add rule ip kube-proxy filter-input ct state new jump service-endpoints-check</pre><p>Same shape of failure, different missing piece (nft_ct this time). I checked IPVS mode next — the core module was present, but every load-balancing scheduler it needs was compiled out. And an eBPF-based replacement (Cilium&#39;s kube-proxy-free mode) was a non-starter from the get-go, because its eBPF programs rely on kernel BTF metadata that this build never generated.</p><p><strong>Every single networking backend Kubernetes has for Services was blocked by the same root cause: a kernel that was trimmed down further than Kubernetes’ Service layer can tolerate.</strong></p><h3>Why this isn’t a Qualcomm problem, or an ARM problem</h3><p>It’s worth being explicit about this, because it’s easy to walk away from a story like this thinking “ah, embedded ARM boards are just bad at Kubernetes.” That’s not the lesson.</p><p>The architecture (ARM64 in this case) had nothing to do with it — ARM64 is a fully supported, heavily used Kubernetes platform. Nor is this specific to any one vendor’s board or BSP. I’ve since seen the same underlying pattern — “Services time out, pods work fine, iptables-restore complains about a missing extension&quot; — reported by people on completely different hardware: Yocto and Buildroot-based industrial devices, hardened/security-focused kernel builds in regulated environments, minimal container-host images trimmed for fast boot, even some heavily pruned custom cloud images.</p><p>The actual pattern is: <strong>any time a kernel is built by someone optimizing for “smaller, faster, more minimal” without Kubernetes networking specifically in mind, there’s a real chance the small set of netfilter/eBPF features kube-proxy depends on got left out.</strong> Nobody sets out to break Kubernetes — these modules just aren’t on by default when your priority is shaving boot time or image size off an embedded target. Kubernetes’ assumption that “any reasonably modern Linux kernel” can run it quietly bakes in an assumption that most people never have to think about, because most people run it on distros that ship those modules unconditionally.</p><p>If you’re evaluating hardware or a custom OS image for a Kubernetes deployment, this is worth checking <em>before</em> you’re debugging it in production:</p><pre>uname -r<br>zcat /proc/config.gz 2&gt;/dev/null | grep -E \<br>  &#39;NETFILTER_XT_MATCH_STATISTIC|NFT_CT|IP_VS_RR|IP_VS_WRR|IP_VS_SH|DEBUG_INFO_BTF&#39;</pre><p>If /proc/config.gz isn&#39;t available, check for the actual module files instead:</p><pre>find /lib/modules/$(uname -r) -name &#39;xt_statistic*&#39; -o -name &#39;nft_ct*&#39; -o -name &#39;ip_vs_rr*&#39;</pre><p>An empty result on either check is your early warning. Run it during hardware bring-up, not after you’ve built an application stack on top of it.</p><h3>What to actually do about it</h3><p>Once you’re sure this is what you’re facing, there are three real paths, in order of how “correct” versus how “fast” they are.</p><h3>1. Fix the kernel (the only permanent fix)</h3><p>If you have any influence over the OS image — you own the BSP, you’re using Yocto/Buildroot and control the recipes, you can request a rebuild — this is the fix that makes everything else in this article unnecessary. You need the relevant kernel config options enabled, either as loadable modules or built in:</p><pre>CONFIG_NETFILTER_XT_MATCH_STATISTIC=m<br>CONFIG_NFT_CT=m<br>CONFIG_IP_VS_RR=m<br>CONFIG_IP_VS_WRR=m<br>CONFIG_IP_VS_SH=m<br>CONFIG_DEBUG_INFO_BTF=y   # only needed if you plan to use an eBPF-based CNI/kube-proxy</pre><p>If you’re on Yocto specifically, meta-virtualization ships kernel config fragments intended for exactly this — pulling it in (and enabling the relevant DISTRO_FEATURES) is usually far less work than hand-assembling a kernel config from scratch. Whatever your build system, the request to whoever owns the image is concrete and small: &quot;enable these five config options and rebuild.&quot; It&#39;s a config change, not a kernel patch.</p><h3>2. Route around kube-proxy for internal traffic: headless Services</h3><p>If a kernel rebuild isn’t available on your timeline, there’s a workaround that’s more durable than it sounds, and it’s worth understanding <em>why</em> it works rather than treating it as a magic incantation.</p><p>A normal Kubernetes Service works by handing out a stable virtual IP (the ClusterIP) and relying on kube-proxy to translate that virtual IP into a real pod IP at the packet level. That translation step is exactly the thing that’s broken.</p><p>A <strong>headless Service</strong> (clusterIP: None) skips that translation entirely. Instead of returning a virtual IP, DNS lookups for a headless Service return the <strong>real IP addresses of the healthy pods behind it</strong>, directly. There&#39;s no kube-proxy step involved — no iptables rule, no rewriting, nothing for the broken layer to fail at.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: my-backend<br>spec:<br>  clusterIP: None      # &lt;- this is the entire fix<br>  selector:<br>    app: my-backend<br>  ports:<br>    - port: 8080</pre><p>Since pod-to-pod networking was proven healthy earlier in this process (that was the whole point of the pod-IP-vs-Service-IP test), and Kubernetes’ DNS automatically keeps a headless Service’s records in sync with which pods are currently Ready, this sidesteps the broken layer cleanly and durably — durably in the sense that it keeps working across pod restarts, not durably in the sense of surviving a Helm upgrade (see the caveats below).</p><p>It’s not free, though, and it’s worth knowing the tradeoffs before you convert a Service and move on:</p><ul><li><strong>Load-balancing changes.</strong> A normal Service round-robins traffic across pods via kube-proxy. A headless Service just hands the <em>client</em> a list of every pod IP — the client decides what to do with that. For a single-replica Service this is a non-issue. For a multi-replica Service, whether you actually get load-balancing now depends on your client library’s behavior (gRPC handles this reasonably; plenty of naive HTTP clients just always use the first address).</li><li><strong>Stale DNS caching.</strong> Some client code resolves a hostname once at startup and never again. With a normal ClusterIP, that’s invisible — the IP never changes even when the backing pod does. With headless, if a pod restarts and its IP changes, a client holding a stale cached IP will start failing until it re-resolves. Well-behaved clients re-resolve periodically; not all clients are well-behaved.</li><li><strong>It reverts on redeploy.</strong> If your Services are templated by Helm (or any GitOps tool), a live kubectl patch or manual edit will be silently overwritten the next time that chart is applied, because the chart still defines a normal ClusterIP. The change needs to live in the chart&#39;s source, not just in the running cluster, or it&#39;s temporary by construction.</li><li><strong>It’s a workaround, not a fix.</strong> Anything that genuinely needs a real ClusterIP — some ingress-controller setups, certain operators, LoadBalancer/NodePort services (see below) — still needs working kube-proxy underneath it. Headless Services solve <em>internal, pod-to-pod-style service discovery</em>. They don’t solve everything Kubernetes networking does.</li></ul><h3>3. For your own debugging/admin access: tunnel straight to a pod IP</h3><p>Separately from fixing application traffic, you’ll likely want to reach the cluster’s API server or an ingress controller from your own machine — for kubectl, a GUI tool, or just testing in a browser. NodePorts and LoadBalancer-type Services are <em>also</em> implemented via kube-proxy, so they&#39;re subject to the exact same breakage; an SSH tunnel aimed at a NodePort will fail with &quot;no route to host&quot; for the same underlying reason everything else did.</p><p>The same principle applies here as with headless Services: skip the layer that’s broken, target something real directly.</p><pre># Find the actual pod behind whatever you&#39;re trying to reach<br>kubectl get pod -n ingress-nginx -l app.kubernetes.io/component=controller \<br>  -o jsonpath=&#39;{.status.podIP}&#39;</pre><pre># Tunnel straight to that pod&#39;s IP and port, not the Service/NodePort<br>ssh -L 8443:&lt;pod-ip&gt;:443 user@your-host</pre><p>The catch is that a pod’s IP changes whenever it restarts, so this needs re-checking periodically — a small price for a connection that actually works. kubectl port-forward is an even better version of the same idea where it fits your workflow, since it goes through the API server&#39;s proxy machinery rather than raw networking, and doesn&#39;t care about kube-proxy at all.</p><h3>4. If you don’t strictly need Kubernetes: consider not using its Service layer</h3><p>This is worth saying plainly, even though it wasn’t the path taken here: if you’re deploying to a <strong>single node</strong> and the appeal of Kubernetes for you is orchestration and self-healing rather than multi-node scheduling, it’s worth asking whether you need the Service abstraction at all. Something like Docker Compose does inter-container discovery over its own bridge network — the exact same kind of direct, non-virtual-IP networking that was proven to work throughout this whole investigation — without depending on any of the kernel features that caused this entire multi-day detour. It’s a legitimate answer for a single-node embedded deployment, not just a fallback.</p><h3>The diagnostic checklist, distilled</h3><p>If you take nothing else from this, take the sequence. It scales down to five minutes for a quick sanity check, or up to a full investigation if something’s actually wrong:</p><ol><li><strong>Isolate the layer</strong> — test a Service by DNS/ClusterIP <em>and</em> by a backing pod’s real IP, side by side (see the comparison above). Pod IP working while the Service fails points straight at kube-proxy.</li><li><strong>Watch kube-proxy’s own logs.</strong> journalctl -u k3s (or kubectl logs on the kube-proxy pod, depending on your distribution) during a sync cycle. A repeating iptables-restore / nftables sync failure naming a specific missing extension is about as direct as root-cause evidence gets.</li><li><strong>Confirm the module is actually absent</strong>, not just unloaded — modprobe &lt;name&gt; failing with &quot;not found&quot; (versus loading successfully) tells you whether this is a five-second modprobe fix or a kernel-image problem.</li><li><strong>Check the kernel config directly</strong> if you can (/proc/config.gz or the module tree under /lib/modules), and check it for <em>all</em> the Service-routing backends, not just the default one — a kernel missing one is often missing several.</li><li><strong>Decide: fix the kernel, or route around kube-proxy.</strong> If you can influence the OS image, that’s the only real fix. If you can’t (yet), headless Services and direct pod-IP access get you a genuinely working system in the meantime — just remember they’re workarounds, and write down what needs to happen for the real fix to land eventually.</li></ol><h3>The takeaway</h3><p>Kubernetes has quietly baked in an assumption most of us never have to think about: that the Linux kernel underneath it has a small, specific set of netfilter or eBPF features available. On any general-purpose distro, that’s true so reliably it’s invisible. On a custom, minimal, or embedded kernel — the kind more and more people are reaching for as Kubernetes creeps onto edge devices, appliances, and industrial hardware — it’s an assumption that can quietly fail, and when it does, the failure looks nothing like “missing kernel module.” It looks like your database client panicking, or a service getting i/o timeout for no visible reason, or a frontend showing a bare 502.</p><p>The way out isn’t to memorize this one incident — it’s to know the one test that separates “your app is broken” from “your kernel is broken”: pod IP versus Service IP. Everything else in this article was just following that thread to its end.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31da3d443377" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[gRPC: For High-Performance Microservices Communication]]></title>
            <link>https://blog.devops.dev/grpc-for-high-performance-microservices-communication-3050b16dbcc0?source=rss-5e90c25b156a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3050b16dbcc0</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[DAMAN]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 17:59:27 GMT</pubDate>
            <atom:updated>2025-03-05T17:59:27.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B87k8HXgp22WMHSg" /></figure><p>gRPC (Google Remote Procedure Call) is a high-performance, open-source RPC framework developed by Google. It enables efficient communication between services in microservices architectures by using <strong>Protocol Buffers (protobufs)</strong> for serialization and HTTP/2 as the transport protocol. Here’s a detailed breakdown of gRPC:</p><h3>What is gRPC?</h3><p>gRPC is a modern, high-performance framework for inter-service communication, supporting multiple programming languages. It is commonly used in <strong>distributed systems</strong> to enable fast and efficient communication between backend services.</p><h3>Why Do We Need gRPC?</h3><p>As applications grow in complexity — especially in <strong>microservices architectures</strong> — efficient communication between services becomes critical. Traditional REST-based APIs, while widely used, have <strong>performance and scalability limitations</strong>. gRPC solves these issues by offering <strong>faster, more efficient, and scalable communication</strong></p><h3>Key Features of gRPC</h3><h4>Uses HTTP/2 Instead of HTTP/1.1</h4><ul><li><strong>Multiplexing:</strong> Multiple requests can be sent in parallel over a single connection, reducing overhead.</li><li><strong>Streaming:</strong> Supports <strong>bidirectional streaming</strong>, where both client and server can send messages independently.</li><li><strong>Header Compression:</strong> Reduces network bandwidth consumption.</li></ul><h4>Protocol Buffers (Protobuf) Instead of JSON</h4><ul><li><strong>Smaller payloads:</strong> Binary serialization is more compact and faster than JSON.</li><li><strong>Stronger typing:</strong> Prevents data structure mismatches between client and server.</li><li><strong>Backward and forward compatibility:</strong> Older clients can communicate with newer servers and vice versa.</li></ul><h4>Supports Multiple Languages</h4><ul><li>Official support for <strong>Go, Python, Java, Node.js, C++, C#, Ruby, PHP</strong>, etc.</li><li>Makes it easier to build <strong>polyglot</strong> (multi-language) microservices.</li></ul><h4>Authentication and Security</h4><ul><li>Supports <strong>TLS</strong> (Transport Layer Security) for encryption.</li><li>Can integrate with <strong>OAuth, JWT, and API Gateway security models</strong>.</li></ul><h3>How gRPC Works?</h3><p><strong>Define a Service using </strong><strong>.proto files</strong></p><p>You define <strong>RPC methods</strong> and <strong>message structures</strong> using Protocol Buffers.</p><p><strong>Generate Code for Client &amp; Server</strong></p><p>The .proto file is compiled using the protoc compiler to generate <strong>gRPC stubs</strong> for the target language.</p><p><strong>Implement the Server</strong></p><p>The server implements the defined RPC methods.</p><p><strong>Client Calls the Server</strong></p><p>The client uses the generated stub to make remote procedure calls.</p><h3>gRPC Communication Types</h3><h4>Unary RPC (Simple Request-Response)</h4><ul><li>Client sends a request → Server processes it → Server sends back a response.</li></ul><h4>Server Streaming RPC</h4><ul><li>Client sends a request → Server streams multiple responses back.</li></ul><h4>Client Streaming RPC</h4><ul><li>Client sends multiple requests → Server processes and sends a single response.</li></ul><h4>Bidirectional Streaming RPC</h4><ul><li>Both client and server <strong>send and receive multiple messages asynchronously</strong>.</li></ul><h3>When to Use gRPC?</h3><p>✅ <strong>Microservices communication</strong><br>✅ <strong>High-performance APIs with low latency</strong><br>✅ <strong>Streaming data (video, chat, real-time updates)</strong><br>✅ <strong>Language-agnostic distributed systems</strong><br>✅ <strong>IoT and mobile apps (efficient data transfer)</strong></p><h3>When NOT to Use gRPC?</h3><p>❌ <strong>Public REST APIs:</strong> Many browsers don’t support gRPC natively (use REST or GraphQL instead).<br>❌ <strong>Client-side apps that require direct browser support:</strong> gRPC-Web is needed for browser support.</p><h3>Best Practices for gRPC in Kubernetes</h3><ol><li><strong>Use gRPC-Aware Proxies</strong>: Prefer Envoy or Nginx over default Kubernetes proxies</li><li><strong>Implement Client-Side Load Balancing</strong>: Distribute connection logic</li><li><strong>Leverage Service Mesh Technologies</strong>: Istio, Linkerd</li><li><strong>Monitor and Tune</strong>: Continuous performance evaluation</li><li><strong>Consider Alternative Deployment Strategies</strong>: StatefulSets, headless services</li></ol><h3>Looking Ahead</h3><p>As cloud-native architectures continue to evolve, gRPC is positioning itself as a critical technology for building interconnected, efficient, and flexible distributed systems. With growing support from major tech companies and an active open-source community, gRPC is more than just a communication protocol — it’s a strategic approach to designing robust, performant microservices.</p><p><strong>Key Takeaway:</strong> In the world of distributed computing, gRPC isn’t just about sending data — it’s about sending data <em>intelligently</em>, <em>efficiently</em>, and <em>seamlessly</em>.</p><p>Whether you’re building the next generation of cloud services, IoT platforms, or complex distributed systems, gRPC offers a powerful toolkit for creating communication layers that are as dynamic and responsive as the applications they support.</p><p>I’ll be diving deep into <strong>gRPC</strong> and why <strong>Kubernetes isn’t always the best fit for gRPC</strong> — along with how to <strong>fix those challenges</strong>! Stay tuned for the full blog. <strong>Follow for more!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3050b16dbcc0" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/grpc-for-high-performance-microservices-communication-3050b16dbcc0">gRPC: For High-Performance Microservices Communication</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Kubernetes Health Checks: Liveness, Readiness, and Keeping Your Pods Resilient]]></title>
            <link>https://medium.com/@sdaman830/mastering-kubernetes-health-checks-liveness-readiness-and-keeping-your-pods-resilient-b6642b0c0bec?source=rss-5e90c25b156a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6642b0c0bec</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[DAMAN]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 08:59:37 GMT</pubDate>
            <atom:updated>2025-03-05T08:59:37.990Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes is a powerful orchestration platform, but even the best systems need safeguards to ensure reliability. Health checks — <strong>liveness</strong> and <strong>readiness probes</strong> — are critical tools to keep your applications running smoothly. In this blog, we’ll break down how these probes work, how Kubernetes restarts failing pods, and how to troubleshoot issues where pods appear “down” but still route traffic.</p><h3>Why Health Checks Matter</h3><p>Imagine a pod crashes silently or becomes unresponsive due to a memory leak. Without health checks, Kubernetes won’t know to intervene, leaving broken pods to serve errors or timeouts. Health probes act as a “doctor” for your pods, constantly checking their status and triggering recovery actions.</p><p>Kubernetes offers two types of probes:</p><ol><li><strong>Liveness Probes</strong>: Determine if a pod is <em>alive</em>. If it fails, Kubernetes restarts the pod.</li><li><strong>Readiness Probes</strong>: Determine if a pod is <em>ready to serve traffic</em>. If it fails, Kubernetes stops sending requests to the pod.</li></ol><h3>How Probes Work</h3><p>Probes are configured in your pod’s YAML definition. You can define them to check endpoints via HTTP requests, TCP sockets, or custom command executions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*NyHier1JlLfJLhQPe_00uQ.png" /><figcaption>Example Configuration</figcaption></figure><h4>initialDelaySeconds</h4><ul><li>The number of seconds Kubernetes waits before running the probe for the first time after the container starts.</li><li>Helps avoid false failures when an app takes time to start.</li><li>Example: initialDelaySeconds: 5 means Kubernetes will wait <strong>5 seconds</strong> before the first check.</li></ul><h4>httpGet</h4><ul><li>Defines an HTTP request to check the container’s health.</li><li>Kubernetes sends a request to the given path and port.</li><li>If the response is <strong>HTTP 200–399</strong>, the probe is considered successful.</li></ul><h4>path</h4><ul><li>Specifies the endpoint the probe will hit.</li><li>Example:</li><li>path: /healthz for the <strong>livenessProbe</strong> means Kubernetes will send a request to http://&lt;pod-ip&gt;:8080/healthz to check if the container is alive.</li><li>path: /ready for the <strong>readinessProbe</strong> means Kubernetes will check http://&lt;pod-ip&gt;:8080/ready to see if the container is ready.</li></ul><h4>periodSeconds</h4><ul><li>Defines how often (in seconds) the probe will run after the first check.</li></ul><h3>How Kubernetes Restarts Pods</h3><ul><li><strong>Liveness Probe Failure</strong>:<br>If the liveness probe fails repeatedly (after failureThreshold attempts), Kubernetes kills the pod and starts a new one. This is called a <strong>restart</strong>.</li><li>Use kubectl describe pod &lt;pod-name&gt; to see restarts (Restart Count).</li><li>Restart policies (in spec.restartPolicy) define how aggressively Kubernetes retries (default: Always).</li></ul><p><strong>Readiness Probe Failure</strong>:<br>If the readiness probe fails, Kubernetes removes the pod’s IP from the <strong>Service’s endpoint list</strong>, stopping new traffic. The pod isn’t restarted — it’s just quarantined until the probe succeeds again.</p><h3>Why “Dead” Pods Might Still Receive Traffic</h3><p>You might notice pods in CrashLoopBackOff or Error state still connected to a Service. Here’s why:</p><p><strong>Termination Grace Period</strong>:<br>When a pod is terminated, Kubernetes gives it time (default: 30s) to shut down gracefully. During this window, the pod may still accept requests</p><p><strong>Probe Misconfiguration</strong>:</p><ul><li>If readiness probes aren’t set up, Kubernetes assumes the pod is always ready.</li><li>Long periodSeconds or high failureThreshold values delay detection.<strong>Stale Endpoints</strong>:<br>The Kubernetes control plane updates endpoints asynchronously. There’s a brief delay between a pod failing and being removed from the Service.</li></ul><h3>Best Practices</h3><ul><li><strong>Use Both Probes</strong>:<br>Liveness for crashes, readiness for temporary unavailability (e.g., loading large data).</li><li><strong>Avoid Overlapping Liveness &amp; Readiness</strong>:<br>A failing liveness probe should indicate an unrecoverable error.</li><li><strong>Test Probe Logic</strong>:<br>Simulate failures (e.g., kubectl exec &lt;pod&gt; -- kill 1) to validate behavior.</li><li><strong>Monitor with Dashboards</strong>:<br>Use tools like Prometheus or the Kubernetes Dashboard to track probe failures.</li></ul><p><em>Follow me on </em><a href="https://x.com/Daman142001"><em>Twitter</em></a><em> for more DevOps insights and case studies.</em></p><p><em>Want to learn more about implementing GitOps in your organization? Let’s connect on </em><a href="https://www.linkedin.com/in/damanpreet-singh-9b25a9225/"><em>LinkedIn</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6642b0c0bec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Multi-Cluster GitOps: From Theory to Production]]></title>
            <link>https://medium.com/@sdaman830/mastering-multi-cluster-gitops-from-theory-to-production-ffae46883e46?source=rss-5e90c25b156a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffae46883e46</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[DAMAN]]></dc:creator>
            <pubDate>Mon, 24 Feb 2025 10:10:00 GMT</pubDate>
            <atom:updated>2025-02-24T10:10:00.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*OVvApfxNcEYi-6FNOzUKAQ.png" /></figure><h3>Introduction</h3><p>In today’s cloud-native landscape, managing multiple Kubernetes clusters across different environments and regions has become increasingly common. GitOps has emerged as a powerful methodology for managing such complex infrastructures, providing consistency, reliability, and automation. This blog post explores how to implement GitOps at scale for multi-cluster environments, covering best practices, challenges, and solutions.</p><h3>Why Multi-Cluster Setup?</h3><ul><li><strong>Resilience</strong>: Avoid single points of failure with distributed clusters.</li><li><strong>Compliance</strong>: Isolate workloads (e.g., PCI-compliant clusters for fintech).</li><li><strong>Scalability</strong>: Serve global traffic by deploying clusters closer to users.</li><li><strong>Cost Optimization</strong>: Mix spot/preemptible instances across clusters.</li></ul><h3>Understanding GitOps at Scale</h3><h4>Core Principles</h4><p>GitOps extends beyond simple infrastructure as code. At its heart, it embraces four key principles:</p><ol><li>Declarative Configuration: All system configurations are defined declaratively and stored in Git</li><li>Version Control: The entire system state is version controlled in Git</li><li>Automated Synchronization: Approved changes are automatically applied to the system</li><li>Continuous Reconciliation: <strong>System drift</strong> is automatically detected and corrected</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BOKI_8JI3wBvpfEY.png" /></figure><h3><strong>GitOps helps with multi-cluster setups in these key ways:</strong></h3><h4>Single Source of Truth</h4><ul><li>All cluster configurations live in Git repositories</li><li>Every change is tracked, versioned, and auditable</li><li>Eliminates configuration drift between clusters</li></ul><h4><strong>Automated Synchronization</strong></h4><ul><li>Changes in Git automatically apply to all relevant clusters</li><li>No manual intervention needed for deployments</li><li>Ensures all clusters stay in sync with desired state</li></ul><h4><strong>Standardized Management</strong></h4><ul><li>Same deployment process across all environments</li><li>Consistent security policies and access controls</li><li>Unified approach for development, staging, and production</li></ul><h4>Easy Scaling</h4><ul><li>New clusters can be bootstrapped automatically</li><li>Reuse configurations across clusters</li><li>Templates and base configurations make adding clusters simple</li></ul><h3>Creating an Optimal GitOps Experience</h3><h4>Core Design Foundations</h4><p>A thoughtfully structured Git repository serves as the cornerstone of successful GitOps implementation. Here’s what makes it effective:</p><h4>Repository Organization</h4><p>Create a transparent, logical folder structure that immediately communicates:</p><ul><li>Where configurations live</li><li>How environments are separated</li><li>Which components are shared</li><li>Where environment-specific kustomizations exist</li></ul><h4>Secure Environment Progression</h4><p>Implement promotion patterns that:</p><ul><li>Allow changes to flow smoothly from dev → staging → production</li><li>Maintain strict separation of environment-specific secrets</li><li>Prevent accidental exposure of sensitive data between environments</li></ul><h4>Centralized Dependency Management</h4><p>Establish a single source of truth by:</p><ul><li>Using a common base configuration for shared resources</li><li>Leveraging Kustomize overlays for environment-specific changes</li><li>Minimizing duplicate configurations</li><li>Maintaining consistent versions across clusters</li></ul><h4>Environment Scalability</h4><p>Design with growth in mind:</p><ul><li>Make new environment creation repeatable and automated</li><li>Use templates and base configurations</li><li>Document environment bootstrapping process</li><li>Keep configuration modular for easy reuse</li></ul><h4>Access Control Design</h4><p>Build in security from the start:</p><ul><li>Define clear RBAC policies</li><li>Separate responsibilities between teams</li><li>Control who can make changes to which environments</li><li>Implement approval workflows for sensitive changes</li></ul><h4>GitOps Repo Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*nNne1wgVbySCCYpll8NFKg.png" /><figcaption>How the Repository holding manifests for Cluster’s look like</figcaption></figure><h3>Understanding GitOps with Kustomize: Base, Overlays, Patches and Kustomization</h3><h4>Base: Your Foundation</h4><p>The base directory is the foundation of your Kubernetes configurations. Think of it as your default configuration that contains all the essential components your application needs to run.</p><h4>What Goes in Base?</h4><ul><li>Core Kubernetes manifests (deployments, services, etc.)</li><li>Common configurations shared across all environments</li><li>Default resource settings</li><li>Cluster-wide configurations</li></ul><h4>Base Structure Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJmJZMV1v7my3RAJx19pKg.png" /><figcaption>Base folder Arrangements</figcaption></figure><h4>Overlays: Environment-Specific Configurations</h4><p>Overlays allow you to create environment-specific variations without modifying your base configuration. Think of overlays as layers that sit on top of your base configuration.</p><h4>Common Overlay Use Cases:</h4><ul><li>Different resource limits for dev/prod</li><li>Environment-specific replicas</li><li>Different ingress configurations</li><li>Environment-specific configs and secrets</li></ul><h4>Overlays Structure Example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MOY6m0-M7hjpNr15zEA0DA.png" /><figcaption>Overlays folder arrangements</figcaption></figure><h4>Patches: Making Targeted Changes</h4><p>Patches are how you modify specific parts of your base configuration for different environments. They’re like surgical tools that allow you to make precise changes.</p><h4>Patch.yml structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nYI2GSLJ4Oy-3SzNKOwoSQ.png" /><figcaption>Typical Patch.yml file</figcaption></figure><h4>Common Patching Scenarios:</h4><ul><li>Modifying resource limits</li><li>Changing replica counts</li><li>Adding environment-specific labels</li><li>Updating image tags</li><li>Modifying configuration values</li></ul><h4>What is kustomization.yaml?</h4><p>The kustomization.yaml file is the heart of Kustomize. It acts as a configuration controller that tells Kustomize:</p><ul><li>Which resources to include</li><li>What modifications to apply</li><li>How to generate new resources</li><li>Which values to substitute</li></ul><h3>Basic Structure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dk6kDtjAdiobKKNpYXISAw.png" /><figcaption>Kustomization.yml for prod env</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uAQ4BQ0_z3Rv-eKACimGow.png" /><figcaption>Kustomization.yml for dev env</figcaption></figure><h4>Challenges &amp; Solutions in Scaling GitOps</h4><p>Add a dedicated section to address real-world hurdles and mitigation strategies:</p><ul><li><strong>Secrets Management</strong>: Use tools like SOPS, Sealed Secrets, or external secret operators (e.g., Vault) to securely store sensitive data in Git.</li><li><strong>Stateful Workloads</strong>: Discuss strategies for databases (e.g., using GitOps for configuration while relying on operator-based backups).</li><li><strong>Network Complexity</strong>: Managing cross-cluster communication (e.g., service meshes like Istio) and ingress/egress policies.</li><li><strong>Drift Detection</strong>: Highlight tools like Argo CD’s kubectl diff integration to detect and resolve configuration drift.</li><li><strong>Rollback Strategies</strong>: Automated rollback via Git revert and health checks.</li></ul><h3>Final Conclusion Draft</h3><p>“GitOps transforms multi-cluster management by enforcing consistency, scalability, and security through declarative workflows. By leveraging tools like Kustomize, Argo CD, and robust secret management practices, teams can efficiently manage global deployments while minimizing drift and maximizing resilience. Start small — automate a single cluster, refine your workflows, and scale confidently. As cloud-native ecosystems evolve, GitOps will remain pivotal in bridging hybrid environments and emerging paradigms like edge computing. Now is the time to embrace GitOps and turn infrastructure complexity into a competitive advantage.”</p><p><em>Follow me on </em><a href="https://x.com/Daman142001"><em>Twitter</em></a><em> for more DevOps insights and case studies.</em></p><p><em>Want to learn more about implementing GitOps in your organization? Let’s connect on </em><a href="https://www.linkedin.com/in/damanpreet-singh-9b25a9225/"><em>LinkedIn</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffae46883e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>