<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Goutham Reddy Kotapalle on Medium]]></title>
        <description><![CDATA[Stories by Goutham Reddy Kotapalle on Medium]]></description>
        <link>https://medium.com/@grkotapalle?source=rss-3bea6fd817ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*C_JYmYngZydIu2BXK5qxew.jpeg</url>
            <title>Stories by Goutham Reddy Kotapalle on Medium</title>
            <link>https://medium.com/@grkotapalle?source=rss-3bea6fd817ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:41:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@grkotapalle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Kubernetes Needed DRA: Understanding the Model It’s Evolving From]]></title>
            <link>https://grkotapalle.medium.com/why-kubernetes-needed-dra-understanding-the-model-its-evolving-from-0557db5612b4?source=rss-3bea6fd817ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/0557db5612b4</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Goutham Reddy Kotapalle]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 09:10:36 GMT</pubDate>
            <atom:updated>2026-03-19T16:59:11.270Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Kubernetes Needed DRA: The Resource Model It Outgrew</h3><figure><img alt="Diagram comparing Count-Based/Node-Scoped device plugins vs Attribute-Aware/Pre-Allocated DRA" src="https://cdn-images-1.medium.com/max/1024/1*XnYJF8h4KiwI5qqDOC149g.png" /><figcaption>From count-based device scheduling to claim-based resource allocation</figcaption></figure><p>Everyone’s talking about Dynamic Resource Allocation, or DRA, in Kubernetes.</p><p>For good reason. DRA introduces a more flexible, scheduler-aware model for managing hardware resources such as GPUs and other accelerators. With DRA reaching GA in Kubernetes 1.34, the ecosystem is moving beyond a world where specialized devices are treated primarily as countable node-level inventory. Resources can now be described, selected, and claimed through first-class APIs. To see why that shift matters, it helps to revisit the model Kubernetes relied on for years.</p><p>At the node level, that story starts with kubelet and plugins.</p><h3>Before DRA: Kubelet, Device Plugins, and Extended Resources</h3><p>Kubelet is responsible for making Pods run on a node, but it is intentionally narrow in what it knows how to manage on its own. It does not come with built-in knowledge of every accelerator, NIC, FPGA, or other vendor-specific device that might exist on a machine.</p><p>Instead, Kubernetes relies on a plugin model at the node boundary. For specialized hardware, that extension point is the device plugin framework. Vendors run device plugins on each node, register them with kubelet, advertise the devices they manage, and report their health. Kubelet then exposes those devices to the rest of the system as extended resources that Pods can request.</p><p>That design was pragmatic. Kubernetes did not need native support for every class of hardware; it needed a consistent way for node-local components to translate local hardware state into something the scheduler and kubelet could understand. In practice, specialized devices became schedulable node inventory.</p><p>That abstraction is important. In the pre-DRA world, hardware was modeled less as something to be described and selected, and more as something to be counted and consumed.</p><h3>How kubelet discovers plugins</h3><p>On every node, kubelet exposes a registration path:</p><p>/var/lib/kubelet/plugins_registry/</p><p>When a plugin starts, usually as part of a DaemonSet, it creates its own Unix socket and reaches out to kubelet’s registration service. Kubelet does not go looking for plugins first. The plugin initiates the handshake. For device resources, the relevant extension point is the device plugin framework.</p><p><strong>Registration flow:</strong></p><pre>Plugin starts<br>  → creates its socket<br>  → connects to kubelet registration socket<br>  → sends Register request</pre><pre>kubelet<br>  → validates plugin<br>  → stores it internally<br>  → connects back to plugin</pre><p>Once that happens, the plugin becomes part of the node’s runtime behavior.</p><p>For device plugins, that means exposing node-local hardware to kubelet. For CSI node plugins, that means serving kubelet’s node-side storage operations, such as staging and publishing volumes.</p><h3>What happens after registration</h3><p>Registration is only the beginning.</p><p>For device plugins, kubelet then asks for device state through ListAndWatch. The plugin sends the full set of devices and keeps streaming updates when something changes. If kubelet restarts, the device plugin is expected to detect that and re-register. Kubernetes’ device plugin docs describe ListAndWatch as a streaming call used to report the set and health of devices.</p><p><strong>Resource discovery flow:</strong></p><pre>kubelet → calls ListAndWatch()<br>plugin → sends full device list<br>plugin → keeps streaming updates on changes</pre><p>For example, a node with four healthy GPUs might end up advertising:</p><pre>nvidia.com/gpu: 4</pre><p>That is the signal kubelet reports through the Node object, and that is what the scheduler uses when making placement decisions. If a device becomes unhealthy, allocatable is reduced for future scheduling.</p><h3>What Pod startup looks like in the old model</h3><p>Once a Pod requests a GPU, the flow looks like this:</p><pre>User creates Pod (requests 1 GPU)<br>  → API server stores Pod<br>  → scheduler picks a node</pre><pre>Pod arrives at kubelet<br>  → kubelet calls Allocate() on the device plugin<br>  → plugin returns device-specific setup details<br>kubelet<br>  → passes those details to the container runtime<br>  → container starts with GPU access</pre><p>At a high level, the model is:</p><p><strong>discover → advertise → schedule → allocate → run</strong></p><p>That separation is important, because it explains both why the model worked well for a long time and why it eventually started to strain.</p><h3>The key tradeoff in the device plugin model</h3><p>Before registration, a node has no GPUs as far as Kubernetes is concerned. After that handshake, those devices become schedulable resources.</p><p>This is not static configuration. It is a dynamic system in which node capabilities are introduced and maintained at runtime. That design was intentional: it worked well in a world where resources were mostly node-local and relatively homogeneous. Keeping resource state close to kubelet avoided pushing too much complexity into the scheduler and let the node remain the source of truth for health and allocation timing.</p><p>That tradeoff made sense early on. It becomes less comfortable once workloads start caring about hardware type, topology, sharing, and allocation semantics across a cluster.</p><h3>Where the old model starts to break down</h3><h4>Loss of resource semantics</h4><p>At its core, the device plugin model reduces resources to a count.</p><pre>nvidia.com/gpu: 4</pre><p>That is useful, but it is also very coarse. The scheduler can see that a node has four GPUs, but it has no native understanding of whether those GPUs are H100s or T4s, how much memory they have, or how they are connected.</p><p>People were not blocked by this. They worked around it.</p><p>In practice, teams often used node labels, node affinity, vendor-specific labeling conventions, or Node Feature Discovery to push richer GPU information into scheduling. Kubernetes’ GPU scheduling docs explicitly call out using Node Feature Discovery to discover hardware and advertise those features as node labels.</p><p>For example:</p><pre>nodeSelector:<br>  gpu-type: a100</pre><p>That can work. But it also comes with clear limits. <strong>Labels are static, manually curated, or separately discovered, and still operate at the node level</strong>. They do not represent per-device allocation state, and they do not control which specific GPU kubelet will assign later.</p><p>So the old model was not unusable. It was workable, but increasingly brittle. The deeper the hardware semantics became, the more teams had to encode them indirectly through labels, affinity, and external conventions.</p><p>A Pod can land on the right node and still get the wrong device.</p><p>DRA allows the scheduler to filter by <strong>attributes</strong> such as compute capability or memory (e.g., “Give me a GPU with cuda.compute.capability: 9.0 and at least 80Gi memory&quot;) instead of relying on indirect label conventions like nodeSelector: gpu-type=h100</p><h4>Late allocation</h4><p>Another limitation is timing.</p><p>The scheduler picks a node based on what kubelet has advertised, but the actual device assignment still happens later on that node during Pod startup, when kubelet calls Allocate().</p><pre>scheduler → picks node<br>kubelet → assigns specific device later</pre><p>For workloads that depend on specific topology or interconnect characteristics, selecting a node first and a device later can produce placements that are syntactically valid but operationally suboptimal. The scheduler may place the workload on a node with enough advertised capacity, while the eventual device assignment fails to match the locality or connectivity assumptions the workload actually depends on.</p><p>That means placement and exact allocation are split across two phases. For simple resources, that is fine. For more constrained resources, that separation becomes harder to reason about.</p><h4>Limited sharing and expressiveness</h4><p>The traditional device plugin model is also limited in how it expresses richer allocation intent.</p><p>Sharing often ends up being implemented through vendor-specific tooling or custom logic outside core Kubernetes. DRA, by contrast, is explicitly designed to request and share resources among Pods.</p><p>That matters in multi-tenant systems, where utilization and isolation both matter, and where “just count devices” is often not enough.</p><h4>Hard-to-debug failures</h4><p>The failure modes are also awkward.</p><p>If a plugin fails to register, kubelet may simply report fewer devices or none at all. From the outside, this often looks like a scheduling issue because the Pod just never lands on that node.</p><ul><li>plugin fails to register</li><li>kubelet reports gpu: 0</li><li>scheduler avoids node</li></ul><p>But the real problem is not in scheduling. It is in node-level capability discovery.</p><p>That debugging split is one of the reasons this layer stays invisible until something goes wrong.</p><h3>Why this led to DRA</h3><p>DRA exists to address exactly these kinds of gaps.</p><p>Kubernetes documents DRA as a way to request and share resources, especially attached devices such as hardware accelerators. With DRA, drivers and cluster admins define <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#deviceclass">DeviceClasses</a> that workloads can claim using <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaims-templates">ResourceClaims</a> (similar to your PersistentVolumeClaims in the storage world), Kubernetes allocates matching devices to specific claims, and then places Pods on nodes that can access those devices.</p><p>The important shift is that DRA does not just add a new claim object. It also adds richer resource information into the API model.</p><p>ResourceClaim is how a workload asks for hardware. <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceslice">ResourceSlice</a> is how drivers can publish structured information about available resources and capacity into Kubernetes, instead of forcing everything through indirect node labels and opaque extended-resource counts. Recent DRA updates around consumable capacity continue that trend.</p><p>One important difference is that allocation becomes its own phase. In the device plugin model, a Pod is scheduled first, and kubelet assigns a specific device later during startup. In DRA, allocation can happen earlier (via a ResourceClaim) during the scheduling cycle, before the Pod is placed onto a node. Kubernetes’ original DRA design explicitly supports both immediate allocation and wait-for-first-consumer style allocation, similar to the storage world.</p><p>DRA also changes the data plane. It was designed around CDI, the Container Device Interface (again, very similar to your CSI, the Container Storage Interface in the storage world), as a standard way to describe devices to the container runtime. That gives drivers a more uniform path for handing GPUs, NICs, or FPGAs to runtimes like containerd and CRI-O. CDI support was introduced alongside early DRA work and later expanded in Kubernetes releases.</p><h3>Old model</h3><pre>Pod → asks for &quot;1 GPU&quot;<br>scheduler → finds a node with gpu: 1<br>kubelet → assigns a specific device later</pre><h3>DRA model</h3><pre>User → creates a ResourceClaim<br>        (describes what is needed)</pre><pre>Driver + Kubernetes APIs<br>  → publish available resources and capacity<br>  → allocate a matching resource to the claim<br>Scheduler<br>  → places Pods on nodes that can access that allocation<br>kubelet<br>  → performs the node-side setup</pre><p>The shift is not just “better scheduling.”</p><p>It is a move from static counting toward claim-based, API-visible resource management. In the old model, advanced hardware semantics were often pushed into labels and scheduling conventions. In the DRA model, more of that information can live in first-class resource APIs instead.</p><p>That makes several things easier in practice.</p><p>A training job can ask for the kind of GPU it actually needs instead of hoping a label convention and a late allocation step line up. A multi-tenant platform can model sharing more directly instead of stitching together vendor-specific workarounds. And workloads that care about topology or connectivity can participate in a richer allocation flow instead of discovering too late that the selected hardware was the wrong fit. Recent Kubernetes docs and release notes explicitly frame DRA around selecting, allocating, sharing, and configuring GPUs, TPUs, NICs, and other devices with richer semantics than the old extended-resource model.</p><h3>Closing thought</h3><p>The device plugin model still matters. It remains foundational in Kubernetes today, and it solved an important class of problems cleanly.</p><p>But it was built around node-local discovery and late allocation. As hardware becomes more specialized and workloads become more demanding, those assumptions start to show their limits.</p><p>DRA does not mean the old model was useless. It means the ecosystem outgrew a model that depended heavily on counts, labels, and node-local decisions for something that increasingly needed richer, shared, and more declarative resource management.</p><p>Understanding the old plugin path makes that shift much easier to reason about.</p><h3>What’s next</h3><p>DRA is easier to appreciate once you see the limits of the model it is replacing. That old plugin path still matters, but it no longer matches the shape of modern accelerator scheduling.</p><p>In the next post, I’ll go deeper into DRA itself: the end-to-end flow, how ResourceClaims and ResourceSlices work in practice, how scheduling changes, and what this means for drivers and platform teams.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0557db5612b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[etcd’s “Time Travel” Architecture: Why MVCC Is the Heartbeat of Kubernetes]]></title>
            <link>https://grkotapalle.medium.com/etcds-time-travel-architecture-why-mvcc-is-the-heartbeat-of-kubernetes-240058b96c83?source=rss-3bea6fd817ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/240058b96c83</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Goutham Reddy Kotapalle]]></dc:creator>
            <pubDate>Wed, 18 Mar 2026 08:41:00 GMT</pubDate>
            <atom:updated>2026-03-18T08:41:00.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDOEdgnrKIXGZzkTLhD4dQ.png" /><figcaption><strong>etcd’s MVCC engine separates logical key history from physical revision storage, enabling consistent snapshots, resumable watches, and restart-safe Kubernetes control loops.</strong></figcaption></figure><p>Kubernetes often feels remarkably resilient during controller restarts, transient partitions, or short-lived control plane disruptions.</p><p>A controller restarts, reconnects, and catches back up. A watch drops briefly, then resumes. The scheduler and other control plane components continue operating against a coherent view of cluster state without having to rediscover the world from scratch.</p><p>That behavior is easy to take for granted.</p><p>But a large part of it comes from one specific design choice in etcd: its <strong>Multi-Version Concurrency Control (MVCC)</strong> engine.</p><p>While etcd is often treated like a simple distributed key-value store, that description misses what makes it so valuable to Kubernetes. etcd does not just store the latest value for each key. It stores state as a <strong>versioned history of revisions</strong>, and that history is what makes watches resumable, snapshots consistent, and controller recovery efficient.</p><p>This is one of the core reasons Kubernetes can remain declarative, level-triggered, and restart-safe by default.</p><h3>etcd is not just storing state, it is storing change over time</h3><p>At a surface level, etcd looks like a key-value database. Kubernetes objects are serialized and stored under well-known paths. Reads fetch current state. Writes update it.</p><p>But the real power of etcd lies in the fact that it does not reduce the datastore to a single mutable snapshot.</p><p>Every write advances a global revision. Instead of only remembering the latest value of a key, etcd maintains enough version history to answer much richer questions:</p><ul><li>what did the keyspace look like at revision X?</li><li>what changed between revision X and revision Y?</li><li>can a watcher resume from the last revision it processed?</li></ul><p>That is the difference between a datastore that merely holds the present and one that preserves a usable history of state transitions.</p><p>For Kubernetes, that distinction is foundational.</p><p>Without it, every brief disconnection would tend to push controllers and informers back toward full relists. resourceVersion would lose much of its meaning. Watches would become much less efficient under ordinary failure.</p><p>With MVCC, the control plane can often catch up by replaying missed deltas rather than rebuilding state from scratch.</p><p>That is what makes etcd feel, in effect, like it supports a controlled form of time travel.</p><h3>The core idea: revisions as the unit of history</h3><p>In etcd, writes are not just updates to keys. They are events in a globally ordered revision history.</p><p>A key may change many times, but those changes are embedded in a larger logical timeline shared across the entire keyspace. This means the system can reason not only about individual objects, but about a consistent view of cluster state at a particular moment.</p><p>That turns out to be exactly what Kubernetes needs.</p><p>Controllers and informers do not just care about current values. They care about <strong>what changed since the last point they observed</strong>. Watches are valuable because they let clients move incrementally through history instead of repeatedly recomputing the present from scratch.</p><p>This is also why resourceVersion matters so much. It is not just a token attached to an object. It represents a point in etcd’s logical history, which means Kubernetes clients can use it to ask for a consistent snapshot or resume from a known boundary.</p><p>None of that works well unless the storage engine underneath treats revisions as a first-class concept.</p><h3>The dual-index architecture that makes MVCC practical</h3><p>One of the most elegant parts of etcd’s design is that it separates the <strong>logical view of keys</strong> from the <strong>physical storage of revisions</strong>.</p><p>That split is what allows etcd to act like both a current-state key-value store and a replayable history engine.</p><h3>The in-memory B-tree: the logical map</h3><p>In memory, etcd keeps a B-tree that maps user-facing keys to their sequence of revisions.</p><p>This is the structure that lets etcd understand the logical keyspace efficiently. Instead of viewing the system as just an undifferentiated stream of writes, etcd can answer questions such as:</p><ul><li>what is the latest version of this key?</li><li>what revisions are associated with it?</li><li>what changed between two revision boundaries?</li></ul><p>This matters because Kubernetes does not interact with etcd only through point reads. It relies heavily on watches, resume points, and consistent snapshots tied to revision numbers.</p><p>The in-memory B-tree is what allows etcd to reason about that logical history without treating every operation like a cold walk through storage.</p><h3>The BoltDB B+ tree: the physical journal</h3><p>On disk, etcd uses BoltDB, but the important detail is not just the storage engine itself. It is how etcd uses it.</p><p>Entries are stored by <strong>revision</strong>, not by user-facing key.</p><p>That is a subtle but powerful choice.</p><p>If data were organized purely by key, updates would behave more like repeated overwrites of the same logical location. By organizing persisted state around revision order, etcd turns writes into something much closer to an append-heavy history of state transitions.</p><p>That layout aligns naturally with MVCC.</p><p>New writes produce new revisions rather than destructively replacing older state in place. As a result, etcd can support consistent point-in-time reads and replay historical changes to watchers that fall behind, as long as those revisions are still retained.</p><h3>Why the split matters</h3><p>This dual structure is the real architectural insight.</p><p>The in-memory B-tree is optimized around the <strong>logical keyspace</strong> that Kubernetes clients think in.<br> The BoltDB backend is organized around the <strong>physical history of revisions</strong> that MVCC depends on.</p><p>Together, they let etcd behave like two systems at once:</p><ul><li>a key-value store that can answer “what is the current state?”</li><li>a versioned history engine that can answer “what changed since revision X?”</li></ul><p>That combination is exactly what makes Kubernetes watches and controller recovery practical at scale.</p><h3>Why Kubernetes watches depend on this</h3><p>A huge amount of Kubernetes scalability depends on not having to relist everything all the time.</p><p>Schedulers, controllers, and shared informer caches all rely on watches so they can process changes incrementally instead of polling full state repeatedly. But a watch mechanism is only useful if the backing store can efficiently answer a very specific question:</p><blockquote><em>What changed after the last revision I processed?</em></blockquote><p>MVCC is what makes that question tractable.</p><p>When a watcher disconnects briefly due to a restart or a transient partition, it does not necessarily need to rebuild its cache from scratch. If it reconnects using a revision that still exists in etcd’s retained history, etcd can replay the missed events directly from MVCC history.</p><p>That behavior is one of the quiet reasons Kubernetes feels robust under normal failure. The system does not always need to rediscover the current world. It can often resume from where it left off.</p><p>Without MVCC-backed history, watches would degrade much more often into full relists, especially under even modest control plane churn.</p><h3>WatchableKV and level-triggered convergence</h3><p>This is where the architecture becomes operationally important.</p><p>Kubernetes controllers are built around <strong>level-triggered reconciliation</strong>. They are not supposed to rely on having observed every event in real time. Instead, they are designed to converge actual state toward desired state as long as they can recover a coherent view of what changed.</p><p>etcd’s watch machinery, often discussed through the watchableKV path, makes that model practical.</p><p>If a watcher falls behind but remains within the retained revision window, etcd can replay missed deltas directly from MVCC history. That allows a controller to continue from a known point instead of paying the cost of a full relist and cache rebuild.</p><p>This has several important consequences for the control plane:</p><ul><li>controllers can remain lightweight and restart-safe</li><li>transient network issues are less disruptive</li><li>shared informer patterns scale better</li><li>reconciliation loops can remain level-triggered rather than becoming event-fragile</li></ul><p>In other words, Kubernetes’ convergence model is not just an API design preference. It is supported by a storage engine that preserves enough history to make recovery cheap.</p><h3>Why resourceVersion is more than metadata</h3><p>It is easy to think of resourceVersion as one of those API fields that mostly matters when debugging watches.</p><p>But it is meaningful precisely because etcd is versioned underneath.</p><p>A resourceVersion is effectively a boundary in logical datastore history. That is what allows Kubernetes clients to say:</p><ul><li>give me a consistent snapshot as of this point</li><li>begin watching after this version</li><li>tell me what changed since then</li></ul><p>If etcd only stored the latest values and discarded the path taken to get there, resourceVersion would lose much of its power. It would no longer serve as a meaningful resume boundary for incremental recovery.</p><p>That is why etcd’s MVCC model does not just improve implementation efficiency. It directly shapes Kubernetes API semantics.</p><h3>The trade-off: history is expensive, but rebuilding is worse</h3><p>Of course, versioned history is not free.</p><p>Keeping old revisions around consumes memory and disk. The in-memory index must track key history, and the persisted backend must retain revisioned entries long enough to support watch replay and consistent snapshots.</p><p>That is why compaction exists.</p><p>Compaction prunes older revisions so history does not grow without bound. But the key design point is that etcd does <strong>not</strong> immediately discard the past after every write. It intentionally keeps a retention window, because that retained history is what allows Kubernetes components to recover efficiently from brief disruptions.</p><p>This is a deliberate trade-off.</p><p>etcd is spending memory and storage so the control plane can spend less time relisting, rebuilding caches, and re-establishing state after ordinary failure.</p><p>That is not an incidental implementation detail. It is part of the architecture that makes Kubernetes stable at scale.</p><h3>Why this design is so elegant</h3><p>What makes etcd especially interesting is that so many higher-level Kubernetes behaviors trace back to this single design decision.</p><p>By preserving state as revision history rather than just latest values, etcd enables:</p><ul><li>consistent point-in-time snapshots</li><li>resumable watches</li><li>meaningful resourceVersion semantics</li><li>restart-safe control loops</li><li>efficient catch-up after transient failures</li></ul><p>A lot of Kubernetes machinery that appears independent on the surface is actually resting on this one foundation.</p><p>Informers, schedulers, controllers, and API machinery all benefit because the backing store preserves enough history to make <strong>catching up cheaper than starting over</strong>.</p><p>That is a powerful property for any control plane.</p><h3>Closing thoughts</h3><p>When people describe etcd as Kubernetes’ backing store, they are not wrong.</p><p>But what Kubernetes really depends on is not just persistence. It depends on <strong>versioned persistence</strong>.</p><p>The important thing about etcd is not merely that it stores cluster state durably. It is that it stores the state in a way that preserves revision history, separates logical key indexing from physical revision storage, and allows clients to resume from known points in time.</p><p>That is what makes watches efficient.<br> That is what gives resourceVersion meaning.<br> That is what allows controllers to recover without constantly relisting the world.</p><p>MVCC is not just an internal storage detail.</p><p>It is one of the core reasons Kubernetes behaves like Kubernetes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=240058b96c83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Stateless to Batch to GPU: How Scheduling Evolves in Kubernetes]]></title>
            <link>https://grkotapalle.medium.com/from-stateless-to-batch-to-gpu-how-scheduling-evolves-in-kubernetes-884a45148e6c?source=rss-3bea6fd817ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/884a45148e6c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Goutham Reddy Kotapalle]]></dc:creator>
            <pubDate>Wed, 18 Mar 2026 08:09:45 GMT</pubDate>
            <atom:updated>2026-03-18T08:09:45.933Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NuMM2ZVoRLvZaDo1kXFKkw.png" /><figcaption><strong>As workloads evolve, Kubernetes scheduling moves from placement to coordination to topology-aware hardware alignment.</strong></figcaption></figure><p>When you first work with Kubernetes, scheduling feels deceptively simple.</p><p>A Pod requests CPU and memory. The scheduler finds a node. Placement happens.</p><p>For stateless services, this model works remarkably well. Availability matters more than precision, and the system absorbs most inefficiencies. A replica starting a little later or landing on a different node rarely changes anything in a meaningful way.</p><p>But this mental model does not hold as workloads evolve.</p><p>Scheduling is not a fixed problem. It changes shape depending on what you run. And as you move from stateless services to batch processing and eventually to GPU-driven systems, the scheduler moves further away from simply placing Pods and closer to managing the behavior of the entire cluster.</p><h3>Batch changes what we are scheduling</h3><p>Stateless workloads are built from largely independent Pods. If one is delayed or restarted, the system continues to function.</p><p>Batch workloads change that completely.</p><p>A Spark or Flink job is not just a set of Pods. It is a coordinated workload with phases, throughput expectations, and bursty resource usage. What matters is not whether a single Pod is scheduled, but whether the job as a whole makes progress.</p><p>This is where scheduling stops being about individual placement and starts becoming about coordination.</p><p>You begin to notice things that were previously invisible, like jobs waiting even when capacity exists, smaller workloads getting starved, and cluster state influencing how long something takes to finish.</p><p>At this point, the scheduler is no longer just answering “where should this run?” It is implicitly answering “how should the system make progress across competing workloads?”</p><h3>Distributed workloads make placement observable</h3><p>As workloads become more distributed, placement decisions start to have visible consequences.</p><p>Where Pods land affects communication patterns. Partial allocation can stall an entire job. Data movement and network topology begin to matter.</p><p>The scheduler is no longer operating in a space where any valid placement is acceptable. It is shaping how the application behaves at runtime.</p><p>This is the point where scheduling starts to feel less like infrastructure plumbing and more like part of the execution model itself.</p><h3>GPUs make the problem physical</h3><p>GPU workloads do not introduce entirely new ideas, but they amplify every existing constraint.</p><p>They are still multi-pod jobs. They are still throughput-driven. They still depend on coordination.</p><p>But now, the underlying hardware stops being abstract.</p><p>GPUs are scarce, expensive, and tied to specific physical layouts. Performance depends on locality. Two placements that look identical from a resource perspective can behave very differently at runtime.</p><p>Ignoring this does not break workloads immediately. It quietly reduces efficiency, which at scale becomes very real cost.</p><h3>Topology is no longer optional</h3><p>With CPUs, we often think in terms of quantities. With GPUs, we are forced to think in terms of layout.</p><p>The scheduler now has to reason about how hardware is arranged, not just how much of it exists. Placement decisions begin to interact with NUMA boundaries, interconnects, and partitioning strategies.</p><p>This changes the nature of scheduling.</p><p>It is no longer sufficient to check whether something fits. The question becomes whether it fits in a way that preserves performance.</p><p>And as clusters grow more heterogeneous, this problem stops being local to a single node and starts becoming a property of the entire fleet.</p><h3>Fragmentation becomes a first-order problem</h3><p>In CPU-heavy environments, fragmentation is usually tolerated. The system has enough flexibility to work around it.</p><p>In GPU environments, fragmentation directly determines what can and cannot run.</p><p>A cluster may have enough aggregate capacity, but if that capacity is scattered in the wrong shapes, large workloads will sit pending indefinitely.</p><p>This introduces a subtle but important shift.</p><p>Short-term efficiency and long-term schedulability begin to conflict. Filling every available gap may improve immediate utilization, but it can degrade the cluster’s ability to handle larger jobs later.</p><p>Over time, scheduling has to account not just for current demand, but for preserving the shape of the cluster itself.</p><h3>The scheduler is not the whole system</h3><p>It is tempting to think that autoscaling or better scheduling logic can solve these problems in isolation.</p><p>In practice, they cannot.</p><p>At scale, scheduling sits in the middle of a larger system. Admission determines what enters the cluster. Provisioning determines what capacity exists. Scheduling decides how that capacity is used.</p><p>If these layers are not aligned, the system behaves poorly even if each individual component looks correct in isolation.</p><p>What begins as a scheduling problem often turns out to be a control plane design problem.</p><h3>Closing thoughts</h3><p>Kubernetes scheduling starts simple because the workloads we begin with are simple.</p><p>But as we move from stateless services to batch systems and then to GPU-driven infrastructure, the problem changes fundamentally.</p><p>It stops being about placing Pods.</p><p>It becomes about coordinating workloads, respecting hardware, and maintaining the shape of the system over time.</p><p>At that point, the scheduler is no longer just a component making decisions. It is part of a larger mechanism that determines how efficiently the entire cluster operates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=884a45148e6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[etcd Compaction and Defragmentation: Their Impact on Leader Election]]></title>
            <link>https://grkotapalle.medium.com/etcd-compaction-and-defragmentation-their-impact-on-leader-election-5e4cf0e94ee3?source=rss-3bea6fd817ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e4cf0e94ee3</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[etcd]]></category>
            <dc:creator><![CDATA[Goutham Reddy Kotapalle]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 10:18:13 GMT</pubDate>
            <atom:updated>2026-03-16T10:18:13.032Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-EWpKAZHQVI2H6sl" /><figcaption>etcd compaction and defragmentation</figcaption></figure><p>If you’ve ever seen a Kubernetes control plane “randomly” stall for a few seconds, there’s a good chance etcd was involved.</p><p>etcd is not just another dependency in Kubernetes. It is the <strong>persistent state store, coordination layer, and write path for nearly every control plane operation</strong>. When etcd is healthy, everything in the control plane feels smooth and responsive. When it isn’t, the symptoms show up everywhere: API latency spikes, controllers behave oddly, and leadership churn appears where you least expect it.</p><p>One of the easiest ways to accidentally make etcd unhappy is during what should be routine maintenance: <strong>compaction and defragmentation</strong>.</p><p>On paper, both are simple housekeeping tasks. In practice, they interact with disk I/O, CPU scheduling, and Raft timing in ways that can trigger <strong>leader elections and temporary write outages</strong> if they’re not handled carefully.</p><p>I’ve seen clusters where a “quick defrag” scheduled during peak hours resulted in a surprise control plane hiccup. Nothing technically failed. But the cluster experienced a brief pause in writes, controllers lost leases momentarily, and alerts started firing.</p><p>This article explains <strong>why that happens</strong> and how to keep etcd maintenance boring, which is exactly how infrastructure maintenance should be.</p><h3>Why etcd Needs Maintenance in the First Place</h3><p>To understand compaction and defragmentation, it helps to understand how etcd stores data.</p><p>etcd uses <strong>MVCC (Multi-Version Concurrency Control)</strong>. Every write to etcd creates a <strong>new revision</strong> rather than overwriting the existing value.</p><p>For example, if a Kubernetes object is updated repeatedly:</p><ul><li>Pod status update</li><li>Lease heartbeat</li><li>Endpoint update</li><li>Controller reconciliation</li></ul><p>Each update creates a <strong>new revision in etcd</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*37V8WzPSiubSwTjOxoHgSQ.png" /></figure><p>Older revisions remain in the database so clients can perform consistent reads at specific points in time. This is essential for Kubernetes watchers, which rely on revision history to track changes in cluster state.</p><p>But the tradeoff is simple: <strong>History accumulates forever unless you remove it. </strong>Even if your cluster size remains stable, the amount of historical data in etcd steadily grows due to:</p><ul><li>controller reconciliation loops</li><li>lease renewals</li><li>status updates</li><li>endpoint changes</li><li>configuration churn</li></ul><p>Left unmanaged, this revision history eventually slows down queries and increases memory and storage pressure.</p><p>This is where <strong>compaction</strong> enters the picture.</p><h3>Compaction: Cleaning Up Historical Revisions</h3><p>Compaction removes <strong>older MVCC revisions that are no longer needed</strong>.</p><p>Conceptually, compaction tells etcd:</p><p>“Any revision older than this point is no longer necessary for reads.”</p><p>After compaction:</p><ul><li>historical revisions are discarded</li><li>watch streams must start from newer revisions</li><li>read performance improves because there is less historical state to traverse</li></ul><p>However, there is an important detail that often surprises operators:</p><p><strong>Compaction does not shrink the database file.</strong></p><p>Internally, etcd stores data using <strong>bbolt</strong>, a memory-mapped B+tree database. When old revisions are removed during compaction, the pages they occupied become <strong>free space inside the database file</strong>, but the file itself remains the same size.</p><p>Think of compaction as:</p><p>Throwing away old paperwork but leaving the filing cabinets exactly the same size.</p><p>Logically, the database is smaller. Physically, the file on disk is unchanged.</p><p>Over time, this creates fragmentation.</p><h3>Defragmentation: Repacking the Database</h3><p>Defragmentation addresses the physical layout of the database.</p><p>After repeated compactions, the bbolt database contains <strong>holes where old pages used to exist</strong>. The file becomes sparse and fragmented, which can reduce cache locality and increase I/O overhead.</p><p>Defragmentation fixes this by:</p><ol><li>Copying all <strong>live pages</strong> into a new compact layout</li><li>Rewriting the database file</li><li>Eliminating unused space</li><li>Shrinking the physical file size</li></ol><p>After defragmentation:</p><ul><li>disk usage decreases</li><li>I/O locality improves</li><li>read performance becomes more predictable</li></ul><p>However, this process is <strong>I/O heavy and temporarily disruptive</strong>.</p><p>While defragmentation runs, the etcd member performing the operation experiences:</p><ul><li>increased disk activity</li><li>reduced responsiveness</li><li>slower request processing</li></ul><p>And that’s where things begin to intersect with Raft.</p><h3>etcd Is Also a Raft Cluster</h3><p>etcd is not just a database; it is also a <strong>Raft consensus cluster</strong>.</p><p>In a typical deployment:</p><ul><li>one node is the <strong>leader</strong></li><li>other nodes are <strong>followers</strong></li><li>the leader processes writes</li><li>followers replicate entries through the Raft log</li></ul><p>Leadership is maintained through <strong>heartbeats</strong>.</p><p>The leader periodically sends heartbeat messages to followers. If followers stop receiving heartbeats within the configured election timeout, they assume the leader has failed and initiate a new election.</p><p>The critical point here is:</p><p><strong>Raft is extremely sensitive to latency and stalls.</strong></p><p>It does not care why the leader stopped responding. It only cares that <strong>heartbeats arrived late or not at all</strong>.</p><p>Disk stalls, CPU starvation, or long blocking operations can all trigger elections even if the node is technically still alive.</p><p>This is exactly what can happen during defragmentation.</p><p><strong>How Maintenance Can Trigger Leader Elections</strong></p><p>During defragmentation, the etcd process may temporarily become slower due to heavy disk I/O and memory activity.</p><p>If this happens on the <strong>current leader</strong>, several things may occur:</p><ul><li>heartbeat messages are delayed</li><li>follower responses are slow</li><li>Raft timing thresholds are exceeded</li></ul><p>Followers then assume the leader is unavailable and start a <strong>new election</strong>.</p><p>The sequence typically looks like this:</p><ol><li>Leader begins defragmentation</li><li>Disk I/O increases</li><li>Heartbeat processing slows</li><li>Followers miss heartbeat deadlines</li><li>Followers trigger a Raft election</li><li>Leadership moves to another node</li></ol><p>During the election window:</p><ul><li>writes are temporarily paused</li><li>API server requests may block</li><li>controllers may lose leadership leases</li><li>monitoring systems raise alerts</li></ul><p>The cluster usually stabilizes quickly, but the brief disruption can look like <strong>random control plane instability</strong>.</p><p>In reality, the system behaved exactly as Raft was designed to behave.</p><h3>The Common Operational Mistake</h3><p>The most common operational mistake is simple:</p><p><strong>Defragmenting the leader directly.</strong></p><p>This creates a scenario where the node responsible for coordinating writes and maintaining cluster timing is simultaneously performing the heaviest maintenance operation.</p><p>The result is predictable:</p><ul><li>heartbeats slip</li><li>followers assume failure</li><li>an election occurs</li></ul><p>Even if the election completes quickly, it introduces unnecessary control plane turbulence.</p><p>This often leads operators to believe etcd is unreliable when the real issue is simply <strong>poor maintenance sequencing</strong>.</p><p><strong>The Safe Way to Run Defragmentation</strong></p><p>A safer operational pattern avoids placing heavy maintenance load on the leader.</p><p>The process typically looks like this:</p><ol><li><strong>Identify the current etcd leader.</strong></li><li><strong>Defragment follower nodes first</strong>, one at a time.</li><li><strong>Transfer leadership</strong> to one of the already-defragmented nodes.</li><li>Defragment the <strong>former leader</strong>, now acting as a follower.</li></ol><p>This sequencing ensures that the active leader is never performing heavy disk operations.</p><p>Instead of risking a random election caused by disk stalls, you perform a <strong>controlled leadership transfer</strong>.</p><p>This keeps the cluster stable while maintenance runs.</p><p><strong>When This Matters Most</strong></p><p>In small clusters with fast SSDs and low control-plane traffic, you may get away with less careful sequencing.</p><p>But as clusters grow and workloads become more dynamic, the risk increases.</p><p>This pattern becomes particularly important when:</p><ul><li>control planes are under heavy load</li><li>clusters have high watch activity</li><li>disks are not ultra-fast</li><li>maintenance is performed during business hours</li><li>previous unexplained leader elections have occurred</li></ul><p>In these environments, careful sequencing can mean the difference between a <strong>routine maintenance window </strong>and a <strong>mysterious control plane blip that triggers alerts across the platform</strong>.</p><h3>The Bigger Picture</h3><p>Compaction and defragmentation solve <strong>two different problems</strong>.</p><p>Compaction manages <strong>logical history</strong> in the MVCC store.</p><p>Defragmentation manages the <strong>physical storage layout</strong> of the database.</p><p>If you skip <strong>compaction</strong>:</p><ul><li>MVCC history grows indefinitely</li><li>read performance degrades</li><li>memory pressure increases</li></ul><p>If you skip <strong>defragmentation</strong>:</p><ul><li>database files grow larger than necessary</li><li>disk locality degrades</li><li>I/O behavior becomes noisier over time</li></ul><p>Both operations are necessary for long-term etcd health.</p><p>But they should always be performed with <strong>Raft timing and leadership behavior in mind</strong>.</p><p>Because etcd health shows up everywhere in Kubernetes:</p><ul><li>API server latency</li><li>controller stability</li><li>leader leases</li><li>cluster upgrades</li><li>recovery times</li><li>watch stream reliability</li></ul><p>Poorly timed maintenance often manifests as <strong>random control plane instability</strong>.</p><p>Well-executed maintenance produces the best possible outcome in infrastructure operations:</p><p>Nothing happens.</p><p>And in distributed systems, “nothing happened” is usually the sign that everything was done right.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e4cf0e94ee3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From PodSpec to a Linux Process: The 150ms Where Your Container Is Born]]></title>
            <link>https://grkotapalle.medium.com/from-podspec-to-a-linux-process-the-150ms-where-your-container-is-born-df0dd4357751?source=rss-3bea6fd817ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/df0dd4357751</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Goutham Reddy Kotapalle]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 10:15:02 GMT</pubDate>
            <atom:updated>2026-03-16T10:15:02.166Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l3P-0x13rp7_bZAv_vaPsg.png" /><figcaption>K8s Pod Creation</figcaption></figure><p>When you deploy something to Kubernetes, it feels almost magical. You apply a YAML, wait a few seconds, and your Pod says <strong>Running</strong>.</p><p>Behind the scenes, we talk about Deployments creating ReplicaSets, schedulers binding Pods to nodes, kubelets reconciling desired state, and readiness probes eventually turning green. It feels like this layered orchestration engine that somehow brings your application to life.</p><p>But there is a part of this process we rarely stop to think about.</p><p>None of those components actually create your container.</p><p>There is a small window of time, usually well under 150 milliseconds, where your container truly comes into existence. It does not happen in etcd. It does not happen in the scheduler. It does not even happen in the kubelet.</p><p>It happens inside a small binary called <strong>runc</strong>.</p><p>That moment is where the Kubernetes abstraction gives way to the Linux kernel.</p><h3>Slowing Down the “Magic”</h3><p>Let’s slow the process down and walk through what really happens.</p><p>When you create a Pod, the first steps are fairly straightforward. The Pod object is written to etcd through the Kubernetes API server. The scheduler then selects a node and records that decision in the Pod specification.</p><p>At this point, the Pod still does not exist as a running workload. It is just desired state stored in the control plane.</p><p>The kubelet on the selected node watches the API server and notices the assignment. Its reconciliation loop detects that a container needs to be created and begins working toward that state.</p><p>If the node is using containerd as its runtime, containerd pulls the container image, unpacks its layers, constructs a root filesystem, and prepares what is called an <strong>OCI bundle</strong>.</p><p>Inside that bundle is a file called <strong>config.json</strong>.</p><p>This file contains the exact specification of how the container should run. It defines the command to execute, environment variables, namespace configuration, mounts, security settings, and resource limits.</p><p>Up until this point, everything is still configuration.</p><p>There is no running process.<br> There is no PID 1.<br> There is no isolated environment.</p><p>There is only a description of one.</p><p>Then containerd makes one final call.</p><pre>runc create</pre><p>That is the moment where the abstraction layer ends and the operating system takes over.</p><h3>The Full Path From PodSpec to Node</h3><p>Before the container process is created, it helps to understand the full control plane path that leads to that moment.</p><p>When a PodSpec is applied:</p><ol><li>The Kubernetes API server stores the Pod object in etcd.</li><li>The scheduler selects a node and sets the Pod’s nodeName.</li><li>The kubelet on that node observes the assignment through its watch stream.</li><li>The kubelet’s reconciliation loop determines that a container must be created.</li><li>The kubelet invokes the container runtime through the <strong>Container Runtime Interface (CRI)</strong>.</li></ol><p>From the kubelet’s perspective, container creation is simply a gRPC request to the runtime.</p><p>For containerd this typically looks like:</p><pre>RunPodSandbox<br>CreateContainer<br>StartContainer</pre><p>Only after these runtime calls does the process creation path begin.</p><p>Kubernetes itself never directly creates containers. It delegates that responsibility to the container runtime.</p><h3>Containers Are Just Processes</h3><p>Once containerd has prepared the OCI bundle, it calls <strong>runc</strong>.</p><p>runc does not spin up a lightweight virtual machine.<br> It does not allocate a tiny operating system.<br> It does not emulate hardware.</p><p>It makes a system call.</p><p>Specifically, it calls <strong>clone()</strong>.</p><p>Many people assume containers originate from the classic Unix fork() call. But fork() simply duplicates a process. It does not create isolation.</p><p>The clone() system call allows the Linux kernel to create a process with a customized set of isolation boundaries.</p><p>With the correct flags, clone() instructs the kernel to create new namespaces for the process:</p><ul><li>a new PID namespace</li><li>a new network namespace</li><li>a new mount namespace</li><li>a new hostname</li><li>a new IPC boundary</li></ul><p>Inside that new PID namespace, the first process becomes <strong>PID 1</strong>.</p><p>That process is your container.</p><p>Not conceptually. Not abstractly.</p><p>Literally.</p><h3>The Namespaces That Form a Container</h3><p>When runc invokes clone() with namespace flags, it creates several Linux namespaces simultaneously.</p><p>Each namespace isolates a different aspect of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzvJM1l0dTzSOnFSaZvRBA.png" /></figure><p>Together, these namespaces create the illusion that the process is running on its own machine.</p><p>But the kernel does not create a new operating system.</p><p>It simply creates a process whose <strong>view of the system has been carefully restricted</strong>.</p><h3>Where Resource Limits Actually Come From</h3><p>When you define CPU and memory limits in a Kubernetes manifest, it is easy to imagine Kubernetes enforcing those limits.</p><p>It does not.</p><p>Before the container process starts, runc assigns it to specific <strong>cgroups</strong> in the Linux kernel.</p><p>Cgroups are where resource constraints are actually enforced.</p><p>They control:</p><ul><li>CPU shares and quotas</li><li>memory limit</li><li>I/O throttling</li><li>process counts</li></ul><p>When a container is <strong>OOMKilled</strong>, that decision is not made by Kubernetes.</p><p>It is made by the <strong>Linux kernel</strong>, enforcing the memory boundary defined in the container’s cgroup.</p><p>Once you understand this, debugging production incidents changes dramatically. Instead of asking what Kubernetes decided, you start asking what the kernel enforced.</p><h3>The Final Transition</h3><p>After namespaces are created, cgroups are assigned, mounts are configured, and security policies are applied, runc performs one final operation.</p><p>It calls <strong>execve()</strong>.</p><p>At that moment, the infrastructure work is finished.</p><p>Your container stops being an idea described in a JSON configuration file and becomes a running process executing your application inside carefully constructed boundaries.</p><p>You can see this process directly from the host.</p><p>It appears in ps.<br> Its state exists in /proc.<br> You can enter its namespaces using nsenter.</p><p>There is no special container object inside the Linux kernel.</p><p>There is only a process with isolation rules applied.</p><h3>Why runc Does Not Stick Around</h3><p>One subtle detail that often surprises people is that runc exits immediately after creating the container.</p><p>It does not supervise the process or remain attached to it.</p><p>runc sets up the kernel state, launches the process, and exits.</p><p>From that point forward, the kernel is responsible for enforcing isolation and resource limits.</p><p>A helper process called <strong>containerd-shim</strong> remains running to manage the container lifecycle and handle input and output streams, but runc itself is gone.</p><p>This design allows higher-level managers, such as containerd or Docker, to avoid remaining attached to every single container process.</p><p>It also explains why containers can survive certain runtime failures.</p><p>A brief containerd restart does not automatically kill running workloads.<br> A kubelet crash does not instantly wipe out containers.</p><p>Once the kernel has created the process and applied its boundaries, the container’s existence depends on Linux itself.</p><h3>Observing This on a Kubernetes Node</h3><p>You can observe this process creation path directly on a Kubernetes node.</p><p>Once a container is running, you can find the container process from the host:</p><pre>ps aux | grep containerd-shim</pre><p>From there, you can inspect the namespaces attached to the process:</p><pre>lsns -p &lt;PID&gt;</pre><p>Or enter the container’s namespaces directly:</p><pre>nsenter -t &lt;PID&gt; -a</pre><p>What you will see is not a special container construct inside the kernel.</p><p>It is simply a process with namespace and cgroup boundaries applied.</p><p>This perspective is extremely useful when debugging container runtime issues.</p><h3>The Mental Model Shift</h3><p>When you step back and look at the stack clearly, the picture changes.</p><p>Kubernetes orchestrates intent.<br> containerd prepares the environment.<br> runc translates that intent into kernel primitives.<br> Linux enforces the rules.</p><p>Understanding this does more than make you better at debugging.</p><p>It changes how you design systems.</p><p>You start to see where abstractions stop and where operating system mechanics begin. You understand which failures originate in the control plane and which originate in the kernel.</p><p>Containers stop feeling magical.</p><p>They start feeling precise.</p><p>Your container is not born when the Pod turns green.</p><p>It is born when the Linux kernel creates a process with new namespaces and attaches it to cgroups.</p><p>That moment happens in a single system call.</p><p>And once it happens, everything else is just Linux.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df0dd4357751" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>