<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gulcan Topcu on Medium]]></title>
        <description><![CDATA[Stories by Gulcan Topcu on Medium]]></description>
        <link>https://medium.com/@gulcantopcu?source=rss-f548dcc00dae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6gmmhqsZDLiXTbbSe34-wQ.png</url>
            <title>Stories by Gulcan Topcu on Medium</title>
            <link>https://medium.com/@gulcantopcu?source=rss-f548dcc00dae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:48:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gulcantopcu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What `os.cpu_count()` Gets Wrong in a CPU-Limited Kubernetes Pod]]></title>
            <link>https://itnext.io/what-os-cpu-count-gets-wrong-in-a-cpu-limited-kubernetes-pod-ab57756a9153?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab57756a9153</guid>
            <category><![CDATA[gunicorn]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 14:08:30 GMT</pubDate>
            <atom:updated>2026-06-27T13:25:25.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxoGsq1P1QgOiSLYD8acZw.png" /></figure><p><em>This article is originally published on my </em><a href="https://www.gulcantopcu.com/posts/what-os-cpu-count-gets-wrong-kubernetes-cpu-limited-pod/"><em>personal blog</em></a><em>.</em></p><p>I gave a pod 500m CPU and then went inside and asked Python how many CPUs it could see. The answer was 20, and that seemed worth understanding.</p><p><strong>TL;DR:</strong> When a Gunicorn config sizes workers from os.cpu_count() inside a 500m-limited pod, it might see the node&#39;s full CPU count instead of what the cgroup actually allows, and most of what those extra workers do is wait to be scheduled.</p><p>I start by checking what Kubernetes reports for the node:</p><pre>kubectl get node minikube -o jsonpath=&#39;{.status.capacity.cpu}{&quot;\n&quot;}{.status.allocatable.cpu}{&quot;\n&quot;}&#39;</pre><pre>20<br>20</pre><p>Both values are 20, which means Kubernetes scheduled the pod onto a node that advertises twenty CPUs as capacity and allocatable CPU. The important trick is that 500m does not remove CPUs from the process view; it limits how much CPU time the cgroup is allowed to spend while those CPUs remain visible.</p><p>And the YAML is where that promise gets made.</p><h2>What the YAML Actually Promises</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/0*GCb1AITn0uHTE1_k.png" /></figure><p>The pod lives in its own namespace, python-cpu-quota-demo, and it&#39;s set up with matching request and limit: both 500m.</p><pre>kubectl get pod cpu-probe -n python-cpu-quota-demo -o yaml</pre><pre>resources:<br>  limits:<br>    cpu: 500m<br>    memory: 128Mi<br>  requests:<br>    cpu: 500m<br>    memory: 128Mi<br>qosClass: Guaranteed</pre><p>These two fields sit next to each other in the spec, but they do very different jobs. The <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits">CPU request</a> is for the scheduler before the pod exists, because Kubernetes needs to know whether the node has room. The <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-kubernetes-applies-resource-requests-and-limits">CPU limit</a> is for the kernel after the pod is running, because the cgroup needs to know how much CPU time this workload may spend per period.</p><p>Memory is crueler when you cross the line.</p><p>That is the gap I want to inspect from inside the pod.</p><h2>What Python Sees vs What the Kernel Enforces</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*9u0NUlFBk0BZ7w69.png" /></figure><p>I want three values next to each other: what Python reports, what the cgroup says, and what Linux affinity allows for the current process.</p><pre>kubectl logs cpu-probe -n python-cpu-quota-demo<br>python 3.13.14<br>os.cpu_count 20<br>os.process_cpu_count 20<br>sched_getaffinity 20<br>cpu.max 50000 100000<br>cpu.cfs_quota_us missing<br>cpu.cfs_period_us missing<br>cpu.stat usage_usec 106789<br>user_usec 81012<br>system_usec 25776<br>nice_usec 0<br>nr_periods 2<br>nr_throttled 2<br>throttled_usec 8150<br>nr_bursts 0<br>burst_usec 0<br>cpus_allowed_list 0-19</pre><p>The CPU-count answers agree with each other. Python says 20, and Linux affinity says the process is allowed to run on CPUs 0-19, so that also comes out as 20. Those answers are not wrong, because the process really can be scheduled on any of those logical CPUs.</p><p>The cgroup is looking at a different limit:</p><pre>kubectl exec deployment/gunicorn-cpu-demo -n python-cpu-quota-demo -- cat /sys/fs/cgroup/cpu.max</pre><pre>50000 100000</pre><p>This pod is using <a href="https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu">cgroup v2</a>, which is why the value lives in cpu.max:</p><pre>kubectl exec deployment/gunicorn-cpu-demo -n python-cpu-quota-demo -- stat -fc %T /sys/fs/cgroup</pre><pre>cgroup2fs</pre><p>On cgroup v2, <a href="https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu">cpu.max</a> has two fields: quota and period. Here the cgroup can spend 50,000 microseconds of CPU time in each 100,000 microsecond period, which works out to half a CPU:</p><pre>50000 / 100000 = 0.5 CPU</pre><p>That is the 500m limit, and now the mismatch is visible: Python sees twenty CPUs, affinity allows twenty CPUs, but the kernel quota allows half a CPU worth of time.</p><p>All three answers are technically true because they are answering different questions. Python’s <a href="https://docs.python.org/3/library/os.html#os.cpu_count">os.cpu_count()</a> is answering &quot;how many logical CPUs are in the system?&quot;, <a href="https://docs.python.org/3/library/os.html#os.process_cpu_count">os.process_cpu_count()</a> and affinity are answering &quot;which CPUs can this thread run on?&quot;, and the cgroup is answering &quot;how much CPU time can this group spend?&quot; Worker sizing for a CPU-bound sync service should start from the third question, but a lot of worker formulas read the first answer.</p><p>So what does Gunicorn do with twenty?</p><h2>What Gunicorn Does With That Number</h2><p>By default, if <a href="https://docs.gunicorn.org/en/stable/settings.html#workers">WEB_CONCURRENCY</a> is not set, this installed version of Gunicorn starts with one worker:</p><pre>kubectl exec deployment/gunicorn-cpu-demo -n python-cpu-quota-demo -- \<br>  python -c &quot;import os; os.environ.pop(&#39;WEB_CONCURRENCY&#39;, None); \<br>  from gunicorn.config import Config; config = Config(); \<br>  print(config.settings[&#39;workers&#39;].default)&quot;</pre><pre>1</pre><p>The problem starts when an application config overrides that default and calculates worker count from the CPU count Python reports. A common Gunicorn config pattern looks like this:</p><pre>import multiprocessing<br>bind = &quot;127.0.0.1:8000&quot;<br>workers = multiprocessing.cpu_count() * 2 + 1</pre><p>Inside this pod, multiprocessing.cpu_count() returns 20, which makes that formula produce 41 workers. The demo startup log prints both the what-if calculation and the value actually used:</p><pre>kubectl logs deployment/gunicorn-cpu-demo -n python-cpu-quota-demo --limit-bytes=5000<br>os.cpu_count=20<br>os.process_cpu_count=20<br>cpu.max=50000 100000<br>quota_cpus=0.50<br>gunicorn_formula_from_quota=2<br>gunicorn_formula_from_os_cpu_count=41<br>WEB_CONCURRENCY=1<br>[2026-06-22 15:17:19 +0000] [1] [INFO] Starting gunicorn 23.0.0<br>[2026-06-22 15:17:19 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)<br>[2026-06-22 15:17:19 +0000] [1] [INFO] Using worker: sync<br>[2026-06-22 15:17:19 +0000] [12] [INFO] Booting worker with pid: 12<br>[22/Jun/2026:15:17:19 +0000] &quot;GET /healthz HTTP/1.1&quot; 200 3 &quot;-&quot; &quot;kube-probe/1.34&quot;</pre><p>gunicorn_formula_from_os_cpu_count=41 is the dangerous what-if number, while WEB_CONCURRENCY=1 is what this run actually uses. The startup lines confirm that Gunicorn boots one worker, which gives us a small baseline before adding more workers to the same half-CPU budget.</p><p>To compare those worker counts, I need an endpoint that spends CPU in a boring and repeatable way.</p><h2>The Endpoint and the Load Setup</h2><p>The service exposes that endpoint as /burn. Each request runs a fixed Python loop and returns the worker pid with elapsed time:</p><pre>import os<br>import time<br>from urllib.parse import parse_qs<br>DEFAULT_LOOPS = int(os.getenv(&quot;BURN_LOOPS&quot;, &quot;1500000&quot;))<br>def burn(iterations):<br>    total = 0<br>    for item in range(iterations):<br>        total = (total + (item * item)) % 1000003<br>    return total<br>def application(environ, start_response):<br>    path = environ.get(&quot;PATH_INFO&quot;, &quot;/&quot;)<br>    if path == &quot;/healthz&quot;:<br>        body = b&quot;ok\n&quot;<br>        start_response(&quot;200 OK&quot;, [(&quot;Content-Type&quot;, &quot;text/plain&quot;), (&quot;Content-Length&quot;, str(len(body)))])<br>        return [body]<br>    if path != &quot;/burn&quot;:<br>        body = b&quot;use /burn\n&quot;<br>        start_response(&quot;404 Not Found&quot;, [(&quot;Content-Type&quot;, &quot;text/plain&quot;), (&quot;Content-Length&quot;, str(len(body)))])<br>        return [body]<br>    query = parse_qs(environ.get(&quot;QUERY_STRING&quot;, &quot;&quot;))<br>    iterations = int(query.get(&quot;n&quot;, [str(DEFAULT_LOOPS)])[0])<br>    started = time.perf_counter()<br>    result = burn(iterations)<br>    duration_ms = (time.perf_counter() - started) * 1000<br>    body = (<br>        f&quot;pid={os.getpid()} loops={iterations} &quot;<br>        f&quot;result={result} duration_ms={duration_ms:.2f}\n&quot;<br>    ).encode()<br>    start_response(&quot;200 OK&quot;, [(&quot;Content-Type&quot;, &quot;text/plain&quot;), (&quot;Content-Length&quot;, str(len(body)))])<br>    return [body]</pre><p>Each benchmark used the same setup. I port-forwarded the service to 127.0.0.1:18080, ran ab for 20 seconds against /burn, and captured /sys/fs/cgroup/cpu.stat from the app container before and after the run. That gave me both sides of the story: what the client saw and what the kernel counted.</p><p>With the workload fixed, the next question is what changed when I changed only the process layout inside the same tiny CPU budget.</p><h2>So Which Worker Count Actually Won?</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*gJnz0KcsykpmnQQi.png" /></figure><p>The pod spec, endpoint, CPU limit, and test duration stayed the same, while WEB_CONCURRENCY changed between runs. I also changed the client pressure: the 1-worker run used -c 5, and the 14-worker run used -c 20. That means this is a stress comparison, not a clean single-variable benchmark, but it still answers the question I cared about: what happens when too many workers fight over the same 0.5 CPU?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*Es6KTc2zThA4OkPRDssasQ.png" /></figure><p>One worker completed 101 responses with a 1,002ms median, while fourteen workers completed 46 responses and pushed the median past 6 seconds. The extra workers did not increase the amount of CPU available to the pod, so they could not buy real throughput, and instead gave the scheduler more runnable processes to pause and resume inside the same 0.5 CPU budget.</p><p>The ab length mismatches are also worth reading carefully.</p><p>The endpoint returns values like pid and duration_ms, so the response body is not a fixed length across requests. That is why ab reports length mismatches as failed requests in the 14-worker run. I still kept the column in the table so the raw client output is not hidden, but the more important numbers are completed requests, request rate, latency, and the kernel counters below.</p><p>With one worker, the full ab output:</p><pre>ab -t 20 -c 5 -s 60 -q http://127.0.0.1:18080/burn</pre><pre>Concurrency Level:      5<br>Time taken for tests:   20.084 seconds<br>Complete requests:      101<br>Failed requests:        0<br>Requests per second:    5.03 [#/sec] (mean)<br>Time per request:       994.273 [ms] (mean)</pre><pre>Percentage of the requests served within a certain time (ms)<br>  50%   1002<br>  95%   1060<br> 100%   1299 (longest request)</pre><p>With 14 workers:</p><pre>ab -t 20 -c 20 -s 60 -q http://127.0.0.1:18080/burn</pre><pre>Concurrency Level:      20<br>Time taken for tests:   21.133 seconds<br>Complete requests:      46<br>Failed requests:        45<br>   (Connect: 0, Receive: 0, Length: 45, Exceptions: 0)<br>Requests per second:    2.18 [#/sec] (mean)<br>Time per request:       9188.476 [ms] (mean)</pre><pre>Percentage of the requests served within a certain time (ms)<br>  50%   6390<br>  95%  11402<br> 100%  12312 (longest request)</pre><p>The client numbers tell us the service got slower, but cpu.stat tells us what the kernel was doing while the clients waited. Before the 14-worker run, the cgroup had 24 throttled periods and just under 1 second of throttled time accumulated since the pod started. After the load test, the same counters looked like this:</p><pre>nr_periods 328<br>nr_throttled 272<br>throttled_usec 285826526</pre><p>That is 248 newly throttled periods and about 285 seconds of newly accumulated throttled time during roughly 20 seconds of wall-clock load. The number can be much larger than wall time because <a href="https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu">cpu.stat</a> accounts throttling across the runnable tasks in the cgroup. With 14 workers competing for the same 0.5 CPU quota, blocked time piles up in parallel.</p><p>The 1-worker run is the same workload with fewer processes trying to spend the quota. In that run, the cgroup had 16 throttled periods and about 0.7 seconds of throttled time before the test. After the load test:</p><pre>nr_periods 830<br>nr_throttled 220<br>throttled_usec 10877871</pre><p>That is 204 newly throttled periods and about 10.2 seconds of newly accumulated throttled time across the run. The single worker still hit the CPU limit, because the endpoint is CPU-bound and the quota is small, but it spent the quota on useful work instead of spreading it across a crowd of workers.</p><p>At this point the kernel counters have already told the story, but dashboards are where people usually look first. So I wanted to know whether Prometheus would show the same failure mode, or whether this would stay hidden unless you went into cpu.stat.</p><h2>The Prometheus View</h2><p>This cluster runs kube-prometheus-stack and scrapes <a href="https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md#prometheus-container-metrics">cAdvisor Prometheus metrics</a> from the kubelet. Before writing a query, I first checked which container CPU metrics were actually available:</p><pre>curl -sG &#39;http://localhost:9090/api/v1/label/__name__/values&#39; \<br>  | python3 -c &quot;<br>import sys, json<br>d = json.load(sys.stdin)<br>cadvisor = [m for m in d[&#39;data&#39;] if &#39;container_cpu&#39; in m]<br>print(&#39;\n&#39;.join(sorted(cadvisor)))<br>&quot;</pre><pre>container_cpu_cfs_periods_total<br>container_cpu_cfs_throttled_periods_total<br>container_cpu_usage_seconds_total</pre><p>Prometheus gives me throttled periods here, while the raw throttled seconds still have to come from cpu.stat. container_cpu_cfs_throttled_periods_total is present, but container_cpu_cfs_throttled_seconds_total is not exposed by the kubelet cAdvisor on this cluster.</p><p>One other detail matters for the query. On this Minikube setup, cAdvisor emits these metrics at pod scope without a container label, which means queries filtering on container=&quot;app&quot; return empty results. The actual label set looks like this:</p><pre>{<br>  &quot;__name__&quot;: &quot;container_cpu_cfs_throttled_periods_total&quot;,<br>  &quot;namespace&quot;: &quot;python-cpu-quota-demo&quot;,<br>  &quot;pod&quot;: &quot;gunicorn-cpu-demo-694589fb97-45cq2&quot;,<br>  &quot;node&quot;: &quot;minikube&quot;<br>}</pre><p>There is no container key there, so the throttle ratio query filters by namespace and pod instead:</p><pre>rate(container_cpu_cfs_throttled_periods_total{<br>  namespace=&quot;python-cpu-quota-demo&quot;,<br>  pod=~&quot;gunicorn-cpu-demo-.*&quot;<br>}[1m])<br>/<br>rate(container_cpu_cfs_periods_total{<br>  namespace=&quot;python-cpu-quota-demo&quot;,<br>  pod=~&quot;gunicorn-cpu-demo-.*&quot;<br>}[1m])</pre><p>During the 14-worker load test, that query showed about 83% of measured periods hitting throttling. Because this is a 1-minute rate() query, the exact value moves as the load window ages out of Prometheus&#39; lookback range:</p><pre>throttle_ratio: 83.4 % | pod: gunicorn-cpu-demo-694589fb97-45cq2</pre><p>Both runs throttle because the workload is CPU-bound and the quota is tight, but the cost of that throttling is different. With 14 workers, the run completed 46 requests while the cgroup spent a large share of measured periods throttled. With 1 worker, the cgroup still throttled, but 101 requests completed cleanly because the single worker spent its quota on the loop instead of sharing it across extra processes.</p><p>Now the fix is less mysterious: choose worker count from the CPU time the cgroup can spend, not from the number of CPUs Python can see.</p><h2>Reading the Quota Before Sizing Workers</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*S6ZnbmbWzsozkFmx.png" /></figure><p>Once you put the pieces next to each other, the chain from YAML to latency is short.</p><p>The safer starting point is to read the quota before sizing workers. A quota-aware helper can read cgroup v2 first and fall back to cgroup v1 <a href="https://docs.kernel.org/scheduler/sched-bwc.html#management">CFS bandwidth</a> files:</p><pre>import math<br>import os<br>from pathlib import Path<br>def cgroup_cpu_quota():<br>    cpu_max = Path(&quot;/sys/fs/cgroup/cpu.max&quot;)<br>    if cpu_max.exists():<br>        quota, period = cpu_max.read_text().strip().split()<br>        if quota != &quot;max&quot;:<br>            return int(quota) / int(period)<br>    for cgroup_cpu_dir in (<br>        Path(&quot;/sys/fs/cgroup/cpu&quot;),<br>        Path(&quot;/sys/fs/cgroup/cpu,cpuacct&quot;),<br>    ):<br>        quota_file = cgroup_cpu_dir / &quot;cpu.cfs_quota_us&quot;<br>        period_file = cgroup_cpu_dir / &quot;cpu.cfs_period_us&quot;<br>        if not quota_file.exists() or not period_file.exists():<br>            continue<br>        quota = int(quota_file.read_text().strip())<br>        period = int(period_file.read_text().strip())<br>        if quota &gt; 0:<br>            return quota / period<br>    return len(os.sched_getaffinity(0))<br>quota_cpus = cgroup_cpu_quota()<br>cpu_bound_workers = max(1, math.floor(quota_cpus))<br>workers = int(os.getenv(&quot;WEB_CONCURRENCY&quot;, cpu_bound_workers))</pre><p>This helper does two separate things on purpose. It reads the kernel-enforced quota first, then still leaves WEB_CONCURRENCY as an override, because the quota is the right starting point but the workload decides the final number.</p><h2>Key Takeaways</h2><ul><li>Inside a CPU-limited Kubernetes pod, Python can report the node’s full CPU count while the kernel enforces a much smaller CPU budget through the cgroup. In this lab, os.cpu_count() returned 20, but cpu.max was 50000 100000, which means the pod was limited to 0.5 CPU.</li><li>A Gunicorn worker formula based on os.cpu_count() can produce a worker count that has nothing to do with the pod&#39;s actual CPU quota. In the demo pod, the common workers = multiprocessing.cpu_count() * 2 + 1 pattern would calculate 41 workers from a container that only had half a CPU worth of time.</li><li>More workers did not mean more completed work for this CPU-bound sync endpoint. Under the same 500m limit, one worker completed 101 requests with a 1,002ms median latency, while fourteen workers completed 46 requests with a 6,390ms median because they were all sharing the same half-CPU quota.</li></ul><p>Until next time!</p><p>G.</p><h3>References</h3><ul><li><a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/">Kubernetes Resource Management for Pods and Containers</a></li><li><a href="https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu">Linux cgroup v2 CPU controller documentation</a></li><li><a href="https://docs.python.org/3/library/os.html">Python </a><a href="https://docs.python.org/3/library/os.html">os module documentation</a></li><li><a href="https://github.com/python/cpython/issues/149452">CPython issue: </a><a href="https://github.com/python/cpython/issues/149452">os.process_cpu_count() should take cgroups CPU limitations into account</a></li><li><a href="https://gunicorn.org/design/">Gunicorn design notes on worker count</a></li><li><a href="https://gunicorn.org/reference/settings/">Gunicorn Settings Reference</a></li><li><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab57756a9153" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/what-os-cpu-count-gets-wrong-in-a-cpu-limited-kubernetes-pod-ab57756a9153">What `os.cpu_count()` Gets Wrong in a CPU-Limited Kubernetes Pod</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubelet Metrics: How cAdvisor and CRI Collect Kubernetes Stats]]></title>
            <link>https://itnext.io/kubelet-metrics-how-cadvisor-and-cri-collect-kubernetes-stats-3c12b7e512ce?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c12b7e512ce</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Tue, 19 May 2026 20:51:51 GMT</pubDate>
            <atom:updated>2026-05-23T13:02:33.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45PM2BOBk-u0Ua262HO-Dw.png" /></figure><p><em>This article was originally published on </em><a href="https://learnkube.com/kubernetes-metrics-cadvisor-kubelet-cri"><em>LearnKube</em></a></p><p><em>TL;DR: This article dissects the Kubernetes metrics pipeline through kubelet, cAdvisor, and CRI to show where your metrics actually come from and what breaks when the defaults change.</em></p><p><strong>This article breaks down how Kubernetes collects container, pod, and node metrics, starting with cAdvisor and the Linux kernel, then shifting to a CRI-native model powered by gRPC.</strong></p><p>You’ll see how kubelet exposes this data, what happens when you flip `PodAndContainerStatsFromCRI`, why container metrics on `/metrics/cadvisor` can be sourced from CRI instead of cAdvisor, and how to trace each metric back to its origin.</p><p>It also explains how kubelet talks to the CRI over gRPC, and why understanding this matters if you rely on Prometheus, Grafana, or any observability stack.</p><h3><strong>How Kubernetes Monitoring Layers Stack Up</strong></h3><p>Kubernetes metrics are the lifeblood of observability in your clusters.</p><p>While tools like Prometheus and Grafana often dominate the monitoring conversation, it’s worth understanding the native mechanisms that Kubernetes uses to collect, expose, and leverage metrics before they ever reach those external systems.</p><p><strong>Kubernetes monitoring works as a multi-layered system which provides insights that span from bare metal to application workloads.</strong></p><p>Each layer builds upon the previous one to create a comprehensive picture of your cluster’s health.</p><p><strong>At the foundation sit node-level metrics.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiBWPSEDUHXkqOgCJzrCYA.png" /><figcaption>Node-level metrics as the foundation of the Kubernetes monitoring stack</figcaption></figure><p>These reveal the utilization of physical and virtual resources like CPU, memory, and disk I/O.</p><p>The <a href="https://github.com/prometheus/node_exporter">Prometheus Node Exporter</a> is commonly used to collect these fundamental metrics, but they originate from the operating system itself.</p><p><strong>One layer up are Kubernetes component metrics.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xJ-HI32_z2MjW17OXrJgA.png" /><figcaption>Kubernetes component metrics layered above node-level metrics</figcaption></figure><p>These expose the health and performance of core services such as kubelet, kube-proxy, and the API server.</p><p>Metrics like pod startup latency or API request throughput can tell you whether your control plane is running efficiently and reliably.</p><p>Zooming out to the object layer, <strong>API resource metrics, often surfaced by tools like </strong>`<a href="https://github.com/kubernetes/kube-state-metrics">kube-state-metrics</a>`<strong>, offer visibility into Kubernetes objects.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnHHBN-JfhNgnpjlL1o0kw.png" /><figcaption>API resource metrics from Kubernetes objects layered above component metrics</figcaption></figure><p>They track details such as the number of pods in a namespace, deployment status, or the number of services running across your cluster.</p><p><strong>Finally, at the top layer are pod and container workload metrics.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHttFFNyy_OHRfvjRfHFWA.png" /><figcaption>Pod and container workload metrics at the top of the Kubernetes monitoring stack</figcaption></figure><p>These focus on the actual performance of your applications.</p><p>This is where critical signals like CPU throttling come into play.</p><p>For instance, knowing how often a container is blocked from using CPU because it’s hit its limit can reveal performance bottlenecks that might otherwise remain hidden.</p><h3><strong>Where Metrics Originate</strong></h3><p><strong>Kubernetes defines resource requests and limits, but the kernel does the actual enforcement.</strong></p><p>It relies on the Linux kernel’s control groups, known as cgroups, to apply those rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YaL4e_ywmVNMzTp8mJv1KQ.png" /><figcaption>In this case, I have a control group for the JVM.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PG7gHGWcGzRuqQEy0Oog1Q.png" /><figcaption>I can create a control group that limits access to CPU, memory, network bandwidth, etc.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5i_aIiX-XYcfwnjbTE6ARQ.png" /><figcaption>Each process can have its control group. I could create a second control group for the Node.js app.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPlAB0RvI7E_GUmQpT8_ww.png" /><figcaption>I can fine-tune the settings for the new control group and further restrict the available resources for that process.</figcaption></figure><p>Cgroups are directories in the `/sys/fs/cgroup/` virtual filesystem.</p><p>They are a live view of resource allocation and enforcement at the kernel level, exposed as files you can read and write.</p><p><strong>These directories define how much CPU time, memory, or I/O bandwidth a process is allowed to consume.</strong></p><p>In this context, a resource is anything the system can allocate, limit, and monitor: CPU cycles, memory usage, disk throughput, network bandwidth, even the number of process IDs a container can spawn.</p><p>But defining resources is only half of the story.</p><p>That’s where controllers make all the difference.</p><p><a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/resource_management_guide/br-resource_controllers_in_linux_kernel"><strong>A controller</strong></a><strong> is a kernel component that enforces resource policies and monitors usage for a specific type of resource.</strong></p><p>For every resource, there’s a controller in cgroups that governs it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*8yG31GNyBCvNvIIk7N7LmQ.png" /><figcaption>A cgroup can be governed by multiple controllers, such as CPU, CPUSET, MEMORY, PIDS, IO, RDMA, HUGETLB, and MISC.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5HOoktVNIJ0ZAJqwUVPcoA.png" /><figcaption>Each controller maps to a concrete resource class: CPU time, CPU cores, RAM, process IDs, IO bandwidth, RDMA resources, huge pages, or miscellaneous kernel-managed resources.</figcaption></figure><p><strong>The kernel reads them, applies the rules they define, and keeps every container within its resource boundaries.</strong></p><p><em>Let’s start a Minikube cluster with containerd as the container runtime, and deploy a Python pod to see this in action:</em></p><pre>minikube start -c containerd<br>kubectl create deployment python \<br>  --image=ghcr.io/learnk8s/python-metrics \<br>  --port=8080 \<br>  -- /usr/local/bin/python3 -m http.server 8080<br><br>kubectl get po -o wide<br>NAME                      READY   STATUS    IP<br>python-66dc9f5c8b-w6x4b   1/1     Running   10.244.0.5</pre><p>The Linux cgroup API has two versions: cgroup v1 and cgroup v2.</p><p>Each version structures resource management differently.</p><p>To understand why cgroup v2 and the systemd driver matter, it helps to start with the older model first: cgroup v1 with the cgroupfs driver.</p><h3><strong>cgroup v1 with cgroupfs: The Legacy Baseline</strong></h3><p>In this model, Kubernetes and the container runtime manage cgroups by writing directly to the cgroup filesystem.</p><p>That works, but it also means the hierarchy is shaped by separate controller trees rather than one unified resource tree.</p><p>In cgroup v1, kubelet and the container runtime can still be configured to use either `systemd` or `cgroupfs`, as long as both sides use the same driver.</p><p>Now let’s step into a cgroup v1 environment and see how Kubernetes builds its QoS-based hierarchies when it uses the `cgroupfs` driver.</p><p>We’ll delete our existing Minikube cluster and reboot into a system where cgroup v1 is enabled:</p><pre>minikube delete</pre><p><em>There are several ways to switch a Linux system back to cgroup v1.</em></p><p>You might pass kernel boot parameters like `systemd.unified_cgroup_hierarchy=0` or disable cgroup v2 entirely, depending on the environment, whether it’s bare metal, a VM, or WSL2.</p><p>Once the node boots into cgroup v1, Kubernetes automatically detects it and adjusts its resource management behavior.</p><p>First, confirm the system is operating under cgroup v1:</p><pre>stat -fc %T /sys/fs/cgroup/<br>tmpfs</pre><p>Now start a fresh Minikube cluster with the containerd runtime:</p><pre>minikube start -c containerd<br>kubectl create deployment python \<br>  --image=ghcr.io/learnk8s/python-metrics \<br>  --port=8080 \<br>  -- /usr/local/bin/python3 -m http.server 8080</pre><p>And deploy the Python pod:</p><pre>kubectl get po -o wide<br>NAME                      READY   STATUS    RESTARTS   AGE   IP<br>python-66dc9f5c8b-4248r   1/1     Running   0          42s   10.244.0.4</pre><p>Now we focus on how Kubernetes structures the cgroups under cgroup v1 with the cgroupfs driver.</p><p>Kubernetes enforces QoS-based resource isolation by creating separate hierarchies for each QoS class under every controller.</p><p>We confirm the kubelet configuration to verify this setting:</p><pre>kubectl proxy --port=8001 &amp;<br>curl -X GET http://127.0.0.1:8001/api/v1/nodes/minikube/proxy/configz | jq . | grep -i qos<br>&quot;cgroupsPerQOS&quot;: true,</pre><p>Per-QoS hierarchy creation is enabled, but which driver is kubelet using to manage these hierarchies?:</p><pre>minikube ssh -- &quot;sudo cat /var/lib/kubelet/config.yaml | grep -i cgroupDriver&quot;<br>cgroupDriver: cgroupfs</pre><p>In cgroup v1 with `cgroupsPerQOS: true`, kubelet’s use of the `cgroupfs` driver results in Kubernetes creating and managing separate cgroup subtrees for QoS classes under each controller.</p><p>Let’s inspect the CPU controller directory structure:</p><pre>minikube ssh -- &quot;ls -la /sys/fs/cgroup/cpu/kubepods/&quot;<br>drwxr-xr-x 5 root root 0 Mar 20 12:10 besteffort<br>drwxr-xr-x 7 root root 0 Mar 20 12:11 burstable<br>drwxr-xr-x 3 root root 0 Mar 20 12:12 guaranteed</pre><p>Each QoS class gets its own directory under each controller.</p><p><strong>Since our Python pod was deployed without resource requests, we can locate it under the </strong>`besteffort`<strong> QoS class:</strong></p><pre>minikube ssh -- &quot;ls -la /sys/fs/cgroup/cpu/kubepods/besteffort/&quot;<br>drwxr-xr-x 4 root root 0 Mar 20 03:51 pod23e59e27-abe5-4529-bf9c-581516ae0c0b<br>drwxr-xr-x 4 root root 0 Mar 20 03:51 pod9f874003-a948-425d-a072-f389dc21bdff<br>drwxr-xr-x 4 root root 0 Mar 20 03:51 podc1d8cd50-b50a-4b3c-a33d-8963242c60ef</pre><p>We find multiple pod directories, named by their UID.</p><p>To correlate the pod directory with the actual python pod let’s retrieve its UID from the Kubernetes API:</p><pre>kubectl get pod python-66dc9f5c8b-4248r -o jsonpath=&#39;{.metadata.uid}&#39;<br>c1d8cd50-b50a-4b3c-a33d-8963242c60ef</pre><p>This matches the directory `podc1d8cd50-b50a-4b3c-a33d-8963242c60ef` under the `besteffort` class.</p><p>Inside this pod directory, each container has its own cgroup, named after the container ID:</p><pre>minikube ssh -- &quot;ls -la /sys/fs/cgroup/cpu/kubepods/besteffort/podc1d8cd50-b50a-4b3c-a33d-8963242c60ef/&quot;<br>-rw-r--r-- 1 root root 0 Mar 20 12:16 cpu.shares<br>-rw-r--r-- 1 root root 0 Mar 20 12:16 cpu.cfs_quota_us<br>drwxr-xr-x 2 root root 0 Mar 20 03:52 ef455b35bf7e2afa0942e25b58cd10858d40ed1d97fffe7f0b6a664d2e64aa54<br>-rw-r--r-- 1 root root 0 Mar 20 04:22 tasks</pre><p>For example, we can inspect the pod’s memory limit in the memory controller:</p><pre>minikube ssh -- &quot;cat /sys/fs/cgroup/memory/kubepods/besteffort/\<br>podc1d8cd50-b50a-4b3c-a33d-8963242c60ef/\<br>memory.limit_in_bytes&quot;<br><br>9223372036854771712</pre><p>This very large value is an effectively unlimited memory ceiling, which is expected for a BestEffort pod.</p><p>At this point, kubelet decides where the pod belongs in the QoS hierarchy, the container runtime helps create and configure the container cgroups, and the kernel enforces the resulting cgroup settings for the processes attached to them.</p><h3><strong>At the crux of how cgroup hierarchy is shaped</strong></h3><p>In cgroup v1, each controller operates in its own separate hierarchy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-esZsl2t7Z7bOUA2amDbaA.png" /><figcaption>cgroup hierarchy layout comparing separate controller trees and unified controller trees</figcaption></figure><p>When we list the mounted cgroup controllers in cgroup v1, we see each one mounted independently as its own filesystem:</p><pre>minikube ssh - &quot;mount | grep cgroup&quot;<br>cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)<br>cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)<br>cgroup on /sys/fs/cgroup/pids type cgroup (rw,relatime,pids)</pre><p><strong>This indicates that each controller, whether CPU, memory, or pids, has its own mount point and hierarchy.</strong></p><p>We can confirm this separation by checking `/proc/cgroups`:</p><pre>minikube ssh -- &quot;cat /proc/cgroups&quot;<br><br>#subsys_name    hierarchy    num_cgroups    enabled<br>cpuset          1            34             1<br>cpu             2            52             1<br>cpuacct         3            34             1</pre><p>When we check the filesystem type of `/sys/fs/cgroup/` in cgroup v1, it reports `tmpfs` instead of `cgroup2fs`:</p><pre>minikube ssh -- &quot;stat -fc %T /sys/fs/cgroup/&quot;<br><br>tmpfs</pre><p>The cgroup fs structure looks like the following:</p><pre>minikube ssh -- &quot;ls -la /sys/fs/cgroup/&quot;<br><br>drwxr-xr-x 15 root root   0 Feb 23 05:17 blkio<br>drwxr-xr-x 15 root root   0 Feb 23 05:17 cpu<br>drwxr-xr-x  2 root root  40 Feb 23 05:17 cpu,cpuacct<br>drwxr-xr-x 23 root root   0 Feb 23 05:17 cpuacct<br>drwxr-xr-x 23 root root   0 Feb 23 05:17 cpuset<br>drwxr-xr-x 18 root root   0 Feb 23 05:17 devices<br>drwxr-xr-x 23 root root   0 Feb 23 05:17 freezer</pre><p>This is the core limitation of cgroup v1: CPU, memory, pids, and other controllers can each have their own hierarchy, so resource management is split across multiple trees.</p><p>cgroup v2 fixes that part by moving controllers into a single unified hierarchy.</p><p>Now let’s switch to a cgroup v2 system and examine the structure of the cgroup filesystem.</p><pre>minikube ssh -- &quot;ls -la /sys/fs/cgroup/&quot;<br><br>-r--r--r-- 1 root root 0 Apr 28 10:51 cgroup.controllers<br>-r--r--r-- 1 root root 0 Apr 28 10:58 cgroup.stat<br>-rw-r--r-- 1 root root 0 Apr 28 10:51 memory.high<br>drwxr-xr-x 5 root root 0 Apr 28 10:51 kubepods.slice<br>...</pre><p><strong>All resource controllers are managed together in a single tree rooted at </strong>`/sys/fs/cgroup/`<strong>.</strong></p><p>To confirm that cgroup v2 is active, we can inspect the mounted cgroup filesystem:</p><pre>minikube ssh -- &quot;mount | grep cgroup&quot;<br><br>cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,...)</pre><p>We can list the active controllers that the kernel has attached to this unified hierarchy by reading `/proc/cgroups`.</p><p>In cgroup v2, all controllers operate within a single hierarchy, and the hierarchy column reflects this by showing `0` for each controller:</p><pre>minikube ssh -- &quot;cat /proc/cgroups&quot;<br><br>#subsys_name    hierarchy       num_cgroups     enabled<br>cpu     0       208     1<br>cpuacct 0       208     1<br>blkio   0       208     1<br>devices 0       208     1</pre><p>To verify the filesystem type for `/sys/fs/cgroup/`, we can run the `stat` utility.</p><p>In cgroup v2, this command reports `cgroup2fs`:</p><pre>minikube ssh -- &quot;stat -fc %T /sys/fs/cgroup/&quot;<br><br>cgroup2fs</pre><p><strong>If it shows </strong>`cgroup2fs`<strong>, we know we’re running cgroup v2.</strong></p><p>So cgroup v2 cleans up the kernel-side hierarchy, but it does not answer the ownership question by itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*9-9DFz6mjxgZvy4gF5MdFQ.png" /><figcaption>cgroup hierarchy layout showing kubelet and runtime ownership across pod and container cgroups</figcaption></figure><p>On a systemd-based node, Kubernetes still needs to decide who owns and manages the cgroup tree: systemd or direct filesystem writes through cgroupfs.</p><p><em>cgroup v1 is now only relevant for legacy systems, and its days are officially numbered.</em></p><p>Modern distributions such as <a href="https://discourse.ubuntu.com/t/performance/29416">Ubuntu 22.04+</a>, <a href="https://fedoraproject.org/wiki/Changes/CGroupsV2">Fedora 31+</a>, and <a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/9.0_release_notes/new-features">RHEL 9+</a> enable cgroup v2 by default.</p><p><strong>Kubernetes has supported cgroup v2 as stable since v1.25, and cgroup v1 has been officially deprecated since Kubernetes v1.35 as part of </strong><a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/5573-remove-cgroup-v1/README.md"><strong>KEP-5573</strong></a><strong>.</strong></p><p>Starting with Kubernetes v1.35, kubelet no longer starts on cgroup v1 nodes by default unless `failCgroupV1` is explicitly set to `false`.</p><p>If you’re running production clusters that still use cgroup v1, you should plan a migration to cgroup v2 and define an upgrade or rollback strategy in advance.</p><p>So far, we’ve seen how cgroup v1 and v2 shape the filesystem layout, and we’ve learned how to verify which mode the node is using.</p><p>But to understand how Kubernetes actually turns that kernel structure into pod and container boundaries, we now need to look at the two decisions kubelet makes next: which cgroup manager it initializes, and which cgroup driver owns the tree.</p><p>And that is where the cgroup driver comes in.</p><h3><strong>How Kubernetes Creates and Manages the Cgroup Hierarchy</strong></h3><p><strong>On a Kubernetes node, kubelet and the container runtime collaborate to build and maintain the cgroup hierarchy used for enforcing pod-level resource constraints.</strong></p><p>Before either component can create or manage any cgroups, kubelet needs to resolve one fundamental question: <em>is the node running cgroup v1 or cgroup v2?</em></p><p>That answer comes early.</p><p><strong>At startup, kubelet queries the kernel to determine the active cgroup mode.</strong></p><p>If it detects cgroup v2, it initializes a v2-specific manager built for the unified hierarchy.</p><p>If the node is using cgroup v1, it falls back to a legacy manager.</p><p><strong>This decision locks in the way kubelet will interact with kernel-level resource controls for the lifetime of the process.</strong></p><p>But the cgroup version is only half the equation.</p><p>The other part is who is responsible for actually managing the cgroup tree within `/sys/fs/cgroup/`.</p><p><strong>This is called the cgroup driver.</strong></p><p>Kubelet supports two drivers: systemd or cgroupfs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ho6dEtbbyKeRbbCFPs0TNg.png" /><figcaption>Kubelet detects whether the node uses cgroup v1 or cgroup v2 and initializes the matching cgroup manager</figcaption></figure><p>It picks one or the other, never both at the same time.</p><p>In cgroup v2, the unified hierarchy makes the `systemd` cgroup driver the recommended choice on systemd-based Linux distributions.</p><p>Kubelet can still be configured to use `cgroupfs`, but Kubernetes recommends <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/">avoiding</a> a setup where systemd and Kubernetes manage cgroups separately.</p><p><strong>If the driver is systemd, kubelet hands cgroup creation to systemd; instead of writing directories itself, it generates logical slice names like </strong>`kubepods.slice`<strong> or </strong>`kubepods-besteffort.slice`<strong>.</strong></p><p>These slices represent pod resource groups.</p><p>After generating the slice names, kubelet asks systemd to instantiate and manage the cgroup structure beneath `/sys/fs/cgroup`.</p><p>This is the part cgroup v2 does not solve alone: ownership of the tree needs to be consistent.</p><p>From that point on, all resource controls for pods are expressed through systemd’s unit model.</p><p><em>Why systemd?</em></p><p><strong>Because when you boot a modern Linux system, systemd is the first userspace process the kernel runs.</strong></p><p>It becomes PID 1.</p><p>As PID 1, systemd takes ownership of process supervision and resource control for the entire system.</p><p>Rather than using shell scripts, systemd defines behavior through typed units.</p><p>Units are structured configuration objects like <a href="https://medium.com/@sebastiancarlos/systemds-nuts-and-bolts-0ae7995e45d3">`.service`, `.scope`, and `.slice`.</a></p><p>A slice is how systemd partitions the system for resource control.</p><p><strong>In Kubernetes slices are automatically created by systemd based on pod QoS classes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Pl8CT7ZtpfVPnn9RIGuhA.png" /><figcaption>Systemd slices organizing Kubernetes pod cgroups under kubepods.slice by quality of service class</figcaption></figure><p>Think of slices like namespaces for CPU and memory budgets, managed for you behind the scenes.</p><p>What matters is you can apply limits at the slice level.</p><p><strong>Services are the more familiar systemd unit type.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*68tssmH3CxYfXGDE0tkZZw.png" /><figcaption>Systemd services such as kubelet.service and containerd.service running under system.slice outside kubepods.slice</figcaption></figure><p>A `.service` represents a process that systemd starts and supervises directly.</p><p>On a Kubernetes node, kubelet and containerd usually run as services:</p><ul><li>`kubelet.service`</li><li>`containerd.service`</li></ul><p>These services live under `system.slice`, not under `kubepods.slice`.</p><p><strong>That distinction matters: kubelet and containerd are host daemons that coordinate pod placement and container startup, but the containers themselves do not become children of </strong>`containerd.service`<strong>.</strong></p><p>The actual container processes are placed into Kubernetes pod cgroups under `kubepods.slice`.</p><p>Scopes are different.</p><p><strong>Scopes are used when systemd needs to manage a process it inherits from another launcher and still wants to control.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P5WlzSSs2NzGoZUwdBX_PQ.png" /><figcaption>Systemd scope units placing individual container processes inside Kubernetes pod slices</figcaption></figure><p>For example when the runtime launches a container, systemd can still take over and manage it.</p><p>It does this by wrapping the container process in a `.scope` unit.</p><p>Then systemd creates a `.scope` unit (such as `cri-containerd-&lt;container-id&gt;.scope`) and places it inside an appropriate slice determined by the pod’s quality of service (QoS) class.</p><p><strong>But this only works if both kubelet and the container runtime agree on the cgroup driver.</strong></p><p>If kubelet generates systemd slice names but containerd uses cgroupfs, the contract breaks.</p><p>If the cgroup driver is cgroupfs, kubelet goes back to the older model: direct filesystem ownership.</p><p><strong>Kubelet interacts with the kernel’s cgroup API through the filesystem to create and manage cgroup directories.</strong></p><p>Let’s step back into our Minikube cluster running cgroup v2 with containerd as the runtime.</p><p>Containerd handles its end of the driver selection agreement through its <a href="https://github.com/containerd/containerd/blob/main/docs/cri/config.md">configuration file</a> in `/etc/containerd/config.toml` through the `SystemdCgroup` parameter:</p><pre>minikube ssh - &quot;sudo cat /etc/containerd/config.toml | grep -i -C2 &#39;SystemdCgroup&#39;&quot;<br>runtime_type = &quot;io.containerd.runc.v2&quot;<br>[plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc.options]<br>SystemdCgroup = true<br>[plugins.&quot;io.containerd.grpc.v1.cri&quot;.cni]</pre><p>This is the config version 2 format used by containerd 1.x.</p><p><strong>Once kubelet and the runtime align on both the cgroup version and the driver, kubelet can safely take ownership of building the pod-level cgroup hierarchy.</strong></p><p><em>But in systemd with cgroup v2, which scope unit goes into which systemd slice?</em></p><p>That’s determined by the pod’s QoS class, which kubelet calculates based on the pod’s resource requests and limits.</p><h3><strong>Kubernetes QoS Classes and cgroup Placement</strong></h3><p><strong>Based on the pod’s resource requests and limits, Kubernetes assigns it to one of three </strong><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/"><strong>Quality-of-Service (QoS) classes</strong></a><strong>, which influences where the pod is placed in the cgroup hierarchy.</strong></p><ul><li>A pod is classified as <strong>Guaranteed</strong> only when every container has CPU and memory requests and limits set, and each request exactly matches its corresponding limit.</li><li>A pod is <strong>Burstable</strong> when it defines at least one CPU or memory request or limit but does not meet the stricter Guaranteed rules.</li><li>A pod is <strong>BestEffort </strong>when none of its containers define CPU or memory requests or limits.</li></ul><p>This QoS-to-cgroup hierarchy behavior is controlled by kubelet’s — cgroups-per-qos` flag, which <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#:~:text=%2D%2Dcgroups%2Dper%2Dqos%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0Default%3A%20true">defaults</a> to `true`.</p><p><strong>When </strong>`cgroupsPerQOS: true`<strong> and systemd manages cgroups on a cgroup v2 node, systemd organizes pods under </strong>`kubepods.slice`<strong> and further into slices based on QoS classes.</strong></p><p>Let’s inspect the root qos directory:</p><pre>minikube ssh - &quot;ls -d /sys/fs/cgroup/kubepods.slice/*/&quot;<br>/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/<br>/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/<br>/sys/fs/cgroup/kubepods-poded2df55a_639e_4beb_aee3_5db422c35910.slice/</pre><p><em>Notice the third entry.</em></p><p>It is not a QoS slice like `kubepods-besteffort.slice` or `kubepods-burstable.slice`.</p><p><strong>This is a pod-level cgroup.</strong></p><p>The `pod…` part maps back to `ed2df55a-639e-4beb-aee3–5db422c35910` Kubernetes UID:</p><p>Let’s verify which pod owns that UID:</p><pre>kubectl get pods -A \<br>-o custom-columns=&#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid&#39; \<br>| grep ed2df55a<br>kube-system kindnet-qkqvh ed2df55a-639e-4beb-aee3–5db422c35910</pre><p>So the third cgroup entry belongs to the `kindnet-qkqvh` pod in the `kube-system` namespace.</p><p>Now let’s verify its QoS class from the Kubernetes API:</p><pre>kubectl get pod kindnet-qkqvh -n kube-system -o jsonpath=&#39;{.status.qosClass}{&quot;\n&quot;}&#39;<br>Guaranteed</pre><p>Now, if we print the QoS class and UID together:</p><pre>kubectl get pod kindnet-qkqvh -n kube-system -o jsonpath=&#39;QoS={.status.qosClass}{&quot;\n&quot;}UID={.metadata.uid}{&quot;\n&quot;}&#39;<br>QoS=Guaranteed<br>UID=ed2df55a-639e-4beb-aee3–5db422c35910</pre><p><strong>We see the mapping is the cgroup for this pod and that pod is classified by Kubernetes as </strong>`Guaranteed`<strong>.</strong></p><p>Now let’s look inside that pod cgroup:</p><pre>minikube ssh - &quot;ls -la /sys/fs/cgroup/kubepods.slice/kubepods-poded2df55a_639e_4beb_aee3_5db422c35910.slice/&quot;<br>cri-containerd-7ae5ffd3996a6ac09031cbf283d6bd9727a24bc723a06e76141132a8e57f1716.scope<br>cri-containerd-d24246f29f54f7adced123bc6194d9e0f15fd3a15c54326cd8c96d39961760c0.scope</pre><p>The two `cri-containerd-*.scope` entries are the container-level systemd scope units running inside the `kindnet-qkqvh` pod.</p><p><strong>We have traced a </strong>`Guaranteed`<strong> pod all the way down from the Kubernetes API to its pod slice and container scopes on disk.</strong></p><p>Simplified to the branch we just inspected, the mapping looks like this:</p><pre>/sys/fs/cgroup/<br>└── kubepods.slice<br>└── kubepods-poded2df55a_639e_4beb_aee3_5db422c35910.slice<br>├── cri-containerd-7ae5ffd3996a6ac09031cbf283d6bd9727a24bc723a06e76141132a8e57f1716.scope<br>└── cri-containerd-d24246f29f54f7adced123bc6194d9e0f15fd3a15c54326cd8c96d39961760c0.scope</pre><p><strong>Now let’s do the same for our Python workload, which lands in a different part of the hierarchy because it has a different QoS class.</strong></p><p>Inside the root slice, systemd further organizes pods into separate slices based on their QoS classes.</p><p>Since our Python pod was deployed without any CPU or memory requests or limits, its resources are managed under `kubepods-besteffort.slice`.</p><p>Let’s confirm the QoS classification of the pod:</p><pre>kubectl get pod python-66dc9f5c8b-2kktd -o jsonpath=&#39;{.status.qosClass}&#39;<br>BestEffort</pre><p>Let’s map our python pod and containers to their systemd-managed cgroup slices and scopes.</p><p>To achieve this we will get the pod UID to map it to the slice name:</p><pre>kubectl get pod python-66dc9f5c8b-2kktd -o jsonpath=&#39;{.metadata.uid}&#39;<br>b60baa0b-1e66–4990–8670–93c5919f09cb</pre><p><strong>Each pod gets its own slice under the qos slices and systemd translates hyphens into underscores when creating pod slice directories (</strong>`kubepods-{qos class}-pod{pod UID with underscores}.slice`<strong>).</strong></p><p>List the available pod slices under `kubepods-besteffort.slice`:</p><pre>minikube ssh - &quot;ls -d /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/*/&quot;<br>/sys/fs/cgroup/…/kubepods-besteffort-pod740242e7_85e5_4369_a8a0_d6101719e386.slice/<br>/sys/fs/cgroup/…/kubepods-besteffort-pod857495d4_07b5_45a2_895b_0298f68797d8.slice/<br>/sys/fs/cgroup/…/kubepods-besteffort-podb60baa0b_1e66_4990_8670_93c5919f09cb.slice/</pre><p>The last pod slice corresponds to our Python pod (its UID matches `b60baa0b-1e66–4990–8670–93c5919f09cb`).</p><p>The other entries are other BestEffort pods on the node, such as kube-system pods like CoreDNS or kube-proxy.</p><p><strong>Within this pod slice, systemd organizes each container into separate </strong>`.scope`<strong> units.</strong></p><p>These scopes are named after the containerd runtime and container ID.</p><p>List the contents of the specific pod slice:</p><pre>minikube ssh - &quot;ls /sys/fs/cgroup/kubepods.slice/\<br>kubepods-besteffort.slice/kubepods-besteffort-podb60baa0b_1e66_4990_8670_93c5919f09cb.slice/ | grep scope&quot;<br>cri-containerd-b21e881ca9d6228281aa32cb1e2ebba5537f2a7b90e860a2f0cc6afec3305229.scope<br>cri-containerd-b8609ccf36f85b5a4fc652317358950861a6f0a538e6c4b4c4243241189fbc11.scope</pre><p>The long hex strings above are the container ID, as assigned by containerd.</p><p><strong>Systemd appends them to the </strong>`.scope`<strong> unit it creates for each container.</strong></p><p><em>So now the question is: which one of these is your Python container?</em></p><p>We query containerd to match the container ID:</p><pre>minikube ssh - &quot;sudo crictl ps - name python&quot;<br>CONTAINER IMAGE NAME POD ID POD<br>b21e881ca9d62 bdbec6b439339 python-metrics b8609ccf36f85 python-66dc9f5c8b-2kktd</pre><p>The container ID `b21e881ca9d62` matches the first `.scope` unit above.</p><p>The other one (`b8609ccf36f85…`) is the pod sandbox, which is the pause container we will inspect next.</p><pre>minikube ssh - &quot;\<br>ls -la \<br>/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/\<br>kubepods-besteffort-podb60baa0b_1e66_4990_8670_93c5919f09cb.slice/\<br>cri-containerd-b21e881ca9d6228281aa32cb1e2ebba5537f2a7b90e860a2f0cc6afec3305229.scope&quot;<br>cpu.max<br>hugetlb.2MB.events<br>memory.high<br>memory.stat</pre><p>At this point, the hierarchy for the Python pod looks like this:</p><pre>/sys/fs/cgroup/<br>└── kubepods.slice<br>└── kubepods-besteffort.slice<br>└── kubepods-besteffort-podb60baa0b_1e66_4990_8670_93c5919f09cb.slice<br>├── cri-containerd-b21e881ca9d6228281aa32cb1e2ebba5537f2a7b90e860a2f0cc6afec3305229.scope<br>│ └── python-metrics container<br>└── cri-containerd-b8609ccf36f85b5a4fc652317358950861a6f0a538e6c4b4c4243241189fbc11.scope<br>└── pod sandbox / pause container</pre><p><strong>We can now dig into its cgroup resource metrics like memory usage statistics.</strong></p><pre>minikube ssh - &quot;cat /sys/fs/cgroup/kubepods.slice/\<br>kubepods-besteffort.slice/kubepods-besteffort-podb60baa0b_1e66_4990_8670_93c5919f09cb.slice/\<br>cri-containerd-b21e881ca9d6228281aa32cb1e2ebba5537f2a7b90e860a2f0cc6afec3305229.scope/\<br>memory.stat&quot; | head -5<br>anon 9601024<br>file 13496320<br>kernel 1056768<br>kernel_stack 16384<br>pagetables 94208</pre><p>Great!</p><p><em>But what about the other scope?</em></p><p><strong>In this setup, even a Pod with a single application container has two active container scopes under the pod slice: one for the application container, one for the pause container.</strong></p><p>The pause container is a <a href="https://learnkube.com/kubernetes-network-packets">sandbox environment</a> that sets up the network namespace, IP address, and IPC for the pod.</p><p>Once the sandbox is running and holding that shared environment, Kubernetes starts the Python container inside that namespace.</p><p>Let’s inspect the pod sandbox `b8609ccf36f85` to confirm the pause container:</p><pre>minikube ssh - &quot;sudo crictl inspectp b8609ccf36f85 | grep image&quot;<br>&quot;image&quot;: &quot;registry.k8s.io/pause:3.10.1&quot;,</pre><p><em>The pause container maps to the other </em>`.scope`<em> unit, but how can we verify it?</em></p><p>We inspect the pod sandbox to retrieve the pause container’s PID:</p><pre>minikube ssh - &quot;sudo crictl inspectp b8609ccf36f85 | grep -E &#39;\&quot;pid\&quot;&#39;&quot;<br>&quot;pid&quot;: &quot;CONTAINER&quot;,<br>&quot;pid&quot;: 1647,</pre><p>PID `1647` corresponds to the pause container.</p><p>We correlate the PID with the running process and its parent shim:</p><pre>minikube ssh - &quot;sudo ps -e -o pid,ppid,cmd | grep -E &#39;\\b1603\\b|\\b1647\\b&#39;&quot;<br>1603 1 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id b8609… -address /run/containerd/containerd.sock<br>1647 1603 /pause<br>1694 1603 /usr/local/bin/python3 -m http.server 8080</pre><p>The second scope is the pause container.</p><p><em>PID </em>`1647`<em> is the </em>`/pause`<em> process, and it shares the same </em>`containerd-shim-runc-v2`<em> parent, PID </em>`1603`<em>, with the Python process </em>`1694`<em>.</em></p><h3><strong>Auto-Detecting cgroup Drivers via KubeletCgroupDriverFromCRI</strong></h3><p>Kubernetes addressed some of the coordination challenges with the `KubeletCgroupDriverFromCRI` feature gate, <a href="https://kubernetes.io/blog/2025/09/12/kubernetes-v1-34-cri-cgroup-driver-lookup-now-ga/">introduced</a> as alpha in v1.28 and graduated to GA in v1.34.</p><p><strong>At startup, kubelet asks the runtime which cgroup driver to use through the CRI </strong><a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers">`RuntimeConfig`<strong> RPC</strong></a><strong>.</strong></p><p>On Kubernetes 1.34+, the feature gate no longer needs to be set explicitly.</p><p>If the runtime lacks the RuntimeConfig RPC, kubelet falls back to the `cgroupDriver` value in its own configuration only in Kubernetes versions that still support this <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#:~:text=for%20more%20details.-,KubeletCgroupDriverFromCRI,-Enable%20detection%20of">fallback</a>.</p><p>Let’s start a new cluster using CRI-O as the container runtime:</p><pre>minikube start -p test-driverfromcri - container-runtime=cri-o</pre><p>When we inspect the `/var/lib/kubelet/config.yaml` file, the kubelet config still shows the configured fallback driver:</p><pre>minikube ssh -p test-driverfromcri - &quot;sudo cat /var/lib/kubelet/config.yaml | grep -A2 cgroupDriver&quot;<br>cgroupDriver: systemd<br>clusterDNS:<br>- 10.96.0.10</pre><p>If the CRI runtime does not implement the `RuntimeConfig` RPC, kubelet falls back to the configured `cgroupDriver`:</p><pre>minikube ssh -p test-driverfromcri - &quot;sudo journalctl -u kubelet | grep -E &#39;RuntimeConfig|CRI implementation&#39;&quot;<br>&quot;RuntimeConfig from runtime service failed&quot; err=&quot;rpc error: code = Unimplemented desc = unknown method RuntimeConfig&quot;<br>&quot;CRI implementation should be updated to support RuntimeConfig. Falling back to using cgroupDriver from kubelet config.&quot;</pre><p><strong>Finally, once kubelet settles on a cgroup driver, it uses that driver consistently when placing pods and containers into the node’s cgroup hierarchy.</strong></p><p>The container runtime then passes the resulting cgroup placement into the OCI runtime layer, where `runc/libcontainer` applies it by writing to the kernel’s cgroup interfaces.</p><p>Whether the hierarchy is represented through systemd slices and scopes or raw cgroupfs directories, the end result is the same: the Linux kernel enforces the configured CPU, memory, and other resource limits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1dvarbUJ6i8BXlXK-BFjg.png" /><figcaption>Without RuntimeConfig discovery, kubelet uses its configured cgroupDriver and the runtime uses its own configuration, so both files must be kept in sync.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8Lu7m4g8GJOVDZJpvCZRw.png" /><figcaption>When the runtime supports the RuntimeConfig RPC, kubelet asks containerd which cgroup driver it uses and can align its own behavior with the runtime.</figcaption></figure><p>At this point, we have seen both sides: cgroup v1 with direct filesystem-managed hierarchies, and cgroup v2 with systemd-managed slices and scopes.</p><p>But enforcement is only half of the story.</p><p>The kernel exposes raw counters, limits, and events through the cgroup filesystem, but Kubernetes still needs a component that can read those low-level files and turn them into useful container and pod-level metrics.</p><p>That is the visibility gap cAdvisor was designed to fill.</p><h3><strong>cAdvisor: Embedded Resource Monitoring in Kubelet</strong></h3><p><strong>Container Advisor, or cAdvisor, is the default kubelet-integrated path for </strong><a href="https://kubernetes.io/docs/reference/instrumentation/node-metrics"><strong>collecting</strong></a><strong> container resource usage statistics on Kubernetes nodes.</strong></p><p>It runs as an embedded component inside the kubelet process and is initialized automatically when kubelet starts.</p><p>Once initialized, it reads resource usage from the cgroup filesystem.</p><p><strong>cAdvisor reads low-level resource data from the cgroup filesystem and attaches labels such as </strong>`pod`<strong>, </strong>`namespace`<strong>, </strong>`container`<strong>, and </strong>`image`<strong>.</strong></p><p>Kubelet then exposes the collected metrics through its own HTTP endpoints: the Summary API and cAdvisor metrics endpoint.</p><p>If `PodAndContainerStatsFromCRI` is enabled and the container runtime supports stats through CRI, kubelet fetches pod and container metrics from the runtime instead of cAdvisor.</p><h3><strong>Kubelet’s Metrics Endpoints</strong></h3><p><strong>Kubelet exposes several distinct metrics and stats endpoints on its HTTP server.</strong></p><p>Each serves a specific purpose and differs in data granularity, format, and source.</p><p><strong>The </strong>`/metrics/cadvisor`<strong> endpoint exposes high-resolution container metrics in Prometheus format.</strong></p><p>These metrics come directly from cAdvisor, and kubelet passes them through as-is to the scraper.</p><p>Prometheus typically scrapes this endpoint to collect detailed per-container metrics such as CPU time, memory usage, and I/O statistics.</p><p><strong>These metrics are useful for low-level monitoring, fine-grained alerting, and capacity planning.</strong></p><p>To query the kubelet’s `/metrics/cadvisor` endpoint, we first need to establish a local proxy to the Kubernetes API server.</p><p>Run the following command and leave it running on another terminal:</p><pre>kubectl proxy - port=8001</pre><p>Once the proxy forwards local HTTP requests to the kubelet’s API on the node, we can access kubelet HTTP endpoints through `http://localhost:8001`.</p><pre>curl -sS http://localhost:8001/api/v1/nodes/minikube/proxy/metrics/cadvisor<br>container_cpu_usage_seconds_total{container=&quot;python-metrics&quot;,cpu=&quot;total&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 0.105818<br>container_memory_usage_bytes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 2.5870336e+07<br>container_fs_reads_bytes_total{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 1.49504e+07<br>container_processes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 1<br>container_spec_cpu_shares{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 2<br>container_spec_memory_limit_bytes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 0</pre><p>Related node, pod, container, and volume stats are also available through kubelet’s Summary API on `/stats/summary`, which returns structured JSON instead of Prometheus-formatted metrics:</p><p>`/stats/summary`<strong> exposes node, pod, container, and volume stats. Metrics Server v0.6.0 and later use </strong>`/metrics/resource`<strong> for </strong><a href="https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server"><strong>CPU and memory metrics instead</strong></a><strong>.</strong></p><p>For example, to inspect our pod’s resource consumption, we can run:</p><pre>curl -sS \<br>  http://localhost:8001/api/v1/nodes/minikube/proxy/stats/summary \<br>  | jq &#39;.pods[] | select(.podRef.name == &quot;python-66dc9f5c8b-2kktd&quot;)&#39;<br>{<br>  &quot;podRef&quot;: {<br>    &quot;name&quot;: &quot;python-66dc9f5c8b-2kktd&quot;,<br>    &quot;namespace&quot;: &quot;default&quot;,<br>    &quot;uid&quot;: &quot;b60baa0b-1e66-4990-8670-93c5919f09cb&quot;<br>  },<br>  &quot;containers&quot;: [<br>    {<br>      &quot;name&quot;: &quot;python-metrics&quot;,<br>      &quot;cpu&quot;: {<br>        &quot;usageNanoCores&quot;: 151695,<br>        &quot;usageCoreNanoSeconds&quot;: 226134000<br>      },<br>      &quot;memory&quot;: {<br>        &quot;usageBytes&quot;: 25870336,<br>        &quot;workingSetBytes&quot;: 22114304,<br>        &quot;rssBytes&quot;: 9596928,<br>        &quot;pageFaults&quot;: 3346,<br>        &quot;majorPageFaults&quot;: 136<br>      },<br>      &quot;rootfs&quot;: {<br>        &quot;usedBytes&quot;: 122880<br>      },<br>      &quot;logs&quot;: {<br>        &quot;usedBytes&quot;: 8192<br>      },<br>      &quot;swap&quot;: {<br>        &quot;swapAvailableBytes&quot;: 0,<br>        &quot;swapUsageBytes&quot;: 0<br>      }<br>    }<br>  ]<br>}</pre><p><strong>If you only need simplified, high-level metrics, </strong>`/metrics/resource`<strong> serves that role.</strong></p><p>It exposes CPU and memory usage in Prometheus format, optimized for lightweight node monitoring.</p><p>We can query this endpoint for aggregated container and pod metrics:</p><pre>curl -sS http://localhost:8001/api/v1/nodes/minikube/proxy/metrics/resource | grep python-metrics<br>container_cpu_usage_seconds_total{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 0.298696 1777623311728<br>container_memory_working_set_bytes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 2.2114304e+07 1777623311728<br>container_start_time_seconds{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 1.7776221060112867e+09<br>container_swap_limit_bytes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 0 1777623324188<br>container_swap_usage_bytes{container=&quot;python-metrics&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 0 1777623324188</pre><p>These metrics provide a point-in-time view of how much CPU and memory the pod and its containers are consuming.</p><p><em>What about if we need to debug kubelet’s performance or runtime interactions?</em></p><p><strong>kubelet exposes its own internal metrics at the </strong>`/metrics`<strong> endpoint.</strong></p><p>These metrics include runtime operation durations, event counters, and error rates that reflect how kubelet interacts with the container runtime and manages node resources.</p><p>For instance, if pods take longer to start or containers fail to stop cleanly, reviewing `kubelet_runtime_operations_duration_seconds` can reveal latency bottlenecks between kubelet and the runtime:</p><pre>curl -sS \<br>  http://localhost:8001/api/v1/nodes/minikube/proxy/metrics \<br>  | grep kubelet_runtime_operations_duration_seconds \<br>  | tail -n 3<br>kubelet_runtime_operations_duration_seconds_bucket{operation_type=&quot;version&quot;,le=&quot;+Inf&quot;} 152<br>kubelet_runtime_operations_duration_seconds_sum{operation_type=&quot;version&quot;} 0.12228928199999994<br>kubelet_runtime_operations_duration_seconds_count{operation_type=&quot;version&quot;} 152</pre><p>The four kubelet metrics endpoints fit together like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1vWZlXFkID9xXs-QIS-JNg.png" /><figcaption>Kubelet metrics endpoints showing /metrics/cadvisor, /stats/summary, /metrics/resource, and /metrics with their data sources and formats</figcaption></figure><p>Historically, cAdvisor was Kubernetes’ primary mechanism for container resource monitoring.</p><p>It provided an efficient mechanism for exposing container metrics when workloads were simpler and observability requirements were limited.</p><p>But as Kubernetes matured, a question appeared.</p><p>If kubelet already talks to the container runtime through CRI, why should it always ask cAdvisor to rediscover the same containers from the host filesystem?</p><p>To answer that, we need to look at cAdvisor’s design first.</p><h3><strong>From cAdvisor to CRI: How Kubelet Collects Metrics Today</strong></h3><p><strong>Originally, cAdvisor collected container metrics by observing the Linux host directly.</strong></p><p>That model worked well for the classic Linux container path, where containers were visible through the host’s cgroup hierarchy.</p><p>But Kubernetes later standardized kubelet-to-runtime communication through the Container Runtime Interface (CRI).</p><p><strong>CRI is a </strong><a href="https://kubernetes.io/docs/concepts/containers/cri/"><strong>gRPC-based API</strong></a><strong> that lets kubelet talk to different container runtimes without being tied to a specific runtime implementation.</strong></p><p>So a natural question appears.</p><p><em>If the runtime already created the containers and already tracks their state, why should kubelet always rely on cAdvisor to rediscover that information from the host?</em></p><p>That is the design reason behind the CRI stats path.</p><p><strong>With this path, kubelet gets pod and container stats directly from the runtime.</strong></p><p>That path avoids collecting the same data twice when the runtime already has it.</p><p>It also helps with runtimes where cAdvisor cannot easily see containers from the host.</p><p><em>But how does kubelet achieve that?</em></p><p>We can verify the exact method names directly from the CRI protobuf definition:</p><pre>curl -sSL https://raw.githubusercontent.com/kubernetes/cri-api/master/pkg/apis/runtime/v1/api.proto \<br>  | grep -E &#39;rpc (ContainerStats|ListContainerStats|PodSandboxStats|ListPodSandboxStats)&#39;<br>    rpc ContainerStats(ContainerStatsRequest) returns (ContainerStatsResponse) {}<br>    rpc ListContainerStats(ListContainerStatsRequest) returns (ListContainerStatsResponse) {}<br>    rpc PodSandboxStats(PodSandboxStatsRequest) returns (PodSandboxStatsResponse) {}<br>    rpc ListPodSandboxStats(ListPodSandboxStatsRequest) returns (ListPodSandboxStatsResponse) {}</pre><p>The runtime exposes stats through <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go">CRI RPC methods</a>.</p><p>These calls return structured <a href="https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto">Protobuf</a> messages containing resource usage data such as CPU, memory, network, process, IO, and per-container stats, depending on the platform and runtime implementation.</p><p>With `PodAndContainerStatsFromCRI` enabled, kubelet can use CRI stats methods such as `ListPodSandboxStats`, `PodSandboxStats`, and `ListContainerStats` to collect pod and container metrics from the runtime.</p><p>Kubelet sends these gRPC requests to the runtime endpoint configured on the node.</p><p>For containerd, that endpoint is commonly `/run/containerd/containerd.sock`.</p><p>For CRI-O, it is commonly `/var/run/crio/crio.sock`.</p><p>Once kubelet receives stats from the runtime, it converts the CRI Protobuf responses into kubelet’s internal stats structures and then exposes the resulting stats.</p><p>But did we bypass cAdvisor completely?</p><p><strong>No.</strong></p><p>Even on the CRI stats path, kubelet can still rely on cAdvisor for node-level and filesystem-related stats that are outside the pod and container stats returned by CRI.</p><p>The two stats paths look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*B70bFmwKGtHrcc89e63ADg.png" /><figcaption>With PodAndContainerStatsFromCRI disabled, kubelet relies on cAdvisor to read pod and container usage from the cgroup filesystem before exposing the stats endpoints.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-c-BkFKVP2YjChCdssk40A.png" /><figcaption>With PodAndContainerStatsFromCRI enabled, kubelet asks containerd for pod and container stats over CRI gRPC calls, while cAdvisor still provides node and filesystem stats.</figcaption></figure><h3><strong>Validating CRI-Based Metrics Collection in Kubelet</strong></h3><p><strong>Now that we understand why Kubernetes shifted metrics collection from cAdvisor to the CRI, let’s validate that kubelet is actually pulling metrics from the runtime.</strong></p><p>We’ll configure kubelet to use CRI-based metrics, confirm it through logs, and compare kubelet’s reported data to what containerd provides directly.</p><p>We start by increasing kubelet’s log verbosity by editing its unit file to pass the ` — v=5` argument.</p><pre>/etc/systemd/system/kubelet.service.d/10-kubeadm.conf</pre><p>Inside the above file, we ensure the `ExecStart` line includes the verbose logging flag.</p><pre>[Unit]<br>Wants=containerd.service<br><br>[Service]<br>ExecStart=<br>ExecStart=/var/lib/minikube/binaries/v1.34.0/kubelet \<br>  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \<br>  --config=/var/lib/kubelet/config.yaml \<br>  --hostname-override=minikube \<br>  --kubeconfig=/etc/kubernetes/kubelet.conf \<br>  --node-ip=192.168.49.2 \<br>  --v=5<br><br>[Install]</pre><p>Once we save the configuration, we reload the systemd daemon and restart kubelet.</p><pre>sudo systemctl daemon-reload<br>sudo systemctl restart kubelet</pre><p>First, validate that the container runtime’s socket is active and listening:</p><pre>minikube ssh - &quot;ss -lx | grep containerd.sock&quot;<br>u_str LISTEN 0 4096 /run/containerd/containerd.sock.ttrpc 80566 * 0<br>u_str LISTEN 0 4096 /run/containerd/containerd.sock 79442 * 0</pre><p>Containerd is exposing its CRI endpoint over `/run/containerd/containerd.sock`.</p><p>Next, verify kubelet is configured to use the correct runtime endpoint:</p><pre>minikube ssh - &quot;sudo cat /var/lib/kubelet/config.yaml | grep -i containerRuntimeEndpoint&quot;<br>containerRuntimeEndpoint: unix:///run/containerd/containerd.sock</pre><p>Kubelet is communicating with the correct CRI runtime over the expected UNIX domain socket.</p><p><strong>Let’s tell kubelet to use the CRI for collecting pod and container stats by enabling the </strong>`PodAndContainerStatsFromCRI`<strong> feature gate.</strong></p><p>Before we flip this switch, one thing is worth knowing.</p><p>Kubelet reports the maturity of every feature gate it knows about through the `/metrics` endpoint, under the `kubernetes_feature_enabled` series.</p><p>Querying that series for `PodAndContainerStatsFromCRI` on a fresh Kubernetes 1.34 cluster gives us:</p><pre>curl -sS http://localhost:8001/api/v1/nodes/minikube/proxy/metrics \<br>| grep &#39;kubernetes_feature_enabled.*PodAndContainer&#39;<br>kubernetes_feature_enabled{name=&quot;PodAndContainerStatsFromCRI&quot;,stage=&quot;ALPHA&quot;} 0</pre><p>`stage=”ALPHA”` and `0` means disabled by default.</p><p>We open kubelet’s `/var/lib/kubelet/config.yaml` configuration file on the minikube node and add the feature gate and ensure the following block is present:</p><pre>…<br>featureGates:<br>PodAndContainerStatsFromCRI: true</pre><p>Then we restart kubelet once more.</p><pre>sudo systemctl restart kubelet</pre><p><strong>At this point, kubelet should be sourcing pod and container metrics directly from containerd over the CRI API.</strong></p><p>When we inspect the kubelet logs with the following command:</p><pre>sudo journalctl -u kubelet | grep -i containerstats<br>May 01 10:27:57 minikube kubelet[4205]: feature gates: {map[PodAndContainerStatsFromCRI:true]}<br>May 01 10:27:57 minikube kubelet[4205]: &quot;PodAndContainerStatsFromCRI&quot;: true</pre><p><em>Great!</em></p><p>We see kubelet successfully loads the `PodAndContainerStatsFromCRI` gate.</p><p>But it’s output doesn’t confirm metrics are being retrieved from the runtime.</p><p>`/stats/summary`<strong> is kubelet’s primary interface for exposing metrics that it collects, whether from cAdvisor or directly from the container runtime through the CRI.</strong></p><p>When `PodAndContainerStatsFromCRI` is enabled, kubelet populates this endpoint with data retrieved from the runtime.</p><p>Let’s query `/stats/summary` endpoint to observe the metrics kubelet is serving and confirm whether they match what the runtime reports.</p><p>We will start the kubelet proxy first if you haven’t already and query the summary stats for our pod:</p><pre>kubectl proxy --port=8001<br>curl -sS \<br>  http://localhost:8001/api/v1/nodes/minikube/proxy/stats/summary \<br>  | jq &#39;.pods[] | select(.podRef.name == &quot;python-66dc9f5c8b-2kktd&quot;)&#39;<br>{<br>  &quot;podRef&quot;: {<br>    &quot;name&quot;: &quot;python-66dc9f5c8b-2kktd&quot;,<br>    &quot;namespace&quot;: &quot;default&quot;<br>  },<br>  &quot;containers&quot;: [<br>    {<br>      &quot;name&quot;: &quot;python-metrics&quot;,<br>      &quot;cpu&quot;: {<br>        &quot;usageNanoCores&quot;: 149575,<br>        &quot;usageCoreNanoSeconds&quot;: 1647087000<br>      },<br>      &quot;memory&quot;: {<br>        &quot;workingSetBytes&quot;: 22114304<br>      }<br>    }<br>  ]<br>}</pre><p>The Summary API reports `22114304` bytes of memory working set, about `22.11 MB`, and `149575` nanocores of current CPU usage for the `python-metrics` container.</p><p><em>But how do we know kubelet sourced this from containerd, not cAdvisor?</em></p><p><strong>We can cross-check by querying containerd directly with </strong>`crictl`<strong>.</strong></p><p>But first, we need to confirm the container ID:</p><pre>kubectl get pod python-66dc9f5c8b-2kktd -o jsonpath=&#39;{.status.containerStatuses[*].containerID}&#39;<br>containerd://9b508d38b441b</pre><p>Now we SSH into the node and run `crictl stats`.</p><pre>minikube ssh -- sudo crictl stats<br><br>CONTAINER           CPU %               MEM                 DISK                INODES<br>...<br>5e63e93291a32       0.21                75.7MB              36.86kB             11<br>62bbd4d869537       0.04                66.93MB             65.54kB             24<br>6cff256e868f3       0.00                37.74MB             65.54kB             24<br>9b508d38b441b       0.02                22.11MB             122.9kB             16<br>```</pre><p>The `python-metrics` container appears as container ID `9b508d38b441b` in `crictl stats`, with `MEM` reported as `22.11MB`.</p><p>That matches the Summary API value.</p><p><strong>CPU is harder to match exactly because both values are point-in-time samples, but they are consistent: kubelet reports </strong>`149575`<strong> nanocores, and </strong>`crictl stats`<strong> shows </strong>`0.02%`<strong> CPU for the same container.</strong></p><p>Next, we query kubelet’s `/metrics/resource` endpoint to see the Prometheus exposition format.</p><pre>curl -sS http://localhost:8001/api/v1/nodes/minikube/proxy/metrics/resource \<br>  | grep -i &quot;python-66dc9f5c8b-2kktd&quot;<br><br>pod_cpu_usage_seconds_total{namespace=&quot;default&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 1.760035 1777632057760<br>pod_memory_working_set_bytes{namespace=&quot;default&quot;,pod=&quot;python-66dc9f5c8b-2kktd&quot;} 2.2421504e+07 1777632057760</pre><p>Again, the working set is in the same range across all three views:</p><ul><li>`/metrics/resource` reports about `22.42 MB`,</li><li>`/stats/summary` and `crictl stats` report about `22.11 MB`.</li></ul><p><strong>Kubelet sources pod and container metrics directly from containerd through the CRI API.</strong></p><p>What happens when we check kubelet’s `/metrics/cadvisor` endpoint:</p><pre>curl -sS http://localhost:8001/api/v1/nodes/minikube/proxy/metrics/cadvisor<br>machine_cpu_cores{machine_id=&quot;a5b246...&quot;,system_uuid=&quot;7bd5a1e2-ea5e-452b-a202-536452caf458&quot;} 20<br>machine_cpu_physical_cores{machine_id=&quot;a5b246...&quot;,system_uuid=&quot;7bd5a1e2-ea5e-452b-a202-536452caf458&quot;} 14<br>machine_cpu_sockets{machine_id=&quot;a5b246...&quot;,system_uuid=&quot;7bd5a1e2-ea5e-452b-a202-536452caf458&quot;} 1<br>machine_memory_bytes{machine_id=&quot;a5b246...&quot;,system_uuid=&quot;7bd5a1e2-ea5e-452b-a202-536452caf458&quot;} 3.338305536e+10<br>machine_swap_bytes{machine_id=&quot;a5b246...&quot;,system_uuid=&quot;7bd5a1e2-ea5e-452b-a202-536452caf458&quot;} 3.4088153088e+10</pre><p><em>Huh!</em></p><p>Before enabling the CRI stats path, `/metrics/cadvisor` exposed detailed container metrics emitted by cAdvisor and labeled by pod, namespace, container, image, and cgroup path.</p><p>Now, in this run, the endpoint only shows machine-level cAdvisor metrics such as CPU topology, installed memory, swap capacity, and machine scrape status.</p><p>In this run, no pod metrics or container-level data appeared in the `/metrics/cadvisor` output.</p><p><em>All the pod and container resource usage?</em></p><p>Those pod and container metrics are now sourced from containerd’s CRI stats implementation.</p><h3><strong>Summary</strong></h3><ul><li>Kubernetes does not directly enforce Linux resource limits; the Linux kernel enforces them through cgroups. Kubelet and the container runtime translate pod resource settings into cgroup configuration, then the kernel applies the actual CPU, memory, pids, and related controls.</li><li>cgroup v2 uses a single unified hierarchy where controllers coexist under `/sys/fs/cgroup/`. cgroup v1 uses separate controller hierarchies, so controllers such as CPU, memory, and pids can be mounted as separate cgroup trees.</li><li>cgroup v1 has been officially deprecated since Kubernetes v1.35. As part of <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/5573-remove-cgroup-v1/README.md">KEP-5573</a>, kubelet now fails by default on cgroup v1 nodes unless `failCgroupV1` is explicitly set to `false`, with full code removal planned no earlier than Kubernetes v1.38.</li><li>Kubelet and the container runtime must use a compatible cgroup driver. With the `systemd` driver, kubelet and the runtime place containers under systemd-managed slices; with `cgroupfs`, they manage cgroup paths directly. For cgroup v2, Kubernetes strongly recommends the `systemd` cgroup driver.</li><li>`KubeletCgroupDriverFromCRI` graduated to GA in Kubernetes v1.34. At startup, kubelet asks the runtime for the cgroup driver through the CRI `RuntimeConfig` RPC when the runtime supports it; otherwise kubelet falls back to its configured `cgroupDriver`.</li><li>cAdvisor is embedded inside the kubelet process and starts as part of kubelet. By default, kubelet uses cAdvisor to collect node, pod, container, volume, and filesystem statistics, then exposes that data through kubelet HTTP endpoints. There is no separate cAdvisor sidecar or daemon in the normal kubelet setup.</li><li>Kubelet exposes several metrics and stats endpoints. `/metrics/cadvisor` exposes cAdvisor-style container and machine metrics in Prometheus format. `/stats/summary` returns structured JSON for node, pod, container, and volume stats. `/metrics/resource` exposes lightweight CPU and memory resource metrics used by modern Metrics Server versions. `/metrics` exposes kubelet’s own internal component metrics, such as operation counters and latencies. Metrics Server 0.6.x and later <a href="https://kubernetes.io/docs/reference/instrumentation/node-metrics">query</a> `/metrics/resource`, not `/stats/summary`.</li><li>CRI is the gRPC API that standardizes kubelet-to-runtime communication. It lets kubelet manage pods and containers through the runtime, and with compatible runtimes it can also collect pod and container metrics directly from the runtime over the runtime socket.</li><li>`PodAndContainerStatsFromCRI` is an Alpha feature gate and is disabled by default. When enabled with a compatible runtime, kubelet collects pod and container stats through CRI instead of relying on cAdvisor for those pod and container stats.</li><li>Even with CRI-based pod and container metrics collection, kubelet still depends on cAdvisor for stats that CRI does not provide, especially node-level, machine-level, volume, and filesystem-related data.</li></ul><h3><strong>References</strong></h3><ul><li><a href="https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/">Kubernetes 1.25: cgroup v2 graduates to GA</a></li><li><a href="https://kubernetes.io/blog/2025/09/12/kubernetes-v1-34-cri-cgroup-driver-lookup-now-ga/">Kubernetes v1.34: KubeletCgroupDriverFromCRI graduates to GA</a></li><li><a href="https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/">kube-state-metrics addon</a></li><li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/features/kube_features.go">pkg/features/kube_features.go</a></li><li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/util.go">pkg/kubelet/cadvisor/util.go</a></li><li><a href="https://minikube.sigs.k8s.io/docs/handbook/config/">minikube Runtime configuration</a></li><li><a href="https://github.com/kubernetes/cri-api">cri-api</a></li><li><a href="https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto">cri protocol definition</a></li><li><a href="https://grpc.io/">gRPC</a></li><li><a href="https://github.com/Mirantis/cri-dockerd">cri-dockerd adapter for docker</a></li><li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go">kubelet.go</a></li><li><a href="https://github.com/google/cadvisor/blob/master/manager/manager.go">manager.go</a></li><li><a href="https://github.com/google/cadvisor/blob/master/container/raw/handler.go">raw handler</a></li><li><a href="https://kubernetes.io/docs/concepts/architecture/cgroups/">cgroup v2</a></li><li><a href="https://github.com/google/cadvisor/issues/2785">cAdvisor issues #2785</a></li><li><a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2371-cri-pod-container-stats/README.md">cAdvisor-less, CRI-full Container and Pod Stats Enhancement</a></li><li><a href="https://kubernetes.io/docs/reference/instrumentation/cri-pod-container-metrics/">PodAndContainerStatsFromCRI feature gate</a></li><li><a href="https://github.com/kubernetes/enhancements/issues/2371">KEP #2371 tracking</a></li><li><a href="https://github.com/containerd/containerd/pull/10691">implement CRI ListPodSandboxMetrics</a></li><li><a href="https://github.com/containerd/containerd/blob/main/docs/cri/config.md">containerd CRI configuration</a></li><li><a href="https://github.com/kata-containers/kata-containers/issues/5391">container-stats exporter to the Kata Containers</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c12b7e512ce" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/kubelet-metrics-how-cadvisor-and-cri-collect-kubernetes-stats-3c12b7e512ce">Kubelet Metrics: How cAdvisor and CRI Collect Kubernetes Stats</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Kubernetes Assigns GPUs: Tracing the Device Plugin Path from Pod Spec to CUDA]]></title>
            <link>https://itnext.io/how-kubernetes-assigns-gpus-tracing-the-device-plugin-path-from-pod-spec-to-cuda-39a4b8ebcca3?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a4b8ebcca3</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[device-plugin]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 22:08:50 GMT</pubDate>
            <atom:updated>2026-06-25T21:16:38.280Z</atom:updated>
            <content:encoded><![CDATA[<p>I asked Kubernetes for one GPU, the pod ran CUDA, and then I asked the API which physical GPU the container got. The answer was still just nvidia.com/gpu: 1, and that seemed worth following all the way down to the node.</p><p><strong>TL;DR:</strong> nvidia.com/gpu: 1 is enough for the scheduler to find a node with GPU capacity, but it is not the physical device assignment. In this minikube run, kubelet got the GPU UUID from the NVIDIA device plugin over nvidia-gpu.sock, wrote it to a local checkpoint file, and the container ended up with NVIDIA_VISIBLE_DEVICES set to that UUID while the pod object still showed only the original one-GPU request.</p><h2>What the Scheduler Sees</h2><p>I start with the one thing the scheduler can see: node capacity.</p><pre>kubectl get nodes -o json | jq &#39;.items[].status.capacity&#39;</pre><pre>{<br>  &quot;cpu&quot;: &quot;20&quot;,<br>  &quot;ephemeral-storage&quot;: &quot;949626612Ki&quot;,<br>  &quot;hugepages-1Gi&quot;: &quot;0&quot;,<br>  &quot;hugepages-2Mi&quot;: &quot;0&quot;,<br>  &quot;memory&quot;: &quot;32603084Ki&quot;,<br>  &quot;nvidia.com/gpu&quot;: &quot;1&quot;,<br>  &quot;pods&quot;: &quot;110&quot;<br>}</pre><p>The node advertises one nvidia.com/gpu, so a pod that asks for one GPU can be placed here. That number is enough for scheduling, but it does not say which GPU UUID, which device node, or which CUDA-visible device the container will receive.</p><p>After the CUDA pod is running, I check the pod object for the same detail:</p><pre>kubectl get pod before-cuda-demo -o json | jq &#39;.spec.containers[].resources&#39;</pre><pre>{<br>  &quot;limits&quot;: {<br>    &quot;nvidia.com/gpu&quot;: &quot;1&quot;<br>  },<br>  &quot;requests&quot;: {<br>    &quot;nvidia.com/gpu&quot;: &quot;1&quot;<br>  }<br>}</pre><p>The pod object still shows the request and limit, not the physical GPU. That means the scheduler has answered only the first question: which node has enough advertised GPU capacity.</p><p>The next question is narrower and more local: which device on that node did the container get?</p><p>That is where the device plugin socket comes in.</p><h2>The Device Plugin Socket</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*XVcQgNoDQJYM-HPJPh49Iw.png" /></figure><p>The first place to look is kubelet’s device plugin directory on the node, because device plugins register with kubelet over Unix sockets:</p><pre>minikube ssh -- sudo ls -la /var/lib/kubelet/device-plugins/</pre><pre>srwxr-xr-x 1 root root 0 Mar 21 14:49 kubelet.sock<br>srwxr-xr-x 1 root root 0 Mar 21 14:49 nvidia-gpu.sock</pre><p>There are two sockets here. kubelet.sock is kubelet&#39;s registration socket, and nvidia-gpu.sock is the socket owned by the NVIDIA device plugin.</p><p>The plugin is running as a DaemonSet pod in kube-system:</p><pre>kubectl get pods -n kube-system | grep nvidia</pre><pre>nvidia-device-plugin-daemonset-b4fh7   1/1   Running   1 (5d ago)   6d</pre><p>When that pod starts, it connects to kubelet.sock and registers itself as the handler for nvidia.com/gpu. I check the kubelet log for the registration line:</p><pre>minikube ssh -- sudo journalctl -u kubelet | grep &quot;Got registration request&quot;</pre><pre>I0321 20:13:22.532584   77328 server.go:160] &quot;Got registration request from device plugin with resource&quot; resourceName=&quot;nvidia.com/gpu&quot;</pre><p>That line is the handoff point. From then on, kubelet knows which local plugin owns nvidia.com/gpu, and the plugin can report devices back through the device plugin API.</p><p>Once kubelet has a plugin, the next useful thing to watch is the moment a pod asks for a GPU.</p><h2>The 5-Millisecond Allocation Path</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*OKhH7e14olBCpihN2DE-Gg.png" /></figure><p>Once the pod lands on this node, kubelet has to turn nvidia.com/gpu: 1 into a real device. That happens through device-plugin gRPC calls over nvidia-gpu.sock.</p><p>Those calls are not visible at kubelet’s default verbosity. I raise kubelet verbosity on the minikube node so the device manager prints the allocation path:</p><pre>minikube ssh -- &quot;sudo sed -i &#39;s/verbosity: 0/verbosity: 4/&#39; /var/lib/kubelet/config.yaml &amp;&amp; sudo systemctl restart kubelet&quot;</pre><p>After restarting kubelet and creating the GPU pod again, I filter for the device manager lines that mention the resource name or the checkpoint write:</p><pre>minikube ssh -- sudo journalctl -u kubelet | grep -E &quot;Looking for needed|Need devices to allocate|GetPreferredAllocation|Making allocation request|Checkpoint file written&quot; | grep -E &quot;nvidia.com/gpu|kubelet_internal_checkpoint&quot;<br>I0322 09:19:09.136353   53696 manager.go:835] &quot;Looking for needed resources&quot; resourceName=&quot;nvidia.com/gpu&quot; pod=&quot;default/before-cuda-demo&quot; containerName=&quot;cuda&quot; needed=1<br>I0322 09:19:09.136380   53696 manager.go:600] &quot;Need devices to allocate for pod&quot; deviceNumber=1 resourceName=&quot;nvidia.com/gpu&quot; podUID=&quot;f2b9f4e5-...&quot; containerName=&quot;cuda&quot;<br>I0322 09:19:09.136391   53696 manager.go:1018] &quot;Issuing a GetPreferredAllocation call for container&quot; resourceName=&quot;nvidia.com/gpu&quot; containerName=&quot;cuda&quot;<br>I0322 09:19:09.137452   53696 manager.go:881] &quot;Making allocation request for device plugin&quot; devices=[&quot;GPU-11ba558d-3437-2335-2fd8-a78c8502f87f&quot;] resourceName=&quot;nvidia.com/gpu&quot; pod=&quot;default/before-cuda-demo&quot; containerName=&quot;cuda&quot;<br>I0322 09:19:09.142098   53696 manager.go:502] &quot;Checkpoint file written&quot; checkpoint=&quot;kubelet_internal_checkpoint&quot;</pre><p>Those five lines are the allocation path in miniature. The first timestamp is .136353, and the checkpoint write lands at .142098, so this run spent about 5.74 milliseconds between &quot;this container needs a GPU&quot; and &quot;the result is written locally.&quot;</p><p>The first two lines are kubelet reading the container’s resource request. It sees nvidia.com/gpu: 1, so the device manager needs one device for the cuda container.</p><p>At .136391, kubelet issues GetPreferredAllocation. The next allocation log line contains the chosen device ID: GPU-11ba558d-3437-2335-2fd8-a78c8502f87f.</p><p>This node has a single GPU, so there is only one candidate in this run.</p><p>The next relevant log line is the Allocate() request for that device. By the last line, kubelet has written the result to a local checkpoint file, and the allocation is done.</p><p>The log tells us kubelet wrote a checkpoint, so the next step is to read that file:</p><pre>minikube ssh -- sudo cat /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint<br>{<br>  &quot;Data&quot;: {<br>    &quot;PodDeviceEntries&quot;: [{<br>      &quot;PodUID&quot;: &quot;f2b9f4e5-ba3b-418e-ab8c-c1751a0619d8&quot;,<br>      &quot;ContainerName&quot;: &quot;cuda&quot;,<br>      &quot;ResourceName&quot;: &quot;nvidia.com/gpu&quot;,<br>      &quot;DeviceIDs&quot;: { &quot;-1&quot;: [&quot;GPU-11ba558d-3437-2335-2fd8-a78c8502f87f&quot;] },<br>      &quot;AllocResp&quot;: &quot;CkIKFk5WSURJQV9WSVNJQkxFX0RFVklDRVMSKEdQVS0xMWJhNTU4ZC0zNDM3LTIzMzUtMmZkOC1hNzhjODUwMmY4N2Y=&quot;<br>    }],<br>    &quot;RegisteredDevices&quot;: {<br>      &quot;nvidia.com/gpu&quot;: [&quot;GPU-11ba558d-3437-2335-2fd8-a78c8502f87f&quot;]<br>    }<br>  }<br>}</pre><p>Now the missing UUID is visible. The checkpoint has the pod UID, the container name, the resource name, and the assigned device ID, all stored on the node rather than in the pod spec.</p><h2>What Gets Injected Into the Container</h2><p>The checkpoint also has an AllocResp field. That base64 string is the serialized AllocateResponse protobuf that the device plugin returned to kubelet, so I decode it before looking inside the container:</p><pre>echo &quot;CkIKFk5WSURJQV9WSVNJQkxFX0RFVklDRVMSKEdQVS0xMWJhNTU4ZC0zNDM3LTIzMzUtMmZkOC1hNzhjODUwMmY4N2Y=&quot; | base64 -d | strings</pre><pre>NVIDIA_VISIBLE_DEVICES<br>GPU-11ba558d-3437-2335-2fd8-a78c8502f87f</pre><p>The decoded response contains NVIDIA_VISIBLE_DEVICES and the same GPU UUID from the checkpoint. That is the first bridge from the Kubernetes resource request to what CUDA will see inside the container.</p><p>The next bridge is the container runtime. On this minikube node, Docker is configured to use the NVIDIA runtime by default:</p><pre>minikube ssh -- cat /etc/docker/daemon.json</pre><pre>{<br>  &quot;default-runtime&quot;: &quot;nvidia&quot;,<br>  &quot;runtimes&quot;: {<br>    &quot;nvidia&quot;: {<br>      &quot;path&quot;: &quot;/usr/bin/nvidia-container-runtime&quot;<br>    }<br>  }<br>}</pre><p>That tells me which runtime path started the container. Now I want to check the result from inside a running container, because the pod spec did not declare any /dev/nvidia* device nodes.</p><p>The vectorAdd sample exits as soon as it finishes, so I create a second pod with the same image and GPU request but a sleep command:</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: gpu-inspect<br>spec:<br>  containers:<br>  - name: cuda<br>    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04<br>    command: [&quot;sleep&quot;, &quot;infinity&quot;]<br>    resources:<br>      limits:<br>        nvidia.com/gpu: 1<br>EOF</pre><p>This cluster has one advertised GPU, so gpu-inspect can run only after the previous GPU pod releases it. In this run, the vectorAdd pod had completed, and the following kubectl exec commands show that gpu-inspect was running.</p><p>First I list the NVIDIA device nodes visible inside the container:</p><pre>kubectl exec gpu-inspect -- ls /dev/ | grep nvidia</pre><pre>nvidia-uvm<br>nvidia-uvm-tools<br>nvidia0<br>nvidiactl</pre><p>Those device nodes were not in the pod YAML. They appeared after kubelet, the device plugin response, Docker, and the NVIDIA runtime finished preparing the container.</p><p>The allocation response contained an environment variable, so I check the NVIDIA variables next:</p><pre>kubectl exec gpu-inspect -- env | grep NVIDIA</pre><pre>NVIDIA_VISIBLE_DEVICES=GPU-11ba558d-3437-2335-2fd8-a78c8502f87f<br>NVIDIA_REQUIRE_CUDA=cuda&gt;=11.7 brand=tesla,driver&gt;=450,driver&lt;451 brand=tesla,driver&gt;=470,driver&lt;471 ...<br>NVIDIA_DRIVER_CAPABILITIES=compute,utility</pre><p>NVIDIA_VISIBLE_DEVICES matches the UUID in the checkpoint. The same device ID now appears in three places: kubelet&#39;s local checkpoint, the decoded allocation response, and the running container&#39;s environment.</p><p>The device nodes are not the whole story, because CUDA also needs driver-related files. I list the NVIDIA-related mount points from /proc/mounts:</p><pre>kubectl exec gpu-inspect -- cat /proc/mounts | grep nvidia | awk &#39;{print $2}&#39;<br>/proc/driver/nvidia<br>/usr/bin/nvidia-smi<br>/usr/bin/nvidia-debugdump<br>/usr/bin/nvidia-persistenced<br>/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.163.01<br>/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.550.163.01<br>/usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.550.163.01<br>/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.163.01<br>/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.163.01<br>/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.163.01<br>/usr/lib/firmware/nvidia/550.163.01/gsp_ga10x.bin<br>/usr/lib/firmware/nvidia/550.163.01/gsp_tu10x.bin<br>/dev/nvidiactl<br>/dev/nvidia-uvm<br>/dev/nvidia-uvm-tools<br>/dev/nvidia0<br>/proc/driver/nvidia/gpus/0000:01:00.0</pre><p>This container has 17 NVIDIA-related mount points. They include host-provided driver binaries, shared libraries, firmware blobs, device nodes, and proc entries.</p><p>At this point the ordinary one-GPU path is visible end to end: the pod asks for a count, kubelet gets a UUID from the plugin, and the runtime makes that UUID usable inside the container.</p><h2>Time-Slicing the Same GPU</h2><p>So far, one advertised GPU has meant one GPU pod at a time. To see what time-slicing changes, I first check the NVIDIA resource capacity before changing the plugin config:</p><pre>kubectl get node -o json | jq &#39;.items[0].status.capacity | to_entries[] | select(.key | startswith(&quot;nvidia&quot;))&#39;</pre><pre>{<br>  &quot;key&quot;: &quot;nvidia.com/gpu&quot;,<br>  &quot;value&quot;: &quot;1&quot;<br>}</pre><p>The node reports nvidia.com/gpu: 1. Now I give the NVIDIA device plugin a time-slicing config with three replicas:</p><pre>cat &lt;&lt;&#39;EOF&#39; | kubectl apply -f -<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: time-slicing-config<br>  namespace: kube-system<br>data:<br>  any: |-<br>    version: v1<br>    sharing:<br>      timeSlicing:<br>        resources:<br>        - name: nvidia.com/gpu<br>          replicas: 3<br>EOF</pre><pre>configmap/time-slicing-config created</pre><p>The ConfigMap does nothing until the plugin reads it, so I redeploy the DaemonSet with that config mounted at /config:</p><pre>kubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system<br><br>cat &lt;&lt;&#39;EOF&#39; | kubectl apply -f -<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: nvidia-device-plugin-daemonset<br>  namespace: kube-system<br>spec:<br>  selector:<br>    matchLabels:<br>      name: nvidia-device-plugin-ds<br>  template:<br>    metadata:<br>      labels:<br>        name: nvidia-device-plugin-ds<br>    spec:<br>      tolerations:<br>      - key: nvidia.com/gpu<br>        operator: Exists<br>        effect: NoSchedule<br>      priorityClassName: system-node-critical<br>      containers:<br>      - image: nvcr.io/nvidia/k8s-device-plugin:v0.17.0<br>        name: nvidia-device-plugin-ctr<br>        env:<br>        - name: CONFIG_FILE<br>          value: /config/any<br>        securityContext:<br>          allowPrivilegeEscalation: false<br>          capabilities:<br>            drop: [&quot;ALL&quot;]<br>        volumeMounts:<br>        - name: device-plugin<br>          mountPath: /var/lib/kubelet/device-plugins<br>        - name: config<br>          mountPath: /config<br>      volumes:<br>      - name: device-plugin<br>        hostPath:<br>          path: /var/lib/kubelet/device-plugins<br>      - name: config<br>        configMap:<br>          name: time-slicing-config<br>EOF</pre><p>The important pieces are CONFIG_FILE=/config/any and the ConfigMap volume mounted at /config. Once the plugin pod is running again, I check the node capacity:</p><pre>kubectl get nodes -o json | jq &#39;.items[0].status.capacity[&quot;nvidia.com/gpu&quot;]&#39;</pre><pre>&quot;3&quot;</pre><p>The scheduler now sees three allocatable GPU slots. I create three pods that each ask for one:</p><pre>for i in 1 2 3; do<br>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: ts-pod-$i<br>spec:<br>  containers:<br>  - name: cuda<br>    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04<br>    command: [&quot;sleep&quot;, &quot;infinity&quot;]<br>    resources:<br>      limits:<br>        nvidia.com/gpu: 1<br>EOF<br>done</pre><pre>pod/ts-pod-1 created<br>pod/ts-pod-2 created<br>pod/ts-pod-3 created</pre><p>All three pods were created. Now the question is whether they received different physical devices or the same one, so I print NVIDIA_VISIBLE_DEVICES from each container:</p><pre>for i in 1 2 3; do<br>  echo &quot;ts-pod-$i: $(kubectl exec ts-pod-$i -- printenv NVIDIA_VISIBLE_DEVICES)&quot;<br>done</pre><pre>ts-pod-1: GPU-11ba558d-3437-2335-2fd8-a78c8502f87f<br>ts-pod-2: GPU-11ba558d-3437-2335-2fd8-a78c8502f87f<br>ts-pod-3: GPU-11ba558d-3437-2335-2fd8-a78c8502f87f</pre><p>Before time-slicing, the node reported nvidia.com/gpu: 1; after this ConfigMap, it reported nvidia.com/gpu: 3. The three containers all printed the same UUID, which means this lab showed a scheduling-count change rather than hardware partitioning.</p><p>Before continuing, clean up the time-slicing setup and restore the default device plugin:</p><pre>kubectl delete pod ts-pod-1 ts-pod-2 ts-pod-3<br>kubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system<br>kubectl delete configmap time-slicing-config -n kube-system<br>kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml</pre><p>Wait for the plugin pod to come back and verify the node reports one GPU again:</p><pre>kubectl get nodes -o json | jq &#39;.items[0].status.capacity[&quot;nvidia.com/gpu&quot;]&#39;</pre><pre>&quot;1&quot;</pre><h2>What Happens When Things Break</h2><p>So far, kubelet can reach the plugin and allocation works.</p><p>The next thing I want to know is what breaks when the plugin disappears. I start by deleting the running NVIDIA device plugin pod:</p><pre>kubectl get pods -n kube-system | grep nvidia</pre><pre>nvidia-device-plugin-daemonset-bnmkq   1/1   Running   1 (5d ago)   6d</pre><pre>kubectl delete pod nvidia-device-plugin-daemonset-bnmkq -n kube-system</pre><pre>pod &quot;nvidia-device-plugin-daemonset-bnmkq&quot; deleted</pre><p>The DaemonSet creates a replacement pod, but the interesting part is what kubelet sees during the gap. I check the kubelet logs for removal and registration:</p><pre>minikube ssh -- sudo journalctl -u kubelet --since &quot;1 min ago&quot; | grep -E &quot;Removed device plugin|Got registration request&quot;</pre><pre>W0322 10:05:41.218903   53696 manager.go:312] &quot;Removed device plugin for resource&quot; resourceName=&quot;nvidia.com/gpu&quot;<br>I0322 10:05:43.894122   53696 server.go:160] &quot;Got registration request from device plugin with resource&quot; resourceName=&quot;nvidia.com/gpu&quot;</pre><p>In this run, there were about 2.7 seconds between kubelet removing the old plugin and receiving a new registration.</p><p>That raises a different question: does a running GPU container lose its device nodes when the plugin pod goes away?</p><p>I check that gpu-inspect can see /dev/nvidia0, then delete the plugin pod while that container is running:</p><pre>kubectl exec gpu-inspect -- ls /dev/nvidia0<br>/dev/nvidia0</pre><pre>kubectl delete -n kube-system $(kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o name | head -1) --force</pre><p>After the plugin pod is deleted, I check the same device path again:</p><pre>kubectl exec gpu-inspect -- ls /dev/nvidia0</pre><pre>/dev/nvidia0</pre><p>/dev/nvidia0 is still there. In this run, removing the plugin pod did not remove device nodes that had already been mounted into a running container.</p><p>That does not mean the plugin is optional. New allocations still need the plugin, so I delete the entire DaemonSet and check what the node reports:</p><pre>kubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system</pre><pre>kubectl get nodes -o json | jq &#39;.items[0].status.allocatable[&quot;nvidia.com/gpu&quot;]&#39;</pre><pre>&quot;0&quot;</pre><p>Allocatable dropped to zero. With no allocatable GPU left, I check the new test pod:</p><pre>kubectl get pod gpu-pending-test</pre><pre>NAME               READY   STATUS    RESTARTS   AGE<br>gpu-pending-test   0/1     Pending   0          9s</pre><p>The new GPU pod is pending because the scheduler sees zero allocatable GPUs.</p><p>The plugin failure path explains new scheduling, but kubelet has another piece of local state: the checkpoint file.</p><p>Kubelet stores device allocation data in /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint, so corrupting that file is a direct way to see how kubelet reacts when its local allocation record is unreadable.</p><p>I back up the checkpoint, write garbage into it, and restart kubelet:</p><pre>minikube ssh -- &quot;sudo cp /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint.bak&quot;<br>minikube ssh -- &quot;echo &#39;CORRUPTED&#39; | sudo tee /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint&quot;<br>minikube ssh -- &quot;sudo systemctl restart kubelet&quot;</pre><p>Then I check kubelet’s checkpoint-related logs:</p><pre>minikube ssh -- sudo journalctl -u kubelet --since &quot;30 sec ago&quot; | grep -i checkpoint</pre><pre>E0328 20:53:56.625089  118745 manager.go:347] &quot;Continue after failing to read checkpoint file. Device allocation info may NOT be up-to-date&quot; err=&quot;invalid character &#39;C&#39; looking for beginning of value&quot;</pre><p>Kubelet found the corrupted file, failed to parse it, and continued with a warning that device allocation info may not be up to date. The running pod still had its GPU mounts because those mounts were created during container startup.</p><p>Restore the backup before continuing:</p><pre>minikube ssh -- &quot;sudo cp /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint.bak /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint&quot;<br>minikube ssh -- &quot;sudo systemctl restart kubelet&quot;</pre><p>One more node-local detail is the socket itself. I delete the plugin pod again:</p><pre>kubectl delete -n kube-system $(kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o name | head -1)</pre><pre>pod &quot;nvidia-device-plugin-daemonset-gvczz&quot; deleted</pre><p>Then I check whether kubelet saw the old gRPC stream end and a new registration arrive:</p><pre>minikube ssh -- &quot;sudo journalctl -u kubelet --since &#39;30 sec ago&#39; | grep -E &#39;ListAndWatch ended|Got registration&#39;&quot;</pre><pre>E0328 20:59:23.736830  118883 client.go:90] &quot;ListAndWatch ended unexpectedly for device plugin&quot; err=&quot;rpc error: code = Unavailable desc = error reading from server: EOF&quot; resource=&quot;nvidia.com/gpu&quot;<br>I0328 20:59:24.920559  118883 server.go:160] &quot;Got registration request from device plugin with resource&quot; resourceName=&quot;nvidia.com/gpu&quot;</pre><p>The first line is kubelet discovering that its gRPC stream to the old socket hit EOF. The plugin on the other end is gone.</p><p>About one second later, the replacement plugin pod has started, created a fresh nvidia-gpu.sock, and registered with kubelet.</p><p>I can check the socket file timestamp to see whether the socket changed. Before deleting the plugin pod:</p><pre>minikube ssh -- &quot;sudo stat -c &#39;%Y %n&#39; /var/lib/kubelet/device-plugins/nvidia-gpu.sock&quot;</pre><pre>1774731875 /var/lib/kubelet/device-plugins/nvidia-gpu.sock</pre><p>After deleting the plugin pod and waiting a few seconds:</p><pre>minikube ssh -- &quot;sudo stat -c &#39;%Y %n&#39; /var/lib/kubelet/device-plugins/nvidia-gpu.sock&quot;</pre><pre>1774731959 /var/lib/kubelet/device-plugins/nvidia-gpu.sock</pre><p>The timestamp changed, so this run saw a fresh nvidia-gpu.sock after the replacement plugin started.</p><p>The pattern is the same as the successful allocation path. The plugin socket, checkpoint file, device assignment, and replacement behavior are all node-local details.</p><h2>Where the Device Mapping Lives</h2><p>Now the original mismatch is easier to explain. The pod object showed the request, while the node had the physical assignment.</p><p>To find the assigned GPU UUID for gpu-inspect, I ask the checkpoint file with the pod UID:</p><pre>POD_UID=$(kubectl get pod gpu-inspect -o jsonpath=&#39;{.metadata.uid}&#39;)</pre><pre>minikube ssh -- &quot;sudo cat /var/lib/kubelet/device-plugins/kubelet_internal_checkpoint&quot; \<br>  | jq --arg uid &quot;$POD_UID&quot; &#39;.Data.PodDeviceEntries[] | select(.PodUID==$uid) | {PodUID, DeviceIDs, ResourceName}&#39;</pre><pre>{<br>  &quot;PodUID&quot;: &quot;60921126-5b78-4e9b-867f-371ba9a65349&quot;,<br>  &quot;DeviceIDs&quot;: {<br>    &quot;-1&quot;: [<br>      &quot;GPU-11ba558d-3437-2335-2fd8-a78c8502f87f&quot;<br>    ]<br>  },<br>  &quot;ResourceName&quot;: &quot;nvidia.com/gpu&quot;<br>}</pre><p>This is the physical GPU UUID that was missing from the pod object.</p><p>The checkpoint is not the only node-local view. Kubelet also exposes the same mapping through the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources">PodResources gRPC API</a>, so I query that socket too:</p><pre>minikube ssh -- &quot;sudo grpcurl -plaintext -unix \<br>  -import-path /tmp -proto api.proto \<br>  /var/lib/kubelet/pod-resources/kubelet.sock v1.PodResourcesLister/List&quot; \<br>  | jq &#39;.podResources[] | select(.name==&quot;gpu-inspect&quot;) | .containers[0].devices&#39;</pre><pre>[<br>  {<br>    &quot;resourceName&quot;: &quot;nvidia.com/gpu&quot;,<br>    &quot;deviceIds&quot;: [<br>      &quot;GPU-11ba558d-3437-2335-2fd8-a78c8502f87f&quot;<br>    ]<br>  }<br>]</pre><p>Same UUID again. The checkpoint file, the PodResources API, and the container environment all point to GPU-11ba558d-3437-2335-2fd8-a78c8502f87f.</p><h2>Key Takeaways</h2><ul><li>A running GPU pod can still look generic from the Kubernetes API. In this lab, kubectl get pod before-cuda-demo -o json showed only &quot;nvidia.com/gpu&quot;: &quot;1&quot; in the container resources, while the physical GPU UUID did not appear anywhere in the pod spec.</li><li>The actual GPU assignment showed up in node-local kubelet data rather than the pod object. For gpu-inspect, the checkpoint file and the PodResources gRPC API both reported GPU-11ba558d-3437-2335-2fd8-a78c8502f87f, while the pod object still only described a one-GPU request.</li><li>The NVIDIA device plugin allocation response was enough to connect the pod request to CUDA-visible hardware. In this run, decoding the checkpoint’s serialized AllocateResponse showed NVIDIA_VISIBLE_DEVICES=GPU-11ba558d-3437-2335-2fd8-a78c8502f87f, and the running container had /dev/nvidia0, /dev/nvidiactl, NVIDIA driver binaries, shared libraries, firmware files, and proc entries mounted in.</li><li>Time-slicing changed the schedulable GPU count, not the physical GPU each container saw. After configuring three replicas, node capacity changed from &quot;nvidia.com/gpu&quot;: &quot;1&quot; to &quot;3&quot;, but ts-pod-1, ts-pod-2, and ts-pod-3 all printed the same NVIDIA_VISIBLE_DEVICES value: GPU-11ba558d-3437-2335-2fd8-a78c8502f87f.</li></ul><p>If you want to understand all of this but aren’t sure where to start, Daniele has an excellent talk that makes the whole GPU-on-Kubernetes story click: <a href="https://www.youtube.com/watch?v=xKLpROR98gs&amp;t=1127s">GPU Sharing on Kubernetes</a>.</p><p>We also gave a <a href="https://kccnceu2026.sched.com/event/2CW1Q/gpus-on-kubernetes-what-actually-happens-when-you-request-nvidiacomgpu-1-gulcan-topcu-daniele-polencic-learnkube?iframe=no">KubeCon EU 2026 talk</a> that walks through the same allocation path live on a cluster. The recording should be up soon.</p><p>Until next time!</p><p>G.</p><h4><strong>References</strong></h4><ul><li><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/">Kubernetes Device Plugins</a></li><li><a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">Schedule GPUs</a></li><li><a href="https://github.com/NVIDIA/k8s-device-plugin">NVIDIA k8s-device-plugin</a></li><li><a href="https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html">NVIDIA Container Toolkit</a></li><li><a href="https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/">Topology Manager</a></li><li><a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/">NVIDIA MIG User Guide</a></li><li><a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html">NVIDIA GPU Sharing with Time-Slicing</a></li><li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/">Dynamic Resource Allocation</a></li><li><a href="https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3573-device-plugin">KEP-3573: Device Plugin API</a></li><li><a href="https://kubex.ai/resources/rightsizing-gpus-in-kubernetes-ebook/?utm_source=LearnKube&amp;utm_medium=newsletter&amp;utm_campaign=LearnKube-fy26">Right-Sizing GPUs in Kubernetes, Kubex eBook</a></li><li><a href="https://www.vcluster.com/gpu-enabled-platforms-on-kubernetes">GPU-Enabled Platforms on Kubernetes: vCluster ebook</a></li><li><a href="https://www.youtube.com/watch?v=xKLpROR98gs&amp;t=1127s">GPU Sharing on Kubernetes, Daniele Polencic (YouTube)</a></li><li><a href="https://kccnceu2026.sched.com/event/2CW1Q/gpus-on-kubernetes-what-actually-happens-when-you-request-nvidiacomgpu-1-gulcan-topcu-daniele-polencic-learnkube?iframe=no">GPUs on Kubernetes: What Actually Happens When You Request nvidia.com/gpu: 1, KubeCon EU 2026</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a4b8ebcca3" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-kubernetes-assigns-gpus-tracing-the-device-plugin-path-from-pod-spec-to-cuda-39a4b8ebcca3">How Kubernetes Assigns GPUs: Tracing the Device Plugin Path from Pod Spec to CUDA</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Intra-Node Pod Traffic in Kubernetes: How Kindnet with PTP Moves Packets]]></title>
            <link>https://itnext.io/inside-intra-node-pod-traffic-in-kubernetes-how-kindnet-with-ptp-moves-packets-ffbbc07612b7?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffbbc07612b7</guid>
            <category><![CDATA[kubernetes-networking]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Tue, 27 May 2025 08:54:54 GMT</pubDate>
            <atom:updated>2025-05-27T18:55:20.897Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BdycuHhje7OPrwpT.png" /></figure><p>When a pod wants to talk to another pod on the same node, each sends traffic to its gateway (`10.244.X.1`) via `eth0` which resolves to the host-side veth on the same node. The host then routes packets to the peer pod’s veth using static entries in it’s routing table. But before that hop, the pod still ARPs for its gateway.</p><p>This article traces packets between two pods on the same node using `tshark`, confirms the ARP exchange, inspects the forwarding path through the host’s veth interfaces, and explores how Kindnet achieves L3 routing — <em>while still relying on ARP resolution on the pod side</em>.</p><p>In the first article, we explored how Kindnet configures pod networking with `ptp`, sets up veth pairs, assigns `/32` host routes, and wires each pod into the host namespace.</p><p>Now let’s start inspecting traffic between two pods on the same node.</p><p><em>You can read this article free on my blog </em><a href="https://podlock.readthedocs.io/blogs/k8s/ptp/ptp.html/"><em>here</em></a><em>.</em></p><p><strong>The Setup: Same-Node Pod Pairs, IPs, and Routes</strong></p><p>We’re working in a two-node Minikube cluster with Kindnet as the CNI, using the default `ptp` plugin. Let’s confirm that our two pods — Pod A and Pod B — are scheduled onto the same node, and retrieve their IPs.</p><pre>kubectl get pods -o wide | grep pod-to-pod<br>pod-to-pod-6d4875f88c-5qcw5 10.244.1.2 minikube-m02<br>pod-to-pod-6d4875f88c-99qwj 10.244.1.3 minikube-m02</pre><p>Both pods land on `minikube-m02`. Their IPs are in the same PodCIDR assigned to that node (`10.244.1.0/24`), and Kindnet has installed per-pod `/32` host routes for them.</p><p>On the host run the following to explore the forwarding path:</p><pre>minikube ssh -n minikube-m02<br>ip route | grep 10.244.1<br>10.244.1.2 dev vethd1a5eead scope host<br>10.244.1.3 dev vethc28b4145 scope host</pre><p>Traffic destined for `10.244.1.3` will be routed via `vethc28b4145` as it is the host-side veth paired with Pod B’s `eth0`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*flRq8PMDb7EiasUX.png" /></figure><p>Now we’ll trace what actually happens when Pod A sends traffic to Pod B on the same node.</p><p><strong>Capture ARP and ICMP Traffic from Pod A to Pod B</strong></p><p>Before we send any pings, we’ll clear the ARP cache inside Pod A to force a fresh request. Inside Pod A:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-5qcw5 - ip neigh flush all</pre><p>We already identified that `vethc28b4145` is the host-side veth connected to Pod B (`10.244.1.3`). From the host (`minikube-m02`), start a packet capture on that interface:</p><pre>sudo tshark -i vethc28b4145 -f &quot;icmp or arp&quot;</pre><p>Leave this running. In a second terminal, trigger the packet flow from Pod A (`10.244.1.2`) to Pod B:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-5qcw5 - ping -c1 10.244.1.3<br>1 packets transmitted, 1 received, 0% packet loss, time 0ms<br>rtt min/avg/max/mdev = 0.121/0.121/0.121/0.000 ms</pre><p>Now switch back to your `tshark` capture. You should see output like this:</p><pre>8a:21:03:db:fe:11 → ff:ff:ff:ff:ff:ff ARP Who has 10.244.1.3? Tell 10.244.1.1<br>6a:9e:4f:c8:4e:83 → 8a:21:03:db:fe:11 ARP 10.244.1.3 is at 6a:9e:4f:c8:4e:83<br>10.244.1.2 → 10.244.1.3 ICMP Echo request<br>10.244.1.3 → 10.244.1.2 ICMP Echo reply<br>6a:9e:4f:c8:4e:83 → ff:ff:ff:ff:ff:ff ARP Who has 10.244.1.1? Tell 10.244.1.3<br>8a:21:03:db:fe:11 → 6a:9e:4f:c8:4e:83 ARP 10.244.1.1 is at 8a:21:03:db:fe:11</pre><p>Before Pod A can send any packet like ping, it must know the MAC address of its next hop, which is `10.244.1.1`. That IP isn’t Pod B — it’s the gateway IP on the other end of the veth pair inside the host. So Pod A issues an ARP request for `10.244.1.1` to figure out what Ethernet address to send the packet to.</p><p>The packet exits Pod A and enters the host through its veth. The host routes it to Pod B using a `/32` rule that maps `10.244.1.3` directly to `vethc28b4145`. Pod B receives the packet, and when preparing the reply, it consults its routing table — default via `10.244.1.1`.</p><p>Pod B doesn’t send the reply directly to Pod A’s IP. Just like Pod A, it always routes through its gateway. So it uses `10.244.1.1` as its next-hop gateway. But before sending the ICMP reply, it ARPs to resolve the MAC address of `10.244.1.1` too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*J5nPoTCqbw4xBc0a.png" /></figure><p><strong>Inspect Routing and Neighbor Tables Inside Each Pod</strong></p><p>We’ve seen the packets on the wire. Now let’s inspect the control plane: how each pod understands where to send traffic and which MACs it has learned.</p><p>Start with <strong>Pod A</strong> (`10.244.1.2`). Check its routing table:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-5qcw5 - ip route<br>default via 10.244.1.1 dev eth0<br>10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.2<br>10.244.1.1 dev eth0 scope link src 10.244.1.2</pre><p>All traffic — including packets to other pods on the same subnet — is sent to `10.244.1.1`, the pod’s declared gateway. This IP is assigned to the host-side veth and acts as the entry point into the node’s routing stack. The pod doesn’t interact with peers directly; it always sends to the gateway. How does the pod know where to send it?</p><p>The last route — `10.244.1.1 dev eth0 scope link` — tells the pod <strong>send packets to `10.244.1.1` directly out `eth0` — it is the other end of the veth pair.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*BhBs_yf1jutMvcJ4.png" /></figure><p>Now check the neighbor table (ARP cache):</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-5qcw5 - ip neigh<br>10.244.1.1 dev eth0 lladdr 72:b5:bb:06:6f:1d STALE</pre><p>The pod has learned the MAC address of its gateway on interface `eth0`, but hasn’t used it recently. No MAC for Pod B — because Pod A never sends directly to it.</p><p>Now flip to <strong>Pod B</strong> (`10.244.1.3`) and run the same commands:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-99qwj - ip route<br>default via 10.244.1.1 dev eth0<br>10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.3<br>10.244.1.1 dev eth0 scope link src 10.244.1.3<br>kubectl exec -it pod-to-pod-6d4875f88c-99qwj - ip neigh<br>10.244.1.1 dev eth0 lladdr 8a:21:03:db:fe:11 STALE</pre><p>Same behavior: each pod talks only to `10.244.1.1`. There is no peer-to-peer ARP — because only the host sees both pods, the pods never see each other directly.</p><p><strong>Trace Host Routing Decisions with `ip route get`</strong></p><p>We’ve confirmed that pods send traffic to their gateway, now let’s explore what the host does when it receives that packet.</p><p>We’ll ask the kernel: “If I send a packet to this IP, what interface and next hop will you use?”</p><pre>minikube ssh -n minikube-m02<br>ip route get 10.244.1.3<br>10.244.1.3 dev vethc28b4145 src 10.244.1.1 uid 1000<br>cache</pre><p>The host routes `10.244.1.3` via `vethc28b4145` — the exact interface we captured earlier. The `src` is `10.244.1.1`, meaning the reply will appear to come from the gateway IP. Finally, the decision is cached, so the kernel won’t recalculate it unless needed.</p><p><strong>Is SNAT Touching Node-Local Pod Traffic?</strong></p><p>We’ve validated that intra-node pod traffic flows through veth pairs, is routed via static `/32` rules (in `ptp`) or switches through a bridge (in `bridge`), and relies on ARP when appropriate. But there’s one last piece to verify: <strong>does Kubernetes ever apply SNAT to pod-to-pod traffic, even when both pods are on the same node?</strong></p><p>Let’s inspect what iptables is doing to traffic on this node. SSH into the node and dump the MASQUERADE rules:</p><pre>minikube ssh -n minikube-m02<br>sudo iptables -t nat -L -n -v | grep MASQUERADE<br>MASQUERADE all - * !docker0 172.17.0.0/16 0.0.0.0/0<br>RETURN all - * * 0.0.0.0/0 10.244.0.0/16 /* kind-masq-agent: local traffic is not subject to MASQUERADE */<br>MASQUERADE all - * * 0.0.0.0/0 0.0.0.0/0 /* kind-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */</pre><p>Let’s go through these, line by line.</p><p>The first one is Docker-specific. It says: if traffic is from `172.17.0.0/16` and <strong>not leaving through `docker0`</strong>, apply SNAT. This is irrelevant for pod traffic, but still active on the node.</p><p>The second line says if the destination is inside the pod network (10.244.0.0/16), don’t SNAT. This is why <strong>no source IP rewrite</strong> happens when one pod talks to another directly by IP.</p><p>The third line is the <strong>default SNAT</strong> rule for all <strong>outbound</strong> traffic. If the packet didn’t match any earlier rule, and it’s leaving the node — <strong>this rule kicks in and rewrites the source IP.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*wOh5WROBinwybGiL.png" /></figure><p><strong>Prove SNAT Does Not Apply to Direct Pod-to-Pod Traffic</strong></p><p>We’ve already seen that the iptables `RETURN` rule excludes traffic within the pod CIDR (`10.244.0.0/16`) from SNAT. Let’s confirm that with a live packet capture — on a pod on the same node. We’ll watch what <strong>Pod B</strong> (`10.244.1.3`) sees when pinged by <strong>Pod A</strong> (`10.244.1.2`). First, open a shell into Pod B and start sniffing:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-4d5wh - tshark -i eth0</pre><p>Now, from Pod A (`10.244.1.2`), send a single ICMP packet to Pod B:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-tj72k - ping -c1 10.244.1.3</pre><p>Now back in the tshark trace inside Pod B, you should see:</p><pre>1 0.000000000 10.244.1.2 → 10.244.1.3 ICMP Echo request<br>2 0.000063461 10.244.1.3 → 10.244.1.2 ICMP Echo reply</pre><p>The packet <strong>arrives with Pod A’s real IP: `10.244.1.2`</strong> which means that the source IP is preserved.</p><p><strong>Kindnet with ptp Key Takeaways</strong></p><p>Pod A sends to Pod B’s IP, but always <strong>routes the packet to its gateway</strong> — `10.244.X.1` — not directly to the peer.</p><p>That “gateway” is just the <strong>host-side veth</strong> on the other end of the pod’s veth pair —<strong>not a real router</strong>.</p><p>Each pod does an <strong>ARP for `10.244.X.1`</strong>, even if the destination pod is on the same node and same subnet.</p><p>The host (Minikube node) sees <strong>/32 static routes</strong> injected by Kindnet that map pod IPs to their respective veth interfaces. So when the packet hits the host, the kernel matches the `/32` route and forwards the packet to the peer pod’s veth.</p><p><strong>Neither pod ever ARPs for the other</strong>, they only see the gateway.</p><p>SNAT is bypassed because iptables has a `RETURN` rule for traffic staying within the pod CIDR.</p><p><strong>Next: What If the Pods Are On Different Nodes?</strong></p><p>Now that we’ve explored intra-node traffic, the next logical step is to go cross-node.</p><p>⬅️ Previously: <strong>Inside a Pod’s Birth: Veth Pairs, IPAM, and Routing with Kindnet CNI</strong> where we teardown veth pairs, `/32` routes, and how `ptp` links pods to the host.</p><p>➡️ Up next: <strong>Inter-Node Pod Traffic with Kindnet: The No-Overlay Walkthrough</strong> where we’ll trace packets as they leave one node and arrive on another using static `/24` routes.</p><p><strong>References</strong></p><ul><li><a href="https://github.com/kubernetes-sigs/kind/issues/2444">https://github.com/kubernetes-sigs/kind/issues/2444</a></li><li><a href="https://www.tkng.io/cni/kindnet/">https://www.tkng.io/cni/kindnet/</a></li><li><a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/">https://kubernetes.io/docs/concepts/cluster-administration/networking/</a></li><li><a href="https://github.com/containernetworking/cni/blob/main/SPEC.md">https://github.com/containernetworking/cni/blob/main/SPEC.md</a></li><li><a href="https://github.com/kubernetes-sigs/kind/tree/main/images/kindnetd">https://github.com/kubernetes-sigs/kind/tree/main/images/kindnetd</a></li><li><a href="https://github.com/kubernetes-sigs/kind/issues/2444">https://github.com/kubernetes-sigs/kind/issues/2444</a></li><li><a href="https://man7.org/linux/man-pages/man8/ip-route.8.html">https://man7.org/linux/man-pages/man8/ip-route.8.html</a></li><li><a href="https://man7.org/linux/man-pages/man8/ip-neigh.8.html">https://man7.org/linux/man-pages/man8/ip-neigh.8.html</a></li><li><a href="https://man7.org/linux/man-pages/man8/iptables.8.html">https://man7.org/linux/man-pages/man8/iptables.8.html</a></li><li><a href="https://wiki.wireshark.org/CaptureSetup/Linux">https://wiki.wireshark.org/CaptureSetup/Linux</a></li><li><a href="https://github.com/containernetworking/plugins/tree/main/plugins/main/ptp">https://github.com/containernetworking/plugins/tree/main/plugins/main/ptp</a></li><li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go">https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffbbc07612b7" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/inside-intra-node-pod-traffic-in-kubernetes-how-kindnet-with-ptp-moves-packets-ffbbc07612b7">Inside Intra-Node Pod Traffic in Kubernetes: How Kindnet with PTP Moves Packets</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside a Pod’s Birth: Veth Pairs, IPAM, and Routing with Kindnet CNI]]></title>
            <link>https://itnext.io/inside-a-pods-birth-veth-pairs-ipam-and-routing-with-kindnet-cni-d6394f3495c5?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6394f3495c5</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-networking]]></category>
            <category><![CDATA[kindnet]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[namespaces]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Thu, 15 May 2025 09:55:22 GMT</pubDate>
            <atom:updated>2026-01-02T13:16:59.952Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*00CnuFh1NIZOBQQWbwMKSQ.png" /></figure><h3><strong>Inside a Pod’s Birth in Kubernetes: Veth Pairs, IPAM, and Routing with Kindnet CNI</strong></h3><p>When a Kubernetes pod starts, it doesn’t just spin up a container. It gets a full, isolated network stack — complete with its own IP address, routing table, and virtual interface. But how exactly does this happen under the hood, especially when using the Kindnet CNI plugin?</p><p>This article breaks down the networking path a pod inherits at creation, using a Minikube cluster running Kubernetes with Kindnet. You’ll observe how the Kindnet CNI assigns IPs from the node’s PodCIDR, creates veth pairs linking the pod to the host network, and installs routing rules that define how the pod communicates within the cluster.</p><p><em>You can find the repo containing the manifests we will use in this article </em><a href="https://github.com/colossus06/kubernetes-networking-with-kindnet"><em>here</em></a><em>.</em></p><p><em>You can read this article free on my blog </em><a href="https://podlock.readthedocs.io/blogs/k8s/ipam/ipam.html"><em>here</em></a><em>.</em></p><p><strong>Kubernetes CNI Requirements Are Simpler Than You Think</strong></p><p>The Container Network Interface (CNI) standard is flexible enough to serve a range of environments — Kubernetes is just one of them. But Kubernetes only requires two core capabilities from a CNI plugin: At its core, Kubernetes just needs the CNI plugin to handle IP assignment and create a network interface. That’s it!</p><p>For IP Address Management, the CNI simply needs to allocate addresses from the node’s PodCIDR range. Think of this like assigning apartment numbers in a building — each pod gets its own unique address within the node’s designated block.</p><p>As for interface configuration, the CNI creates a virtual network interface inside the pod’s network namespace. This is installing the network “doorway” that connects the pod to the outside world.</p><p>To achieve this Kindnet uses well-known CNI reference plugins.</p><p><strong>Kindnet’s Approach</strong></p><p>Kubernetes itself doesn’t handle IP assignments or interface configurations directly. Instead, it delegates this task to a Container Network Interface (CNI) plugin. Kindnet uses a configuration directory to define how networking is set up. Let’s inspect this directory:</p><pre>minikube ssh<br>cd /etc/cni/net.d/ &amp;&amp; ls -la<br>-rw-r - r - 1 10-kindnet.conflist<br>-rw-r - r - 1 100-crio-bridge.conf.mk_disabled<br>-rw-r - r - 1 200-loopback.conf<br>-rw-r - r - 1 87-podman-bridge.conflist.mk_disabled<br>cat 10-kindnet.conflist<br>{<br>&quot;cniVersion&quot;: &quot;0.3.1&quot;,<br>&quot;name&quot;: &quot;kindnet&quot;,<br>&quot;plugins&quot;: [<br>{<br>&quot;type&quot;: &quot;ptp&quot;,<br>&quot;ipMasq&quot;: false,<br>&quot;ipam&quot;: {<br>&quot;type&quot;: &quot;host-local&quot;,<br>&quot;dataDir&quot;: &quot;/run/cni-ipam-state&quot;,<br>&quot;routes&quot;: [<br>{ &quot;dst&quot;: &quot;0.0.0.0/0&quot; }<br>],<br>&quot;ranges&quot;: [<br>[ { &quot;subnet&quot;: &quot;10.244.0.0/24&quot; } ]<br>]<br>}<br>,<br>&quot;mtu&quot;: 1500<br>},<br>{<br>&quot;type&quot;: &quot;portmap&quot;,<br>&quot;capabilities&quot;: {<br>&quot;portMappings&quot;: true<br>}<br>}<br>]<br>}</pre><p>The `ptp` plugin creates the veth pair connecting the pod to the host — think of this as a virtual network cable stretched between two points. One end lives in the pod’s network namespace while the other stays in the host network. The host-side veth gets assigned the IP `.1` in the pod’s subnet (used as gateway).</p><p>The `host-local` plugin handles IP allocation, drawing addresses from the node’s PodCIDR range. This ensures each pod gets its own unique address without stepping on any network toes.</p><p><strong>Tracking a Pod’s Network Lifecycle</strong></p><p>Let me walk you through what happens when you run a simple deployment. In our two node minikube cluster, we have three pods:</p><pre>minikube control-plane v1.32.0 192.168.49.2<br>minikube-m02 &lt;none&gt; v1.32.0 192.168.49.3<br>pod-to-pod-6d4875f88c-5qcw5 10.244.1.2 minikube-m02<br>pod-to-pod-6d4875f88c-99qwj 10.244.1.3 minikube-m02<br>pod-to-pod-6d4875f88c-mfm7t 10.244.0.3 minikube</pre><p>Two pods might be on the same physical machine — but can they interfere with each other? No. Because Kubernetes guarantees isolation using <strong>Linux network namespaces</strong>.</p><p><em>You can find the repo containing the manifests we will use in this article </em><a href="https://github.com/colossus06/kubernetes-networking-with-kindnet"><em>here</em></a><em>.</em></p><p><strong>Namespaces</strong></p><p>Each Kubernetes pod runs inside its own isolated Linux network namespace. This namespace has its own interfaces, routing table, and iptables rules. From the pod’s perspective, it’s the entire network stack — isolated from every other pod and the host.</p><p>Let’s get a shell inside the pod running on minikube node and list network interfaces:</p><pre>k exec -it pod-to-pod-6d4875f88c-mfm7t - bash<br>ls /sys/class/net<br>eth0 lo</pre><p>Within the pod’s isolated namespace, two interfaces are present by default: `lo`, the loopback used for intra-pod communication (e.g., `localhost` bindings), and `eth0`, which handles all ingress and egress traffic outside the pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uCq6CRdCcmsRh4Q4J1w7-Q.png" /></figure><p>But `eth0` isn’t backed by a physical interface. Instead, it’s one end of a virtual Ethernet device — <em>a veth pair</em>.</p><p>What is a veth pair?</p><p><strong>The Veth Pair: Your Pod’s Pipe to the Host</strong></p><p>When a pod is created, the container runtime invokes the CNI plugin with an `ADD` command. Kindnet responds by generating a <strong>veth pair</strong>— a linked virtual Ethernet device.</p><p>An interface index (or ifindex) is a unique integer assigned by the Linux kernel to every network interface on the system. When you create a veth pair, each side gets its own ifindex:</p><ul><li>One end of the pair becomes `eth0` <strong>inside</strong> the pod’s private network namespace.</li><li>The other end lives in the <strong>host</strong> namespace and is assigned the gateway IP `10.244.X.1/32`.</li></ul><p>Let’s get the interface index for `eth0` on the pod:</p><pre>cat /sys/class/net/eth0/ifindex<br>2</pre><p>We can identify the host veth paired with eth0 using ifindex correlation by running the following:</p><pre>ip -br a<br>lo UNKNOWN 127.0.0.1/8 ::1/128<br>eth0@if3 UP 10.244.0.2/24 fe80::683b:1eff:fe76:37ad/64</pre><p>`if3` means `eth0` is inside the pod, and it’s peer-linked to <strong>interface with index 3 on the host</strong>. So, that index (3) refers to the host-side interface’s ifindex!</p><p>This brings to me to a question then: can i match this ifX to Host vethXXXX?</p><p>To know it we rely on two facts:</p><ul><li>pod’s `eth0` is the inside end of a veth pair and it is linked to host interface index 3.</li><li>`vethX` is interface index 3 on the minikube node, and it’s linked to interface index 2, which is in the pod as we’ve seen above.</li></ul><p>If we list all host-side veth interfaces and interface index 3 on the Minikube node, we should see `ifindex=2`:</p><pre>m ssh<br>ip -br addr | grep veth<br>vethffa2ff74@if2 UP 10.244.0.1/32 fe80::e4e5:97ff:feea:4a03/64<br>veth4fc93ebb@if2 UP 10.244.0.1/32 fe80::6074:acff:fefa:853a/64<br>ip -o link | grep ^3:<br>3: vethffa2ff74@if2: UP link-netnsid 1</pre><p>The host-side veth that’s paired with the pod’s `eth0` is interface index 2 inside the pod!</p><p><strong>The host-side veth IP</strong></p><p>Notice those `10.244.0.1/32` addresses assigned to the veth interfaces? They’re the gateway IPs our pods are configured to use and it exists inside each pod’s routing table.</p><p>The `/32` subnet mask (meaning a single-host network) is Kindnet’s way of handling pod-to-host routing without complex bridge devices. Each veth pair becomes a simple point-to-point link, creating a direct pipeline for packets to flow between the pod and the host. Inside the pod, look at the routing table:</p><pre>kubectl exec -it pod-to-pod-6d4875f88c-mfm7t - sh<br>apt update &amp;&amp; apt install -y iproute2<br>ip route<br>default via 10.244.0.1 dev eth0<br>10.244.0.0/24 via 10.244.0.1 dev eth0 src 10.244.0.2<br>10.244.0.1 dev eth0 scope link src 10.244.0.2</pre><p>Let’s take a look at them one by one.</p><p>`default via 10.244.0.1 dev eth0` means if a packet is destined for any IP not explicitly matched by a more specific route, kernel sends it to 10.244.0.1 using the `eth0` interface. So if we curl `1.1.1.1` or hit `8.8.8.8` from the pod, this is the route that applies. The packet will go to 10.244.0.1.</p><p>`10.244.0.0/24 via 10.244.0.1 dev eth0 src 10.244.0.2` says to reach any IP in the range `10.244.0.0/24` (for example other pods on this node), send traffic via the gateway on `eth0`, using `10.244.0.2` as the source IP. Even though the destination is “local”, it still sends the packet to `10.244.0.1`.</p><p>Both of the other routes in the pod’s table — the `/24` and default — rely on forwarding to `10.244.0.1`. But for Linux to send anything to 10.244.0.1, it needs to understand how to reach it.</p><p>`10.244.0.1 dev eth0 scope link src 10.244.0.2` says to reach `10.244.0.1`, send traffic directly over `eth0` because it is on the same Ethernet segment as me.</p><p>Now jump to the minikube host:</p><pre>ip route | grep 10.244<br>10.244.0.2 dev vethffa2ff74 scope host # Pod mfm7t (local) → its veth interface<br>10.244.1.0/24 via 192.168.49.3 dev eth0 # Route to node minikube-m02 (Pod 5qcw5 and 99qwj)</pre><p>`10.244.0.2 dev vethffa2ff74 scope host` maps Pod mfm7t’s IP to its veth on the host. When traffic comes in from outside the pod, the kernel delivers it through this interface.</p><p>`10.244.1.0/24 via 192.168.49.3 dev eth0` tells the host to forward traffic for any pod on `minikube-m02` out via its `eth0`, using the internal IP `192.168.49.3` of `minikube-m02`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sDf1ZirqFw01wCO2iQEsjA.png" /></figure><p>To summarize, when Kubernetes schedules a pod, it hands off setting up the networking to the CNI. Kindnet takes over and wires the pod into the cluster network. This setup gives the pod a routable IP, a default gateway, and a clean L3 path out through the host. But what happens when it starts talking to another pod?</p><p>➡️ Up next: <strong>Inside Intra-Node Pod Traffic in Kubernetes: How Kindnet with PTP Moves Packets</strong> where we trace packets from one pod to another on the same node.</p><p><strong>References</strong></p><ul><li><a href="https://github.com/containernetworking/cni/blob/main/SPEC.md">CNI Specification v1.0.0 — Container Networking Interface</a></li><li><a href="https://github.com/kubernetes-sigs/kind/tree/main/images/kindnetd">Kindnetd Plugin Source Code</a></li><li><a href="https://kubernetes.networking.guide/cni/kindnet/">The Kubernetes Networking Guide — Kindnet</a></li><li><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/">Kubernetes CNI Plugin Architecture</a></li><li><a href="https://man7.org/linux/man-pages/man8/ip-route.8.html">iproute2 Linux Manual Pages</a></li><li><a href="https://man7.org/linux/man-pages/man7/namespaces.7.html">Understanding Linux Network Namespaces and veth Pairs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6394f3495c5" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/inside-a-pods-birth-veth-pairs-ipam-and-routing-with-kindnet-cni-d6394f3495c5">Inside a Pod’s Birth: Veth Pairs, IPAM, and Routing with Kindnet CNI</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Makes a Computer Computer]]></title>
            <link>https://itnext.io/what-makes-a-computer-computer-d6eb304c9eb0?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6eb304c9eb0</guid>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[computer-repair]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 20:15:45 GMT</pubDate>
            <atom:updated>2026-01-02T13:17:26.938Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a60Ww_SQ_RwTzg2hFy1vEg.png" /></figure><p><em>My system died mid-project. I opened the case and tried to learn what really makes a computer a computer.</em></p><p><em>My system died mid-project. I opened the case and tried to learn what really makes a computer a computer.</em></p><p>A few weeks ago, in the middle of a critical project, my workstation went silent.</p><p>Just a faint crackle when I hit the power button.</p><p>I swapped the cable, tried a different wall outlet but still dead.</p><p>I figured it might be RAM. I reseated the sticks, moved them to different DIMM slots.</p><p>Nothing changed.</p><p>In that moment, staring at a lifeless box, I realized something obvious and uncomfortable: I had no idea what had actually failed.</p><p>So I opened the case and started pulling parts.</p><blockquote>This walkthrough is what I learned along the way.</blockquote><p>And yes — I’ll tell you at the end exactly what went wrong and how I fixed it.</p><p>First, I had to stop treating it like a black box and learn what actually makes it tick.</p><h4>What Actually Defines a Computer</h4><p>What makes a computer different from every other electronic device?</p><p>It’s not the screen, not the keyboard, and not just the CPU.</p><p>What defines a computer is simple: it can run arbitrary programs. You don’t change its function by rewiring it — you change it by loading new instructions.</p><p>A microwave can’t do that. It runs one fixed program baked into hardware.</p><p>A computer reads instructions and acts on them, line by line. It’s not the hardware itself — it’s how that hardware responds to code.</p><p>That design unlocks everything.</p><p>At the foundation, a computer needs three things: a processor to execute instructions, memory to hold those instructions and data, and a way to talk to the outside world — input and output.</p><p>Let’s walk up to a real desktop machine. What’s the first thing you notice?</p><p>A case. Steel or plastic.</p><h4><strong>First Contact: The Case</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2GHvrpqofPQ5v587-Qh9FA.png" /></figure><p>That’s not just for looks — it’s a structural chassis. It holds everything in place, controls airflow, and grounds the system.</p><p>Let’s open it up, and see the real machine.</p><p>Before anything can run, it needs clean, reliable power.</p><p>That’s the job of the Power Supply Unit.</p><h4><strong>Power First: The PSU</strong></h4><p>It takes wall voltage and converts it into low-voltage DC power.</p><p>It powers the motherboard, disks, fans, and peripherals.</p><p>Everything inside connects to the motherboard. This is the electrical and logical backbone of the entire system.</p><p>Every signal, every voltage, every data line runs through it.</p><p>Now, look closely at the board.</p><h4><strong>The Backbone: The Motherboard</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKds4aHAe8VmnUBm3kNLPw.png" /></figure><p>The CPU socket holds the processor. RAM slots (DIMM slots) hold the memory modules, the chipset handles I/O coordination, PCI lanes, SATA, USB, and more.</p><p>PCIe slots connect graphics cards, NVMe SSDs, network cards. SATA headers connect hard drives and older SSDs.</p><p>Power connectors link the board and CPU to the PSU.</p><p>The motherboard defines what the system supports — CPU architecture, RAM capacity and speed, I/O bandwidth, and physical expandability. It also constrains cooling and power delivery, directly influencing thermal and performance limits.</p><p>Now look at the center of the motherboard — usually near the top, below the PSU and between the RAM slots and PCIe lanes.</p><p>Slotted into a square socket with a locking mechanism, covered by a heatsink and cooling fan, sits the CPU. It’s mounted directly onto the board to ensure tight signal timing and thermal contact.</p><h4><strong>Execution Core: The CPU</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*riKrQTzE1vq_0eLfoaR6FA.png" /></figure><p>This is the machine’s execution core. It doesn’t store data permanently. It doesn’t generate output. It reads instructions from memory, processes them, and moves on — billions of times per second.</p><p>At the core, you’ve got something called the ALU — Arithmetic Logic Unit.</p><p>It’s the calculator. Every time your program adds numbers, compares values, flips bits — that’s the ALU doing the work.</p><p>Next to that, you have registers.</p><p>Think of them as ultra-fast scratchpads.</p><p>When the CPU is doing math, it doesn’t go to RAM — it loads values into registers, does the operation, and then writes the result back to memory if needed. They’re tiny, but lightning fast.</p><p>Then there’s the control unit. This part is the coordinator.</p><p>It reads each instruction from memory, decodes what the instruction is telling the CPU to do, and then signals the rest of the chip: “<em>Fetch this value,</em>” “<em>Do this operation</em>,” “<em>Store the result here.</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ko3QFoA_tPfZkLB7FdUfrw.png" /></figure><p>But all this would grind to a halt if the CPU had to go to RAM every time it needed data.</p><p>So there’s cache: L1, L2, and sometimes L3. These are layers of high-speed memory built directly into the CPU.</p><p>When a program accesses data, the CPU first checks the cache.</p><p>If it’s there (a cache hit), it’s fast. If not, it has to go all the way to RAM (a cache miss), which takes longer.</p><p>Finally, everything runs on a clock.</p><p>Not a wall clock — a pulse, millions or billions of times per second.</p><p>Every operation inside the CPU is synchronized to this clock.</p><p>One tick might fetch an instruction. The next might decode it. Another executes it. The faster the clock, the more operations per second the CPU can push through.</p><p>Now here’s where things scale.</p><p>Most CPUs today aren’t just one execution unit. They’re multicore.</p><p>That means inside that single physical chip, there are multiple cores — two, four, eight, sometimes dozens.</p><p>Each one can run a different thread of execution. So one core might be running your browser, another running your container runtime, another serving API requests , all in parallel.</p><p>Each core has its own control unit, ALU, and set of registers.</p><p>Some cache layers are shared, others are private. But the idea is this: one chip, multiple brains, each running their own tasks at the same time.</p><p>Now look just to the right of the CPU socket , those long vertical slots lined up side by side.</p><p>That’s where the RAM modules go.</p><h4><strong>Fast but Volatile: RAM</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YXwBlQZUE6ARcb1pPqVdEQ.png" /></figure><p>You push them in until the clips click into place.</p><p>These sticks hold your system’s short-term memory.</p><p>The OS, running apps, the data they’re working on live here.</p><p>When you launch a program, it doesn’t run from disk, the system copies it into RAM.</p><p>When the CPU fetches an instruction, it’s pulling it from RAM.</p><p>But there’s a catch — RAM is volatile.</p><p>If you power it off, and everything inside it disappears.</p><p>RAM is organized as a grid of addressable bytes where each address points to exactly one byte.</p><p>The CPU accesses these bytes using its address bus. Modern CPUs can address terabytes of RAM , if the motherboard supports it and the tech behind this is usually DRAM, dynamic RAM.</p><p>It stores bits as electrical charges in capacitors, which leak constantly. That’s why DRAM needs to refresh every few milliseconds to keep your data alive.</p><p>The performance of RAM comes down to latency and bandwidth.</p><p>Latency is how fast it can respond to a request, bandwidth is how much data it can push per second.</p><p>Both are influenced by the memory’s clock speed, the number of channels (single, dual, quad), and how the controller is configured in the chipset.</p><p>But as fast as RAM is it forgets, you need a place to store things long term.</p><h4><strong>Long-Term Memory: Storage</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asR9dJvvF0Ji-CD0pOIUuA.png" /></figure><p>Now look lower on the case, or sometimes tucked behind the motherboard tray in modern builds. You’ll find storage devices: hard drives and SSDs.</p><p>Let’s start with HDDs. You’ll know them by their size and weight.</p><p>Inside them are spinning magnetic platters, a moving read/write head.</p><p>They’re cheap and high-capacity, but slow and you can hear them working.</p><p>Then there are SSDs — solid-state drives.</p><p>There are no moving parts, just NAND flash memory. They’re quiet, shockproof, and orders of magnitude faster than HDDs for random access.</p><p>And above that, there’s NVMe.</p><p>These drives don’t even bother using SATA; instead they plug directly into PCIe lanes — either via M.2 slots on the board or U.2 cables.</p><p>NVMe SSDs talk directly to the CPU with high-speed queues, low latency, and massive throughput.</p><p>The kernel, the bootloader?</p><p>They’re sitting on that disk.</p><p>Now glance just beneath the storage cage or beside the front fans, depending on the case layout — you’ll see the cooling setup.</p><h4><strong>Keep It Cool: Thermal Management</strong></h4><p>Look at the heatsink over the CPU. A large metal block, sometimes finned, always paired with a fan. Its job is to pull heat away from the processor and push it into the surrounding air.</p><p>That fan blows it out toward the exhaust or up into the top of the case.</p><p>But that’s not the only hot spot. Your graphics card has its own fans, maybe even a vapor chamber or blower design.</p><p>RAM generates heat, the chipset gets warm.</p><p>Voltage regulators near the CPU socket?</p><p>Definitely need airflow.</p><p>If the system doesn’t move that heat out, it builds up — and once temperatures hit thermal limits, performance drops.</p><p>That’s throttling.</p><p>Your CPU slows down to protect itself.</p><p>Some high-end systems go further: custom liquid loops, all-in-one coolers with radiators and pumps, or heat pipes that transfer heat across the board to a larger fin stack.</p><p>Now look at the middle or bottom section of the motherboard — long rectangular slots perpendicular to the RAM. These are the PCIe slots.</p><p>Most systems have at least one x16 slot populated. That’s where the GPU lives — your graphics card.</p><h4><strong>Parallel Power: The GPU</strong></h4><p>Some machines don’t have a discrete GPU.</p><p>If the CPU has integrated graphics, it handles display output internally. But if you’re gaming, rendering, running simulations, training models — or just need raw parallel compute power — you need a dedicated GPU.</p><p>The card locks into the PCIe slot and often takes up two or even three expansion slots on the back of the case.</p><p>It connects directly to the motherboard lanes and draws additional power from the PSU via 6, 8, or 12-pin cables.</p><p>Inside, the GPU has its own processor cores, known as shaders.</p><p>Its own VRAM. Its own memory controller. Even its own scheduler. It offloads rendering, parallel math, and massive compute workloads from the CPU.</p><p>Now look around the edge of the motherboard and back panel. This is where you see the I/O.</p><h4><strong>Talking to the Outside: I/O Interfaces</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3vVHTRU2fM13bYGEO97ig.png" /></figure><p>USB ports — stacked in rows — for keyboards, mice, gamepads, external drives, and everything else. Ethernet ports for network connectivity — whether you’re SSH-ing into this box or letting it act as a node in a cluster.</p><p>HDMI, DisplayPort, DVI — these send display signals to your monitor, powered by either the GPU or integrated graphics.</p><p>Audio jacks for microphones and speakers, sometimes even Wi-Fi antennas or optical audio out, depending on the board.</p><p>These I/O ports are wired into the motherboard’s chipset, which acts as a bridge to the CPU.</p><p>They communicate through buses and memory-mapped registers.</p><p>When you plug something in, the OS doesn’t talk directly to the device.</p><p>It talks to the controller, which exposes the device as a resource — files, sockets, devices in `/dev`, whatever abstraction the OS uses.</p><p>Now step back and look at the system as a whole.</p><h4><strong>Boot and Bring-Up: Firmware and OS</strong></h4><p>When you hit the power button, everything starts from firmware.</p><p>The BIOS — or UEFI — lives on a chip soldered to the board. It’s the first code that runs.</p><p>It initializes the CPU, checks the RAM, sets up system buses, enables features like virtualization or hyperthreading, and begins the boot process.</p><p>Then it finds the bootloader on the storage drive. That bootloader loads the OS kernel into RAM.</p><p>From there, the kernel takes over: brings up drivers, mounts filesystems, starts services, initializes runtime environments, and begins executing userspace.</p><p>Now your system is live.</p><p>Let’s tie it all together.</p><h4><strong>Key Takeaways: What You’re Really Looking At</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KROaECcLeSw5gz86Z8nZyA.png" /></figure><p>The case holds everything in place, managing structure, airflow, and grounding. The power supply delivers clean, reliable voltage to every component.</p><p>The motherboard ties it all together — routing signals, distributing power, and defining the system’s capabilities.</p><p>The CPU executes instructions at high speed, pulling data and code from RAM, which acts as the system’s working memory.</p><p>Storage holds everything that matters ready to be loaded at boot.</p><p>If a discrete GPU is present, it takes on the heavy lifting for graphics and massive parallel workloads, freeing the CPU to do more.</p><p>Cooling keeps the entire system in check, dissipating heat so performance doesn’t throttle under load.</p><p>I/O ports give the machine its connection to the outside .</p><p>Firmware gets the system moving and the OS brings it to life.</p><p>And the workloads?</p><p>That’s the part only you control.</p><p><strong>So What Was the Problem?</strong></p><p>The power supply.</p><p>When I finally pulled it out and looked, it was obvious. The capacitors were swollen, tops puffed out, brown sludge leaking down the sides. One had blown entirely.</p><p>Dead.</p><p>I bought a brand-new PSU. Plugged it in, hit the power button —</p><p>Nothing.</p><p>Same silence. No spin. No light.</p><p>Still, I knew it had to be the PSU. So I borrowed one from a friend and plugged it in.</p><p>The system came alive.</p><p>Turns out the new part was dead on arrival!</p><p>Even hardware lies. Even fixes can fail.</p><p>But the real fix wasn’t the part.</p><p>It was learning how the system works, even if I’m not always right.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6eb304c9eb0" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/what-makes-a-computer-computer-d6eb304c9eb0">What Makes a Computer Computer</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Atomic ConfigMap Updates in Kubernetes: How Symlinks and Kubelet Make It Happen]]></title>
            <link>https://itnext.io/atomic-configmap-updates-in-kubernetes-how-symlinks-and-kubelet-make-it-happen-21a44338c247?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/21a44338c247</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[configmap]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubelet]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 07:28:33 GMT</pubDate>
            <atom:updated>2026-01-02T13:18:10.049Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHMnCdMJkg8X9uRezPB75Q.png" /></figure><p>TL;DR <em>If you mount a ConfigMap as a volume, Kubernetes automatically updates the file system when the ConfigMap changes. However, if you’re using environment variables or subPath mounts, updates aren’t reflected unless the pod is restarted.</em></p><p>When working with Kubernetes and ConfigMaps, one thing becomes immediately apparent: ConfigMaps are a flexible way to inject configurations into your application without rebuilding your container images. The key takeaway here is that the behavior of ConfigMaps, especially in terms of updates, depends entirely on how they are mounted and how Kubernetes manages those updates. This becomes critical when you need real-time updates for applications running in a Kubernetes cluster.</p><p>ConfigMaps aren’t encrypted and are not designed to store sensitive information. They are purely a way to inject configuration data into your application, keeping it separate from the container image.</p><p>So, how do you consume these ConfigMaps in your applications? Kubernetes provides a few options for using ConfigMaps, which include:</p><ul><li>Injecting them as environment variables.</li><li>Passing them as command-line arguments.</li><li>Mounting them as files in a volume within the container’s filesystem.</li></ul><h4><strong>Consuming ConfigMaps in Pods</strong></h4><p>When a pod is created that consumes a ConfigMap (either as environment variables or mounted volumes), the pod’s specification includes references to the ConfigMap. You can see these references in the pod’s YAML definition under `spec.volumes[]` for volumes and `spec.containers[].env[]` for environment variables.</p><p>Upon scheduling the pod to a node, the kubelet on that node retrieves the ConfigMap data from the API server to populate the environment variables or mount the ConfigMap as a volume.</p><p>If the ConfigMap is mounted as a volume, the kubelet sets up the volume by creating a directory structure in the pod’s file system where the ConfigMap data will reside.</p><p>The kubelet then establishes a watch connection with the Kubernetes API server for each ConfigMap or Secret used by the pods on its node. When the ConfigMap or Secret is updated, the API server notifies the kubelet through this connection.</p><h4><strong>ConfigMap as Environment Variables</strong></h4><p>When you mount a ConfigMap as environment variables, the pod starts with the values present in the ConfigMap at the time of pod creation. However, updates to the ConfigMap do not propagate to the pod’s environment variables unless the pod is restarted.</p><p>You can create a configmap with the following command:</p><pre>kubectl create configmap demo-config - from-literal=example.key=&quot;Initial Value&quot;</pre><p>Then, mount it in an nginx pod:</p><pre>env:<br>- name: EXAMPLE_KEY<br>  valueFrom:<br>    configMapKeyRef:<br>      name: demo-config<br>      key: example.key</pre><p>If you want to verify the environment variable injection:</p><pre>kubectl exec &lt;one-of-the-pods&gt; -- env | grep EXAMPLE_KEY</pre><p>You’ll see:</p><pre>EXAMPLE_KEY=Initial Value</pre><p>Now, let’s update the ConfigMap:</p><pre>kubectl patch configmap demo-config -p &#39;{&quot;data&quot;:{&quot;example.key&quot;:&quot;First Update:env variables&quot;}}&#39;</pre><p>If you recheck the environment variable, it still shows the `Initial Value`. This is because the pod doesn’t dynamically pick up changes to the ConfigMap. Environment variables are fixed for the pod’s lifetime: to update the environment variables, you need to restart the pod.</p><p>If you’re using a Deployment, you can delete the pods manually, and the ReplicaSet controller will automatically recreate them:</p><pre>33s Normal SuccessfulDelete ReplicaSet/nginx-5cd5c97db5 Deleted pod: nginx-5cd5c97db5-kkvgn # initial pod 1<br>33s Normal SuccessfulCreate ReplicaSet/nginx-bd485bdfc Created pod: nginx-bd485bdfc-lzfsg<br>33s Normal Killing Pod/nginx-5cd5c97db5-kkvgn Stopping container nginx<br>31s Normal Pulled Pod/nginx-bd485bdfc-lzfsg Successfully pulled image &quot;bitnami/nginx&quot; in 1.773s (1.773s including waiting). Image size: 184551081 bytes.<br>31s Normal Created Pod/nginx-bd485bdfc-lzfsg Created container nginx<br>31s Normal Started Pod/nginx-bd485bdfc-lzfsg Started container nginx<br>30s Normal SuccessfulDelete ReplicaSet/nginx-5cd5c97db5 Deleted pod: nginx-5cd5c97db5-4fw9t # initial pod 2<br>...</pre><p>Alternatively, you can perform a rolling restart of the Deployment using the following command and check the configmap value:</p><pre>kubectl rollout restart deployment nginx<br>#kubectl exec nginx-8dff55589-hqnb4 - env | grep EXAMPLE_KEY<br>EXAMPLE_KEY=First Update:env variables</pre><h4><strong>Mounting a ConfigMap as a Regular Volume</strong></h4><p>Unlike environment variables or `subPath` mounts, which don’t reflect updates without restarting the pod, mounting a `ConfigMap` as a volume in Kubernetes allows for dynamic updates. This means any changes to the `ConfigMap` are automatically applied to the pod without needing restarts or manual intervention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOPrk-f0ip8djzoFXn-ETw.png" /></figure><p>If you mount the `config-volume` ConfigMap to the `/etc/config` directory in your pod, your deployment YAML might look like this:</p><pre>...<br>  volumeMounts:<br>  - name: config-volume<br>    mountPath: /etc/config<br>volumes:<br>- name: config-volume<br>  configMap:<br>  name: demo-config<br>...</pre><p>If you add this `ConfigMap` volume mount request to the deployment manifest, the kubelet performs several actions. We’ll take a quick look at them in the following three sections:</p><p><strong>Fetching the ConfigMap data from the Kubernetes API</strong></p><p>kubelet constantly monitors specific paths and resources it’s interested in. Watches typically have a timeout after which they close automatically, and a new watch is established.</p><p>You can log these watch close notifications if you start a minikube cluster with higher verbosity and ssh into it:</p><pre>journalctl -u kubelet | grep &#39;Watch&#39;<br><br>#example output<br>Oct 02 12:17:21 minikube kubelet[2451]: I1002 12:17:21.444106 2451 apiserver.go:52] &quot;Watching apiserver&quot;<br>Oct 02 12:23:23 minikube kubelet[2451]: I1002 12:23:23.536954 2451 reflector.go:808] object-&quot;kube-system&quot;/&quot;kube-root-ca.crt&quot;: Watch close - *v1.ConfigMap total 8 items received<br>Oct 02 12:23:25 minikube kubelet[2451]: I1002 12:23:25.463764 2451 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.CSIDriver total 8 items received<br>Oct 02 12:24:19 minikube kubelet[2451]: I1002 12:24:19.474887 2451 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.RuntimeClass total 8 items received<br>Oct 02 12:24:32 minikube kubelet[2451]: I1002 12:24:32.208463 2451 reflector.go:808] object-&quot;kube-system&quot;/&quot;kube-proxy&quot;: Watch close - *v1.ConfigMap total 7 items received<br>Oct 02 12:25:09 minikube kubelet[2451]: I1002 12:25:09.357705 2451 reflector.go:808] object-&quot;kube-system&quot;/&quot;coredns&quot;: Watch close - *v1.ConfigMap total 8 items received</pre><p>When a pod mounts a ConfigMap as a volume, the kubelet first fetches the ConfigMap data from the Kubernetes API.</p><p>If you ssh into a minikube node and check kubelet logs:</p><pre>journalctl -u kubelet | grep -i &quot;Received&quot;<br>#example output<br>Oct 02 12:53:30 minikube kubelet[2451]: I1002 12:53:30.557615 2451 configmap.go:211] Received configMap default/demo-config containing (1) pieces of data, 26 total bytes<br>...</pre><p>You can manually verify the byte size. First, view the ConfigMap content with:</p><pre>kubectl get configmap demo-config -o yaml<br>#example output<br>data:<br>example.key: First Update:env variables<br>...</pre><p>Each character (including spaces) is typically 1 byte in most text encodings like UTF-8. It’s obvious already but let’s calculate it:</p><pre>echo -n &quot;First Update:env variables&quot; | wc -c<br>26</pre><p>You’ll see the string’s length in bytes matches the total number of bytes in the kubelet logs.</p><p><strong>Translating the data and storing in a node local directory</strong></p><p>After retrieving the ConfigMap, kubelet translates the key-value pairs from the ConfigMap into actual files that it stores in a local directory. Each key in the ConfigMap becomes a file, and the corresponding value becomes the file’s contents.</p><p>If you pick either of the pod id:</p><pre>kubectl get pod &lt;pod-name&gt; -n &lt;namespace&gt; -o jsonpath=&#39;{.metadata.uid}&#39;<br># example output<br>721a7048-bd0e-4a74-96e3-12f425f9d38e</pre><p>Then SSH into the node and check the contents of the pod’s volumes directory:</p><pre>minikube ssh<br>sudo ls /var/lib/kubelet/pods/721a7048-bd0e-4a74-96e3-12f425f9d38e/volumes/kubernetes.io~configmap/<br># example output<br>config-volume<br># ls config-volume<br>example.key</pre><p>If you check the current value from the mounted `ConfigMap`:</p><pre># k exec &lt;nginx-pod-name&gt; -- cat /etc/config/example.key<br>First Update:env variables</pre><p>You’ll get one file named `example.key` in the `/etc/config/` directory and its value will be the last patch we made: `First Update:env variables`.</p><h4><strong>Mounting the volume into the pod’s filesystem</strong></h4><p>Finally, kubelet mounts the volume into the pod’s filesystem at the path specified in the pod’s manifest.</p><p>If you inspect the pod’s filesystem:</p><pre>kubectl exec -it &lt;pod-name&gt; -- sh<br>ls -la /etc/config<br># example output<br>drwxr-xr-x 2 root root 4096 Sep 25 17:58 ..2024_09_25_17_58_43.3550472942<br>lrwxrwxrwx 1 root root 32 Sep 25 17:58 ..data -&gt; ..2024_09_25_17_58_43.3550472942<br>lrwxrwxrwx 1 root root 18 Sep 25 17:50 example.key -&gt; ..data/example.key</pre><p>You should see the files and symbolic links created from the ConfigMap data which means the volume has been mounted successfully.</p><h4><strong>Updating ConfigMaps and symlink switch</strong></h4><p>A symlink (symbolic link) is a file system pointer that references another file or directory. It doesn’t contain the actual data but points to where the data is located. The applications running in the pods accesses the ConfigMap through this symlink.</p><p>When kubelet mounts a ConfigMap as a volume in a pod, it sets up two levels of symlinks:</p><ul><li>The file you access in the pod (`/etc/config/example.key`) pointing to `..data/example.key`.</li><li>and the `..data` directory pointing to the latest timestamped directory that holds the actual ConfigMap data.</li></ul><p>When the ConfigMap is updated, kubelet doesn’t modify the file directly: creates a new directory with the updated files and changes the `..data` symlink to point to this directory.</p><p>If you patch the `ConfigMap` once again with the following command:</p><pre>kubectl patch configmap demo-config -p &#39;{&quot;data&quot;:{&quot;example.key&quot;:&quot;Now, Second update for symlinks&quot;}}&#39;</pre><p>If you recheck the directory structure:</p><pre>kubectl exec -it &lt;pod-name&gt; -- sh<br>ls -la /etc/config<br><br>drwxr-xr-x 2 root root 4096 Sep 25 18:09 ..2024_09_25_18_09_31.1466192921<br>lrwxrwxrwx 1 root root 32 Sep 25 18:09 ..data -&gt; ..2024_09_25_18_09_31.1466192921<br>lrwxrwxrwx 1 root root 18 Sep 25 17:50 example.key -&gt; ..data/example.key</pre><p>You’ll see the `..data` symlink now points to (`2024_09_25_18_09_31.1466192921`). The application continues to use the old version until kubelet switches the symlink.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e4rbwJLa9aDF2ji_w6Eeig.png" /></figure><p><em>You can find the SVG </em><a href="https://kuberada.devtechops.dev/_images/symlink.svg"><em>here</em></a><em>.</em></p><p>Symlink switch is an atomic operation managed by the kubelet as part of its volume reconciliation process.</p><p>If you recheck the kubelet logs and compare the bytes manually:</p><pre>journalctl -u kubelet<br><br>#example output<br>Oct 02 13:07:11 minikube kubelet[2451]: I1002 13:07:11.492226 2451 configmap.go:211] Received configMap default/demo-config containing (1) pieces of data, 26 total bytes<br>Oct 02 13:07:28 minikube kubelet[2451]: I1002 13:07:28.495492 2451 configmap.go:211] Received configMap default/demo-config containing (1) pieces of data, 31 total bytes<br><br>kubectl get configmap demo-config -o yaml<br>echo -n &quot;Now, Second update for symlinks&quot; | wc -c<br>#example output<br>31</pre><p>You verify this number matches with kubelet logs.</p><h4><strong>Kubelet Sync and Caching Behavior</strong></h4><p>The kubelet uses a sync loop to keep each node in your Kubernetes cluster in line with your desired state. By default, this loop runs once every minute. During each cycle, kubelet handles tasks like starting or stopping pods, mounting or unmounting volumes, updating symbolic links, and performing routine maintenance.</p><p>To manage ConfigMaps efficiently, the kubelet caches them in memory with a one-minute default Time-To-Live (TTL). If your application requests a ConfigMap within this minute, the kubelet serves it from the cache, which helps reduce unnecessary calls to the API server. Once the TTL expires, the kubelet fetches the latest version of the ConfigMap from the API server in the next sync cycle.</p><p>There are three strategies the kubelet can use to detect changes to ConfigMaps :</p><ul><li>Watch (Default): The kubelet watches the Kubernetes API server for any changes to ConfigMaps and Secrets.</li><li>Cache: Here, kubelet relies solely on its local cache, which refreshes based on the TTL.</li><li>Get: kubelet fetch the latest ConfigMap or Secret from the API server every time it runs the sync loop.</li></ul><p>The speed at which an updated ConfigMap appears in your pod depends on two main factors: Kubelet Sync Period (`syncFrequency`) and ConfigMap Cache TTL (`configMapAndSecretChangeDetectionStrategy`).</p><p>You can adjust the sync behavior to better suit your application’s needs tweaking the `syncFrequency`.</p><p>In the worst-case scenario, an updated ConfigMap might take up to two minutes to show up in your pod. This means that if you update a ConfigMap right after a sync cycle, your pod may not see the change until both the sync loop and cache TTL have passed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8l7nyAVSuWkj1zR8dvefEA.png" /></figure><p><em>You can find the SVG </em><a href="https://kuberada.devtechops.dev/_images/sync.svg"><em>here</em></a><em>.</em></p><p>While ConfigMaps can dynamically update mounted files, not all applications handle these updates smoothly. Some applications only load their configuration during startup, which means they won’t recognize changes unless you manually refresh them, such as by sending a SIGHUP signal. Others use file watchers to detect changes in real-time. This works well for applications designed to handle configuration changes on the fly. However, if the app isn’t built for this, you must manage manual reloads or rely on complete pod restarts.</p><p>Even applications capable of dynamic reloads face another challenge: asynchronous propagation of ConfigMap updates across pods. Not all pods may receive the update simultaneously. The default sync delay (up to two minutes) might be acceptable for non-critical systems. Still, such delays might cause cascading failures in production environments if a wrong configuration spreads unevenly.</p><p>Tools like Reloader can help manage this risk. For example, Reloader monitors ConfigMaps and triggers rolling restarts of pods whenever a configuration change is detected.</p><p>Alternatively, Kustomize configMapGenerator allows you to quickly revert to a previous ConfigMap version using an earlier configuration.</p><p>You can also trigger immediate refreshes by updating a pod’s annotations. This forces the pod to recognize that something has changed and prompts Kubernetes to refresh or restart the pod right away.</p><ul><li>Mounting Strategy Matters. If you mount a ConfigMap as a volume, Kubernetes automatically updates the file system when the ConfigMap changes. However, if you’re using environment variables or subPath mounts, updates aren’t reflected unless the pod is restarted.</li><li>When updating ConfigMaps mounted as volumes, you always have a reliable view of configuration data before or after an update.</li><li>To project the new keys immediately, you can simply update one of the pod’s annotations.</li><li>With the Watch strategy set as default, Kubernetes strikes a good balance between timely updates and system performance.</li><li>Although ConfigMaps supports dynamic updates, you might also need native or third-party tools for more control and safety.</li></ul><h4><strong>Final Words</strong></h4><p>If you still need to figure out why we’re using config maps, see the twelve-factor app in the references section.</p><p>Choosing the right way to mount ConfigMaps depends on your configuration needs.</p><p>If you need environment variables or static files that won’t change during the pod’s lifecycle, mounting as an environment variable or using `subPath` might work fine. However, if you need dynamic configuration updates, mounting the ConfigMap as a volume helps your application react to changes in real-time without pod restarts.</p><h4><strong>References</strong></h4><ul><li><a href="https://kubernetes.io/docs/concepts/configuration/configmap/">ConfigMaps</a></li><li><a href="https://minikube.sigs.k8s.io/docs/">Minikube</a></li><li><a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/">Kubelet Configuration</a></li><li><a href="https://12factor.net/config">Twelve-Factor App Methodology</a></li><li><a href="https://github.com/stakater/Reloader">Reloader</a></li><li><a href="https://kustomize.io/">Kustomize</a></li><li><a href="https://external-secrets.io/">External Secrets Operator</a></li><li><a href="https://helm.sh/docs/">Helm</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21a44338c247" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/atomic-configmap-updates-in-kubernetes-how-symlinks-and-kubelet-make-it-happen-21a44338c247">Atomic ConfigMap Updates in Kubernetes: How Symlinks and Kubelet Make It Happen</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes: The Art of Zero-Downtime Deployments]]></title>
            <link>https://itnext.io/kubernetes-the-art-of-zero-downtime-deployments-fa92c8ec5646?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa92c8ec5646</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Mon, 05 Aug 2024 23:14:41 GMT</pubDate>
            <atom:updated>2025-07-02T03:00:44.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omalAbMuPpHIhuvDIAX19g.png" /></figure><p>Downtime is a nightmare for businesses, costing money and eroding customer trust. With its powerful container orchestration capabilities, Kubernetes offers a solution: seamless deployments that keep your application running smoothly even during updates. However, achieving this zero-downtime magic requires a strategic approach and an understanding of advanced techniques.</p><p>This blog post will guide you through the techniques you need to achieve seamless deployments in Kubernetes. We’ll also put these strategies into practice using a knowledge-powered customer support chatbot.</p><p>We’ll explore various deployment strategies, such as Blue/Green and Canary, sharing real-world examples and key metrics to monitor along the way.</p><p>When you finish reading, you’ll have the confidence and knowledge to upgrade your applications seamlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*ky5ahswXHtdrSdEOQcw8XA.png" /></figure><p><em>You can find this mindmap in SVG </em><a href="https://kuberada.devtechops.dev/_images/deploy_diagram_1.svg"><em>here</em></a><em>.</em></p><p><strong>Kubernetes Built-in Deployment Strategies: Smooth Sailing with Rolling Updates</strong></p><p>Kubernetes Deployments offer a powerful, declarative way to manage your applications. Simply specify how many instances (replicas) of your app should be running and the container image to use. Kubernetes handles the rest, automatically creating and managing a ReplicaSet to keep things running smoothly even if nodes fail or other disruptions occur.</p><p><strong>Matching Labels: The Key to Pod Management</strong></p><p>A key aspect of Deployments is pinpointing which Pods to manage. This is achieved using labels.</p><p>The `spec.selector.matchLabels` field in your Deployment acts like a filter, specifying which labels Kubernetes should look for to identify the Pods it’s responsible for. These labels must match the ones you’ve attached to your Pods in the `spec.template.metadata.labels` section.</p><p>It’s like tagging your Pods and then telling your Deployment to only manage Pods with specific tags. It’s critical that the labels in `spec.selector.matchLabels` and `spec.template.metadata` match exactly.</p><p><strong>Rolling Deployments</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*U3Vj_0-aw5x-CR-gO-5qOw.png" /></figure><p>Kubernetes comes with two built-in deployment strategies; the default is `RollingUpdate .`This strategy is ideal for updating your application with minimal user disruption. It gradually replaces old Pods with new ones, ensuring your application remains available throughout the update process.</p><p>Imagine you’re running a customer support chatbot named “Kuberada Chatbot” that utilizes a knowledge base. You want to upgrade this chatbot to version 2.0, which includes essential feature enhancements and bug fixes, without causing any downtime for your users. Let’s see how a Rolling Deployment makes this possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQFXC7-gJuw8fP8-VPa21A.png" /></figure><p>Our current YAML file defines a Deployment named `chatbot` that manages two replicas of the `kuberada-chatbot` container, currently running version 1.0 of the image:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: chatbot<br>spec:<br>  replicas: 2 <br>  selector:<br>    matchLabels:<br>      app: chatbot<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>    spec:<br>      containers:<br>      - name: kuberada-chatbot<br>        image: kuberada-chatbot:v1.0<br>        ports:<br>      - containerPort: 5000</pre><p>The Deployment controller automatically adds a `pod-template-hash` label to each ReplicaSet it creates. This ensures that child ReplicaSets don’t overlap, as seen in the following output:</p><pre>kubectl get pods --show-labels<br>NAME                                   READY   STATUS    RESTARTS   AGE     LABELS<br>chatbot-748fd587d8-hrjmp   1/1     Running   0          2m23s   app=chatbot,pod-template-hash=748fd587d8<br>chatbot-748fd587d8-xdmpt   1/1     Running   0          4m26s   app=chatbot,pod-template-hash=748fd587d8</pre><p><strong>Maintaining service availability while upgrading to version 2.0</strong></p><p>Now, let’s upgrade our chatbot to version 2.0. We want to do this with minimal disruption, ensuring that at least one instance of the chatbot is always available to handle user requests. We’ll achieve this by fine-tuning the rolling update strategy and `maxUnavailable` and `maxSurge` parameters.</p><p>The impact of these parameters becomes more significant in our small-scale deployment. Since we have two replicas and decided to allow one Pod to be unavailable during the update, half of our capacity could be down anytime.</p><p>Similarly, setting `maxSurge: 1` permits one additional Pod to be created above the desired replica count, temporarily increasing the total number of Pods to three during the update.</p><pre>...<br>  strategy:<br>    type: RollingUpdate # Explicitly set to RollingUpdate (default)<br>    rollingUpdate:<br>      maxUnavailable: 1<br>      maxSurge: 1        # Allow one extra pod during the update<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>    spec:<br>      containers:<br> - name: kuberada-chatbot<br>        image: kuberada-chatbot:2.0 # Updated image tag<br>...</pre><p>Kubernetes then orchestrates the rolling update. You can monitor its progress with `kubectl rollout status deploy chatbot`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CO4ecPdLnaTNxEOrzzZzA.png" /></figure><p>This command provides real-time updates, showing how many pods are updating, how many are available, and if any errors have occurred.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcYC1ZrCFT2WZd0YdBbKDQ.png" /></figure><p><strong>Handling Rollout Issues</strong></p><p>But what if the rollout gets stuck? What if the new pod fails to start due to issues like image pull errors, application misconfigurations, resource constraints, or bugs in the latest version?</p><p>If a rollout encounters problems, the Deployment controller automatically intervenes and halts it, preventing the new ReplicaSet from scaling up further. This safeguard prevents a faulty deployment from taking down your entire service. The existing pods continue to handle the load, ensuring continuous service availability.</p><p>If you enable rollback mechanisms (by setting `revisionHistoryLimit` to a non-zero value), Kubernetes can automatically revert to the previous stable state if the deployment fails.</p><p><strong>Rollback to Stability and revisionHistoryLimit</strong></p><p>Kubernetes maintains a history of your changes to a Deployment, recording each change as a separate revision through `spec.revisionHistoryLimit`. This parameter provides a valuable audit trail and allows for easy rollbacks to previous versions if necessary.</p><p>Let’s say we want to roll back to the last stable version. We can use the following command:</p><pre>k rollout undo deploy chatbot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LaARyOF_MCLS4HC4eDvNRA.png" /></figure><p>This ensures that at least one pod (the old version) remains operational, providing uninterrupted service while we address the issues in the new version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cG4VJL682vWb1X7G7YDTpg.png" /></figure><p>By setting `maxUnavailable` to 1 and having two replicas, we balance high availability and efficient updates. Even in the face of potential deployment issues, Kubernetes’ built-in safeguards and rollback mechanisms ensure minimal disruption to your service.</p><p>We can further enhance our deployment by configuring a readiness probe for our chatbot service container. This allows Kubernetes to accurately assess the health of each pod and ensure that client traffic is directed only to those that are fully operational and ready to handle requests.</p><p><strong>Fixed Deployments (Recreate)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*wlySV4_wIssvQIef2HIb1w.png" /></figure><p>Let’s consider a scenario in which our data engineers plan to update our AI model, incorporating more extensive data sources to enhance response quality and accuracy. Before deploying these changes to production, we must thoroughly test them in a development environment.</p><p>Due to the significant nature of these updates, we’ll utilize the fixed deployment (Recreate) strategy in the dev environment to ensure consistency throughout the transition. This strategy will terminate all existing Pods before creating new ones with the updated configuration.</p><p>While this approach is more straightforward, it results in downtime. However, it’s well-suited for non-production environments or scenarios where a brief service interruption is acceptable.</p><p><strong>Deployment Configuration</strong></p><p>Here’s a Deployment configuration example using the Recreate strategy:</p><pre>spec:<br>  replicas: 2<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: chatbot-dev</pre><p>This configuration ensures that all existing pods are terminated before new ones, incorporating the latest updates, are created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*XpCczFI9FeEWl8VnrKAhUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*OeIHZORwo_FsaNMNdn2S2w.png" /></figure><p><strong>Production Rollout</strong></p><p>Once testing in the development environment is successful, we can schedule the production rollout during off-peak hours, such as late at night or early in the morning, to minimize any impact on users.</p><p><strong>Choosing the Right Strategy</strong></p><p>The ideal deployment strategy depends on your specific needs. Rolling deployments are generally preferred for production environments where minimizing downtime is critical. Fixed deployments, on the other hand, are more swift, effective, and suitable for development or testing environments or when brief downtime is acceptable.</p><p><strong>Limitations of Basic Deployments and the Need for Advanced Strategies</strong></p><p>While rolling updates in Kubernetes offer some control, they don’t inherently guarantee zero downtime. Even the briefest service interruption can be detrimental for applications with strict uptime requirements or intricate dependencies. We’ve seen that Kubernetes provides some control through `maxSurge` and `maxUnavailable` settings, but these alone might not eliminate downtime.</p><p>Furthermore, with proper configuration of session affinity (ensuring a user’s requests are directed to the same pod throughout their session), users might experience smooth transitions between the old and new versions of the application during an update. This can lead to a poor user experience or errors, primarily if the two versions handle data or behave differently.</p><p>We must go beyond Kubernetes’ built-in capabilities and implement specific techniques to achieve near-zero downtime deployments. Let’s explore one such technique: graceful shutdown and readiness probes.</p><p><strong>Implementing Graceful Shutdown</strong></p><p>A graceful shutdown lets your application finish processing ongoing requests before the pod terminates. This is essential for preventing “connection reset by peer” errors, which can occur during a rolling update when the load balancer sends requests to shutting down pods.</p><p>During rolling updates, Kubernetes replaces old pods with new ones. By ensuring the old pods gracefully shut down, we reduce the chances of dropping or mishandling requests mid-transition.</p><p>When a pod receives a termination signal, Kubernetes removes it from the service’s list of endpoints. The `preStop` hook gives Kubernetes time to update its internal state and prevent new connections from being routed to the terminating pod. A brief sleep in the `preStop` hook further ensures that in-flight requests can be completed and no new requests are sent to the pod, further minimizing the risk of connection reset errors.</p><p>We have integrated the shutdown process directly into our application code for a clean and efficient solution.</p><pre>def handle_shutdown_signal(signum, frame):<br>    print(&quot;Graceful shutdown initiated.&quot;)<br> time.sleep(10) #or <br> sys.exit(0)<br><br>signal.signal(signal.SIGTERM, handle_shutdown_signal)<br>signal.signal(signal.SIGINT, handle_shutdown_signal)</pre><p>However, if you prefer to let Kubernetes handle the shutdown delay, you can use a `preStop` hook to introduce a brief pause (e.g., 10 seconds) before the container terminates:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 10&quot;]</pre><p>Remember, even with services like NodePort, a short `preStop` delay is beneficial because it allows Kubernetes to update networking rules and stop sending traffic to the pod before it’s terminated.</p><p>Here is a go-to article if you want to dive deep into <a href="https://learnk8s.io/graceful-shutdown">graceful shutdowns for long-running tasks</a>.</p><p><strong>Readiness Probes: Ensuring Pod Readiness</strong></p><p>Let’s configure the readiness probe to ensure that Kubernetes only directs traffic to fully initialized pods and is ready to handle requests. This proactive measure further enhances the stability and reliability of our deployments.</p><pre>#spec.template.spec.containers<br>readinessProbe:<br>  httpGet:<br>    path: /health<br>    port: 5000<br>  initialDelaySeconds: 250  # Wait 2 minutes before starting readiness checks<br>  periodSeconds: 20  # Check every 20 seconds<br>  timeoutSeconds: 10  # Allow up to 10 seconds for the response<br>  failureThreshold: 3  # Mark the pod as unready after 3 consecutive failures</pre><pre>#app.py<br>...<br>@app.route(&#39;/health&#39;)<br>def health_check():<br>    # Basic check to see if the application has loaded the necessary components<br>    if embeddings is not None and model is not None:<br>        return &quot;Healthy&quot;, 200<br>    else:<br>        return &quot;Unhealthy&quot;, 500<br>...</pre><p><strong>What We Have Achieved with readinessProbe</strong></p><p>With the addition of the readinessProbe and the corresponding `/health` endpoint in our Python code, we’ve now established a mechanism for Kubernetes to intelligently determine when a pod is genuinely ready to receive traffic.</p><p>Now, Kubernetes will only send requests to pods that have successfully passed their readiness checks, ensuring that users are never directed to a pod that is still in the process of starting up or experiencing issues.</p><p>The readinessProbe will act as an additional safeguard during rolling updates. Kubernetes will terminate an old pod when a new pod with the updated version has passed its readiness checks and is fully prepared to handle traffic.</p><p><strong>Preventing incomplete initialization: initialDelaySeconds</strong></p><p>The `initialDelaySeconds` parameter, set to approximately 4 minutes, is crucial for our knowledge-based chatbot because it requires a significant amount of time to load its underlying knowledge base and language models into memory upon startup. Setting a longer initial delay allows the chatbot to complete this initialization process before Kubernetes checks its readiness and potentially sends traffic.</p><p><strong>The Need for Advanced Strategies</strong></p><p>While Kubernetes’ built-in deployment strategies offer a good starting point, they may only sometimes be sufficient for applications where even the slightest downtime can significantly impact them. For many organizations, the cost of downtime is too high. Lost revenue, damaged reputation, and potential compliance violations can all result from even brief service interruptions. This is where advanced deployment strategies, such as Blue/Green and Canary deployments, come in. These strategies offer more fine-grained control and flexibility, ensuring your applications remain consistently available and reliable, even during updates.</p><p><strong>Deep Dive into Zero-Downtime Deployment Strategies</strong></p><p><strong>Blue/Green Deployments ( A/B deployments)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*U3Vj_0-aw5x-CR-gO-5qOw.png" /></figure><p>Blue/Green deployments offer a seamless way to update your application without any downtime. Think of it as having two identical production environments, one labeled “Blue” (your current live version) and the other “Green” (where you deploy the new version). Once you’ve thoroughly tested and validated the latest version in the Green environment, you switch all the traffic from Blue to Green, making Green the new live production environment.</p><p>Let’s walk through how to set up a Blue/Green deployment for our chatbot application in Kubernetes.</p><p>Create two deployment files for the current environment (blue) and green.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: chatbot-blue<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: chatbot<br>      version: blue<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>        version: blue<br>    spec:<br>      containers:<br>       - name: chatbot<br>              image: kuberada-chatbot:blue<br>              ports:<br>       - containerPort: 5000</pre><p>The Green environment will host the new version of the chatbot.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: chatbot-green<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: chatbot<br>      version: green<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>        version: green<br>    spec:<br>      containers:<br>      - name: chatbot<br>              image: kuberada-chatbot:green<br>              ports:<br>      - containerPort: 5000</pre><p>We have two deployments running and one Service (blue deployment svc), which routes traffic to the Blue environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9mUWQqn7qXhpjkpL6PMhA.png" /></figure><pre>gulcan@topcu:~$ k get po,rs,deploy,svc<br>NAME                                 READY   STATUS    RESTARTS   AGE<br>pod/chatbot-blue-6bd7dfb695-4zjv2    1/1     Running   0          13m<br>pod/chatbot-blue-6bd7dfb695-hcf96    1/1     Running   0          13m<br>pod/chatbot-green-797c56675c-dqdgj   1/1     Running   0          4m45s<br>pod/chatbot-green-797c56675c-pc9wx   1/1     Running   0          4m45s<br><br>NAME                                       DESIRED   CURRENT   READY   AGE<br>replicaset.apps/chatbot-blue-6bd7dfb695    2         2         2       13m<br>replicaset.apps/chatbot-green-797c56675c   2         2         2       4m45s<br><br>NAME                            READY   UP-TO-DATE   AVAILABLE   AGE<br>deployment.apps/chatbot-blue    2/2     2            2           13m<br>deployment.apps/chatbot-green   2/2     2            2           4m45s<br><br>NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE<br>service/chatbot      NodePort    10.96.126.2   &lt;none&gt;        80:30007/TCP   13m</pre><p><strong>Switching to the Green</strong></p><p>After validating the blue environment (version blue of the chatbot), update the Service to point to the Green environment, label of the green deployment:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: chatbot-service<br>spec:<br>  selector:<br>    app: chatbot<br>    version: green<br>...<br><br>...<br>#k describe svc chatbot<br>gulcan@topcu:~$ k describe svc chatbot <br>Name: chatbot<br>Namespace: default<br>Labels: &lt;none&gt;<br>Annotations: &lt;none&gt;<br>Selector: app=chatbot,version=green<br>Type: NodePort<br>...</pre><p>After successfully switching traffic to the Green environment, you may decommission the Blue environment to free up resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zslatn2oVS65zNOKtfNsg.png" /></figure><p><strong>Key Takeaways</strong></p><p>Blue/Green deployments provide a reliable strategy for zero-downtime implementations, particularly for large-scale applications where downtime is not an option. The ability to quickly switch back to the previous version (Blue) if any issues arise makes it a highly resilient approach by simply updating the service routing to point back to the Blue environment.</p><p>This deployment approach is especially valuable for mission-critical applications like our chatbot and industries such as finance, healthcare, and e-commerce, where maintaining uninterrupted service is paramount.</p><p>While Blue/Green deployments are powerful, they come with some potential challenges:</p><ul><li>When you have database changes, it’s important to handle them carefully to avoid disrupting your live production environment. Techniques like dual writes or schema versioning can help you achieve this. Tools like Flyway or Liquibase (Java-based) can manage database migrations effectively in such scenarios.</li><li>You’ll want to make sure user sessions are maintained smoothly across both environments. This can be achieved by using external session stores like Redis (typically used with Python, Node.js, Java) to store session data centrally, allowing sessions to persist even when traffic is switched between environments.</li><li>Before switching traffic, thoroughly test your new version in the Green environment. Automated testing tools like Selenium (Java, Python, C#) and Cypress (JavaScript) for functional testing and Apache JMeter (Java) or k6 (JavaScript) for load testing are your allies in validating your new deployment. You can also use Locust to test Python-based applications.</li></ul><p><strong>Canary Deployments</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*f3UZvDBOgqZoHdCmuMmtJw.png" /></figure><p>Canary deployments involve gradually rolling out a new version of your application to a small subset of users while most continue using the stable version. This approach allows you to monitor the latest release in a real-world production environment and ensure it functions as expected before gradually increasing traffic.</p><p>Suppose our team has developed a new, improved AI model for our chatbot. To test its performance in real-world conditions before a full rollout, we’ll deploy it using a canary strategy, gradually exposing the new model to a subset of our users.</p><p>We’ll begin with deploying our stable version with three replicas.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: chatbot-stable<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: chatbot<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>    spec:<br>      containers:<br>      - name: kuberada-chatbot<br>        image: kuberada-chatbot:stable<br>...<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: chatbot<br>spec:<br>  selector:<br>    app: chatbot<br>  ports:<br> - protocol: TCP<br>      port: 5000<br>      targetPort: 5000</pre><p>Next, we’ll create a new deployment for the canary version, using a smaller number of replicas (e.g., 1) and the new image `kuberada-chatbot:canary`.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: chatbot-canary<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: chatbot<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>    spec:<br>      containers:<br>      - name: kuberada-chatbot<br>        image: kuberada-chatbot:canary</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*mQnKeHoMjLKftVQR65SEBA.png" /></figure><p>What about the traffic splitting?</p><p>Both deployments now share the same `app: chatbot` label, allowing the service to target pods from both versions. Kubernetes will distribute incoming traffic to the `chatbot` service between the pods in both deployments proportionally to their replica counts.</p><p><strong>Testing the traffic Splitting</strong></p><p>We can test which Pods are receiving traffic using `curl`:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZL3wM0d2DQyNaSk4pNvPQ.png" /></figure><p>With 3 replicas for the stable version and 1 for the canary, roughly 75% of traffic will go to the stable version and 25% to the canary.</p><p><strong>Monitoring and Gradual Rollout</strong></p><p>To ensure a smooth canary deployment, you’ll want to actively monitor your stable and canary deployments using tools like Prometheus and Grafana. This lets you make informed decisions about the rollout and quickly respond to potential issues.</p><p>We’ve configured our application to expose metrics such as request latency and request count, and we’ve annotated both versions of the chatbot deployment to tell Prometheus where to scrape these metrics from.</p><pre># Increase in HTTP 5xx errors:<br># This query calculates the rate of HTTP 5xx status errors over a 5-minute window.<br># It&#39;s useful for detecting spikes in server errors, indicating potential issues with your application or infrastructure.<br>rate(http_requests_total{status=~&quot;5..&quot;}[5m])<br><br># Overall response times:<br># This query calculates the 95th percentile of response times over a 5-minute window.<br># The &#39;histogram_quantile&#39; function is used to extract the 0.95 quantile from the histogram data, giving you insight into the response times experienced by the slowest 5% of users.<br>histogram_quantile(0.95, sum(rate(request_latency_seconds_bucket[5m])) by (le))<br><br># Chatbot-specific response time:<br># This query focuses on the response times for a specific application component (chatbot) running the &quot;canary&quot; version.<br># It calculates the 95th percentile of response times, helping to monitor the performance of this specific part of the application.<br>histogram_quantile(0.95, sum(rate(chatbot_response_time_seconds_bucket{app=&quot;chatbot&quot;, version=&quot;canary&quot;}[5m])) by (le))<br><br># System load:<br># This query calculates the rate of requests over a 5-minute window, providing an overview of system load.<br># Monitoring the request rate helps in understanding traffic patterns and scaling needs.<br>rate(request_count_total[5m])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcRyC-KNdolDVwKMYjqX6Q.png" /></figure><p>Grafana offers a powerful way to visualize these metrics collected by Prometheus. Consider setting up separate dashboards for the canary and stable versions to compare their performance easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AiRBJ-sZu5PMW5Le_c1Rkw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9ZNarHSvmMDqPL04jsApA.png" /></figure><p><strong>Making the Call: Full Rollout or Not?</strong></p><p>Before you fully roll out the canary version, carefully evaluate these key indicators:</p><p>1. <strong>Error Rate:</strong> Is the canary’s error rate at or below the stable version’s? Any spikes are a major red flag.</p><p>2. <strong>Latency:</strong> Is the canary version maintaining or improving response times compared to the stable version? Significant increases in latency could indicate underlying problems.</p><p>3. <strong>Resource Usage:</strong> Is the canary version using more CPU or memory than the stable version? Monitor this to avoid resource bottlenecks.</p><p>4. <strong>User Feedback:</strong> If possible, gather qualitative data, such as user feedback or support tickets, to complement your metrics.</p><p>5. <strong>Steady-State Performance: </strong>Allow the canary version to run significantly under real-world conditions. This will help you catch any delayed or intermittent issues that might not appear immediately.</p><p><strong>Key Decisions Before Rollout</strong></p><ul><li><strong>Thresholds:</strong> Define clear thresholds for error rates, latency, and resource usage that the canary must meet before you fully roll it out.</li><li><strong>Stability Over Time:</strong> Before increasing traffic to the canary version, make sure it maintains consistent performance over a set period.</li><li><strong>Risk Assessment:</strong> Consider the potential impact of a full rollout. If the canary introduces significant new functionality or changes, be more cautious in your assessment.</li><li><strong>Automated Rollback: </strong>Have automated rollback procedures in place just in case the canary version doesn’t meet your performance expectations during the rollout.</li></ul><p>With these checks in place, you can confidently roll out your canary version, knowing it will likely meet your performance and stability goals.</p><p><strong>Gradual Rollout and Rollback with Kubernetes</strong></p><p>Once you’re confident in the canary, gradually increase its traffic share by scaling up its replica count while scaling down the stable version. If any problems arise during this gradual rollout, Kubernetes makes it easy to roll back to the stable version by simply adjusting the replica counts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*RPedpOvwBEeSeO7I7sA7xA.png" /></figure><p><strong>Argo Rollouts</strong></p><p>Argo Rollouts is a Kubernetes controller that enables advanced deployment strategies like canary and blue-green deployments. It also offers features like automated rollbacks and metric-based analysis, making it a powerful tool for complex implementations.</p><p>For our knowledge-powered chatbot, we let’s use Argo Rollouts to implement a canary deployment strategy.</p><p>Instead of using a standard Deployment, you define a Rollout resource, which allows for controlled traffic shifting to the new version based on predefined steps. This approach allows you to monitor the canary release closely and roll back if necessary.</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Rollout<br>metadata:<br>  name: chatbot-canary<br>spec:<br>  replicas: 2<br>  strategy:<br>    canary:<br>      steps:<br>        - setWeight: 10<br>        - pause: {duration: 5m}<br>        - setWeight: 50<br>        - pause: {duration: 10m}<br>        - setWeight: 100<br>  selector:<br>    matchLabels:<br>      app: chatbot<br>  template:<br>    metadata:<br>      labels:<br>        app: chatbot<br>        version: canary<br>    spec:<br>      containers:<br>      - name: kuberada-chatbot<br>        image: kuberada-chatbot:canary<br>        ports:<br>        - containerPort: 5000</pre><p>Now Argo Rollouts will gradually shift traffic to the canary version, providing a controlled environment for monitoring and quick rollback if necessary. This process can minimize downtime and ensure that the new version is stable before a full rollout.</p><p><strong>Beyond Basic Canary: Advanced Techniques</strong></p><p>While we’ve explored manual traffic splitting and progressive delivery with Argo Rollouts, the world of canary deployments offers even more advanced techniques. For instance, Istio, a service mesh, provides fine-grained control over traffic routing and precise percentage-based traffic splitting between application versions. On the other hand, Flagger, a Kubernetes operator, automates the entire canary process, handling traffic shifting and metric analysis to determine if the canary is healthy enough for a full rollout. If you want even more granular control, feature flags allow you to selectively activate specific features for a subset of users. Tools like LaunchDarkly or Split can help you manage this.</p><p><strong>The Power of Canary Deployments</strong></p><p>Canary deployments are a powerful tool in your DevOps arsenal. They offer a safe and controlled way to introduce new versions of your application, significantly minimizing the risk of widespread issues. By carefully monitoring and analyzing key metrics during the canary phase, you can make informed decisions about the rollout and quickly revert if necessary. Remember, the best canary deployment technique will depend on the specific needs of your application and infrastructure. For more complex scenarios, tools like Istio, Flagger, and Argo Rollouts offer additional automation and control.</p><p><strong>Comparing Blue-Green and Canary Deployments</strong></p><p>Let’s take a closer look at how Blue/Green and Canary deployments stack up against each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*IPOaJncZbAwkCaAoe0G8gw.png" /></figure><p><strong>Blue-Green Deployments</strong></p><ul><li><strong>Parallel Environments:</strong> In a Blue/Green deployment, both the old (blue) and new (green) versions of your application run side-by-side, each with its own set of replicas.</li><li><strong>Traffic Switch:</strong> Initially, all traffic is directed to the blue environment. Once the green environment is thoroughly tested and ready, you simply flip a switch and route all traffic to the green environment.</li><li><strong>Quick Rollback:</strong> If any issues arise with the new version, you can instantly switch traffic back to the blue environment, minimizing user impact.</li><li><strong>Resource Intensive:</strong> The downside is that Blue/Green deployments require more resources since you’re essentially running two complete environments simultaneously.</li><li><strong>Ideal Use Cases: </strong>Blue/Green deployments are well-suited for major updates where even a brief period of downtime is unacceptable.</li></ul><p><strong>Canary Deployments</strong></p><ul><li><strong>Gradual Rollout: </strong>With Canary deployments, you introduce the new version to a small subset of users while the majority continue using the stable version. Traffic is gradually shifted from the stable to the canary version based on predefined success criteria.</li><li><strong>Controlled Rollback: </strong>If any issues arise, you can easily halt the canary deployment and redirect all traffic back to the stable version.</li><li><strong>Resource Efficient:</strong> Canary deployments are generally more resource-efficient than Blue/Green deployments since you’re only running a limited number of replicas for the new version.</li><li><strong>Ideal Use Cases:</strong> Canary deployments are perfect for A/B testing, gradually rolling out new features, and gathering real-world user feedback without impacting the majority of your users.</li></ul><p><strong>Choosing the Right Strategy</strong></p><p>The ideal deployment strategy will depend on your specific application, its requirements, and your risk tolerance. Blue/Green deployments offer a straightforward rollback path and ensure zero downtime, but they come with a higher resource cost. Canary deployments are more resource-efficient and offer greater control over the rollout process, but they might introduce a small risk of exposing some users to potential issues with the new version.</p><p><strong>Other Strategies and Tools for Zero-Downtime Deployments</strong></p><p>Beyond Blue/Green and Canary deployments, several other techniques and tools can help you achieve seamless Kubernetes deployments with minimal or zero downtime. Here’s a breakdown of these advanced strategies:</p><p><strong>A/B Testing</strong></p><p>A/B testing involves deploying two or more versions of an application or feature simultaneously to different user segments. This allows you to compare user behavior and outcomes between the versions, helping you determine which one performs better in terms of engagement or conversion rates. While not traditionally focused on zero downtime, A/B testing can be used alongside other deployment strategies to ensure that only the best-performing version reaches all users. It’s more about feature effectiveness and user experience rather than purely about risk mitigation.</p><p><strong>Shadow Deployments</strong></p><p>Shadow deployments allow you to run a new version of your application alongside the current version, without exposing it to users. The new version processes the same traffic as the live version, but its responses are discarded. This is ideal for testing performance, load handling, and identifying potential issues in a production-like environment without affecting the user experience. Shadow deployments are especially useful in scenarios where you need to validate application behavior under real load before a full rollout, helping you achieve a smoother transition with minimal risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*JidNDvIUDdCkR9DxsOvSKw.png" /></figure><p><strong>Progressive Delivery with Feature Flags</strong></p><p>Progressive delivery allows teams to release features gradually, controlling the exposure of new features to users while monitoring performance and collecting feedback. Tools like Flagger and Istio enable automated canary deployments with dynamic traffic shifting based on real-time metrics. This method reduces the risk of widespread issues by ensuring that only stable, high-performing features are fully deployed.</p><p><strong>AI-Powered Monitoring and Automation</strong></p><p>AI and machine learning are increasingly integrated into deployment pipelines to predict potential issues and automate responses. AI can help detect anomalies, optimize resource allocation, and ensure that deployments are smooth and downtime is minimized. Integrating AI-powered tools like Datadog’s anomaly detection with your Kubernetes deployments can automatically trigger alerts or rollbacks, maintaining application stability.</p><p><strong>Security-First Deployment Practices</strong></p><p>Security is a crucial aspect of Kubernetes deployments. Tools like OPA (Open Policy Agent) and Kyverno enforce security policies during the deployment process, ensuring that security is integral to the pipeline. By implementing these tools, you can enforce security best practices at the time of deployment, reducing vulnerabilities and ensuring compliance with organizational standards.</p><p><strong>Key Takeaways</strong></p><p>Let’s recap the essential lessons from this guide:</p><p>- Zero-Downtime deployments are crucial for both user experience and business continuity. They prevent financial losses and maintain your reputation.</p><p>- Kubernetes’ built-in strategies, like Rolling Updates, offer a foundation for smooth deployments but may need careful configuration for high-availability applications.</p><p>- Advanced strategies like Blue/Green and Canary deployments provide more control and flexibility for minimizing downtime.</p><p>- Real-time monitoring of key metrics during deployments is essential for making informed decisions and ensuring a successful rollout.</p><p>- GitOps practices enhance deployment management by providing automation, version control, and traceability.</p><p>- Graceful shutdowns and readiness probes help your applications handle updates smoothly and prevent disruptions for users.</p><p>- Tailor your deployment strategy to your specific needs and risk tolerance.</p><p>- Advanced techniques like A/B testing and shadow deployments offer additional flexibility for testing and experimentation.</p><p>- Security and compliance must be integrated into your deployment pipeline to minimize vulnerabilities.</p><p><strong>Conclusion</strong></p><p>Kubernetes offers a variety of advanced deployment strategies and tools, each with its unique advantages. From A/B testing to shadow deployments and progressive delivery, the goal remains the same: minimize disruption, enhance user experience, and maintain high availability. These strategies, combined with continuous monitoring and automated rollbacks, form the foundation of a robust, zero-downtime deployment approach.</p><p>Choose wisely and deploy with confidence!</p><p><strong>Enjoyed this read?</strong></p><p>If you found this guide helpful, folow me on:</p><ul><li><a href="https://www.linkedin.com/in/gulcantopcu/">LinkedIn</a> to get the latest updates.</li><li><a href="https://medium.com/@gulcantopcu">Medium</a> for even more Kubernetes insights and discussions.</li><li><a href="https://podlock.readthedocs.io/">Podlock</a>, for in-depth articles.</li></ul><p>Until next time, happy deploying!</p><p><a href="https://podlock.readthedocs.io/">Hi</a></p><p><strong>Resources and References</strong></p><ul><li><a href="https://kuberada.devtechops.dev/blogs/devops/cicd/cicd.html">Canary Deployments Made Easy: A CI/CD Journey with GitHub Actions and Argo CD Rollouts</a></li><li><a href="https://kuberada.devtechops.dev/blogs/k8s/istio/istio.html">Unlocking Istio’s Power: A Step-by-Step Guide to Seamless Microservices Management</a></li><li><a href="https://learnkube.com/graceful-shutdown">LearnK8s Guide on Graceful Shutdowns</a></li><li>Kubernetes Documentation on <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment">Rolling Updates</a></li><li><a href="https://prometheus.io/docs/introduction/overview/">Prometheus</a> Documentation</li><li><a href="https://grafana.com/docs/">Grafana</a> Documentation</li><li><a href="https://argoproj.github.io/argo-rollouts/">Argo Rollouts</a> Documentation</li><li><a href="https://flagger.app/">Flagger</a> Documentation</li><li>Kubernetes Documentation on <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">Readiness Probes</a></li><li><a href="https://learnkube.com/troubleshooting-deployments">A visual guide on troubleshooting Kubernetes deployments</a></li><li><a href="https://www.openpolicyagent.org/docs/latest/">Open Policy Agent Documentation</a></li><li><a href="https://kyverno.io/docs/">Kyverno Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa92c8ec5646" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/kubernetes-the-art-of-zero-downtime-deployments-fa92c8ec5646">Kubernetes: The Art of Zero-Downtime Deployments</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Perfect Kubernetes Workload: A Practical Guide for Application Success]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://itnext.io/choosing-the-perfect-kubernetes-workload-a-practical-guide-for-application-success-fe905c40788a?source=rss-f548dcc00dae------2"><img src="https://cdn-images-1.medium.com/max/1920/1*RZHSjrO8SDM91r3EVir2tQ.png" width="1920"></a></p><p class="medium-feed-snippet">Choosing the optimal Kubernetes workload can be a critical factor in the success of your application deployment.</p><p class="medium-feed-link"><a href="https://itnext.io/choosing-the-perfect-kubernetes-workload-a-practical-guide-for-application-success-fe905c40788a?source=rss-f548dcc00dae------2">Continue reading on ITNEXT »</a></p></div>]]></description>
            <link>https://itnext.io/choosing-the-perfect-kubernetes-workload-a-practical-guide-for-application-success-fe905c40788a?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe905c40788a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 13:48:13 GMT</pubDate>
            <atom:updated>2025-05-26T16:53:51.681Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Transforming MLOps with Kubeflow & KServe: Churn Prediction]]></title>
            <link>https://medium.com/@gulcantopcu/transforming-mlops-with-kubeflow-kserve-churn-prediction-2b554a3bd356?source=rss-f548dcc00dae------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b554a3bd356</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kserve]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Gulcan Topcu]]></dc:creator>
            <pubDate>Wed, 03 Jul 2024 14:39:03 GMT</pubDate>
            <atom:updated>2025-05-26T16:54:38.896Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EJA4NU4IMqDNZll-KeB5A.png" /></figure><p>As a Kubernetes wrangler, you have probably heard the buzz about MLOps and its potential to revolutionize data-driven businesses.</p><p>While the potential ROI is significant (reducing churn by a mere 5% can boost profits by 25-95% [<a href="https://hbr.org/2014/10/the-value-of-keeping-the-right-customers">^1</a>]) , building and maintaining effective MLOps pipelines can be daunting, especially if you&#39;re new to the machine learning world.</p><p>In this guide, I&#39;ll show you how to effectively use Kubeflow, KServe, and Flask to predict customer churn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8qz9xM2U_6flIN35G1s1rw.png" /></figure><p>Traditional MLOps workflows often grapple with several pain points, such as:</p><ul><li>Managing the growing volume of data and models.</li><li>Maintaining and scaling the underlying infrastructure.</li><li>Seamlessly integrate and deploy models into production.</li><li>Managing operational costs effectively.</li></ul><p>Enter <a href="https://www.kubeflow.org/">Kubeflow</a>, an end-to-end MLOps platform that streamlines machine learning workflows, and <a href="https://github.com/kserve/kserve">KServe</a>, its serverless model deployment component.</p><p>Together, they offer a robust solution to these challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*_gtWjBrE3GMIfPWECGSOnw.png" /></figure><p><strong>Why MLOps Matters</strong></p><p>Efficient MLOps practices can significantly impact businesses by:</p><ul><li>Accelerating the time from model development to deployment.</li><li>Enhancing the ability to retrain and optimize models.</li><li>Facilitating the handling of large-scale data and model training.</li><li>Leading to more informed decision-making and competitive advantage.</li></ul><p><strong>Why Kubernetes is a popular choice for MLOps?</strong></p><p>Kubernetes is a popular choice for MLOps because it:</p><ul><li>Efficiently manages resources and scales applications.</li><li>Ensures optimal usage of computing resources.</li><li>Allows for easy deployment across different environments.</li></ul><p><strong>Why Kubeflow?</strong></p><p>Kubeflow is a comprehensive platform that supports the entire machine learning lifecycle. It comprises several key components:</p><ul><li>Pipelines automates the workflow from data ingestion to model deployment, streamlining the process and ensuring consistency.</li><li>Notebooks provides an interactive environment for data scientists to experiment, develop, and fine-tune models efficiently.</li><li>Experiments tracks and manages different model versions and experiments, facilitating better model management and iteration.</li></ul><p>These components work together to simplify and enhance the MLOps process, making it easier to manage, deploy, and scale machine learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*dnpTrxy3LN6AXc3VXuGh7A.png" /></figure><h4><strong>KServe &amp; Serverless MLOps</strong></h4><p>KServe simplifies model deployment and scaling on Kubernetes with its serverless architecture, automatically adjusting resources based on demand.</p><p>Minimized operational overhead, automatic scaling, and cost-effectiveness makes KServe a powerful and straightforward choice for deploying, managing, and scaling machine learning models.</p><h3><strong>Case Study: Churn Prediction Project</strong></h3><p>In this project, my focus is on the MLOps pipeline, not the specific churn prediction algorithm.</p><p>I used Flask to create an API for the model, simplifying deployment and management with KServe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BnZJ_oCCuyUwP5UiDr7vqQ.png" /></figure><p><strong>What is Churn?</strong></p><p>Churn prediction identifies customers likely to leave, allowing businesses to take proactive measures.</p><p>Using a telco dataset with customer data and churn status, I designed a Kubeflow pipeline to automate the entire process.</p><pre>import kfp<br>from kfp import dsl<br><br>def preprocess_op():<br>    return dsl.ContainerOp(<br>        name=&#39;Preprocess Data&#39;,<br>        image=&#39;myrepo/preprocess:latest&#39;,<br>        command=[&#39;python&#39;, &#39;preprocess.py&#39;]<br>    )<br><br>def train_op():<br>    return dsl.ContainerOp(<br>        name=&#39;Train Model&#39;,<br>        image=&#39;myrepo/train:latest&#39;,<br>        command=[&#39;python&#39;, &#39;train.py&#39;]<br>    )<br><br>@dsl.pipeline(<br>    name=&#39;Customer Churn Prediction Pipeline&#39;,<br>    description=&#39;An example pipeline for customer churn prediction.&#39;<br>)<br>def churn_prediction_pipeline():<br> preprocess_task = preprocess_op()<br> train_task = train_op().after(preprocess_task)<br><br>if __name__ == &#39;__main__&#39;:<br> kfp.compiler.Compiler().compile(churn_prediction_pipeline, &#39;churn_prediction_pipeline.yaml&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uaZUFhFSg1YOl4oybLNlLw.png" /></figure><p><strong>What does the Flask Web App do?</strong></p><p>The Flask web app:</p><ul><li>takes a JSON file from the user,</li><li>communicates with the model,</li><li>analyzes file,</li><li>finally returns the churn prediction result, chur or no churn.</li></ul><pre>from flask import Flask, request, jsonify<br>import requests<br><br>app = Flask(__name__)<br><br>@app.route(&#39;/predict&#39;, methods=[&#39;POST&#39;])<br>def predict():<br> data = request.json<br> response = requests.post(&#39;http://kserve-model:8080/v1/models/customer-churn:predict&#39;, json=data)<br> result = response.json()<br>    if result[&#39;predictions&#39;][0] == 1:<br>        return jsonify({&quot;prediction&quot;: &quot;😢 Churn&quot;})<br>    else:<br>        return jsonify({&quot;prediction&quot;: &quot;🙂 No churn&quot;})<br><br>if __name__ == &#39;__main__&#39;:<br> app.run(debug=True)</pre><p>Let’s simulate a customer profile that is more likely to churn.</p><pre>      {<br>        &quot;gender&quot;: 0,<br>        &quot;SeniorCitizen&quot;: 1,<br>        &quot;Partner&quot;: 0,<br>        &quot;Dependents&quot;: 0,<br>        &quot;tenure&quot;: 1,<br>        &quot;PhoneService&quot;: 1,<br>        &quot;MultipleLines&quot;: 0,<br>        &quot;InternetService&quot;: 1,<br>        &quot;OnlineSecurity&quot;: 0,<br>        &quot;OnlineBackup&quot;: 0,<br>        &quot;DeviceProtection&quot;: 0,<br>        &quot;TechSupport&quot;: 0,<br>        &quot;StreamingTV&quot;: 0,<br>        &quot;StreamingMovies&quot;: 0,<br>        &quot;Contract&quot;: 0,<br>        &quot;PaperlessBilling&quot;: 1,<br>        &quot;PaymentMethod&quot;: 2,<br>        &quot;MonthlyCharges&quot;: 95.75,<br>        &quot;TotalCharges&quot;: 95.75<br>      }</pre><p>This customer exhibits a profile that is likely to churn.</p><p>Why?</p><p>We have observed that the customer is still in the early stages of their relationship with the company and may not be fully committed and has the flexibility to leave at any time, making them more likely to churn.</p><p>It’s crucial to remember that this is a prediction based on a single data snapshot.</p><p>For more accurate predictions, analyzing long-term behavior and other factors like customer service interactions, promotions, and competitor offerings is essential.</p><p>You can expand this study including other classifiers such as Logistic Regression, Random Forest, Gradient Boosting Machines, and Neural Networks.</p><p><strong>Why KServe?</strong></p><p>I want to focus on our model’s business logic and integration with other applications rather than worrying about the underlying infrastructure details.</p><p>Deploying the Flask-based model on KServe is straightforward:</p><pre>apiVersion: &quot;serving.kserve.io/v1beta1&quot;<br>kind: &quot;InferenceService&quot;<br>metadata:<br>  name: customer-churn-model<br>  namespace: kubeflow<br>spec:<br>  predictor:<br>    serviceAccountName: sa-minio-kserve<br>    sklearn:<br>      storageUri: &quot;s3://mlops-data/model.pkl&quot;</pre><p>Once I port forward the model endpoint, it is accessible for predictions.</p><p>We can interact with it programmatically (e.g., from our application’s backend) or through tools like curl or Postman.</p><pre>curl -X POST \<br>  http://localhost:8080/v1/models/customer-churn-model:predict \<br>  -H &#39;Content-Type: application/json&#39; \<br>  -d @input.json</pre><p>Using flask based webapp:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ypl0vri7UTMgXUPMTiHtBA.png" /></figure><p><strong>Key Takeaways from Our Practical Implementation</strong></p><ul><li>Manual management of ML workflows was time-consuming, prone to errors, and required significant developer effort. With Kubeflow Pipelines, we automated the entire process, reducing workflow management time and decreasing errors. This automation freed up developers to focus on model development and optimization.</li><li>Deploying models manually resulted in inconsistent performance, frequent downtime, and scalability issues during high load periods. KServe provided automatic scaling and efficient resource utilization, resulting in improved model response time and reduced downtime during peak usage.</li><li>Integrating the model with applications was cumbersome, with slow response times affecting user experience. The Flask API integration streamlined interactions, improving response times and simplifying the integration process, which reduced development time.</li><li>Over-provisioning resources led to high operational costs, with many underutilized resources needing to be utilized. Implementing a serverless architecture with KServe reduced resource costs by 40%, as resources were automatically scaled based on demand, ensuring optimal utilization.</li><li>The ML lifecycle stages were disjointed, causing delays and inconsistencies across data ingestion, preprocessing, training, and deployment. Kubeflow’s end-to-end automation reduced the time to deploy a new model version from weeks to days, improving consistency and reducing manual intervention.</li></ul><p><strong>Enjoyed this read?</strong></p><p>If you found this guide helpful, check my blog archive for more challenging projects <a href="https://kuberada.devtechops.dev/">Kuberada</a> 📚✨</p><ul><li>Follow me on <a href="https://www.linkedin.com/in/gulcantopcu/">LinkedIn</a> to get updated.</li><li>Read incredible Kubernetes Stories on <a href="https://medium.com/@gulcantopcu">Medium</a>.</li></ul><p>Until next time!</p><p>[¹]: <a href="https://hbr.org/2014/10/the-value-of-keeping-the-right-customers">Harvard Business Review: The Value of Keeping the Right Customers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b554a3bd356" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>