<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ivan Porta on Medium]]></title>
        <description><![CDATA[Stories by Ivan Porta on Medium]]></description>
        <link>https://medium.com/@gtrekter?source=rss-f58cb680fdbf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1RjkE8VZ5Cuide--vmkkxw.jpeg</url>
            <title>Stories by Ivan Porta on Medium</title>
            <link>https://medium.com/@gtrekter?source=rss-f58cb680fdbf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:19:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gtrekter/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What Platform Teams Can Expect From Crossplane v2.2]]></title>
            <link>https://medium.com/todea-engineering/what-platform-teams-can-expect-from-crossplane-v2-2-1cc7bbec57d4?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cc7bbec57d4</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 16:19:53 GMT</pubDate>
            <atom:updated>2026-09-13T16:19:53.056Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BuTuOIbRjc_mf0tcvYpKRQ.avif" /></figure><p>Crossplane is a <a href="https://www.cncf.io/projects/crossplane/">CNCF</a> control plane that manages cloud infrastructure through Kubernetes APIs, reconciling declared resources continuously the way Kubernetes reconciles pods. v2.2 is a release aimed at production gaps: an alpha pipeline inspector, CEL validation beyond spec, and dependency-aware runtime configuration. Use it when infrastructure should be a platform API; reach for Terraform when plan-and-apply is enough.</p><p>A developer submits a ticket to request a database. Three days later, the platform team responds, but the configuration isn’t quite right. The developer files another ticket and waits again. By the third week, the database might finally be ready. This cycle repeats across every team and environment, creating a daily reality that most platform teams recognize: developers lose significant time waiting for infrastructure, while platform teams struggle to keep up with the constant flow of requests.</p><p>There are plenty of good tools for provisioning. <a href="https://developer.hashicorp.com/terraform">Terraform</a> is platform-agnostic and widely used. CloudFormation is the go-to option on AWS, and every cloud provider offers its own console and CLI. Each tool does its job well. This article isn’t about choosing the best one. Instead, it looks at the problem from a different perspective.</p><p><a href="https://www.crossplane.io/">Crossplane</a> offers a Kubernetes-native approach to managing cloud resources. Instead of setting up infrastructure outside the cluster and then linking it back, Crossplane brings that infrastructure under the same control loop as your applications. This fits naturally with how many teams already work. Paired with GitOps, a pull request becomes the primary way to manage changes, and the cluster continuously reconciles toward the desired state.</p><p>The project has moved quickly over the past year. In August 2025, Crossplane v2 introduced big changes, like removing Claims and adding namespaced composite and managed resources. The latest release, <a href="https://github.com/crossplane/crossplane/releases">v2.2</a>, adds an alpha Pipeline Inspector for troubleshooting, broader CEL validation, and more improvements.</p><h3>What is Crossplane, actually?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xG-RchiGXeS1czJRoM_tdw.png" /></figure><p>Crossplane is a control plane framework for platform engineering. You install it into a Kubernetes cluster, known as the management cluster, and that cluster becomes the control plane for everything outside it: cloud accounts, SaaS APIs, internal tools, and even other Kubernetes clusters. All of these are managed through the same Kubernetes API your applications already use. The management cluster itself must be set up separately; Crossplane does not create it for you. Once Crossplane is running, there is no state file and no separate workflow. Drift is fixed by the same reconciliation loop that keeps your Deployments healthy.</p><p>Crossplane has four major components. You can use all four or only the ones you need.</p><p><strong>1. Managed resources (MRs)</strong> map directly to external cloud resources in Kubernetes. For example, an S3 from AWS or a ResourceGroup from Azure is considered an MR. Crossplane uses spec.forProvider as the main reference and keeps the actual cloud resource in sync with it. You create MRs using kubectl, and the provider handles provisioning and reconciliation.</p><p><strong>2. Composition</strong> lets you create custom APIs using a function pipeline. There are three main parts to understand:</p><ul><li>A CompositeResourceDefinition (XRD) defines a schema. It tells Kubernetes, “here’s a new custom API kind I’m creating, and these are its fields.” You can think of it as a CRD with added features for Crossplane.</li><li>A Composition acts as a recipe. It says, “when someone creates an XR of kind Foo, run this set of functions to create these MRs or other Kubernetes resources.” In version 2, this always uses a function pipeline.</li><li>A Composite Resource (XR) is an instance of the API you defined with an XRD. When a user creates an XR, Crossplane uses the matching Composition’s pipeline to generate the needed resources. You can write functions in YAML, KCL, Python, or Go.</li></ul><p><strong>3. Operations</strong> run function pipelines to completion, similar to a Kubernetes Job. There are three modes: Operation (one-time), CronOperation (scheduled), and WatchOperation (event-driven). Operations are currently in alpha.</p><p><strong>4.</strong> The <strong>package manager</strong> handles installing and updating providers, configurations, and functions.</p><h3>How a Crossplane request flows</h3><p>There are two entry points into this flow, depending on what you’re applying.</p><p>When a developer or a pipeline creates an XR in a namespace, the composition engine watches it, runs the configured function pipeline, and creates the needed resources. These resources can be other Kubernetes resources, managed resources, or both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kkzji_Q0UFTbENGBlXFcBg.png" /></figure><p>When a user applies an MR directly, either by itself or as part of a Composition, the provider takes over. It monitors the MR through the Kubernetes API, calls the external system to create or update the real resource, and updates the status. After that, it keeps checking: if the real resource changes from spec.forProvider, the provider fixes it. All state is stored in etcd, so there is no separate state file.</p><h3>When should you use traditional IaC instead?</h3><p>Crossplane and tools like Terraform or CloudFormation overlap in scope (both can provision a cloud database) and differ in how. The right choice depends on where your platform already lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zL_Es6SwGo198gdhh77BgQ.png" /></figure><p>If your team does not use Kubernetes, Crossplane is not the best place to start. Terraform is simpler and does not need a control plane. But if you are on Kubernetes, especially if you already use Argo CD or Flux, it is easy to manage your infrastructure in the same way. Crossplane is the closest option for writing infrastructure as code and handling it like the rest of your declarative cluster state.</p><h3>What’s new in v2.2?</h3><p>v2.2 adds five things you’ll notice in practice and one that quietly improves reliability. Each one closes a specific gap that platform teams have been hitting in production.</p><ul><li><strong>Pipeline inspector (alpha)</strong>: Composition functions are powerful, but they have always been hard to debug. If a pipeline acts up on a running control plane, you could only see what each function got and returned by writing tests, running crossplane render locally, or adding your own instrumentation. v2.2 adds the pipeline inspector. When you turn on the feature flag, the Crossplane controller intercepts every RunFunctionRequest and RunFunctionResponse and forwards them over gRPC to a Unix socket you set up. A sidecar read from this socket and handle the data however you need: stream it to stdout during development or send it to an audit pipeline in production. To use it, add --enable-pipeline-inspector to Crossplane. The default socket path is /var/run/pipeline-inspector/socket, but you can change it with --pipeline-inspector-socket.</li></ul><pre># Enable the pipeline inspector feature flag<br>args:<br>  - --enable-pipeline-inspector<br>  - --pipeline-inspector-socket=/var/run/pipeline-inspector/socket<br> <br># Inject the pipeline inspector sidecar<br>sidecarsCrossplane:<br>  - name: pipeline-inspector<br>    image: xpkg.crossplane.io/crossplane/inspector-sidecar:v0.0.3<br>    args:<br>      - --socket-path=/var/run/pipeline-inspector/socket<br>      - --max-recv-msg-size=8388608  # 8MB<br>    volumeMounts:<br>      - name: pipeline-inspector-socket<br>        mountPath: /var/run/pipeline-inspector<br>    resources:<br>      requests: { cpu: 10m, memory: 64Mi }<br>      limits:   { cpu: 100m, memory: 128Mi }<br> <br># Add the shared volume for Unix socket communication<br>extraVolumesCrossplane:<br>  - name: pipeline-inspector-socket<br>    emptyDir: {}<br> <br>extraVolumeMountsCrossplane:<br>  - name: pipeline-inspector-socket<br>    mountPath: /var/run/pipeline-inspector</pre><ul><li><strong>XRD validation outside </strong>spec: XRD validation outside x-kubernetes-validations, (which are Kubernetes&#39; CEL-based validation rules) used to only work on fields under an XR&#39;s spec. If you wanted to enforce rules like &quot;all Database names must start with db-&quot;, you had to use an external admission controller such as Kyverno, OPA/Gatekeeper, or a custom webhook. With v2.2, that restriction is gone. Now, you can write CEL rules outside of spec, and the API server enforces them at admission time.</li></ul><pre>apiVersion: apiextensions.crossplane.io/v1<br>kind: CompositeResourceDefinition<br>metadata:<br>  name: databases.platform.example.org<br>spec:<br>  group: platform.example.org<br>  names:<br>    kind: Database<br>    plural: databases<br>  versions:<br>    - name: v1alpha1<br>      served: true<br>      referenceable: true<br>      schema:<br>        openAPIV3Schema:<br>          type: object<br>          x-kubernetes-validations:<br>            - rule: &quot;self.metadata.name.startsWith(&#39;db-&#39;)&quot;<br>              message: &quot;Database names must start with &#39;db-&#39;&quot;<br>          properties:<br>            spec:<br>              type: object<br>              properties:<br>                region:<br>                  type: string</pre><ul><li>ImageConfig<strong> runtime for dependencies</strong>: A Crossplane package, including Providers, runs as a Deployment. To customize the Deployment, such as by adding service account annotations, pod labels, or container arguments, use a DeploymentRuntimeConfig and reference it from the package.</li></ul><pre>kind: Provider<br>spec:<br>  package: xpkg.crossplane.io/crossplane-contrib/provider-azure-network:v1.0.0<br>  runtimeConfigRef:<br>    name: azure-workload-identity</pre><p>This approach works well when you install the package directly. However, Crossplane can also install packages as dependencies. In that case, you could not get Workload Identity or any other runtime customization onto providers installed as dependencies.</p><p>ImageConfig is a cluster-scoped resource that matches packages based on their image prefix, not on which Provider or Configuration object created them. In v2.2, a new field was added: spec.runtime.configRef. With this change, Crossplane applies the DeploymentRuntimeConfig to any package whose image is matched, regardless of how it was installed.</p><pre>apiVersion: pkg.crossplane.io/v1beta1<br>kind: ImageConfig<br>metadata:<br>  name: azure-workload-identity<br>spec:<br>  matchImages:<br>    - prefix: xpkg.crossplane.io/crossplane-contrib/provider-azure-<br>    - prefix: xpkg.crossplane.io/crossplane-contrib/provider-family-azure<br>  runtime:<br>    configRef:<br>      name: azure-workload-identity</pre><p>Every Azure family provider, whether installed directly or added as a dependency, receives the runtime config.</p><ul><li>RequiredSchemas<strong> for functions</strong>: Composition and functions sometimes need the OpenAPI schema of a resource to validate inputs, make schema-aware decisions, or generate resources dynamically. Before v2.2 you could ask Crossplane for the corresponding CRD as a RequiredResource, parse it, and extract the schema yourself, but only for custom resources, since built-in kinds like Deployment don&#39;t have a CRD. v2.2 introduces RequiredSchemas on the RunFunctionResponse thich returns the schema for any kind, built-in or custom.</li><li>crossplane beta trace<strong> improvements</strong>: You can now pass a kind (and optionally a namespace) instead of a single resource and get the dependency tree for every instance. And --watch (alias -w) keeps the output live, the way kubectl get -w does.</li><li><strong>Function packages no longer install bundled CRDs</strong>: CRDs included in a function package are not applied to the cluster anymore. Also, packages with unknown or disallowed kinds now install successfully and simply skip those objects. Previously, the install would fail in these cases.</li><li><strong>The package cache layout has changed</strong>: Cache filenames now come from the package’s OCI source and digest instead of the PackageRevision’s Kubernetes name. This change affects side-loading used by some provider e2e suites.</li></ul><h3>Operational reality</h3><ul><li><strong>The management cluster is your state.</strong> Crossplane does not use an external state file. All XRDs, Compositions, XRs, and managed resources are stored in the management cluster’s etcd. If you lose that cluster without backups, your cloud resources keep running, but Crossplane loses track of them and stops reconciling. Silent drift can build up. Treat the management cluster like any production-critical Kubernetes cluster: use a highly available control plane, back up etcd, and avoid running it on your laptop. Local k3s or kind clusters are fine for learning, demos, or the Get Started guide, but not for important state. This is the trade-off for not having a Terraform state file: you solve one operational problem but gain another that is easier to overlook.</li><li><strong>Upgrade through v2.1, not directly.</strong> Crossplane performs CRD migrations with each minor version upgrade, so skipping versions can cause you to miss important migrations. If you are on v1.x, use the Crossplane v2 upgrade guide. If you are on v2.1, upgrade directly to v2.2.</li><li><strong>v1.20 is not yet end-of-life.</strong> v1.20 is still supported and has not reached end-of-life yet. However, since you are on a maintenance-only branch, it’s a good time to start planning your upgrade to v2.x.</li><li><strong>Pipeline inspector is alpha.</strong> The flag is off by default and the contract may still change. Sidecar image versioning is also not stable yet. Try it in development, since function pipelines are much easier to understand when you can see them, but do not add it to your incident-response runbook yet.</li><li><strong>Namespaced MRs are not yet universal.</strong> AWS managed resources are fully namespaced. The Upbound Azure provider, which is widely used, and GCP are currently rolling out this feature.</li><li><strong>v2 removed several things.</strong> Native patch-and-transform composition, the ControllerConfig type, external secret stores, composite resource connection details, and the default registry for packages are no longer available. Most users can upgrade without breaking changes, but if you use these features, you will need to do some cleanup. Before upgrading, run kubectl get pkg and make sure every package uses a fully qualified image, such as registry.example.com/repo/package:tag.</li></ul><h3>A practical recommendation</h3><p>If you are considering a control plane for your Kubernetes platform and do not have a strong reason to stick with Terraform, try Crossplane v2.2 first. The <a href="https://docs.crossplane.io/">Get Started guide</a> can be completed in an afternoon on any Kubernetes cluster. If Crossplane meets your needs, you can manage both application and infrastructure workflows with one declarative model. If not, you will have a clear, documented reason to keep your current tools.</p><p>If you are already using Crossplane v2.1, upgrade to v2.2. Features like server-side apply on the MRD controller, dependency-aware runtime config, schema access for functions, and better trace output are valuable even if you do not use the pipeline inspector. If you are still on v1.x, pin to v1.20, migrate any deprecated features, then upgrade to v2.x and continue from there. v2 offers good backward compatibility, but the deprecations are real.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cc7bbec57d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/what-platform-teams-can-expect-from-crossplane-v2-2-1cc7bbec57d4">What Platform Teams Can Expect From Crossplane v2.2</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KubeCon China 2026: Capacity Recovered From The Same Hardware]]></title>
            <link>https://medium.com/todea-engineering/kubecon-china-2026-capacity-recovered-from-the-same-hardware-8a2083cea310?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a2083cea310</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[openstack]]></category>
            <category><![CDATA[cncf]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Sun, 13 Sep 2026 16:16:26 GMT</pubDate>
            <atom:updated>2026-09-13T16:16:26.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1_35ISZXo_GnVSaxPn9pw.png" /></figure><p><a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/">KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026</a> in Shanghai ran on two ideas. The first was a workload shift the opening keynote put analyst figures behind: AI compute is now weighted to inference rather than training. Serving it has strained the abstractions Kubernetes was built on, and that complaint ran right through the program: a Deployment does not describe an inference replica, and time to first token is not a request rate. The second idea was enforcement, a layer below the first and largely indifferent to it. HAMi and DRA split a card and hold each workload to its share, whether the work above is training or serving. The two met at the spending line: teams that needed more capacity got it by changing the unit they allocate, not by buying hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*9Hb8_9PXwdEoaB4TxwQ5Gg.avif" /></figure><h3>What the keynotes framed</h3><p>The keynotes put that shift in numbers. On the analyst figures cited from the stage, inference now takes 67% of AI compute against 33% for training, and the reason given was frequency rather than size: training happens sporadically, inference happens on every prompt, and agent workloads are inference hungry. The same keynote projected inference power demand reaching 93.3 GW by 2030. It also mapped each cloud-native primitive onto an AI-era counterpart: containers plus sandboxes for isolation, microservices plus agents, service mesh plus agentic protocols such as MCP, declarative APIs plus model-as-a-service, and immutable infrastructure plus SLA-aware infrastructure.</p><p>The <a href="https://pytorch.org/">PyTorch</a> Foundation keynote argued that the frontier is now set by how fast teams iterate. The old linear pipeline, train a model and then serve it, has been replaced by a loop in which production behaviour feeds back into the weights, and the evidence offered was day-zero serving support in <a href="https://docs.vllm.ai/">vLLM</a> and SGLang for Kimi K3, MiniMax M3 and Qwen 3.8. The keynote’s worked example was Shopify, running that loop daily on its commerce agent: production failures become training data, a calibrated judge critiques, repairs and validates the trajectories, PyTorch retrains on them with supervised fine-tuning followed by reinforcement learning, and vLLM serves the updated weights. On their GraphQL evaluation the distilled model passed the frontier reference baseline somewhere between 46,000 and 61,000 training examples. The headline figure, a ~96% serving-cost reduction at up to 2,000 requests per minute, was presented as a modeled comparison of serving cost, not a measured total including training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SCteCjU3psG36GkUIyo6Gg.png" /></figure><p><a href="https://project-hami.io/">HAMi</a> is a clear example of this lower layer taking shape. It allows multiple workloads to <a href="https://medium.com/en/blog/hami-explained">share a single GPU card</a>, with each getting its own memory and compute quota. It sits as an open GPU resource layer beneath the schedulers rather than beside them, with Volcano, Koordinator and Kueue above it and a KAI Scheduler integration below. A DRA driver now expresses the same slicing through Kubernetes’ own claim API, translating the older device-plugin requests rather than sitting next to them, and the project is explicit that the two modes should not run in one cluster at once. HAMi was named in five sessions on the opening day, and three of the production reports below run it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoMiMMghqPHb5iGarhTSkw.png" /></figure><h3>What the enterprise talks actually showed</h3><p>China Merchants Bank faced the problem in its plainest form: one accelerator pool, a training workload that wants stable capacity and an inference workload that wants rapid elasticity, both competing for the same cards. Rather than buy more accelerators, they scheduled both on one shared base. Training arrives through Kueue for admission and quota; inference runs vLLM or SGLang and scales on KEDA against Prometheus metrics. Beneath both sit the Kubernetes scheduler with HAMi for placement, and Fluid for datasets, checkpoints and cached weights. The tidal behaviour is the point: when inference demand rises, KEDA scales out, and training jobs checkpoint themselves and release their cards, resuming when demand falls. They reported average accelerator compute utilization rising from 35% to over 60%, inference cost per one million input plus output tokens down by more than 60%, fivefold multi-tenant training density, and Qwen3–14B loading in under two seconds from Fluid-cached weights, against production targets of TTFT at or below 5 seconds and TPOT at or below 50 milliseconds. Their own summary line was that the value comes from coordination across layers rather than from any single component. They were equally clear about the cost of assembling this rather than buying a platform: ongoing integration and compatibility testing. The work won CNCF’s End User Case Study Contest at the event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RbkjtHTCC-xsH4OTOT564w.png" /></figure><p>Intsig, the company behind CamScanner, met the same problem at fleet scale: a 10,000-card estate, self-built plus multi-cloud, carrying more than 1,000 online inference services and more than 1,000 offline training jobs in parallel, with the training side above 90% utilization over 24 hours. Their constraint was not placement but queue time, people waiting for a card. The practices they credited are slicing first, with whole cards kept for heavy and large-model work, binpack placement to fill partly used GPUs rather than scatter half-card fragments, affinity pairing so two high-load workloads never share a card, a monitoring loop, and elastic scaling. Together those are credited with 50% higher GPU utilization at 30% lower overall cost, and inference performance degradation held under 10%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSJj9hmgffJ-8ygxMTxlGw.png" /></figure><p>Viettel put it most plainly. The Vietnamese operator had about 2,500 GPUs of many generations sitting behind team-by-team assignment, and one flagship card running a single inference service at full load reached roughly 10% utilization. Their diagnosis was that the limit had never been the hardware but the unit of allocation. The four-layer stack they built on that reading provisions bare metal with OpenStack Ironic and Nova, orchestrates with Kubernetes plus HAMi slicing and KEDA, serves with vLLM, SGLang, KServe and llm-d, and meters tokens through Envoy AI Gateway. A team now gets an API key and an OpenAI-compatible endpoint in minutes, instead of waiting weeks for a server. On one production card, same workload and same service level, utilization went from 13% to 59%. They presented that as the cause and the rest as what follows from it: 3.5x more work per card, which is the same result stated as 71% fewer GPUs for the same throughput on the workload they measured, and separately 2.1x work delivered per Watt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6h8dAylVOhSRYtRmV8ZgqA.png" /></figure><p>More useful still was where they said not to do it. Sharing returned 3.3x on a forecasting service whose request size was capped upstream and never moved past about 10% of a card. It ran 1.6x worse on a model whose engine already batched continuously, and came out at 0.77x aggregate on compute-bound training, worth doing only for isolation. Their caveat is that the figures are theirs. What transfers is the shape of the workload: where the engine already keeps the card busy, sharing only divides it.</p><p>Alibaba made the same trade at the cluster layer rather than the card. Qwen is served across Karmada federations, where the migration plan they showed moves online units one at a time, each round waiting a five-minute stability window, reserving capacity on the target before it moves a unit and holding the old slot so a failed round can roll back. A rebalancer re-splits batch inference as spot capacity moves between clusters. They reported roughly 15% higher allocation rate, roughly 30% higher spot GPU utilization, and cross-cluster migrations that took days finishing in minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yD31vlP_cWQq9J1fsd6oYA.png" /></figure><h3>The replica became a group</h3><p>One inference replica is no longer reliably one pod, and that is what makes these workloads hard to schedule. The <a href="https://lws.sigs.k8s.io/">LeaderWorkerSet (LWS)</a> maintainer session, run by DaoCloud and Huawei Cloud, set out a table of model sizes: Kimi K3 at 2.8 trillion parameters, DeepSeek V4 Pro at 1.6 trillion, GLM 5.3 at around 750 billion. Their point was that model weights of that size now exceed the HBM of a single node, so one replica becomes a group of pods spread across machines. Neither a Deployment nor a StatefulSet expresses a leader plus its workers as that unit of replication, which is what LWS provides: replicas multiplied by group size, injected leader addresses, and a restart policy that by default recreates the whole group when one pod fails. DisaggregatedSet sits above it, one service definition producing slices that each contain every role, each child LWS named by slice, revision and role so a rollout never leaves a new prefill talking to an old decode. The scheduling failure underneath is specific: schedule those pods independently and every GPU slot can fill with leaders while no replica ever becomes ready. Volcano answers it by making an LWS group a PodGroup, so the group is admitted as one unit or not at all, with the scheduler-plugins coscheduler doing the same and YuniKorn reaching it through its own task groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*gzvYt-pvTiTQoI-YTZEPuw.avif" /></figure><p>Huawei brought the same structure to a consumer assistant. The inference plane behind its Celia assistant is three nested levels: a ModelServing holds serving groups, and each group holds roles, so a multimodal model runs vit, prompt and decode roles with different pod shapes, held at a working ratio. Gang policy compiles to a Volcano PodGroup, topology policy pins prefill and decode to the same rack, and a ratio constraint holds the prefill to decode ratio between 1 and 3 as each role scales on its own metrics. Eviction protection was the detail worth keeping: a validating webhook checks each logical unit ahead of the native disruption budget, and where a single group holds the only instance of a role the eviction is rejected until an operator intervenes or capacity is added. NPU utilization moved from 30.1% to 40.7%, and the automated operations and maintenance rate for NPU resources from 63% to 87%. The Kthena router, a Volcano-community project rather than a Huawei one, was benchmarked against llm-d’s KV-cache-aware routing across seven configurations, with the losses shown: two came out worse than the baseline on every measure, two lost only on time to first token, and the best three improved on all six, the strongest reaching 25.1% higher request throughput and 23.8% lower TTFT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a88Q9hZ3NIZWZShx-HgRpg.png" /></figure><h3>The layer that was not about AI</h3><p>Samsung SDS took the same question to the physical layer. They offer heterogeneous bare metal and VM infrastructure as a service from 17 cloud data centres, with operations across 40 countries, and every new product had been landing on the same underlay cluster. The cost was never the object count, it was the coupling. One node ended up running Multus, Calico and OVN together, two CSI drivers and two container runtimes, with isolation stopping at the namespace. Kubernetes and driver versions moved together, so one version bump meant a full revalidation, and a data-plane failure in one workload took the others down with it. Their answer was to split the underlay into purpose-built clusters and carry the cost of running many with Cluster API. Each purpose cluster’s control plane runs as pods in a shared control cluster, etcd included, with Konnectivity tunneling API server traffic between two overlay networks with no route to each other. Version and CNI are then chosen per cluster. What ties this back to the rest of the event is what it buys them. Because node composition is declarative, one commit moves nodes from an idle cluster to a busy one, and the use case they showed was a daily cycle: GPUs weighted to inference during business hours and to training overnight. The economics they gave for it are blunt, that an idle node is paid-for hardware earning nothing and utilization is the margin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*cZMTI6da_jpyuCBY3qj7nA.avif" /></figure><h3>The themes worth taking home</h3><p>Four things are worth acting on. First, measure what one card actually delivers under production load before buying another, because three of the teams on stage started somewhere between roughly 10% and 35% utilization, and the one that said so most plainly got past it without buying anything. Second, change the unit of allocation before the budget, whether that means a slice of a card with HAMi, a leader-worker group with LeaderWorkerSet, a role inside a serving group, or a node moved between clusters. Third, test sharing rather than assuming it. Viettel’s three results, 3.3x on a workload that could not fill a card, 1.6x worse on one that already could, and 0.77x aggregate on compute-bound training, are the same question asked of three shapes of workload, and what decides it is whether the engine already keeps the card busy. Fourth, if inference already spans more than one node, move to a group-shaped workload API and gang scheduling now rather than retrofitting them later, because separating prefill from decode has become an API surface of its own in DisaggregatedSet and in Huawei’s ModelServing, above what LeaderWorkerSet gives you. Allocation was not the only lever in Shanghai, since caching, compression and routing each produced large gains of their own, but it was the one that recurred, and the resource underneath all of them is accelerator time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a2083cea310" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/kubecon-china-2026-capacity-recovered-from-the-same-hardware-8a2083cea310">KubeCon China 2026: Capacity Recovered From The Same Hardware</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Ingress to Inference Gateway: Gateway API and the Inference Extension]]></title>
            <link>https://medium.com/todea-engineering/from-ingress-to-inference-gateway-gateway-api-and-the-inference-extension-e7cde870b00e?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7cde870b00e</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Thu, 27 Aug 2026 06:45:06 GMT</pubDate>
            <atom:updated>2026-08-27T06:45:06.816Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfA9JG6EkM1wiI5VllxHUw.png" /></figure><p>Traditional load balancing assumes all backends are the same, so it does not matter which one handles a request. LLM traffic changes this. For example, one vLLM replica might have a warm KV cache for an ongoing conversation, while another would need to start from scratch. Some requests are short questions, while others are huge codebases. Round-robin balancers cannot tell the difference, so requests may go to busy replicas while others with cached work sit idle. This leads to higher latency for users and wasted GPU resources.</p><p>Kubernetes addresses this with two layers. The Gateway API is replacing Ingress as the main way traffic enters a cluster. On top of that, the Gateway API Inference Extension, developed by the same community (SIG-Network together with WG-Serving), adds a set of APIs and a protocol. This lets a gateway route traffic to model servers based on factors like queue depth, cache state, and loaded adapters, rather than treating them as generic HTTP backends.</p><h3>From Ingress to Gateway API</h3><p>Before we get into inference, let’s quickly go over what the Gateway API is and why it matters for Kubernetes networking today. The main difference from Ingress controllers is how it supports collaboration. With Ingress, there’s nothing in the API to prevent multiple teams from using the same hostname. Kubernetes will accept these conflicting objects, and what happens next depends on the controller. For example, <strong>ingress-nginx</strong> merges all Ingress resources that use the same host, and if there are duplicate paths, the oldest rule takes priority. This means one team could add routes or settings to a hostname that another team thinks they own. There’s also the issue of namespace binding: an Ingress can only use a TLS secret from its own namespace. So, platform engineers either have to copy certificates into every application namespace, which risks exposing private keys, or each team has to manage their own certificate lifecycle. Finally, the Ingress spec doesn’t cover things like timeouts, authentication, rate limits, rewrites, or canaries. These features are usually handled with vendor-specific annotations at different layers, like application, infrastructure, or operations.</p><p>The SIG-Network team <a href="https://www.youtube.com/watch?v=cduG0FrjdJA">proposed the Gateway API at KubeCon San Diego 2019</a> to replace the confusing mix of annotations with a typed, role-aware, and extensible API. It reached general availability in October 2023. However, the real turning point was when the widely used NGINX Ingress Community controller was retired. With <a href="https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/">no releases, no fixes, no security patches</a> expected, teams worldwide realized that moving to the Gateway API was no longer optional, but an urgent migration they needed to plan and complete.</p><p>The Gateway API is designed to break up the overloaded Ingress object by dividing responsibilities based on who owns each decision:</p><ul><li>A GatewayClass declares a kind of load balancer the platform offers (internal L7, external L7, a vendor’s implementation).</li><li>A Gateway instantiates one; creating it is what provisions the listener, the hostname, TLS, and so on.</li><li>HTTPRoute objects, owned by application teams in their own namespaces, attach to that Gateway and declare “requests matching this go to my backend.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSfvJ-BzH7M5LDY536x0Sg.avif" /></figure><h3>How a gateway becomes an inference gateway</h3><p>At its core, the Gateway API inference extension is a set of new CRDs and controllers built on top of Envoy’s external processing filter (<a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter">ext-proc</a>).</p><p>Every incoming request passes through Envoy’s chain of HTTP filters. When it reaches the ext-proc filter, Envoy opens a two-way gRPC stream with an external server that acts as the extension’s controller. Envoy sends the request’s headers, and, if the filter’s processing mode requires it, the body as it arrives. The external server then responds with instructions, and Envoy applies them, such as changing headers, modifying the body, adding metadata, or sending an immediate response. Because the processing logic runs on a separate server, it can be deployed, scaled, and updated independently, and it can be written in any language. The ext-proc contract is defined by Envoy, but it is not limited to Envoy. Other gateways that follow the same external-processing protocol include Envoy-based Istio, Agentgateway, and NGINX Gateway Fabric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMYjSxeH7BlXFzC1A-e5HQ.avif" /></figure><p>When a request reaches Envoy, normal route matching happens first. However, the matched HTTPRoute now points to an InferencePool instead of a regular Service. The InferencePool is a new CRD that lists the model server pods and the ext-proc server managing them. Envoy’s role ends there. The routing logic is handled by the ext-proc server, called the <strong>Endpoint Picker (EPP)</strong>, which collects metrics, scores the candidate pods running the serving engine, and decides which one should handle the request.</p><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: llama3-route<br>  namespace: llm<br>spec:<br>  parentRefs:<br>  - name: inference-gateway<br>  rules:<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /<br>    backendRefs:<br>    - group: inference.networking.k8s.io<br>      kind: InferencePool<br>      name: vllm-llama3-8b<br>---<br>apiVersion: inference.networking.k8s.io/v1<br>kind: InferencePool<br>metadata:<br>  name: vllm-llama3-8b<br>  namespace: llm<br>spec:<br>  selector:<br>    matchLabels:<br>      app: vllm-llama3-8b<br>  targetPorts:<br>  - number: 8000<br>  endpointPickerRef:<br>    name: vllm-llama3-8b-epp<br>    port:<br>      number: 9002<br>    failureMode: FailClose</pre><p>Pods that join a pool are expected to use the project’s <a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/docs/proposals/003-model-server-protocol/README.md">model server protocol</a>. This protocol is simple: it requires serving OpenAI-compatible Completions and Chat APIs and exposing Prometheus metrics like queue depth, running requests, and KV-cache utilization. The EPP uses these metrics to make routing decisions. As of now, vLLM, SGLang, and Triton’s TensorRT-LLM backend all support this, each with their own metric names.</p><h3>How the endpoint picker (llm-d-router) works</h3><p>As of 2026, the EPP codebase has moved to the llm-d project’s <a href="https://github.com/llm-d/llm-d-router">llm-d/llm-d-router</a> (previously called llm-d-inference-scheduler). The version in the Gateway API Inference Extension will soon be archived. For this article, we will focus on the llm-d-router EPP, which has seen significant improvements in its scheduling logic since the move.</p><h4>EPP and plugins</h4><p>The EPP is highly configurable and now supports more than 80 plugins across five categories: scheduling, request control, flow control, request handling, and data layer. These plugins can be enabled in the configuration and are loaded at startup. For example:</p><pre>kind: ConfigMap<br>apiVersion: v1<br>metadata:<br>  name: vllm-llama32-1b-epp<br>data:<br>  default-plugins.yaml: |<br>    apiVersion: llm-d.ai/v1alpha1<br>    kind: EndpointPickerConfig<br>    plugins:<br>    - type: queue-scorer<br>    - type: kv-cache-utilization-scorer<br>    - type: prefix-cache-scorer<br>    - type: metrics-data-source<br>      parameters:<br>        scheme: &quot;http&quot;<br>        path: &quot;/metrics&quot;<br>        insecureSkipVerify: true<br>    - type: core-metrics-extractor<br>    schedulingProfiles:<br>    - name: default<br>      plugins:<br>      - pluginRef: queue-scorer<br>        weight: 2<br>      - pluginRef: kv-cache-utilization-scorer<br>        weight: 2<br>      - pluginRef: prefix-cache-scorer<br>        weight: 3</pre><p>Besides the plugins you define in the configuration, the EPP also loads several default plugins. For example, it adds max-score-picker if a profile does not specify a picker, and it includes the OpenAI, Anthropic, and vLLM-HTTP parsers if no parser is set. It also loads global-strict-fairness-policy, static-usage-limit-policy, and others, including any plugins needed to provide data for those you have configured. For instance, the prefix-cache plugin needs a tokenized prompt, which the EPP creates at startup even if it is not defined in the ConfigMap.</p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1785950733.484222,&quot;caller&quot;:&quot;loader/configloader.go:164&quot;,&quot;msg&quot;:&quot;Instantiated all plugins and applied system defaults. Effective raw configuration&quot;,&quot;config&quot;:&quot;{Plugins: [{Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\&quot;insecureSkipVerify\&quot;:true,\&quot;path\&quot;:\&quot;/metrics\&quot;,\&quot;scheme\&quot;:\&quot;http\&quot;}} {Name: core-metrics-extractor, Type: core-metrics-extractor} {Name: single-profile-handler, Type: single-profile-handler} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: &lt;nil&gt;}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}&quot;}</pre><p>The EPP builds a dependency graph based on the data keys each plugin produces and uses. For every request, it runs the data-producing plugins in the correct order according to this graph.</p><h4>What the EPP does out of band</h4><p>Outside of the main request flow, the EPP sets up watches on the Kubernetes API for InferencePool, Pod, InferenceObjective, and InferenceModelRewrite resources. It scrapes metrics from each pod in the pool, emits its own metrics, and, when precise prefix-cache awareness is enabled, subscribes to each pod’s KV-cache event stream.</p><p>Precise prefix-cache awareness is especially useful because Prometheus metrics only show how full the KV-cache is, not which blocks are on which pod. If you subscribe to the serving engine notifications via a ZeroMQ PUB socket, you can receive updates about blocks (vLLM’s unit of KV-cache by default fixed to 16 tokens and identified by a chained hash of its contents and everything before it) being added or evicted. These updates are decoded by an engine adapter stored in the EPP’s memory. Remember, the serving engine must be started with KV-event publishing enabled. vLLM does not send these events by default, so each pod needs a flag like — kv-events-config ‘{“enable_kv_cache_events”: true, “publisher”: “zmq”, “endpoint”: “tcp://*:5557”}’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1lqL9t4VEq5pSKqAtovLA.avif" /></figure><h4>When a request arrives</h4><p>When a new request arrives at the EPP over the ext-proc stream, the EPP waits for the full body to arrive, buffering the chunks until the end-of-stream flag. This way, it has the complete JSON to process. Next, it selects a parser based on the URL path suffix. By default, three parsers are set up, each handling its own suffixes. If none match, the request is rejected:</p><ul><li>openai-parser claims completions, chat/completions, embeddings, responses, and their kind;</li><li>anthropic-parser claims messages and messages/count_tokens;</li><li>vllmhttp-parser claims inference/v1/generate.</li></ul><p>The selected parser converts the raw bytes into a typed request, including the model name, messages or prompt, and the stream flag, and stores this in a per-request state object.</p><pre>{&quot;caller&quot;:&quot;handlers/server.go:435&quot;,&quot;msg&quot;:&quot;EPP received request&quot;,&quot;x-request-id&quot;:&quot;a8a70838-...&quot;}<br>{&quot;caller&quot;:&quot;handlers/server.go:448&quot;,&quot;msg&quot;:&quot;Incoming body chunk&quot;,&quot;EoS&quot;:false}<br>{&quot;caller&quot;:&quot;handlers/server.go:448&quot;,&quot;msg&quot;:&quot;Incoming body chunk&quot;,&quot;EoS&quot;:true}<br>{&quot;caller&quot;:&quot;handlers/server.go:454&quot;,&quot;msg&quot;:&quot;decoding&quot;}</pre><p>One consequence of this design is memory: the EPP holds each request body fully in memory while it processes it, and it buffers non-streaming response bodies the same way, with no size cap of its own, so very large prompts cost gateway memory. Streaming responses are unaffected; the EPP forwards SSE chunks to the client as they arrive.</p><p>Next, the EPP looks for an InferenceModelRewrite resource matching the pool and the requested model, and applies its rules. The resource lets you define multiple weighted targets; think of it as HTTPRoute-style traffic splitting, but for models. For example:</p><pre>apiVersion: llm-d.ai/v1alpha2<br>kind: InferenceModelRewrite<br>metadata:<br>  name: canary-model-split<br>spec:<br>  poolRef:<br>    name: production-llm-pool<br>  rules:<br>  - matches:<br>    - model:<br>        value: &quot;llama3&quot;  <br>    targets:<br>    - modelRewrite: &quot;llama3-stable&quot;<br>      weight: 90<br>    - modelRewrite: &quot;llama3-canary&quot;<br>      weight: 10</pre><p>In case of match, the EPP changes the model field in the request body to the chosen target and re-serializes the body. On the way back, it replaces the model name in the response body with the original name, so the client never sees the rewrite. In the logs, incomingModelName is what the client requested, and targetModelName is what is actually used after any rule is applied:</p><pre>{&quot;caller&quot;:&quot;requestcontrol/director.go:220&quot;,&quot;msg&quot;:&quot;No associated InferenceObjective found, using default&quot;,&quot;objectiveKey&quot;:&quot;&quot;}<br>{&quot;caller&quot;:&quot;requestcontrol/director.go:281&quot;,&quot;msg&quot;:&quot;LLM request assembled&quot;,&quot;incomingModelName&quot;:&quot;meta-llama/Llama-3.2-1B-Instruct&quot;,&quot;targetModelName&quot;:&quot;meta-llama/Llama-3.2-1B-Instruct&quot;,&quot;priority&quot;:0}</pre><p>Next, before the EPP performs tasks such as tokenization, hashing, or scoring, the default admission controller checks if the pool is saturated, and rejects requests with a priority below 0, set by an InferenceObjective with 429 status code. Requests with priority 0 or higher, which is the default if no `InferenceObjective` matches, always get through this check.</p><p><strong>Note</strong>: This accept-or-reject behavior is just the default. With the experimental flow-control feature enabled, requests are queued by priority band and handled fairly within each band. They are dispatched in order of priority as capacity becomes available. The saturation signal is also a plugin, called `utilization-detector` by default, and its thresholds can be configured.</p><pre># PASS<br>{&quot;level&quot;:&quot;trace&quot;,&quot;caller&quot;:&quot;requestcontrol/admission.go:116&quot;,&quot;msg&quot;:&quot;Executing LegacyAdmissionController&quot;,&quot;objectiveKey&quot;:&quot;sheddable-batch&quot;,&quot;priority&quot;:-1,&quot;fairnessID&quot;:&quot;default-flow&quot;}<br>{&quot;level&quot;:&quot;trace&quot;,&quot;caller&quot;:&quot;requestcontrol/admission.go:127&quot;,&quot;msg&quot;:&quot;Request admitted&quot;,&quot;requestID&quot;:&quot;4eb39a51-...&quot;}<br>Rejection, mid-burst. Probes 1–5 got HTTP 429, and each produced this pair:<br> <br># DROPPED<br>{&quot;level&quot;:&quot;trace&quot;,&quot;caller&quot;:&quot;requestcontrol/admission.go:76&quot;,&quot;msg&quot;:&quot;Request rejected: system saturated and request is sheddable&quot;,&quot;x-request-id&quot;:&quot;98bbfafe-...&quot;,&quot;objectiveKey&quot;:&quot;sheddable-batch&quot;,&quot;priority&quot;:-1}<br>{&quot;level&quot;:&quot;error&quot;,&quot;caller&quot;:&quot;handlers/server.go:478&quot;,&quot;msg&quot;:&quot;Error handling request&quot;,&quot;error&quot;:&quot;inference error: ResourceExhausted - system saturated, sheddable request dropped&quot;}</pre><p>If the request is admitted, the EPP evaluates the pods in the pool. It first takes a snapshot of all ready pods, then runs each plugin in order based on the configuration and dependency graph. With the previous ConfigMap, the process starts with tokenization: the EPP splits the prompt into 4-byte pseudo-tokens, groups them into blocks, and hashes each block together with all previous blocks. This way, each block’s hash represents the entire prefix up to that point, using the same chaining method as vLLM.</p><p><strong>Note:</strong> A different configuration can change or skip this phase. If you use the precise prefix-cache plugin, the estimate becomes a lookup in the KV-event index from the out-of-band section. If you remove the prefix scorer entirely, nothing happens in this step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFTeyP0AEH7dSZ5xvBFMlA.avif" /></figure><p>The EPP then checks these block hashes in order against its index. It stops at the first block no pod is recorded for, counts how many blocks the index attributes to each pod, and records these counts for each candidate pod.</p><p>Then the scheduler takes over. The profile handler picks which scheduling profile applies (single-profile-handler always answers default), and the profile’s filters prune the snapshot:</p><pre>{&quot;caller&quot;:&quot;scheduling/scheduler.go:69&quot;,&quot;msg&quot;:&quot;Running profile handler, Pick profiles&quot;,&quot;plugin&quot;:&quot;single-profile-handler/...&quot;}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:162&quot;,&quot;msg&quot;:&quot;Completed running filter plugins&quot;,&quot;remainingEndpoints&quot;:2}</pre><p>Each scorer then assigns a score between 0 and 1 to every remaining pod. For example:</p><ul><li>queue-scorer: ranks pods by waiting queue relative to each other, with the shortest queue in the snapshot getting 1, the longest getting 0, and if all queues are equal, everyone gets 1;</li><li>kv-cache-utilization-scorer: rewards free cache space; a pod reporting 83% KV-cache usage scores 0.17, and an idle one scores 1;</li><li>prefix-cache-scorer: rewards cached work with the share of this prompt’s blocks the pod already holds, so 40 of 50 blocks scores 0.8, none scores 0.</li></ul><pre>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;queue-scorer/...&quot;,&quot;endpoint&quot;:&quot;...8p55s...&quot;,&quot;score&quot;:1}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;queue-scorer/...&quot;,&quot;endpoint&quot;:&quot;...4r286...&quot;,&quot;score&quot;:1}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;kv-cache-utilization-scorer/...&quot;,&quot;endpoint&quot;:&quot;...8p55s...&quot;,&quot;score&quot;:1}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;kv-cache-utilization-scorer/...&quot;,&quot;endpoint&quot;:&quot;...4r286...&quot;,&quot;score&quot;:1}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;prefix-cache-scorer/...&quot;,&quot;endpoint&quot;:&quot;...8p55s...&quot;,&quot;score&quot;:0}<br>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:210&quot;,&quot;msg&quot;:&quot;Calculated score&quot;,&quot;plugin&quot;:&quot;prefix-cache-scorer/...&quot;,&quot;endpoint&quot;:&quot;...4r286...&quot;,&quot;score&quot;:0}</pre><p>Each score is multiplied by its weight from the ConfigMap and then summed. The picker, which is max-score-picker in this example, shuffles the candidates to break ties randomly and selects the one with the highest total score.</p><pre>{&quot;caller&quot;:&quot;scheduling/scheduler_profile.go:303&quot;,&quot;msg&quot;:&quot;Candidate pods for picking&quot;,&quot;endpoints-weighted-score&quot;:[{&quot;...8p55s...&quot;,&quot;Score&quot;:4},{&quot;...4r286...&quot;,&quot;Score&quot;:4}]}<br>{&quot;caller&quot;:&quot;maxscore/picker.go:88&quot;,&quot;msg&quot;:&quot;Selecting endpoints from candidates sorted by max score&quot;,&quot;max-num-of-endpoints&quot;:1,&quot;num-of-candidates&quot;:2}<br>{&quot;caller&quot;:&quot;requestcontrol/director.go:488&quot;,&quot;msg&quot;:&quot;Request handled&quot;,&quot;endpoint&quot;:&quot;10.20.0.2:8000&quot;}</pre><p>The EPP sends the chosen pod’s address to Envoy in the x-gateway-destination-endpoint header and in dynamic metadata with the same key. Envoy then forwards the request body to that pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UbCD-__pNNRqr6HSgMREkg.avif" /></figure><h3>Fine-Tuning , LoRA adapters and the Inference Extension</h3><p>More companies are now running inference on their own infrastructure, using open-weight models instead of relying only on services like Claude or ChatGPT. This approach offers practical benefits: lower costs at scale, more control over data, and the ability to adapt general-purpose models to specific company needs without having to fully fine-tune and deploy a separate model for each use case. Projects like Meta’s llama-cookbook, Microsoft’s LoRA, and Hugging Face’s PEFT have made this kind of specialization much more accessible, so regular engineering teams can now use techniques that once required a lot of compute, data, and ML expertise.</p><h4>What is LoRA</h4><p>The main idea behind LoRA is straightforward. Instead of updating every weight in a large model, you freeze the base model and train only small pairs of low-rank matrices, called “adapters,” which are attached to selected layers for a specific use case. This usually creates artifacts that are tens to hundreds of megabytes, not gigabytes. At each targeted layer, the input goes through both the frozen original weights and the adapter’s low-rank path, and the outputs are added together to produce the specialized result.</p><p>This method avoids having to create a separate multi-gigabyte copy of the model for each specialization. During inference, the adapter can be merged into the base weights or kept separate. This lets a LoRA-aware serving engine like vLLM dynamically apply different adapters on top of a single shared base model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9LHqMnsWB_dnn_wlN1EhA.avif" /></figure><p>When the serving engine starts, vLLM sets up memory for LoRA adapters using several configuration options. For example, --max-loras sets how many adapters can be used in the same GPU batch, and --max-cpu-loras sets how many adapter weights can stay cached in host memory and be moved to the GPU as needed. The adapters are registered with --lora-modules, so a request can select one through the model field while still sharing the same loaded base model.</p><pre>spec:<br>  template:<br>    spec:<br>      containers:<br>      - name: vllm<br>        image: vllm/vllm-openai:latest<br>        args:<br>        - --model<br>        - Qwen/Qwen2.5-3B-Instruct<br>        - --port<br>        - &quot;8000&quot;<br>        - --max-model-len<br>        - &quot;2048&quot;<br>        - --gpu-memory-utilization<br>        - &quot;0.85&quot;<br>        - --max-num-seqs<br>        - &quot;16&quot;<br>        - --enable-lora<br>        - --max-loras<br>        - &quot;2&quot;<br>        - --max-cpu-loras<br>        - &quot;3&quot;<br>        - --max-lora-rank<br>        - &quot;32&quot;<br>        - --lora-modules<br>        - medical=calebking/qwen2.5-3b-instruct-medical-lora<br>        - korean-law=kim0924/qwen2.5-3b-korean-law-lora<br>        - reasoning=namanadep/qwen2.5-3b-instruct-reasoning-lora-bf16</pre><h4>EPP and LoRA</h4><p>When you send a request for a LoRA adapter, the start of the EPP flow stays the same. The request goes to Envoy, is passed to the EPP over the ext-proc stream, and the OpenAI parser extracts the model from the model field and applies any matching InferenceModelRewrite, just as before.</p><pre>curl -s http://35.216.6.64/v1/chat/completions -H &#39;Content-Type: application/json&#39; -d &#39;{&quot;model&quot;:&quot;korean-law&quot;,&quot;messages&quot;:[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:&quot;전세 계약이 무엇인가요?&quot;}],&quot;max_tokens&quot;:64}&#39;</pre><p>The difference comes when the EPP evaluates the endpoints in the InferencePool. Not every replica can serve a LoRA adapter request equally well. One replica might already have the korean-law adapter active, another might have space to load it, and another might be at its LoRA capacity. The EPP uses the lora_requests_info metric from the serving engine to find out which adapters are active or waiting on each replica, and the maximum number a replica can hold.</p><pre>{&quot;msg&quot;:&quot;Refreshed metrics&quot;, &quot;ts&quot;:1787719680.300, &quot;endpoint&quot;:{&quot;name&quot;:&quot;vllm-qwen25-3b-lora-6888d54787-wjcb4-rank-0&quot;}, &quot;metrics&quot;:[&quot;…&quot;, &quot;vllm:lora_requests_info&quot;, &quot;…&quot;], &quot;updated&quot;:&quot;{ActiveModels:map[korean-law:0] WaitingModels:map[korean-law:0] MaxActiveModels:2 …}&quot;}</pre><p>To do this, the lora-affinity-scorer plugin has to be added to the EndpointPickerConfig. Once it is set up, the plugin assigns a score to each candidate endpoint:</p><ul><li>1.0 the adapter is already active on this endpoint</li><li>0.8 a free adapter slot: active + waiting &lt; max_lora</li><li>0.6 the adapter is already queued to load on this endpoint</li><li>0.0 endpoint full: the request would wait for a slot to free</li></ul><p><strong>Note:</strong> This does not replace the other scheduling signals. The scheduler still considers queue depth, KV-cache utilization, prefix-cache locality, and other configured scorers. LoRA affinity just adds an extra score that shows how suitable each replica is for serving the requested adapter.</p><p>After the scorers have run, nothing new happens. Their values are multiplied by the configured weights and summed; the picker selects the winning endpoint, and the EPP returns that pod’s address to Envoy in x-gateway-destination-endpoint.</p><h3>One gateway, many models</h3><p>According to the <a href="https://www.datadoghq.com/state-of-ai-engineering/">State of AI Engineering 2026</a>, using multiple models is now standard. Across thousands of organizations running AI in production, more than 70 percent use three or more models, and the share running more than six has nearly doubled in a year. Teams are adding models instead of replacing them. In the same cluster, you might have one gateway in front of a Llama pool, a Qwen pool, and a Mistral pool, each focused on different tasks.</p><p>However, routing is more complicated because the OpenAI API schema puts the model name in the request body, and a body field is not something gateways can normally route on: they usually match only paths and headers. <strong>Body-based routing</strong> solves this problem. This extension extracts the model name from the JSON body and sets it as the X-Gateway-Model-Name header, which HTTPRoute can match like any other header. You can then use normal HTTPRoutes to route to the right InferencePool. For example:</p><pre>rules:<br>- matches:<br>  - path:<br>      type: PathPrefix<br>      value: /<br>    headers:<br>    - type: Exact<br>      name: X-Gateway-Model-Name<br>      value: meta-llama/Llama-3.1-8B-Instruct<br>  backendRefs:<br>  - group: inference.networking.k8s.io<br>    kind: InferencePool<br>    name: vllm-llama3-8b</pre><h3>Operational reality</h3><ul><li><strong>It is best to use the llm-d router, even though the extension still includes its own.</strong> The Gateway API Inference Extension still has a working endpoint picker, and nothing will break if you keep using it. However, the code has moved to <a href="https://github.com/llm-d/llm-d-router">llm-d/llm-d-router</a>, which is where new scheduling work is happening, and the old version will be archived. Point new pools to the llm-d EPP from the start, and treat switching an existing pool as a planned migration, not just an image update, since the plugin configuration does not transfer directly.</li><li><strong>You may be running plugins you did not declare.</strong> No matter what your ConfigMap lists, the loader adds its own plugins on top, such as a picker for any profile without one, the parsers, the policies, and any plugins needed by the ones you requested. The actual configuration is only known at runtime. The EPP prints the effective configuration once at startup, and logs any data producers it auto-creates as separate “auto-created default producer” lines just after it. Capture both, save them, and compare after every upgrade.</li><li><strong>Default prefix scoring is only an estimate, not an exact lookup.</strong> Without KV-event tracking, the EPP cannot see inside a pod’s cache. It fingerprints the prompt in 4-byte pseudo-tokens and scores each pod based on how much of that fingerprint it thinks the pod has. This ranking is helpful but not guaranteed. Check the hit rate from the serving engine, not the scorer, and switch to precise tracking if the two do not match.</li></ul><h3>A practical recommendation</h3><p>If you already run several replicas of one model behind a plain Service, move that pool behind an `InferencePool` this week and let your own traffic guide the rest. You can pilot this in an afternoon: deploy the EPP with just queue-scorer and kv-cache-utilization-scorer, point one HTTPRoute at the pool, and keep the Service route so you can switch back easily. Replay your production prompt-length distribution against both setups and compare p95 TTFT, output tokens per second, and the engine’s prefill cache-hit rate. Add the prefix scorer in a second run, or you will not know which change made the difference. A prefix hit does not just speed up the cached part of prefill; it skips that computation entirely, and only the uncached tail of the prompt still runs prefill. On an H100 node that costs six figures a year, that difference matters. If you do not see an improvement in your traffic, do not adopt it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7cde870b00e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/from-ingress-to-inference-gateway-gateway-api-and-the-inference-extension-e7cde870b00e">From Ingress to Inference Gateway: Gateway API and the Inference Extension</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kyverno Explained: Why Modern Platforms Need Policy-as-Code]]></title>
            <link>https://medium.com/todea-engineering/kyverno-explained-why-modern-platforms-need-policy-as-code-e941ff76a6cf?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e941ff76a6cf</guid>
            <category><![CDATA[kyverno]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[policy-as-code]]></category>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[cncf]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 15:01:50 GMT</pubDate>
            <atom:updated>2026-07-24T15:01:50.339Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XC8hQ1rORwCK3Kzsoq_PZQ.avif" /></figure><p>Kyverno is a <a href="https://www.cncf.io/projects/kyverno/">CNCF-graduated</a> policy engine that validates, mutates, generates, and cleans up Kubernetes resources with policies written as Kubernetes YAML and CEL, with no new policy language required. Use it to enforce security and operational standards at admission time and in background scans; for a handful of simple validations, Kubernetes’ built-in ValidatingAdmissionPolicy may be enough.</p><p><a href="https://kubernetes.io/docs/concepts/security/pod-security-standards/">Running pods as root</a>. Deploying images with the latest tag everywhere, unpinned, unsigned, and not traced to a digest. Relying on a Bash script to bootstrap new namespaces with default NetworkPolicies, ResourceQuotas, and pull secrets, all but certain to fail on the next update. Spinning up costly EKS LoadBalancers for demos that linger, forgotten, draining budgets.</p><p>Platform teams used to handle this operational overhead manually, and each new team or cluster just added more technical debt. Kyverno helps by bringing all of this under one system.</p><p><a href="https://kyverno.io/">Kyverno</a> is an open-source, Kubernetes-native policy engine. It lets you write policies in YAML, and now also in CEL, which are checked when resources are created and regularly audited against running workloads. Companies like Coinbase, LinkedIn, and Spotify, as well as the U.S. Department of Defense, use Kyverno in production. Kyverno reached <a href="https://github.com/cncf/toc/issues/1967">CNCF Graduation</a> at KubeCon Amsterdam in March 2026.</p><h3>What is Kyverno, actually?</h3><p>Kyverno is an <a href="https://github.com/kyverno/kyverno">open-source</a> policy engine for Kubernetes that lets you write policies as Kubernetes resources. Unlike OPA or Gatekeeper, which require you to learn Rego before writing rules, Kyverno uses YAML and supports CEL, so you can work with the same API objects you already know.</p><p>Developers or SREs can write a ValidatingPolicy just like they write a Deployment, apply it with kubectl, and see violations as standard Kubernetes events and PolicyReport resources. There’s no need to learn a new language, query model, or way of thinking.</p><h3>How it works</h3><p>Kyverno is built on Kubernetes’ dynamic admission controller architecture, with a background scanner layered on top. Policies are evaluated at three points in a resource’s lifecycle:</p><ol><li><strong>Admission</strong>: When a developer, GitOps tool, or CI/CD pipeline submits a resource, the API server triggers admission webhooks. Kyverno’s webhook checks the request, applies matching policies, and returns allow, deny, or a mutated resource in the request path.</li><li><strong>Reconciliation</strong>: Background controllers watch for resources that should exist (cleanup policies, generation rules) and reconcile them in a loop, separate from admission.</li><li><strong>The new reality</strong>: Kyverno’s reporting system regularly scans existing resources against current policies on a schedule, surfacing violations in workloads that were created before a policy was in place or that changed later.</li></ol><p>Your application code stays the same. Every resource that goes through the API server is checked, mutated, and recorded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSnTOCkwVVTwfofc1Atohw.png" /></figure><h3>Policy types</h3><p>Kyverno’s CEL-native policy types were introduced incrementally, beginning with v1.14 and expanding through v1.17 with the addition of Namespaced versions. This brought the policies.kyverno.io group to a total of eleven CRDs (five cluster-scoped, five namespaced, plus PolicyException). Each type instructs the API server on how to handle the policy outcome:</p><ul><li>ValidatingPolicy: Validates resources against a CEL expression.</li></ul><pre>apiVersion: policies.kyverno.io/v1<br>kind: ValidatingPolicy<br>metadata:<br>  name: require-team-label<br>spec:<br>  validationActions: [Deny]<br>  matchConstraints:<br>    resourceRules:<br>      - apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        operations: [&quot;CREATE&quot;, &quot;UPDATE&quot;]<br>        resources: [&quot;pods&quot;]<br>  validations:<br>    - expression: &quot;has(object.metadata.labels) &amp;&amp; &#39;team&#39; in object.metadata.labels&quot;<br>      message: &quot;Pods must carry a &#39;team&#39; label.&quot;</pre><ul><li>MutatingPolicy: Mutates resources during admission using CEL-based applyConfiguration or jsonPatch expressions, similar to Kubernetes MutatingAdmissionPolicy.</li></ul><pre>apiVersion: policies.kyverno.io/v1<br>kind: MutatingPolicy<br>metadata:<br>  name: default-run-as-non-root<br>spec:<br>  matchConstraints:<br>    resourceRules:<br>      - apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        operations: [&quot;CREATE&quot;]<br>        resources: [&quot;pods&quot;]<br>  mutations:<br>    - patchType: ApplyConfiguration<br>      applyConfiguration:<br>        expression: &gt;<br>          Object{<br>            spec: Object.spec{<br>              securityContext: Object.spec.securityContext{<br>                runAsNonRoot: true<br>              }<br>            }<br>          }</pre><ul><li>GeneratingPolicy: Creates downstream resources when a trigger resource is created, namespace defaults, network policies, or RBAC bindings.</li></ul><pre>apiVersion: policies.kyverno.io/v1<br>kind: GeneratingPolicy<br>metadata:<br>  name: default-deny-on-namespace<br>spec:<br>  evaluation:<br>    synchronize:<br>      enabled: true<br>  matchConstraints:<br>    resourceRules:<br>      - apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        operations: [&quot;CREATE&quot;]<br>        resources: [&quot;namespaces&quot;]<br>  variables:<br>    - name: targetNs<br>      expression: &quot;object.metadata.name&quot;<br>    - name: downstream<br>      expression: &gt;-<br>        [<br>          {<br>            &quot;kind&quot;: dyn(&quot;NetworkPolicy&quot;),<br>            &quot;apiVersion&quot;: dyn(&quot;networking.k8s.io/v1&quot;),<br>            &quot;metadata&quot;: dyn({<br>              &quot;name&quot;: &quot;default-deny&quot;<br>            }),<br>            &quot;spec&quot;: dyn({<br>              &quot;podSelector&quot;: dyn({}),<br>              &quot;policyTypes&quot;: dyn([&quot;Ingress&quot;, &quot;Egress&quot;])<br>            })<br>          }<br>        ]<br>  generate:<br>    - expression: generator.Apply(variables.targetNs, variables.downstream)</pre><ul><li>ImageValidatingPolicy: Checks container images against trusted attestors like Cosign, Notary, or sigstore TUF, and supports in-toto attestations. SBOMs are also supported as one type of attestation payload.</li></ul><pre>apiVersion: policies.kyverno.io/v1<br>kind: ImageValidatingPolicy<br>metadata:<br>  name: require-signed-images<br>spec:<br>  validationActions: [Deny]<br>  webhookConfiguration:<br>    timeoutSeconds: 30<br>  failurePolicy: Fail<br>  matchConstraints:<br>    resourceRules:<br>      - apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        operations: [&quot;CREATE&quot;, &quot;UPDATE&quot;]<br>        resources: [&quot;pods&quot;]<br>  matchImageReferences:<br>    - glob: &quot;ghcr.io/kyverno/test-verify-image*&quot;<br>  attestors:<br>    - name: kyvernoCosign<br>      cosign:<br>        key:<br>          data: |<br>            -----BEGIN PUBLIC KEY-----<br>            MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM<br>            5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==<br>            -----END PUBLIC KEY-----<br>  validations:<br>    - expression: &gt;-<br>        images.containers.map(image, verifyImageSignatures(image, [attestors.kyvernoCosign])).all(e, e &gt; 0)<br>      message: &quot;Images from ghcr.io/kyverno/test-verify-image must be signed by the Kyverno project key.&quot;</pre><ul><li>DeletingPolicy: Removes resources that match a selector on a set schedule. This is helpful for temporary environments and cleanup tasks.</li></ul><pre>apiVersion: policies.kyverno.io/v1<br>kind: DeletingPolicy<br>metadata:<br>  name: cleanup-stale-ephemeral-pods<br>spec:<br>  schedule: &quot;0 2 * * *&quot;<br>  matchConstraints:<br>    resourceRules:<br>      - apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        operations: [&quot;*&quot;]<br>        resources: [&quot;pods&quot;]<br>        scope: &quot;Namespaced&quot;<br>    namespaceSelector:<br>      matchLabels:<br>        environment: test<br>  conditions:<br>    - name: olderThan48h<br>      expression: &quot;time.now() - timestamp(object.metadata.creationTimestamp) &gt; duration(&#39;48h&#39;)&quot;<br>    - name: ephemeralOrFinished<br>      expression: &gt;-<br>        (has(object.metadata.labels) &amp;&amp; &#39;ephemeral&#39; in object.metadata.labels &amp;&amp; object.metadata.labels.ephemeral == &#39;true&#39;) || (has(object.status) &amp;&amp; has(object.status.phase) &amp;&amp; (object.status.phase == &#39;Succeeded&#39;</pre><p>The Namespaced versions (NamespacedValidatingPolicy, NamespacedMutatingPolicy, NamespacedGeneratingPolicy, NamespacedImageValidatingPolicy, NamespacedDeletingPolicy) are important for multi-tenancy. Teams can create and manage policies in their own namespaces without needing cluster-admin rights, while platform owners use the cluster-scoped types for policies that must apply everywhere.</p><p>These CEL-based policies are about as mature as the ValidatingAdmissionPolicy in Kubernetes, with the API server handling more of the work. Kyverno adds features like a policy engine, reporting, exceptions, generation, and image validation, which VAP does not include.</p><p>The older ClusterPolicy and CleanupPolicy types still work for now, but they are being phased out. <a href="https://kyverno.io/blog/2026/02/02/announcing-kyverno-release-1.17/">Kyverno 1.17</a> marked them as deprecated, and they will be removed later in 2026. Plan to migrate to the CEL-native CRDs soon, and write new policies using these types unless you have a specific reason not to.</p><h3>When should you use ValidatingAdmissionPolicy instead?</h3><p>Kubernetes 1.30 and later includes <a href="https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/">ValidatingAdmissionPolicy</a> as a built-in CEL validator that doesn’t need a separate controller. For some teams, this is enough, but for most, it isn’t.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKHyBmo5GW19c8mPAe8q_g.png" /></figure><p>If you only need to validate resources at admission time, VAP is a lighter option and is the right choice. If you need more features, you’ll want a policy engine, and Kyverno offers the closest match between writing policies in YAML and treating them as code.</p><h3>Operational reality</h3><ul><li><strong>Rollout, not install.</strong> Starting Kyverno in audit mode is recommended. Use validationActions: [Audit] for CEL-native types or failureAction: Audit for legacy ClusterPolicy (legacy Enforce corresponds to Deny). Avoid deploying deny policies immediately, as this can disrupt controllers if their ServiceAccounts do not meet label requirements. Run in audit mode for at least a week, review PolicyReport resources, address violations, and then switch to deny.</li><li><strong>Mutation is not a security control.</strong> MutatingPolicy works well for setting defaults, image pull secrets, resource requests, and standard labels, but it’s not a replacement for validation. A user who can submit a pod spec might submit one that your mutation never processes, due to admission order or exemptions. Anything that must always be true should use a ValidatingPolicy.</li></ul><h3>A practical recommendation</h3><p>If you are evaluating a policy engine for a Kubernetes platform and you do not already have a strong reason to pick OPA/Gatekeeper, pilot Kyverno first. The install fits in an afternoon. If it does what you need, you just saved a quarter of operational work. If it doesn’t, you have a clear, documented reason to go somewhere else.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e941ff76a6cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/kyverno-explained-why-modern-platforms-need-policy-as-code-e941ff76a6cf">Kyverno Explained: Why Modern Platforms Need Policy-as-Code</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubecost Explained: Kubernetes FinOps That Moves the Bill]]></title>
            <link>https://medium.com/todea-engineering/kubecost-explained-kubernetes-finops-that-moves-the-bill-4a589badfdf2?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a589badfdf2</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 15:00:47 GMT</pubDate>
            <atom:updated>2026-07-24T15:00:47.312Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHKe1jrrnz-OiybPOEGGyQ.avif" /></figure><p>Kubecost is a Kubernetes cost-allocation and optimization platform, built on OpenCost, that joins cluster telemetry with cloud-provider billing data to tie spend to a real owner: a namespace, workload, or team. Deploy it when the bill needs owners, not estimates; its numbers only become trustworthy after cloud-billing reconciliation, and only an operating cadence turns them into savings.</p><p>Most platform teams are familiar with Kubernetes costs. The monthly cloud bill arrives, finance asks why it’s higher, and engineering can only respond with “more workloads.” This gap between what the bill shows and what platform teams can explain is exactly what FinOps aims to address to help optimize operational costs. The real question is whether you need an enterprise platform for this. For most teams, the answer is no. <a href="https://www.opencost.io/">OpenCost</a> and <a href="https://www.kubecost.com/">Kubecost</a> can give platform teams the visibility they need, as long as the tool is paired with an operating cadence.</p><h3>The pressure is real</h3><p>Kubernetes accounting is no longer just about a single cluster or a single cloud. Most teams now manage fleets of clusters, often across multiple cloud providers, and sometimes combine on-premises control planes with managed services. Containers move between nodes, nodes move between zones, and the same workload might run in several regions to meet latency or compliance needs, making the cost attribution even harder.</p><p>Industry data has shown the same problem for years. Back in 2021, a <a href="https://www.cncf.io/blog/2021/06/29/finops-for-kubernetes-insufficient-or-nonexistent-kubernetes-cost-monitoring-is-causing-overspend/">CNCF FinOps survey</a> found that most teams couldn’t reliably measure their Kubernetes spending, with over-provisioning and lack of accountability as the main issues. The same story happened in 2025, with the <a href="https://cast.ai/reports/kubernetes-cost-benchmark/">Cast AI 2025 Kubernetes Cost Benchmark</a> using real cluster data from over 2,100 organizations on AWS, GCP, and Azure, showing average CPU utilization at <strong>10%</strong> and memory utilization at <strong>23%</strong>. These numbers come from production telemetry, not just survey responses. The problem hasn’t changed in four years; if anything, it’s become clearer as fleets have grown.</p><p>The real issue isn’t that Kubernetes is too expensive. It’s that most teams don’t have enough visibility into what they’re spending.</p><h3>What is Kubecost, actually?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qv6eUcf--KE01YhSaJaYbg.avif" /></figure><p>Kubecost is a Kubernetes platform for cost allocation, optimization, and governance. After IBM acquired it in 2024, Kubecost now offers both open-source and enterprise versions. At its core, it sits on top of <strong>OpenCost</strong>, and runs as an in-cluster agent stack with several microservices: data collectors, a cloud-cost ingestor, a forecasting service, a per-node network-costs DaemonSet, and a fast ClickHouse-backed aggregator. By combining Kubernetes telemetry with cloud-provider billing data, Kubecost provides detailed cost allocation views, and accuracy improves when cloud billing integrations are configured and reconciled.</p><p>The platform is built around three primary pillars:</p><ul><li><strong>Cost allocation</strong> rolls spend up by namespace, label, service, workload, or <em>Collection</em>. Kubecost v3 adds a grouping concept that bundles Kubernetes and external cloud-side costs into a single deduplicated unit. Costs are tracked across CPU, memory, persistent volumes, GPUs, and network traffic.</li><li><strong>Optimization recommendations</strong> suggest right-sized requests based on real usage, propose cheaper node types, and let teams configure quantile-based controls instead of accepting a one-size default. Recommendations can be archived for historical reference and exported as CSV or PDF.</li><li><strong>Alerts and governance</strong> ship as configurable budget actions, scheduled reports, and Slack/email notifications. Alerts live next to the budget they belong to rather than as a separate alerting subsystem.</li></ul><h3>How a Kubecost install flows</h3><p>Cost data collection begins inside your clusters. When it starts, the <strong>FinOps agent</strong> sets up a watch on the Kubernetes API for the pricing ConfigMap, so any custom pricing rules can be applied without a restart. It also resolves node pricing per node (falling back to a computed value when the cloud provider’s price is unavailable). It then collects metrics from each <strong>Network Costs</strong> DaemonSet pod, creates a new binary snapshot, and, if configured, writes it to external storage such as Azure Blob Storage or AWS S3.</p><p>The <strong>Network Costs</strong> DaemonSet subscribes to the kernel’s conntrack table via a netlink socket, parsing each flow per-direction byte and packet counters, and maintains an in-memory map of pod, node, service, and endpoint state via Kubernetes API watches. It uses this map to link observed connections to specific workloads.</p><p>While these agents send internal snapshots to shared storage, the <strong>Cloud Cost Ingestor</strong> manages external financial data. It runs on a schedule, connects to cloud provider billing exports, pulls daily CSVs, and backfills historical data. Because cloud providers release billing data with a several-hour delay, Kubecost reconciles cluster data with cloud billing after a short wait. This means the most recent day or two of cost data is only an estimate, while older data is fully reconciled (assuming a working cloud-billing integration).</p><p>The <strong>Aggregator</strong> is the main engine and uses an embedded ClickHouse database. In a multi-cluster deployment, it fans in snapshots from several agent clusters. It checks multiple ConfigMaps for configuration and falls back to defaults when none are present. It ingests agent snapshots and, when configured, external billing CSVs, then drives them through a multi-stage SQL pipeline that reconciles and de-duplicates overlapping costs and produces the final cost tables that other microservices consume. The Aggregator also manages data retention by setting per-table, per-resolution TTLs in ClickHouse, so fine-grained windows expire within days while rollups are kept for weeks or months.</p><p>Finally, the <strong>Forecasting Service</strong> serves as a predictive cost-monitoring tool, using this data to generate cost forecasts. At the same time, the <strong>Cluster Controller</strong> uses the Aggregator’s optimization insights to take actions, such as applying right-sizing recommendations directly in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRcfCIH6DfnBNNSWi-YVqg.png" /></figure><h3>Allocate to a real owner first</h3><p>Kubecost spreads node costs across the pods running on each node, typically weighted by resource requests, and rolls the result up by namespace, label, service, workload, or Collection, covering CPU, memory, PV, GPU, and network.</p><p>What makes this approach effective is good organization, not just technical setup. Each namespace should match a real owner, like a team, product, or department. When this mapping is in place, the allocation view shows which team is responsible for each cost, without needing a spreadsheet. Teams that already use labels like team, cost-center, or product can use these for the same purpose, and Collections help make label-based views easy to use.</p><p>Labels can change over time, namespaces can increase, and someone will eventually deploy into default. Reviewing the unlabeled bucket each week helps keep the data accurate and useful.</p><h3>Right-size requests against actual usage</h3><p>This is the lever that moves the bill the most. Often, developers set CPU and memory requests defensively, never revisit them, and the gap between request and use shows up directly on the invoice. The 10%-CPU / 23%-memory benchmark cited above is a useful authority anchor when finance asks for a number.</p><p>A pattern that reliably finds savings: plot <em>requested vs. actual</em> CPU and memory per workload over a few weeks, then walk each workload’s request down to what it actually uses. One practical case from a service mesh deployment had proxy sidecars set to 100 millicores each. The node could host roughly 200 pods on paper, but the scheduler exhausted allocatable CPU at around 90 pods because every pod carried a 100 millicores sidecar request on top of its own. After the request-rightsizing pass, pod density per node tripled with no application change and no node fleet change.</p><p>Kubecost 3.0 makes this loop tighter. Container Request Sizing Insights show usage visualizations directly in the UI, recommendations can be archived, CSV/PDF exports include labels, and quantile-based controls let you set tighter recommendation percentiles for predictable services and looser ones for bursty workloads. The enterprise tier adds an Automated Container Request Sizing UI that operates across clusters with custom profiles, suspension controls, audit history, and a comparison between recommended and realized savings; the open-source tier gets a free allowance up to 250 cores on EKS primary clusters.</p><p>Using a percentile-based recommendation policy is usually the most effective approach in production.</p><ul><li>Set the CPU request to the 90th percentile of actual CPU usage from the past week or month, and then add a safety margin. The Kubernetes VPA default is about 15% for CPU. Because CPU is time-shared, the kernel lets bursty workloads use extra capacity when it is available. Adding more padding for rare spikes usually just increases requests without much benefit.</li><li>Set the memory request to a high percentile of peak usage, and then add a safety margin. Memory is not time-shared, so going over the limit can cause OOM kills instead of graceful degradation. Aim for about the 90th percentile of peaks, then add a margin. The VPA default is about 20% for memory.</li></ul><p>These settings decide the QoS class. A pod is considered Guaranteed only if every container has CPU and memory requests set equal to their limits. If this is not the case for any container, the whole pod is treated as Burstable, or as BestEffort if no container sets any requests or limits. This setup works well for most workloads. Reserve the fully Guaranteed class for critical workloads with strict latency SLAs, where CPU and memory requests match their limits. In those cases, you might waste some headroom, but you get the best eviction protection and, if needed, exclusive CPU pinning.</p><p>If you want an additional feedback before automating any of this, you can run <a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler">VPA</a> in recommendation-only mode for several weeks, or use <a href="https://github.com/robusta-dev/krr">KRR</a> open-source. Comparing recommendations across KRR, VPA-recommendation-mode, and Kubecost is more reliable than trusting any single tool’s number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iaE-WOn1AOOofnT1IfV3Cw.avif" /></figure><h3>Capacity-versus-request as the North Star metric</h3><p>The ratio that tells you most about cluster efficiency is <strong>total pod requests ÷ total node allocatable capacity</strong>, across CPU and memory: how much of what you pay for is even claimed by a pod. It is also what Kubecost’s request right-sizing is built on. Kubecost ships with several targets against which recommendations are computed: Production 0.65, Development 0.80, High Availability 0.50 (Cluster Right-Sizing API). Below it you carry capacity nothing asks for; above it you’ve spent the headroom that cluster class should keep. Kubecost also picks the utilization it sizes against by context; development the trending 85th-percentile, production the 98th, HA the 99.9th; and only on a one-day window; longer windows use maximum usage. The often-quoted “85th percentile” is just the development one-day default, not a universal setting.</p><p>The ratio is what you watch; an autoscaler (Karpenter, Cluster Autoscaler) moves it, but only if requests are honest, which is why Kubecost’s request right-sizing sits upstream of any autoscaling story. The autoscaler reacts to requests; Kubecost tells you whether they reflect reality.</p><p>Recent days are directional by design: reconciliation needs a full day of billing data, so for a roughly 48-hour window costs stay at public on-demand pricing unless a node is provably not on-demand; Spot is accurate sooner only through a separately configured AWS Spot data feed (Cloud Billing Integrations). Read efficiency (independent of reconciled pricing) separately from cost. Node Group Sizing (formerly Cluster Right-Sizing, rebuilt in v3.0) turns this into an action: it analyzes in-cluster CPU, RAM, and GPU utilization against node capacity over a configurable window and recommends, per node group, changing the node count or switching the instance type. It runs from a preset profile or a custom metric (usage.max/p95/p85/avg or request.max/avg), with a target-utilization threshold per resource, never below average requested resources. It detects node groups by each provider’s standard label, so it works across EKS, AKS, and GKE without setup (<a href="https://docs.kubecost.com/">v3.x docs</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glF5-USdfd8mhqTceOoMLg.avif" /></figure><h3>Find the always-on workloads that don’t need to be</h3><p>Once you’ve handled allocation and right-sizing, look at workloads that run all day, every day, even when they don’t have to. In one platform team’s review, 31% of workloads used less than 25% CPU for almost the entire day, yet Kubernetes costs still went up by about 18% over the year. This happened because engineers spent a lot of time tuning capacity and dealing with alerts, and because each team set up its own autoscaling rules differently.</p><p>The triage falls into three buckets. Production services that are genuinely over-spec’d belong in the right-sizing loop above. Non-production environments (dev, integration, demo) rarely need to run on weekends or overnight; a scheduled scale-to-zero is the highest-ROI change in this category. Batch and stateless workloads with retry tolerance are candidates for Spot instances, which trade roughly a 90% discount for a two-minute interruption notice.</p><p>Kubecost helps you find underused workloads. With Kubecost 3.0’s Advanced Filters, you can quickly sort workloads by namespace, label, or service using AND/OR conditions right in the UI, instead of having to do it elsewhere.</p><h3>Track commitment coverage and utilization separately</h3><p>Reserved capacity and savings commitments are common sources of unnecessary cloud costs. Teams often either ignore them and pay full on-demand prices, or buy them and forget to check if they are being used, leaving discounts unused on resources that are no longer needed. There are two important metrics to watch, and they are easy to mix up:</p><ul><li><strong>Coverage</strong> means the portion of your regular usage that is protected by a commitment.</li><li><strong>Utilization</strong> is how much of your commitment you actually use.</li></ul><p>Each cloud provider has different tools, but they all fit into three main types, and the calculations work the same way everywhere:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iv1JsBL7nixUnd-6d7DNeg.png" /></figure><p>A good approach is to aim for commitment utilization between 80% and 95%, instead of trying to reach 100%. Going for 100% leaves no room for normal changes, like removing unused instances, changing instance types, or handling a drop in traffic. It may look efficient in a quarterly review, but it can cause problems day-to-day. For coverage, aiming for 60% to 75% is reasonable. This range is high enough to get a good discount, but low enough to allow for changes each quarter. These ranges are based on practical experience, not rules set by the cloud provider.</p><p>With Kubecost, costs are first estimated using public on-demand cloud provider prices until the actual cloud bill is ready. When the bill becomes available, usually within about 48 hours, Kubecost updates its estimates with the real costs. This update includes Reserved Instances, Savings Plans, committed-use discounts, and Spot pricing, along with any special rates you might have, such as Enterprise Discount Programs.</p><h3>When should you use a commercial FinOps platform instead?</h3><p>Most teams should start with the open-source chart. You can look at the commercial tiers later, once your needs grow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S_qaKGKZnboKRBDSOZJXUg.png" /></figure><p>If you just need per-namespace allocation, basic recommendations, and Slack alerts for a few clusters, the open-source version is enough. But if you manage many clusters across different clouds, need automated fixes, want vendor-managed history, or need to give your finance team detailed, reconciled discount numbers, then a commercial platform is worth it. The decision should be based on these needs, not just on how the dashboard looks.</p><h3>Operational reality</h3><ul><li><strong>ClickHouse and a unified agent replace the old stack.</strong> In v3, the 2.x DuckDB store is replaced with a ClickHouse database. This change makes allocation and cloud-cost API queries much faster and more reliable at scale. It also removes the need for Prometheus, which cuts down on memory use and makes deployment easier, while still providing OpenCost-standard metrics.</li><li><strong>History is a deliberate choice, not a default.</strong> Whatever the storage backend, the retention window is the upper bound on the period-over-period reporting you can produce. Monthly reporting requires at least 30 days; year-over-year requires a year. Tier cold data to object storage if on-cluster retention gets expensive faster than the engineering time to set up the tiered pipeline.</li><li><strong>Reconciliation lag is structural.</strong> The 24–48 hour billing-reconciliation delay is a property of cloud-provider billing exports, not of Kubecost. Build the operating model around it: argue about last week, not yesterday.</li><li><strong>Multi-cluster needs a story.</strong> Open-source Kubecost can federate across clusters, but the experience is rougher than the commercial multi-cluster aggregator. Beyond five or six clusters, decide early whether to run per-cluster Kubecost and aggregate externally (into your own warehouse, for example) or pay for the commercial multi-cluster path. Either is defensible; drifting between the two is not.</li><li><strong>The EKS add-on offers a quick way to get started.</strong> The Kubecost v3 free tier has a $100k USD spend limit over 30 days, while the Amazon EKS optimized Kubecost bundle is listed by AWS as exempt from that spend limit.</li><li><strong>The operating model is the deliverable.</strong> If a team installs Kubecost but does not set up regular reviews, they will drift just like a team without any FinOps tools. The standard approach is to have a small FinOps group, such as a platform engineer, a finance analyst, and an SRE on rotation, meet each week to review the capacity-versus-request ratio, identify the most over-provisioned workloads, and check any namespace with a significant change in monthly cost. For smaller teams, a 30-minute review every two weeks with the platform engineer and CTO can achieve the same results.</li></ul><h3>A practical recommendation</h3><p>If you are considering a FinOps approach for a Kubernetes platform and do not have a contractual obligation to choose a commercial product, start by piloting open-source Kubecost 3.x. Installation can be completed in an afternoon. Assign at least one namespace to a designated owner, provide a request-versus-usage dashboard to one team for two weeks, and share the capacity-versus-request ratio in a channel visible to the platform team. If regular reviews of these metrics become routine, you have achieved FinOps. If not, adopting a commercial platform will not resolve the underlying issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a589badfdf2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/kubecost-explained-kubernetes-finops-that-moves-the-bill-4a589badfdf2">Kubecost Explained: Kubernetes FinOps That Moves the Bill</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KubeCon India 2026: Sovereign AI On A Cloud-Native Stack]]></title>
            <link>https://medium.com/todea-engineering/kubecon-india-2026-sovereign-ai-on-a-cloud-native-stack-bd9ec9804c93?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd9ec9804c93</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 22:18:26 GMT</pubDate>
            <atom:updated>2026-07-14T22:18:26.298Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UTGKDSlWVrLVyU5ZYi3Hg.avif" /></figure><p><a href="https://www.cncf.io/kubecon-cloudnativecon-events/">KubeCon + CloudNativeCon India 2026</a> in Mumbai ran on one throughline: sovereign AI as an architectural need rather than a slogan, with population-scale platforms like Sarvam and NPCI running on the same cloud-native tools as commercial ones, and a recurring CNCF-native stack of Prometheus, Thanos, Argo CD, and vLLM. Here is what the keynotes actually showed and which patterns travel.</p><p>A few weeks back, we covered <a href="https://todea.co.kr/en/blog/aws-summit-2026-seoul">AWS Summit Seoul 2026 </a>and how Korean companies are bringing agentic AI into production. At KubeCon + CloudNativeCon India 2026, a similar theme emerged, but with unique challenges. India’s vast population and 22 official languages mean serving citizens can involve answering a farmer in Marathi or handling a loan question in Hindi, often over the phone instead of through an app. This diversity creates a different engineering challenge, which shaped the focus of the Mumbai keynotes.</p><h3>The throughline the keynotes shared</h3><p>The word ‘sovereign’ came up often in the KubeCon India keynotes, and two talks highlighted it directly. Sarvam discussed building sovereign AI for all of India, while the National Payments Corporation of India (NPCI) emphasized that cloud-native technologies are now essential infrastructure for large-scale, sovereign AI. This trend isn’t limited to India. At Naver Cloud’s DAN25 conference in Seoul in November 2025, CEO Kim Yuwon and others also focused on sovereign AI, describing it as vertical models built by companies that understand their own country’s language, data, and industry. Seeing this idea appear independently in both Seoul and Mumbai shows that sovereign AI is now an architectural need, not just a slogan.</p><p>Scale was expected at this event, but it wasn’t the main theme. Every keynote addressed large-scale operations, though not all in the same way. Sarvam and NPCI focused on serving the entire population, while Flipkart, Rapido, and JioHotstar represented big commercial platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpQVYS7YrN4Lu9jAe2LRyw.avif" /></figure><p>Sarvam’s approach to sovereign AI is rooted in policy. It is one of twelve organizations chosen under the Innovation Center pillar of the IndiaAI Mission, receiving ₹246.72 crore in funding and computing resources. Sarvam is building core language and speech models like Bulbul (text-to-speech for 11 Indian languages with 39 voices) and Saaras (speech-to-text for all 22 scheduled languages).</p><h3>What the enterprise keynotes actually showed</h3><p>Sarvam’s Avi Kothari and Vinayak Gavariya described a conversational-AI platform that India needs to speak in 22 languages, from Marathi to a farmer confirming her crops, to Tamil for a citizen accessing a government scheme, to Hindi for a loan query. The scale they cited was millions of API calls and voice conversations daily, at sub-second latency, while remaining auditable, and they grounded it in a concrete deployment: a national crop-verification program that reached 17 million farmers in a single 24-day campaign, with a roadmap to 80 million farmers the following season. The stack was entirely <a href="https://landscape.cncf.io/">CNCF</a>: Linkerd for a zero-trust service mesh, Strimzi-managed Kafka for event streaming, Argo CD and Helm for GitOps delivery across roughly 20 components, Prometheus and Thanos for unified multi-cluster observability, and CloudNativePG as the database of record.</p><p>Tittu Varghese from NPCI highlighted what population-scale means for payments. NPCI processes over 24 billion transactions per month and 700 million payments per day. They also run real-time fraud detection at sub-second speeds using Kubernetes, Kubeflow, Prometheus, vLLM, OpenStack, and NVIDIA AI platforms. This was the clearest example of AI inference being part of essential national infrastructure.</p><p>Flipkart’s session focused on reliability rather than AI inference, with its Reliability Engineering team building a centralized, multi-tenant chaos-engineering platform using the LitmusChaos project to strengthen hundreds of microservices. They customized Litmus in four ways: a hybrid multi-tenancy setup, a DaemonSet-based high-availability model for chaos injection, a Script Runner for dynamic target selection, and a hybrid extension for VM workloads outside Kubernetes. Their efforts were recognized on stage by winning the CNCF End User Case Study Contest. They moved about 90% of the chaos experiments into staging clusters and contributed 5 fixes upstream, including improvements to database index and duplicate-name validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Amx3El5M5O6fDXCPG27lLQ.avif" /></figure><p>Rapido and JioHotstar provided more examples at the consumer scale. Rapido’s CTO, Srivatsa Katta, and Adarsh K Kumar talked about India’s largest taxi platform, which handles over 4 million rides daily by matching riders and drivers in real time. Their system runs on more than 150 microservices and processes 200,000 requests per second using Kubernetes, Istio, Prometheus, Thanos, Cert Manager, and External DNS. Pradeep Bishnoi from JioHotstar explained how they stream live cricket to 50–60 million users at once during big events, using Kubernetes, Amazon EKS, OpenTelemetry, Karpenter, and Envoy Gateway across many clusters worldwide. He also previewed new AI-driven features, like conversational Kubernetes troubleshooting and smarter SRE automation.</p><h3>The GPU question kept coming up</h3><p>Another key theme was that the main challenge in AI workloads is not getting enough GPUs, but using them efficiently. Saiyam Pathak from vCluster explained that while containers, autoscaling, multi-tenancy, and RBAC are solved, Kubernetes still requires whole GPUs even if a workload only needs 10% of one. His solution is HAMi, which splits a physical GPU into memory-based slices, and DRA, which lets Kubernetes treat GPUs as more than just simple counts. He showed this live, with a MacBook controlling an NVIDIA DGX Spark, where a single Blackwell GPU ran two open-source LLMs for two teams simultaneously. In another session, Ravindra Patil from Red Hat addressed the same issue at the routing layer with llm-d. He compared it to a basic round-robin balancer that spreads repeated prompts across pods, causing unnecessary cache reloads. Instead, llm-d sends each request to a pod that already has the needed cache (cache-aware) or to the least-loaded pod if starting fresh (load-aware), using a pipeline of scrapers, filters, and scorers to pick the best target. This approach led to fewer redundant cache loads and better P95/P99 latency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AYVAQr4fV8kNbKBxt2rWQ.avif" /></figure><h3>The CNCF-native stack that recurred</h3><p>Several projects appeared in different keynotes. Prometheus was mentioned in almost every talk, often with Thanos for multi-cluster observability. Argo CD was the main tool for GitOps delivery. vLLM handled inference, and the discussion about splitting GPUs (HAMi, DRA, KAI-Scheduler) echoed the Dynamic Resource Allocation theme from AWS Summit Seoul. Service meshes were widely used, with companies choosing between Linkerd and Istio.</p><h3>The themes worth taking home</h3><p>A few key patterns emerged. First, sovereignty has become a core architectural need, not just a buzzword, and this idea is emerging across regions, from Sarvam and NPCI in Mumbai to Naver’s DAN25 in Seoul. Second, when the stakes were truly population-scale, like Sarvam’s millions of farmers or NPCI’s 700 million daily payments, the same cloud-native tools used by Flipkart, Rapido, and JioHotstar for commercial traffic handled the load. Third, the GPU challenge has shifted from getting enough hardware to making better use of it, with solutions like HAMi, DRA, and KAI helping Kubernetes manage fractional, capability-aware GPUs. The main takeaway is that none of this relies on new infrastructure built just for AI; it’s about using the existing CNCF stack well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd9ec9804c93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/kubecon-india-2026-sovereign-ai-on-a-cloud-native-stack-bd9ec9804c93">KubeCon India 2026: Sovereign AI On A Cloud-Native Stack</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HAMi Explained: Why Your GPUs Sit Idle While Your Bill Doesn’t]]></title>
            <link>https://medium.com/todea-engineering/hami-explained-why-your-gpus-sit-idle-while-your-bill-doesnt-3b9b489c9780?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b9b489c9780</guid>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 16:45:43 GMT</pubDate>
            <atom:updated>2026-11-17T22:59:37.294Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQDfJhJ_ETJlLHLLMOoAYA.png" /></figure><p>Until Dynamic Resource Allocation arrived, Kubernetes could only hand out GPUs as whole cards. A pod that needs 4 GB of an 80 GB H100 still reserves the entire device for as long as it runs. The result is a familiar pattern: inference services idling on dedicated cards, dashboards showing every GPU allocated while nvidia-smi shows them mostly unused, and budget requests for more GPUs because the cluster looks full. DRA stops a device from being over-committed, but it does not enforce limits once a pod is running. That is the gap HAMi fills.</p><h3>What HAMi does</h3><p>HAMi is an open-source GPU virtualization layer for Kubernetes, now a CNCF Incubating project. A pod asks for an exact share of a card’s memory and compute (on NVIDIA, nvidia.com/gpumem and nvidia.com/gpucores next to the usual nvidia.com/gpu), and a library inside the container holds it to that share without changes to the application or image. Unlike time-slicing, which has no per-replica memory caps, HAMi refuses an allocation past the limit with an out-of-memory error and leaves the other tenants running. Unlike MIG, its boundary is software, not hardware: it keeps workloads within budget but cannot contain a driver reset or an XID error on the shared card.</p><h3>How a request flows</h3><p>Four components handle a shared-GPU request. The device plugin publishes each card’s capacity on the node. A mutating webhook hands the pod to HAMi’s scheduler. The scheduler extender picks a node and a slice and records it on the pod. At container start, the device plugin injects the limits and a preloaded library that enforces them and makes nvidia-smi report the slice instead of the whole card.</p><h3>Scheduling policies that move the bill</h3><p>HAMi offers binpack and spread at both the node and the GPU level. Node binpack lets the autoscaler drain and remove empty nodes, at the cost of a larger blast radius; node spread keeps every node warm. GPU binpack keeps whole cards free for large models but concentrates noisy neighbors; GPU spread suits latency-sensitive inference but fragments memory. Placement only happens when pods are scheduled, so the savings arrive as pods churn.</p><h3>HAMi and MIG together</h3><p>HAMi can also manage MIG, re-carving instances to fit incoming jobs, so one pool can serve both software slices and hardware-isolated instances.</p><h3>Before you roll it out</h3><p>HAMi replaces the NVIDIA device plugin, so disable the GPU Operator’s plugin on HAMi nodes and roll out pool by pool. Its scheduler image is pinned to your Kubernetes version at install time, so add it to your upgrade runbook. And gpucores is a ceiling, not a reservation: benchmark p99 latency under a noisy neighbor before committing to an SLO.</p><p>The full article covers each step in detail, with the architecture diagram, logs from a working cluster, the MIG trade-offs and the cost arithmetic for a pilot. <a href="https://todea.co.kr/en/blog/hami-explained">Read it on the Todea blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b9b489c9780" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/hami-explained-why-your-gpus-sit-idle-while-your-bill-doesnt-3b9b489c9780">HAMi Explained: Why Your GPUs Sit Idle While Your Bill Doesn’t</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Linkerd on OpenShift: SCCs, CNI, and the Edges That Bite]]></title>
            <link>https://medium.com/todea-engineering/running-linkerd-on-openshift-sccs-cni-and-the-edges-that-bite-29935ffc99da?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/29935ffc99da</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[linkerd]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Thu, 21 May 2026 07:21:00 GMT</pubDate>
            <atom:updated>2026-07-14T22:12:14.085Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CMnYGdUUhugteqFisLJ8Q.avif" /></figure><p>Installing Linkerd on a vanilla Kubernetes cluster is a ten-minute job. Installing it on OpenShift is not. The Helm chart is the same, the control plane is the same, the proxies are the same, and yet the first helm install you run against OpenShift will leave you with a linkerd namespace full of deployments that never scale past zero:</p><pre>oc get deployment -n linkerd<br>NAME                     READY   UP-TO-DATE   AVAILABLE   AGE<br>linkerd-destination      0/1     0            0           2m40s<br>linkerd-identity         0/1     0            0           2m40s<br>linkerd-proxy-injector   0/1     0            0           2m40s</pre><p>None of that is a bug. OpenShift’s security model is doing exactly what it was designed to do. The trick is knowing which parts of Linkerd collide with it, and which of the available fixes you actually want to live with.</p><h3>Why the vanilla install fails</h3><p>OpenShift gates every pod through Security Context Constraints (SCCs). An SCC is a cluster-scoped policy that decides which `securityContext` settings a pod is allowed to use: which UIDs it can run as, which Linux capabilities it can request, which volume types it can mount, whether it can access the host network.</p><pre>oc get scc<br>NAME                              PRIV    CAPS                   SELINUX     RUNASUSER          FSGROUP     SUPGROUP    PRIORITY     READONLYROOTFS   VOLUMES<br>anyuid                            false   &lt;no value&gt;             MustRunAs   RunAsAny           RunAsAny    RunAsAny    10           false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>hostaccess                        false   &lt;no value&gt;             MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;hostPath&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>hostmount-anyuid                  false   &lt;no value&gt;             MustRunAs   RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;hostPath&quot;,&quot;nfs&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>hostmount-anyuid-v2               false   &lt;no value&gt;             RunAsAny    RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;hostPath&quot;,&quot;nfs&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>hostnetwork                       false   &lt;no value&gt;             MustRunAs   MustRunAsRange     MustRunAs   MustRunAs   &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>hostnetwork-v2                    false   [&quot;NET_BIND_SERVICE&quot;]   MustRunAs   MustRunAsRange     MustRunAs   MustRunAs   &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>insights-runtime-extractor-scc    true    [&quot;CAP_SYS_ADMIN&quot;]      RunAsAny    RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;*&quot;]<br>machine-api-termination-handler   false   &lt;no value&gt;             MustRunAs   RunAsAny           MustRunAs   MustRunAs   &lt;no value&gt;   false            [&quot;downwardAPI&quot;,&quot;hostPath&quot;]<br>node-exporter                     true    &lt;no value&gt;             RunAsAny    RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;*&quot;]<br>nonroot                           false   &lt;no value&gt;             MustRunAs   MustRunAsNonRoot   RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>nonroot-v2                        false   [&quot;NET_BIND_SERVICE&quot;]   MustRunAs   MustRunAsNonRoot   RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>privileged                        true    [&quot;*&quot;]                  RunAsAny    RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;*&quot;]<br>privileged-genevalogging          true    [&quot;*&quot;]                  RunAsAny    RunAsAny           RunAsAny    RunAsAny    &lt;no value&gt;   false            [&quot;*&quot;]<br>restricted                        false   &lt;no value&gt;             MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]<br>restricted-v2                     false   [&quot;NET_BIND_SERVICE&quot;]   MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    &lt;no value&gt;   false            [&quot;configMap&quot;,&quot;csi&quot;,&quot;downwardAPI&quot;,&quot;emptyDir&quot;,&quot;ephemeral&quot;,&quot;persistentVolumeClaim&quot;,&quot;projected&quot;,&quot;secret&quot;]</pre><p>At admission, the plugin enumerates every SCC on the cluster, sorts them by priority, and for each one checks two things: whether the pod’s ServiceAccount has RBAC access to it, and whether the pod spec validates against its rules. The pod is admitted against the first SCC that passes both. If none do, it is rejected, and the event log lists every SCC it tried and why each one said no.</p><p>When the pod is rejected, the event log spells out exactly what the plugin tried: one line per SCC, with either Forbidden: not usable by user or serviceaccount (failed the RBAC check) or an Invalid value: … detail (passed RBAC, failed validation). Reading that list is how you debug.</p><pre>oc get events -n linkerd<br>LAST SEEN   TYPE      REASON         OBJECT                                         MESSAGE<br>7m47s       Warning   FailedCreate   replicaset/linkerd-destination-5fd5f7b7f7      Error creating: pods &quot;linkerd-destination-5fd5f7b7f7-&quot; is forbidden: unable to validate against any security context constraint: [provider &quot;anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 65534: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_ADMIN&quot;: capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_RAW&quot;: capability may not be added, provider restricted-v2: .containers[0].runAsUser: Invalid value: 2102: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .containers[1].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .containers[2].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .containers[3].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider &quot;restricted&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;machine-api-termination-handler&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostaccess&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;insights-runtime-extractor-scc&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;node-exporter&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged-genevalogging&quot;: Forbidden: not usable by user or serviceaccount]<br>106s        Warning   FailedCreate   job/linkerd-heartbeat-29446087                 Error creating: pods &quot;linkerd-heartbeat-29446087-&quot; is forbidden: unable to validate against any security context constraint: [provider &quot;anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider &quot;restricted&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;machine-api-termination-handler&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostaccess&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;insights-runtime-extractor-scc&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;node-exporter&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged-genevalogging&quot;: Forbidden: not usable by user or serviceaccount]<br>7m47s       Warning   FailedCreate   replicaset/linkerd-identity-688fff88b4         Error creating: pods &quot;linkerd-identity-688fff88b4-&quot; is forbidden: unable to validate against any security context constraint: [provider &quot;anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 65534: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_ADMIN&quot;: capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_RAW&quot;: capability may not be added, provider restricted-v2: .containers[0].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .containers[1].runAsUser: Invalid value: 2102: must be in the ranges: [1000750000, 1000759999], provider &quot;restricted&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;machine-api-termination-handler&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostaccess&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;insights-runtime-extractor-scc&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;node-exporter&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged-genevalogging&quot;: Forbidden: not usable by user or serviceaccount]<br>7m47s       Warning   FailedCreate   replicaset/linkerd-proxy-injector-5f654db4db   Error creating: pods &quot;linkerd-proxy-injector-5f654db4db-&quot; is forbidden: unable to validate against any security context constraint: [provider &quot;anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 65534: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_ADMIN&quot;: capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: &quot;NET_RAW&quot;: capability may not be added, provider restricted-v2: .containers[0].runAsUser: Invalid value: 2102: must be in the ranges: [1000750000, 1000759999], provider restricted-v2: .containers[1].runAsUser: Invalid value: 2103: must be in the ranges: [1000750000, 1000759999], provider &quot;restricted&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;nonroot&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostmount-anyuid-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;machine-api-termination-handler&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork-v2&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostnetwork&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;hostaccess&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;insights-runtime-extractor-scc&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;node-exporter&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged&quot;: Forbidden: not usable by user or serviceaccount, provider &quot;privileged-genevalogging&quot;: Forbidden: not usable by user or serviceaccount]</pre><p>Linkerd installs without any custom SCC bindings, so its ServiceAccounts inherit only the cluster-wide RBAC grant that every authenticated SA gets on OpenShift 4.11+: access to restricted-v2, and nothing else. The event log confirms this. Every other provider returns Forbidden: not usable by user or serviceaccount. Linkerd’s pods are validated against restricted-v2 alone, and they fail.</p><p>As its name suggests, restricted-v2 is strict:</p><ul><li>No added Linux capabilities.</li><li>Pods must run as a non-root UID <strong><em>within the project’s assigned UID range</em></strong>. Each OpenShift project gets its own range via the openshift.io/sa.scc.uid-range namespace annotation, and pods must fall inside it.</li><li>No host paths, no host network, no privilege escalation.</li></ul><p>The Linkerd pods violate this in two ways. The first hits every container in the install; the second only applies to linkerd-init:</p><ul><li><strong>The UIDs don’t fit the project range.</strong> restricted-v2 requires every UID to fall inside the project’s assigned range: [1000750000, 1000759999] in this example, a different window in every project. linkerd-init runs as 65534 (nobody), and the control-plane containers and proxy sidecar default to 2102 and 2103. None of those are in range, and chasing them through Helm isn’t a fix: ranges are reassigned whenever a namespace is recreated, and meshed application pods live in <strong><em>other </em></strong>projects with ranges of their own. No single UID works everywhere.</li><li><strong>linkerd-init needs capabilities </strong><strong>restricted-v2 won’t grant.</strong> Its whole job is rewriting iptables so the pod’s traffic is redirected through the sidecar, which takes NET_ADMIN and NET_RAW. Neither is on the allow-list.</li></ul><h3>Two ways out</h3><p>There are two supported paths. They solve the same problem from opposite ends, and the choice has real operational consequences.</p><h4>Path 1: Linkerd CNI (recommended)</h4><p>The Linkerd CNI plugin moves the iptables setup <strong>out of the pod</strong> and into a node-level CNI chain. The linkerd-cni DaemonSet drops a binary and a config into the node’s CNI directory, and Multus chains it into every pod’s CNI setup. The plugin is a no-op for pods that aren’t meshed. The pod itself no longer needs NET_ADMIN or NET_RAW.</p><p>The CNI DaemonSet still does privileged work on the node, so its ServiceAccount needs a SCC that allows it:</p><pre>apiVersion: security.openshift.io/v1<br>kind: SecurityContextConstraints<br>metadata:<br>  name: linkerd-cni-scc<br>allowPrivilegedContainer: true<br>allowPrivilegeEscalation: true<br>defaultAllowPrivilegeEscalation: true<br>allowHostNetwork: false<br>allowHostPorts: false<br>allowHostPID: false<br>allowHostIPC: false<br>allowHostDirVolumePlugin: true<br>volumes:<br>  - hostPath<br>  - configMap<br>  - projected<br>  - downwardAPI<br>  - emptyDir<br>seccompProfiles:<br>  - &#39;*&#39;<br>runAsUser:<br>  type: RunAsAny<br>seLinuxContext:<br>  type: RunAsAny<br>fsGroup:<br>  type: RunAsAny<br>supplementalGroups:<br>  type: RunAsAny<br>users:<br>  - system:serviceaccount:linkerd-cni:linkerd-cni</pre><p>One more OpenShift-specific detail: the CNI paths are not the upstream Kubernetes defaults. OpenShift puts CNI binaries under /var/lib/cni/bin and configs under /etc/kubernetes/cni/net.d, and Linkerd CNI has to drop its files in the right place or Multus will never see it. Set the Helm values accordingly at install time:</p><pre>helm install linkerd2-cni linkerd2-edge/linkerd2-cni \<br>  --namespace linkerd-cni \<br>  --set destCNIBinDir=/var/lib/cni/bin \<br>  --set destCNINetDir=/etc/kubernetes/cni/net.d \<br>  --set privileged=true</pre><p>Once the DaemonSet is healthy, install the control plane with --set cniEnabled=true. The linkerd-init container is then omitted from every meshed pod, and the proxy sidecar runs without elevated capabilities.</p><p>CNI shifts the privileged work from pods to the node, but it doesn’t solve the UID problem for the control plane itself. Bind the Linkerd control plane ServiceAccounts to a minimal, non-privileged SCC:</p><pre>apiVersion: security.openshift.io/v1<br>kind: SecurityContextConstraints<br>metadata:<br>  name: linkerd-scc<br>allowPrivilegedContainer: false<br>allowPrivilegeEscalation: false<br>defaultAllowPrivilegeEscalation: false<br>allowHostNetwork: false<br>allowHostPorts: false<br>allowHostPID: false<br>allowHostIPC: false<br>allowHostDirVolumePlugin: false<br>volumes:<br>  - configMap<br>  - projected<br>  - downwardAPI<br>  - emptyDir<br>  - secret<br>seccompProfiles:<br>  - &#39;*&#39;<br>runAsUser:<br>  type: MustRunAsNonRoot<br>seLinuxContext:<br>  type: RunAsAny<br>fsGroup:<br>  type: RunAsAny<br>supplementalGroups:<br>  type: RunAsAny<br>users:<br>  - system:serviceaccount:linkerd:linkerd-destination<br>  - system:serviceaccount:linkerd:linkerd-identity<br>  - system:serviceaccount:linkerd:linkerd-proxy-injector<br>  - system:serviceaccount:linkerd:linkerd-heartbeat</pre><p>Meshed application workloads also carry a linkerd-proxy sidecar running as UID 2102, so their ServiceAccounts need the same grant. Add each one to the users: list, or run oc adm policy add-scc-to-user linkerd-scc -z &lt;sa&gt; -n &lt;namespace&gt;.</p><h4>Path 2: Custom SCC for proxy-init</h4><p>If you prefer to keep linkerd-init, deploy the following custom SCC:</p><pre>apiVersion: security.openshift.io/v1<br>kind: SecurityContextConstraints<br>metadata:<br>  name: linkerd-scc<br>allowPrivilegedContainer: false<br>allowPrivilegeEscalation: false<br>defaultAllowPrivilegeEscalation: false<br>allowHostNetwork: false<br>allowHostPorts: false<br>allowHostPID: false<br>allowHostIPC: false<br>allowHostDirVolumePlugin: false<br>requiredDropCapabilities:<br>  - ALL<br>allowedCapabilities:<br>  - NET_ADMIN<br>  - NET_RAW<br>volumes:<br>  - configMap<br>  - projected<br>  - downwardAPI<br>  - emptyDir<br>  - secret<br>seccompProfiles:<br>  - &#39;*&#39;<br>runAsUser:<br>  type: MustRunAsNonRoot<br>seLinuxContext:<br>  type: RunAsAny<br>fsGroup:<br>  type: RunAsAny<br>supplementalGroups:<br>  type: RunAsAny<br>users:<br>  - system:serviceaccount:linkerd:linkerd-destination<br>  - system:serviceaccount:linkerd:linkerd-identity<br>  - system:serviceaccount:linkerd:linkerd-proxy-injector<br>  - system:serviceaccount:linkerd:linkerd-heartbeat</pre><p>The difference from the previous path is that this SCC grants NET_ADMIN and NET_RAW to the control plane ServiceAccounts. The grant applies only to the users listed in the SCC, so you’ll need to add every new application ServiceAccount as you onboard it.</p><p>This looks contained, but its blast radius is bigger than it appears. linkerd-init runs inside every meshed pod, not just the control-plane ones, so every meshed namespace needs those capabilities available to its ServiceAccounts. In practice you end up binding this SCC (or a sibling) to system:serviceaccounts:&lt;app-namespace&gt; for every application namespace you onboard. That operational tax never really goes away.</p><h3>One more trap: the policy controller leases</h3><p>Whichever path you pick, there’s one more OpenShift-specific detail that catches people out. OpenShift enables the OwnerReferencesPermissionEnforcement admission plugin by default, which isn’t the case on a stock Kubernetes cluster. That plugin requires anyone setting an ownerReference on an object to also hold delete permission on it, so that garbage collection can later remove it. When the policy controller tries to claim the policy-controller-write lease in leases.coordination.k8s.io and attach an ownerRef, the call fails: the default linkerd-policy ClusterRole grants neither update nor delete on leases. Patch in the missing verbs:</p><pre>oc get clusterrole linkerd-policy -o json \<br>  | jq &#39;(.rules[] | select(.apiGroups==[&quot;coordination.k8s.io&quot;] and .resources==[&quot;leases&quot;]) | .verbs) |= (. + [&quot;update&quot;,&quot;delete&quot;] | unique)&#39; \<br>  | oc apply -f -</pre><h3>When the xtables modules aren’t available</h3><p>On some OpenShift clusters, the RHCOS kernel doesn’t autoload every xtables compatibility module that linkerd-init relies on. When that happens, linkerd-init fails to insert its iptables rules and the control-plane components end up stuck in `Init:CrashLoopBackOff`.</p><pre>oc logs -n linkerd deploy/linkerd-destination -c linkerd-init --previous<br>time=&quot;2025-12-27T04:59:58Z&quot; level=info msg=&quot;/usr/sbin/iptables-nft-save -t nat&quot;<br>time=&quot;2025-12-27T04:59:58Z&quot; level=info msg=&quot;# Generated by iptables-nft-save v1.8.11 (nf_tables) on Sat Dec 27 04:59:58 2025\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\n:PROXY_INIT_REDIRECT - [0:0]\nCOMMIT\n# Completed on Sat Dec 27 04:59:58 2025\n&quot;<br>time=&quot;2025-12-27T04:59:58Z&quot; level=info msg=&quot;/usr/sbin/iptables-nft -t nat -F PROXY_INIT_REDIRECT&quot;<br>time=&quot;2025-12-27T04:59:58Z&quot; level=info msg=&quot;/usr/sbin/iptables-nft -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191,4567,4568 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191,4567,4568&quot;<br>time=&quot;2025-12-27T04:59:58Z&quot; level=info msg=&quot;Warning: Extension multiport revision 0 not supported, missing kernel module?\niptables v1.8.11 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain PROXY_INIT_REDIRECT\n&quot;<br>Error: exit status 4</pre><p>iptables-nft isn’t a pure nftables translator: matchers like -m multiport, -m owner, and -m comment, and the REDIRECT target go through nft_compat, which needs the legacy xt_* modules to be loadable. If any of those modules isn’t available, rule insertion fails. To make sure they’re loaded at boot, use this MachineConfig:</p><pre>apiVersion: machineconfiguration.openshift.io/v1<br>kind: MachineConfig<br>metadata:<br>  name: load-linkerd-xt-modules<br>  labels:<br>    machineconfiguration.openshift.io/role: worker<br>spec:<br>  config:<br>    ignition:<br>      version: 3.2.0<br>    storage:<br>      files:<br>      - path: /etc/modules-load.d/linkerd-xt.conf<br>        mode: 0644<br>        overwrite: true<br>        contents:<br>          source: data:,xt_multiport%0Axt_comment%0Axt_REDIRECT%0Axt_owner</pre><p>The data:, URL is the four module names separated by URL-encoded newlines, the format systemd-modules-load expects. Applying this drains and reboots every node in the worker MachineConfigPool.</p><h3>Which one to pick</h3><p>The real choice is about where the privilege lives and how many SCCs you have to maintain.</p><p>The <strong>CNI path</strong> concentrates the privilege in one place: a single DaemonSet in a dedicated linkerd-cni project, governed by one linkerd-cni-scc bound to one ServiceAccount. The control plane gets a minimal, non-privileged linkerd-scc. Application workloads pick up the same minimal SCC for their proxy sidecars, but nothing in the cluster ever needs NET_ADMIN or NET_RAW inside an application pod.</p><p>The <strong>proxy-init path</strong> spreads the privilege out: every meshed application namespace needs its ServiceAccounts bound to an SCC that grants NET_ADMIN and NET_RAW. You are operating that grant forever, for every new app team that onboards.</p><p>Pick CNI unless your cluster administrators forbid DaemonSets that write to host CNI paths. In that case, proxy-init with a custom SCC is the honest fallback. Just build the per-namespace SCC grant into your namespace-provisioning automation from day one, or it will bite you six months in.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29935ffc99da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/running-linkerd-on-openshift-sccs-cni-and-the-edges-that-bite-29935ffc99da">Running Linkerd on OpenShift: SCCs, CNI, and the Edges That Bite</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Summit Seoul 2026: Korean Enterprises And Agentic AI]]></title>
            <link>https://medium.com/todea-engineering/aws-summit-seoul-2026-korean-enterprises-and-agentic-ai-0bd9db90100b?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/0bd9db90100b</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Thu, 21 May 2026 02:37:45 GMT</pubDate>
            <atom:updated>2026-07-14T22:12:42.277Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JxA2uC2VRR2F41cYPEpZZA.avif" /></figure><p>Over the last few years, people have been asking the same question about AI: with so much money going into models, GPUs, and data centers, when will it pay off? Earnings calls and analyst reports have often mentioned a possible bubble, and companies that tried generative-AI pilots in 2024 have been waiting to see real production results.</p><p>At AWS Summit Seoul 2026, Samsung Electronics, Yogiyo, and Yanolja presented production-oriented agentic-AI/AIOps cases, while KB Kookmin Bank presented a production-scale KBaaS API-infrastructure modernization case. Together, the sessions showed how Korean enterprises are applying AI-era cloud architecture to operations, development, and embedded finance. Here’s a summary of how the event was organized, what the enterprise sessions showed, which AWS tools were most common, and the key patterns you can apply to your own plans.</p><h3>The shift the day was framed around</h3><p>AWS Korea CEO Ham Ki-ho opened the event by outlining three stages: Generative AI, Agentic AI, and Physical AI. He described Agentic AI as the active phase, where large language models act as reasoning engines that plan, use tools, and take action, instead of just answering prompts. Physical AI was presented as the next step, with a new Physical AI Frontier Program for Korean robotics, AI-chip, and manufacturing companies. This program will support everything from data collection to edge inference and help with global expansion.</p><p>AWS CFO Jon Felton announced a cumulative ₩12.6 trillion investment plan in Korea by 2031, described publicly as the largest-ever investment in Korea by a global cloud provider, with an estimated ₩15.06 trillion contribution to Korea’s GDP between 2023 and 2027. The Industry Day talks that followed showed how this commitment will play out in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tgRVV0n8LQJqP-9dsRdk8Q.png" /></figure><p>The keynote also highlighted the main agentic tools AWS featured at the event, with Vice President Jason Bennett explaining each one:</p><ul><li><strong>Kiro:</strong> a new agentic IDE for software development, described on stage as a peer to Claude Code and Copilot-style coding assistants</li><li><strong>Amazon Quick Suite:</strong> an agentic AI workspace for finding insights, conducting research, automating tasks, visualizing data, and taking actions across apps</li><li><strong>AWS Transform and AWS Transform Custom:</strong> agentic modernization tools for mainframe, VMware, Windows, and legacy-code modernization, including version upgrades, runtime migrations, language translations, and architecture changes</li></ul><h3>What the enterprise talks actually showed</h3><p>Across the agentic-AI sessions, a pattern emerged, but it was not universal. Samsung and Yanolja both used a supervisor or core-agent structure that delegates work to specialist agents. Yogiyo showed a Bedrock AgentCore-based AIOps workflow connected to operational data and tool functions. KBaaS was different: an EKS-based API platform and gateway modernization for embedded finance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYMFTi4ISTK-j4l19PlU-A.png" /></figure><p>The numbers made it clear these were real deployments, not just experiments. Samsung Electronics used Samsung Account, which supports 2.1 billion users, over 50 services, more than 2.7 million requests per second, and 200,000 transactions per second across four regions on EKS with over 70 namespaces. Samsung set Day-1 AIOps goals including 90%+ MTTR reduction, 99% incident detection within 10 minutes, and reducing human-in-the-loop operational work below 20%, as part of a roadmap toward ‘Toil 0% / Human First.’ KB Kookmin Bank’s KBaaS platform now handles about 1,800 APIs and 200 million calls per day after switching from a third-party API gateway to their own on AWS EKS. Yogiyo showed how they moved from checking multiple consoles and logs during incidents to using an AgentCore-based AIOps loop, which brings metric observation, correlation analysis, change-history comparison, and RCA grounding into one workflow. Yanolja shared the most technical details, describing how their domain-specialist agents for DevOps and SRE are managed by a Core Agent on Bedrock AgentCore Runtime, using Strands Agent as the SDK and end-to-end authentication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJm7Km0wfoR-dmqhXqGzmQ.png" /></figure><p>Samsung, Yogiyo, and Yanolja all centered their agentic-AI cases on AIOps or infrastructure operations rather than customer-facing copilots. KB Kookmin Bank’s case showed a different but related production pattern: modernizing a high-volume API platform on EKS for embedded finance.</p><p>On the development side, the counterpart is AI-DLC (AI-Driven Development Lifecycle), which is AWS’s approach to AI-led development with humans checking at validation and final approval. Samsung saw a 70% reduction in lead time using this method, and LG Electronics’ MS division reported double the productivity. The main idea is to separate intent from execution: humans decide what should happen and check the results, while agents handle the steps in between.</p><h3>The AWS-native stack that recurred</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kwB7qTNrrRgnOHkC-rafwQ.png" /></figure><p>A few key components showed up again and again in the enterprise sessions:</p><ul><li><strong>Amazon Bedrock AgentCore</strong> appeared across Samsung, Yogiyo, and Yanolja. Yanolja explicitly presented Bedrock AgentCore Runtime with Strands Agent and authentication. Yogiyo showed an AIOps architecture using Amazon Bedrock AgentCore with runtime, memory, gateway, and MCP tool functions. Samsung showed an AgentCore-based AIOps architecture with Supervisor, Domain, and Task agents.</li><li><strong>Kiro</strong> was used as the agentic IDE, and Samsung included it in their toolset along with Amazon Quick Suite and Bedrock AgentCore.</li><li><strong>Amazon EKS</strong> was the hosting platform for both Samsung Account and KBaaS, the two cases where the host was clearly specified.</li></ul><h3>The themes worth taking home</h3><p>Several patterns stood out during the day. First, the supervisor pattern appeared twice independently. Samsung and Yanolja reaching the same structure without coordination is a strong signal. Second, in three of the four cases (Samsung, Yogiyo, Yanolja), the entry point was AIOps or infrastructure operations rather than customer-facing copilots. KB Kookmin Bank instead modernized its API platform on EKS for embedded finance. Third, humans are intentionally kept in the loop. Datadog’s session made this clear, stating that sovereignty and judgment remain with people, while agents handle execution via the Datadog AI Agent Builder. Samsung’s AI-DLC approach on the development side followed the same principle. Fourth, infrastructure needs to catch up. GS Neotek’s session highlighted that the real GPU issue in agent workloads is not a shortage but poor utilization, excessive idle time, and over-allocation. They suggested using Dynamic Resource Allocation (DRA) on EKS as a next-generation operations model that considers workload needs, sharing policies, and topology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KnENRm9HHy-cSokcvzUD9g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0bd9db90100b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/aws-summit-seoul-2026-korean-enterprises-and-agentic-ai-0bd9db90100b">AWS Summit Seoul 2026: Korean Enterprises And Agentic AI</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Service Mesh Explained: Why Modern Platforms Need It]]></title>
            <link>https://medium.com/todea-engineering/service-mesh-explained-why-modern-platforms-need-it-cd9c48a3bea2?source=rss-f58cb680fdbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd9c48a3bea2</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Ivan Porta]]></dc:creator>
            <pubDate>Tue, 19 May 2026 05:35:42 GMT</pubDate>
            <atom:updated>2026-07-14T22:13:04.667Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSyNgS3twuIEeXoOjjpqNQ.avif" /></figure><p>If you run more than a handful of microservices on Kubernetes, your developers have probably already reinvented half of a service mesh. Retry and timeout logic wired into every HTTP client. Certificate rotation held together by a CronJob and a script. Service-to-service auth improvised with bearer tokens. A mix of library versions, language idioms, and “we’ll fix it later” config spread across every repo.</p><p>Every team solves the same problems slightly differently. A service mesh moves those concerns out of application code and into infrastructure your platform team owns.</p><h3>The duties developers shouldn’t own</h3><p>When you split a monolith into microservices, you also split everything that used to happen in-process: error handling, retries, timeouts, auth, metrics. Without a mesh, every team rebuilds the same patterns, usually in their own language’s idiom, often misconfigured, rarely consistent:</p><ul><li><strong>Retry and timeout logic</strong> in every HTTP client, often misconfigured</li><li><strong>TLS certificate management</strong> with shared secrets passed around in headers</li><li><strong>Service-to-service authentication</strong> improvised with API tokens</li><li><strong>Distributed tracing instrumentation</strong> added by hand to every endpoint</li><li><strong>Circuit breakers</strong> copied between services with subtle differences</li><li><strong>Load balancing and retry budgets</strong> scattered across application code</li><li><strong>Network policies</strong> expressed as IP allowlists nobody dares to change</li></ul><p>Multiply that by every service, every language you ship in, and every team maintaining it. The cost is enormous, the consistency is poor, and the reliability suffers.</p><p>A service mesh takes these concerns off your developers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXTec0WvxMmnZEPBjm708w.png" /></figure><p>The developer writes business logic. The mesh handles the cross-cutting plumbing.</p><h3>How it works: the proxy architecture</h3><p>Every service mesh is built on the same two components:</p><ol><li>A <strong>data plane</strong>: Network proxies that intercept every byte of service-to-service traffic, handle mTLS, and enforce policy.</li><li>A <strong>control plane</strong>: Controllers that configure those proxies, issue workload identities, and aggregate their telemetry.</li></ol><p>What differs between meshes is <strong><em>where the proxy runs</em></strong>. There are two dominant models.</p><h4>Sidecar proxies</h4><p>In the sidecar model, the proxy runs as an extra container inside every application pod. Linkerd works this way by default; so does classic Istio. At pod startup, iptables rules (installed either by an init container or by a CNI plugin) transparently redirect the pod’s inbound and outbound TCP traffic through the sidecar.</p><p>When service A calls b.default.svc.cluster.local, A’s application container makes a normal HTTP call to a Kubernetes Service. The iptables redirect funnels it through A’s sidecar, which handles mTLS, retries, timeouts, and routing, then forwards the request to B’s sidecar. B’s sidecar enforces authorization policy, records metrics, and forwards to B on loopback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RNr5vL8c6v0a7a8JJF8r6w.png" /></figure><h4>Sidecarless architectures</h4><p>The major players are:</p><ul><li><strong>Istio ambient mode</strong> runs an L4 tunnel (ztunnel) as a DaemonSet, one per node, handling mTLS for all local pods. L7 features (HTTP routing, authorization on paths, etc.) are layered on by deploying optional per-namespace <strong>waypoint proxies</strong>. You pay for L7 only where you need it.</li><li><strong>Cilium Service Mesh</strong> pushes L4 mTLS and policy into <strong>eBPF </strong>in the Linux kernel, and runs Envoy <strong>per node</strong> for L7 concerns.</li></ul><p>The trade-offs invert: no per-pod overhead, faster pod startup, and fewer moving parts per workload. But a shared proxy means tenancy concerns bleed between workloads on the same node, and debugging a single service’s traffic now means reasoning about a node-scoped component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZYoiP_fUsNyslacYd1Myw.png" /></figure><p>From the application’s perspective, both models look identical: it makes a normal HTTP call to a Kubernetes Service. From the operator’s perspective, every byte of that traffic is now authenticated, encrypted, observable, and governable by policy, without a single line of application code changing.</p><h3>What you get at the platform level</h3><p>A properly deployed service mesh gives you, without application changes:</p><ul><li><strong>Mutual TLS between every service</strong>: The mesh handles certificate issuance, rotation, and revocation. No PKI code in app teams’ repos.</li><li><strong>Workload identity</strong>: Based on cryptographic certificates, not IP addresses or shared secrets. The foundation of zero-trust networking.</li><li><strong>Traffic management</strong>: Retries, timeouts, circuit breaking, canary releases. All declared in configuration.</li><li><strong>Distributed tracing</strong>: Every request gets a trace ID propagated automatically.</li><li><strong>Consistent metrics</strong>: Request rate, error rate, p50/p95/p99 latency. For every service, with no instrumentation work.</li><li><strong>Multi-cluster service discovery</strong>: Service in cluster A can call a service in cluster B as if they were neighbors.</li></ul><h3>Common pitfalls</h3><p>Service mesh adoption fails in predictable ways:</p><ol><li><strong>Treating it as a switch.</strong> Treating install as the finish line. Getting the mesh running is the easy part. Operating it, rotating the trust anchor, debugging proxy issues in the request path, staying current with upgrades, tuning resource limits as the cluster grows, is where the real investment lives. Budget for it up front, or the mesh becomes shelfware.</li><li><strong>Choosing for features instead of fit.</strong> Istio, Linkerd, Consul, and Cilium Service Mesh have very different operational profiles. The mesh with the longest feature list is not the one you want if your team is three people. Pick the one whose operational surface you can actually own.</li><li><strong>Skipping observability work.</strong> A service mesh produces enormous telemetry. Without dashboards and SLOs, the data is useless.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd9c48a3bea2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/todea-engineering/service-mesh-explained-why-modern-platforms-need-it-cd9c48a3bea2">Service Mesh Explained: Why Modern Platforms Need It</a> was originally published in <a href="https://medium.com/todea-engineering">Todea Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>