<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nithin katla on Medium]]></title>
        <description><![CDATA[Stories by Nithin katla on Medium]]></description>
        <link>https://medium.com/@nithinkatla19?source=rss-756b3ce5fcd2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VmNpd9aDW-d98nHUqYqH0A.png</url>
            <title>Stories by Nithin katla on Medium</title>
            <link>https://medium.com/@nithinkatla19?source=rss-756b3ce5fcd2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:05:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nithinkatla19/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Hidden Half of AI Engineering]]></title>
            <link>https://medium.com/@nithinkatla19/the-hidden-half-of-ai-engineering-6f4cd4075cc7?source=rss-756b3ce5fcd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f4cd4075cc7</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Nithin katla]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 07:12:26 GMT</pubDate>
            <atom:updated>2026-03-10T07:12:26.727Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>How to Run Large Language Models Reliably in Production. </em>A deep dive into the infrastructure stack that makes real-world AI possible</strong></h4><p>There is a moment every AI team eventually hits.</p><p>The model is trained. The benchmarks look good. The demo runs perfectly on a laptop. Everyone is excited. And then someone asks the question that changes everything:</p><p><em>“Okay, but how do we actually run this for real users?”</em></p><p>That question doesn’t have a simple answer. And the silence that often follows it reveals a gap that is quietly becoming one of the most important problems in engineering today.</p><p>Training a large language model gets most of the attention. It is where the research happens, where the papers get written, where the talent wants to work. But training is, in a sense, the easier part. You do it once. You iterate. You improve. Eventually you have a model you are proud of.</p><p>Running that model, reliably, at scale, for thousands or millions of users, is an entirely different problem. It requires a different way of thinking, a different set of tools, and increasingly, a different kind of engineer.</p><p>This article is about that second problem. It is about the infrastructure stack that serious AI teams are quietly building, the layers that sit between your model and your users, and why understanding them is becoming one of the most valuable skills in the industry.</p><h3>Why Production AI Is a Distributed Systems Problem</h3><p>If you squint at a modern LLM deployment, it starts to look less like a machine learning system and more like a distributed database.</p><p>You have workloads that need to be split across multiple machines. You have state that needs to be managed carefully. You have failures that need to be recovered from automatically. You have traffic that spikes unpredictably and hardware that is expensive to leave idle.</p><p>The model is one component. But the system around it is what determines whether your AI product actually works.</p><p>Consider what happens during a single inference request:</p><ul><li>A user sends a prompt</li><li>The request is routed to an available worker</li><li>GPU memory is allocated to hold the model weights and the context</li><li>Tokens are generated one by one, each dependent on the last</li><li>The response is streamed back to the user</li><li>Memory is released for the next request</li></ul><p>Each of these steps is a potential point of failure. Each is a potential bottleneck. At scale, even tiny inefficiencies compound into serious problems.</p><p>This is why the infrastructure layer is not an afterthought. It is where the real engineering work happens in production AI.</p><h3>The Modern LLM Infrastructure Stack</h3><p>Over the past few years, a reasonably standard production stack has emerged. It is not the only way to run LLMs, but it represents the thinking of teams who have actually wrestled with these problems at scale.</p><p>It looks like this, from the bottom up:</p><blockquote><strong><em>Kubernetes → KubeRay → Ray → vLLM → llm-d → LLM Model</em></strong></blockquote><p>Each layer solves a specific problem. Each one would be difficult to replace with something else without losing something important. Together they form a system that is more than the sum of its parts.</p><p>Let’s go through each layer in detail.</p><h3>Layer 1: Kubernetes — The Foundation</h3><p>Kubernetes is not an AI technology. It was built to manage containerized applications of all kinds, long before large language models existed. But it turns out to be exactly what AI infrastructure needs.</p><p>Think of Kubernetes as the operating system for your cluster. It answers questions like: where should this workload run? What happens if a node fails? How do I add capacity when demand increases? How do I know if my services are healthy?</p><h4>What Kubernetes actually does</h4><p>When you describe a workload to Kubernetes, you tell it what you want, not how to achieve it. You say: I want three replicas of this service running at all times, each with access to two GPUs. Kubernetes figures out the rest.</p><p>If a node goes down, Kubernetes notices, reschedules the workload on a healthy node, and updates the routing so requests stop going to the failed instance. The whole thing can happen in seconds, often without anyone being paged.</p><p>This is called declarative infrastructure, and it is one of the most powerful ideas in modern systems engineering.</p><h4>Why it matters for AI</h4><p>GPU nodes are expensive. They are also more likely to fail than regular compute because they run hotter, draw more power and are under heavier load. Kubernetes handles those failures gracefully.</p><p>LLM workloads are also highly variable. A system that needs five replicas at 2pm might only need one at 2am. Kubernetes autoscaling handles that automatically, saving significant infrastructure cost.</p><p>Without Kubernetes, or something equivalent, you would be managing all of this manually. That is not scalable.</p><h3>Layer 2: Ray — Distributed Execution at Scale</h3><p>One GPU is not enough to run a serious LLM. One node is rarely enough to handle serious traffic. Ray is the layer that makes distributing work across many machines feel almost simple.</p><p>Ray is a Python-native framework for distributed computing. It was originally built at UC Berkeley for reinforcement learning research, but it has since become one of the most popular tools for any kind of distributed Python workload — including LLM inference.</p><h4>The core abstraction</h4><p>Ray’s central idea is that you should be able to take almost any Python function and run it in parallel, across many machines, with minimal changes to your code. You decorate a function with @ray.remote, and Ray handles the distribution, the scheduling, the fault tolerance and the result aggregation.</p><p>For LLM serving, this means you can run model replicas across dozens of workers, distribute inference batches across them and aggregate results, all with relatively little glue code.</p><h4>Why this matters</h4><p>Without something like Ray, you would need to build your own distributed job scheduling system. You would need to handle worker failures, load balancing and communication between machines from scratch.</p><p>Ray gives you all of that for free, and it integrates well with the rest of the Python ML ecosystem. You can use it with PyTorch, with HuggingFace, and critically, with vLLM.</p><h3>Layer 3: KubeRay — The Bridge Between Worlds</h3><p>Ray and Kubernetes are both powerful tools, but they were not designed to work together out of the box. KubeRay is the project that bridges them.</p><p>KubeRay is a Kubernetes operator for Ray. It lets you define Ray clusters as Kubernetes resources, which means you get all the benefits of both systems: Kubernetes scheduling, health checking and scaling, combined with Ray’s distributed execution model.</p><h4>What it manages</h4><p>KubeRay handles the full lifecycle of a Ray cluster: starting it, scaling it up and down based on demand, monitoring the health of Ray workers and cleaning up after failures. It also integrates with Kubernetes autoscaling APIs, so you can scale Ray workers in response to GPU utilization metrics or queue depth.</p><p>In practice, this means your AI serving infrastructure becomes much more self-managing. You define the desired state and KubeRay works to maintain it.</p><h4>Why this is harder than it sounds</h4><p>Ray clusters have specific requirements around networking and shared state. Getting them to work well in Kubernetes, especially with GPUs involved, requires careful configuration. KubeRay encapsulates a lot of that complexity, so your team doesn’t have to rediscover it from scratch.</p><h3>Layer 4: vLLM — Inference That Doesn’t Waste Your Hardware</h3><p>You could run an LLM without vLLM. PyTorch alone can do inference. But if you care about serving real users efficiently, you want vLLM.</p><p>vLLM is an inference engine built specifically for large language models. It was introduced in a 2023 paper from UC Berkeley, and it introduced one idea that changed how people think about LLM serving: PagedAttention.</p><h4>The memory problem it solves</h4><p>When you generate text with an LLM, the model needs to hold intermediate computations in GPU memory called the KV cache. The size of this cache grows with the length of the context. And it needs to be held for the entire duration of a request.</p><p>The naive approach is to pre-allocate memory for the maximum possible context length, for every request. This wastes enormous amounts of memory, because most requests are shorter than the maximum, and that wasted memory cannot be used for other requests.</p><p>PagedAttention solves this by managing KV cache memory in pages, similar to how operating systems manage virtual memory. Memory is allocated dynamically, as needed and released when the request completes. The result is dramatically better GPU utilization.</p><h4>Continuous batching</h4><p>vLLM also implements continuous batching, which is a technique for processing multiple requests simultaneously on a single GPU. Rather than waiting for one request to complete before starting the next, vLLM manages a dynamic batch that is constantly being updated as requests arrive and complete.</p><p>This sounds like a minor optimization, but at scale it can increase throughput by an order of magnitude compared to naive sequential processing.</p><h4>Why this matters for cost</h4><p>GPUs are the most expensive component of your inference infrastructure. vLLM is specifically designed to squeeze more useful work out of each one. In practice, teams report being able to serve several times more users with the same hardware after switching to vLLM. At the economics of GPU compute, that translates directly into meaningful cost savings.</p><h3>Layer 5: llm-d — Distributed Intelligence for the Entire Inference Stack</h3><p>If vLLM solves inference efficiency on a single node, llm-d solves the harder problem of coordinating inference intelligently across an entire cluster.</p><p>llm-d is a Kubernetes-native distributed inference framework, launched in 2025 with founding contributions from Red Hat, Google Cloud, IBM Research, NVIDIA and CoreWeave. It builds directly on top of vLLM, extending it with capabilities that only become necessary once you are running at serious scale and that are genuinely difficult to build yourself.</p><p>The project’s ambition is stated plainly: to make production-grade LLM serving as ubiquitous and reliable as Linux. That is a bold goal, but the engineering behind it is concrete and practical.</p><h4>The key insight: inference has two very different phases</h4><p>To understand what llm-d does, you first need to understand something about how LLMs generate text. Every inference request goes through two distinct phases.</p><p>The first is prefill. This is where the model reads and processes the user’s prompt. It is compute-intensive — the GPU is working hard, doing a lot of matrix multiplication to build an understanding of the input. The output of this phase is a set of key-value pairs (the KV cache) that represent the model’s understanding of the context so far.</p><p>The second is decode. This is where the model generates the response, one token at a time. It is memory-bandwidth-intensive rather than compute-intensive. The GPU is mostly reading from the KV cache and producing the next token, then reading again, then producing the next.</p><p>These two phases have very different hardware requirements. Prefill wants raw compute. Decode wants fast memory access. Running them on the same hardware is a compromise — you are never fully optimizing for either.</p><h4>Disaggregated serving: splitting prefill and decode</h4><p>llm-d’s signature innovation is called disaggregated serving. Instead of running prefill and decode on the same pod, it routes them to specialized workers: dedicated prefill nodes that are optimized for compute-heavy prompt processing, and dedicated decode nodes that are optimized for the memory-intensive generation phase.</p><p>Think of it like a well-run kitchen. One team handles prep — chopping, measuring, building mise en place. Another team handles the actual cooking. Separating the roles lets each team specialize and keeps the whole kitchen running faster.</p><p>In practice, this approach can dramatically reduce time-to-first-token — the latency between a user submitting a prompt and seeing the first word of the response. Early benchmarks from Google Cloud show roughly 2x improvements in time-to-first-token for use cases like code completion.</p><h4>Intelligent, LLM-aware routing</h4><p>Traditional load balancers route requests using simple strategies: round-robin, least-connections, random. These strategies were designed for stateless web traffic. LLM requests are not stateless.</p><p>Two requests that share a long system prompt have already done a significant portion of the same computation — the KV cache for that shared prefix. If you route both requests to the same worker, the second request can reuse that cached computation. If you route them to different workers, the computation has to be done twice.</p><p>llm-d’s inference scheduler is aware of this. It tracks which prefixes are cached on which workers, monitors queue depth and GPU utilization across the cluster, and makes routing decisions that maximize cache reuse and minimize latency. This is called prefix-aware or KV-cache-aware routing, and it is one of the things that generic load balancers simply cannot do.</p><h4>Multi-tier KV cache</h4><p>KV cache data does not have to live exclusively in GPU memory. llm-d introduces a multi-tier caching system that allows KV cache to be offloaded to CPU memory, local disk or even shared storage across servers.</p><p>This matters for two reasons. First, it expands the effective cache capacity beyond what fits on the GPU, allowing longer contexts and more reuse of previously computed prefixes. Second, it reduces storage costs — GPU memory is the most expensive storage tier, and offloading aggressively to cheaper tiers can meaningfully reduce the cost of running large models.</p><h3>Why this is the next frontier</h3><p>vLLM made single-node inference efficient. llm-d makes cluster-wide inference intelligent. These are complementary rather than competing contributions, which is why llm-d is built on top of vLLM rather than replacing it.</p><p>As models get larger, as context windows get longer and as agentic workflows that generate many sequential inference requests become more common, the cluster-level optimizations that llm-d provides become increasingly important. Teams that are serving large models in production today are already hitting the limits of single-node optimization. llm-d is where the industry is heading.</p><h3>How the Stack Works Together</h3><p>The real power of this stack comes from how the layers interact.</p><p>Consider what happens when your LLM service experiences a traffic spike:</p><ul><li>vLLM detects that its request queue is growing and reports high utilization</li><li>llm-d’s inference scheduler observes queue depth and KV cache pressure across the cluster</li><li>KubeRay, monitoring these metrics, sees that more Ray workers are needed</li><li>Kubernetes responds by scheduling new GPU nodes from the cluster autoscaler</li><li>KubeRay launches new Ray workers on those nodes and connects them to the cluster</li><li>llm-d begins routing new prefill requests to the freshly provisioned prefill workers</li><li>Users experience no degradation in response time</li></ul><p>The entire process can happen in a few minutes, automatically, without human intervention. When the traffic spike subsides, the same process runs in reverse and the extra nodes are released to save cost.</p><p>This is what infrastructure-as-code, applied to AI serving, actually looks like in practice.</p><h3>What You Can Build With This Stack</h3><p>When these layers are in place, capabilities that previously required heroic engineering effort become relatively straightforward.</p><h4>High-throughput inference</h4><p>With vLLM’s continuous batching and PagedAttention, combined with Ray’s ability to distribute across multiple nodes, you can handle thousands of concurrent inference requests on a reasonably sized cluster. This is not theoretical — teams at companies of all sizes have demonstrated this in production.</p><h4>Automatic failure recovery</h4><p>Kubernetes health checking, combined with Ray’s fault tolerance, means that most failure scenarios are handled automatically. A GPU node crashes at 3am? Kubernetes detects it, Ray redistributes the affected workers and new nodes are scheduled. Your on-call engineer sleeps through it.</p><h4>Multi-model serving</h4><p>The same cluster can run multiple models simultaneously. A coding assistant, a summarization model, a customer service bot — each can be deployed as separate Ray deployments, sharing the underlying infrastructure, scaling independently based on their own traffic patterns.</p><h4>Efficient GPU utilization</h4><p>Perhaps most importantly, this stack makes it practical to actually use the GPUs you are paying for. Without intelligent memory management and batching, it is easy to find yourself with expensive hardware sitting largely idle while you wait for requests to arrive. vLLM, properly configured, keeps utilization consistently high.</p><h3>The New Kind of AI Engineer</h3><p>There is a skills gap opening up in the industry that does not get talked about enough.</p><p>ML researchers know how to train models. Software engineers know how to build applications. But the person who understands both the model and the infrastructure — who can debug a GPU memory fragmentation issue and also understand why the model is producing inconsistent outputs — is genuinely rare.</p><p>This is the emerging role of the AI infrastructure engineer. Or, increasingly, the expectation that any senior engineer working on AI products should have at least a working understanding of this stack.</p><p>If you are a machine learning engineer who has never had to think about Kubernetes, now is the time to learn. If you are a backend engineer who has been watching AI from the sidelines, the distributed systems skills you already have are surprisingly applicable to this domain.</p><p>The companies building serious AI products are not just hiring researchers. They are hiring people who can keep those products running.</p><h3>What to Learn First</h3><p>If you are new to this stack and want to build practical skills, here is a path that makes sense:</p><p><strong>1. Start with Kubernetes basics. </strong>You don’t need to become a Kubernetes expert, but you need to understand pods, deployments, services and autoscaling. The official documentation is excellent and there are many good interactive tutorials available.</p><p><strong>2. Experiment with vLLM locally. </strong>Install vLLM, download a small open-source model and serve it. Try sending concurrent requests and watch how it handles them. Read the documentation on PagedAttention to understand what is happening under the hood.</p><p><strong>3. Learn Ray’s core abstractions. </strong>The Ray documentation has good tutorials on remote functions and actors. The Ray Serve component, which handles model serving specifically, is worth particular attention.</p><p><strong>4. Explore llm-d. </strong>If you want to understand where cluster-level inference optimization is heading, llm-d is the project to watch. The quickstart guide is well-documented and even running it in a small test cluster will teach you a lot about disaggregated serving and KV-cache-aware routing.</p><p><strong>5. Deploy something end to end. </strong>The best way to understand this stack is to actually build with it. Set up a small Kubernetes cluster, deploy a Ray cluster with KubeRay and serve a model with vLLM. You will learn more in an afternoon of debugging than in a week of reading.</p><p>The tooling in this space is evolving quickly. What matters more than knowing the current tools is understanding the problems they solve. Memory management, request batching, fault tolerance, autoscaling — these are durable concepts that will remain relevant regardless of which specific tools the industry converges on.</p><p>The AI is dominated by models. Which foundation model is best? How many parameters does it have? What score did it get on the benchmark?</p><p>These are not unimportant questions. But they are not the only questions that matter if you are actually trying to build something real.</p><p>The teams that will build the most impactful AI products are the ones who treat the infrastructure with the same seriousness as the model. They understand that a great model on unreliable infrastructure is just an expensive demo. And a good model on excellent infrastructure can be a product that people actually use.</p><blockquote><strong><em>The model is important. The infrastructure around the model is what makes it real.</em></strong></blockquote><p>That, in the end, is what the hidden half of AI engineering is about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f4cd4075cc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>