<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shivank Chaudhary on Medium]]></title>
        <description><![CDATA[Stories by Shivank Chaudhary on Medium]]></description>
        <link>https://medium.com/@shivank1128?source=rss-51548f5ef95------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_l3cnEmiG4fxmRT2KhVihw.jpeg</url>
            <title>Stories by Shivank Chaudhary on Medium</title>
            <link>https://medium.com/@shivank1128?source=rss-51548f5ef95------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 03:56:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shivank1128/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[8x Faster LLM Cold Starts on Kubernetes With NVIDIA Dynamo Snapshot, your guide to scale to 0 and…]]></title>
            <link>https://medium.com/@shivank1128/8x-faster-llm-cold-starts-on-kubernetes-with-nvidia-dynamo-snapshot-your-guide-to-scale-to-0-and-c55ee2b7bff7?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/c55ee2b7bff7</guid>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[ai-inferencing]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 05:32:27 GMT</pubDate>
            <atom:updated>2026-10-17T05:33:46.373Z</atom:updated>
            <content:encoded><![CDATA[<h3>8x Faster LLM Cold Starts on Kubernetes With NVIDIA Dynamo Snapshot, your guide to scale to 0 and overcommit your GPUs</h3><p><em>Testing NVIDIA Dynamo Snapshot on H200s: the numbers, what broke, and where it stops helping.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m9eHOS53q-rvwfqOont-qg.png" /></figure><h3>TL;DR</h3><p>I tested NVIDIA Dynamo Snapshot (CRIU + cuda-checkpoint) on a gpt-oss-20b vLLM worker, scaling from 0 → 1 replica.</p><p><strong>106.1 s → 13.3 s</strong> time to first response (median of 5). Serving latency and throughput stayed the same.</p><p>Biggest surprise: weight loading was only 5 s of the cold start. CUDA graph capture was 47 s.</p><p>Verdict: works well for single-GPU, text-only vLLM workers. Not for MIG slices, RDMA KV-cache connectors or multimodal (yet).</p><h3>Why I tried this</h3><p>Scaling an LLM replica from zero took close to two minutes, even with weights on local NVMe.</p><p>During a traffic spike that’s two minutes of queued requests, or a lot of idle GPUs kept warm “just in case”.</p><p>NVIDIA published a blog on Dynamo Snapshot claiming big startup wins. I wanted my own numbers, per stage.</p><h3>Setup</h3><pre>Component            Version<br>-------------------  ------------------------------------------<br>Kubernetes           RKE2 v1.35.6, single node<br>GPUs                 8x NVIDIA H200 141 GB (test used 1-2)<br>NVIDIA driver        580.126.20<br>GPU Operator         v26.3.3<br>Container runtime    containerd 2.2.5 (RKE2 build)<br>Kernel               6.17<br>Dynamo               1.4.2 (operator + snapshot chart)<br>vLLM                 0.26.0 (inside vllm-runtime:1.4.2)<br>CRIU                 4.2<br>Model                openai/gpt-oss-20b (MXFP4, 13 GiB), TP=1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9TPtnySE6Y3afyeXv7Zdug.png" /></figure><p>The pieces. ①–③ happen once (checkpoint). ④–⑧ happen on every scale-up (restore).</p><p>How it works, in short:</p><ul><li>A checkpoint Job starts the worker normally once, then puts vLLM to sleep (frees the KV cache).</li><li>A privileged snapshot-agent DaemonSet dumps the process with cuda-checkpoint + CRIU to a PVC.</li><li>On scale-up, a webhook swaps the new pod’s command for a placeholder, and the agent restores the saved process into it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H8ByBeuAbjy27egwrIZQpw.png" /></figure><p>Checkpoint once, restore on every scale-up.</p><h3>How I measured</h3><ul><li>A small test script scales the worker with kubectl patch (replicas 0 → 1) and records T0.</li><li>It polls the frontend every 0.5 s with a one-token chat request. First HTTP 200 = time to first response.</li><li>It also collects 18 timestamps from Kubernetes, vLLM and Dynamo logs, so every stage gets its own number.</li><li>After each start: a 60 s load test (8 concurrent streams, ~512 tokens in / 128 out), then scale back to 0.</li><li>Same image, same worker args, same node for both modes. 5 repeat runs each, 60 s cool-down between runs.</li></ul><h3>Steps</h3><h3>1. Turn on checkpointing in the operator</h3><pre>$ helm upgrade dynamo-platform dynamo-platform-1.4.2.tgz -n dynamo-system \<br>    --reuse-values --set dynamo-operator.checkpoint.enabled=true --force-conflicts</pre><p>One value. The rendered diff was a 5-line ConfigMap change plus a pod restart of the operator.</p><p>I needed --force-conflicts because of a label someone had added by hand (see &quot;What broke&quot;).</p><h3>2. Install the snapshot agent</h3><p>The snapshot chart deploys the agent DaemonSet, a checkpoint PVC and a seccomp profile that blocks io_uring (CRIU can’t dump it).</p><p>First thing I checked once it was up:</p><pre>$ kubectl exec -n demo ds/snapshot-agent -- criu check<br>Warn  (criu/kerndat.c:1285): Can&#39;t keep kdat cache on non-tempfs<br>Looks good.</pre><p>“Looks good” on kernel 6.17 was the thing I was most worried about. Good start.</p><p>The agent image is big. Pulling it took a while:</p><pre>Successfully pulled image &quot;nvcr.io/nvidia/ai-dynamo/snapshot-agent:1.4.2&quot; in 15m41.68s (15m41.68s including waiting).</pre><h3>3. Build a placeholder image</h3><p>The restore runs inside the new worker container, so the worker image needs CRIU, cuda-checkpoint and nsrestore in it.</p><p>NVIDIA doesn’t publish one. You build it from your exact runtime image (the restored process expects the same files on disk):</p><pre>$ make docker-build-placeholder \<br>    PLACEHOLDER_BASE_IMG=nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.2 \<br>    PLACEHOLDER_IMG=registry.example.com/team/dynamo-vllm-placeholder:1.4.2</pre><p>Took 36 min, mostly a slow apt mirror and one lossy Go module download. The compile itself was minutes.</p><h3>4. Opt the worker in</h3><pre>experimental:<br>  checkpoint:<br>    enabled: true<br>    startupPolicy: WaitForCheckpoint   # every measured start is a restore<br>    deletionPolicy: Retain</pre><p>With WaitForCheckpoint, worker replicas stay at 0 until the checkpoint is ready.</p><p>The checkpoint Job came up on its own:</p><pre>$ kubectl get dckpt -n demo<br>NAME                                          MODEL                     BACKEND   PHASE   CHECKPOINTID                       AGE<br>checkpoint-1cf68a75aefb7b1a1f2c47f613d178ae   demo/gptoss20b-snapshot   vllm      Ready   1cf68a75aefb7b1a1f2c47f613d178ae   4d11h</pre><p>What the agent logged for the dump (IDs and timestamps trimmed):</p><pre>CRIU dump completed  {&quot;duration&quot;: 8.538566562}<br>Checkpoint timing summary  {&quot;checkpoint&quot;: {&quot;duration&quot;:&quot;9.887322655s&quot;,&quot;phases&quot;:{&quot;criu_dump_duration&quot;:&quot;8.538566562s&quot;,&quot;cuda_duration&quot;:&quot;1.164179823s&quot;,&quot;overlay_capture_duration&quot;:&quot;126.386055ms&quot;,&quot;prepare_duration&quot;:&quot;57.949386ms&quot;}}}</pre><pre>$ kubectl exec -n demo ds/snapshot-agent -- du -sh /checkpoints/1cf68a75aefb7b1a1f2c47f613d178ae<br>21G	/checkpoints/1cf68a75aefb7b1a1f2c47f613d178ae</pre><p>21 GiB, not the ~125 GiB I expected. vLLM’s 111 GiB KV cache is freed (sleep mode) before the dump.</p><p>Whole checkpoint Job: 125 s (cold start ~109 s + dump ~10 s + finalize).</p><h3>5. Scale up and time it</h3><p>Cold start (no snapshot), vLLM’s own stage logs from one run (log prefixes trimmed):</p><pre>19:10:52  Loading weights took 3.36 seconds<br>19:10:53  Model loading took 13.03 GiB memory and 4.287618 seconds<br>19:10:56  torch.compile took 2.30 s in total<br>19:11:01  Initial profiling/warmup run took 5.00 s<br>19:11:12  Available KV cache memory: 111.44 GiB<br>19:11:59  Graph capturing finished in 38 secs, took 0.53 GiB<br>19:12:01  init engine (profile, create kv cache, warmup model) took 67.18 s (compilation: 2.30 s)</pre><p>With snapshot, the agent restores instead (pod and container IDs trimmed):</p><pre>Restore timing summary  {&quot;restore&quot;: {&quot;duration&quot;:&quot;6.892109443s&quot;,&quot;phases&quot;:{&quot;criu_restore_duration&quot;:&quot;4.089530533s&quot;,&quot;cuda_duration&quot;:&quot;2.723554388s&quot;,&quot;host_inspect_duration&quot;:&quot;14.73978ms&quot;,&quot;nsrestore_setup_duration&quot;:&quot;64.284742ms&quot;}}}<br>INFO lifecycle.run_lifecycle: Restore sentinel detected<br>INFO abstract.wake_up: It took 0.698674 seconds to wake up tags {&#39;weights&#39;, &#39;kv_cache&#39;}.<br>INFO _core: Registered base model &#39;openai/gpt-oss-20b&#39; MDC</pre><p>Test script output for the snapshot runs:</p><pre>[S2-1] first response after 12.8s<br>[S2-2] first response after 12.3s<br>[S2-3] first response after 13.8s<br>[S2-4] first response after 13.7s<br>[S2-5] first response after 13.3s</pre><p>And the namespace at the end of testing (trimmed):</p><pre>$ kubectl get po -n demo<br>NAME                                           READY   STATUS      RESTARTS   AGE<br>gptoss20b-baseline-frontend-6c5cbf7978-zvj8l   1/1     Running     0          4d11h<br>gptoss20b-snapshot-frontend-7ff4b6c4-jq8wq     1/1     Running     0          4d11h<br>snapshot-agent-zbcmd                           1/1     Running     0          4d14h<br>...</pre><p>Workers are at 0 here, which is the point. Only frontends and the agent stay up.</p><h3>What broke (and the fix)</h3><p><strong>1. The snapshot chart assumes stock containerd paths. RKE2 isn’t stock.</strong></p><pre>$ ls -d /var/lib/containerd<br>ls: cannot access &#39;/var/lib/containerd&#39;: No such file or directory</pre><p>The chart hard-codes /var/lib/containerd as a hostPath. On RKE2 it lives under /var/lib/rancher/rke2/agent/containerd, and the socket is in /run/k3s/containerd.</p><p>The agent reads each container’s overlay upperdir from /proc/&lt;pid&gt;/mountinfo and tars it. So the host path has to be mounted at the same path inside the agent.</p><p>Fix: set runtime.socketPath, then patch the rendered manifest (two lines) before applying.</p><p><strong>2. Helm upgrade conflict on a hand-added label.</strong></p><pre>Apply failed with 1 conflict: conflict with &quot;kubectl-label&quot; using monitoring.coreos.com/v1: .metadata.labels.release</pre><p>Someone had relabeled the operator’s ServiceMonitor so Prometheus would scrape it. The chart hard-codes a different value.</p><p>Fix: --force-conflicts, then re-apply the label right after.</p><p><strong>3. Restarting the operator crashed the planners.</strong></p><pre>conversion webhook for nvidia.com/v1alpha1, Kind=DynamoGraphDeployment failed</pre><p>The operator also serves the CRD conversion webhook. While it restarts (~30 s), planners reading their deployment get a 500 and exit.</p><p>They came back on their own in about a minute. No worker pod was touched. Still, do operator changes in a quiet window.</p><p><strong>4. My own frontend crash-looped.</strong></p><pre>ERROR main.parse_args: Unknown arguments specified: [&#39;-m&#39;, &#39;dynamo.frontend&#39;]</pre><p>If a component has command but no args, the operator injects default args, so it ran -m dynamo.frontend twice.</p><p>Fix: pass explicit args (--model-name ...).</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jc4s8ydQ4bOkZ4fQ5OFAPA.png" /></figure><p>Results: cold start vs Dynamo Snapshot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJq-hYZfMN77Wh9DNmB4wA.png" /></figure><p>Where the 106 seconds go</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUESaEYdEE_TjCQuXhxdug.png" /></figure><p>Time per stage</p><p>What stood out:</p><ul><li>8x faster to first response, 10x to pod Ready.</li><li>Serving after a restore was identical: TTFT p50 ~31 ms, inter-token ~3.4 ms, no JIT spikes.</li><li>The checkpoint was taken on one H200. 4 of 7 restores landed on a different H200, and restore times were the same either way (12.3–13.8 s). Driver 580 handles the GPU migration.</li><li>The stage chart is the real lesson. Weights were 5 s. Graph capture (47 s) and Python/runtime boot (27 s) were the cold start.</li></ul><h3>Limits / where it doesn’t apply</h3><ul><li><strong>First-ever start isn’t faster.</strong> Checkpoint Job (125 s) + first restore (13 s) is a bit slower than one cold start (127 s). The win starts at the second scale-up.</li><li><strong>Warm node, local NVMe.</strong> Repeat restores likely hit the page cache. Network storage or a cold node will be slower.</li><li><strong>MIG isn’t supported.</strong> The agent only understands whole GPUs (nvidia.com/gpu, GPU-… UUIDs). My read of the code: it has no way to map a checkpoint onto a MIG slice.</li><li><strong>RDMA / InfiniBand KV-cache connectors</strong> (Mooncake in my case) are out. Dynamo’s snapshot code forces NCCL_IB_DISABLE=1 because CRIU and cuda-checkpoint can&#39;t restore InfiniBand state.</li><li><strong>Multimodal, embedding, diffusion workers aren’t validated.</strong> Multi-GPU is experimental.</li><li><strong>Needs driver 580+ and a privileged node agent.</strong></li><li><strong>Small things:</strong> restored pods took ~34 s to scale down (they ignored SIGTERM until the grace period ran out), and the operator waited ~200 s before creating the checkpoint Job. I haven’t tracked that one down yet.</li></ul><h3>Would I use this in prod?</h3><p>Yes, for the right models. Not as a blanket fix.</p><p><strong>Pros:</strong></p><ul><li>8x faster scale-up, same serving performance.</li><li>~8x fewer GPU-seconds wasted per scale-up.</li><li>Opt-in per worker. Workloads without the checkpoint block are untouched.</li></ul><p><strong>Cons:</strong></p><ul><li>Experimental API (it lives under experimental:).</li><li>Narrow fit today: one whole GPU, no RDMA connectors, no multimodal.</li><li>Extra moving parts: privileged agent, per-runtime placeholder image, checkpoint storage.</li></ul><p><strong>What I’d do before (or alongside) snapshot:</strong></p><ul><li>Capture fewer CUDA graph sizes. That stage alone was 47 s.</li><li>Size the KV cache explicitly instead of profiling.</li><li>Shorten startup probe periods. Some of our deployments use 60 s.</li><li>Pre-pull images on GPU nodes.</li></ul><h3>Cleanup</h3><pre>$ kubectl delete dgd gptoss20b-snapshot gptoss20b-baseline -n demo<br>$ kubectl delete dckpt --all -n demo<br>$ helm uninstall snapshot -n demo     # or kubectl delete -f of the rendered manifest<br>$ helm upgrade dynamo-platform ... --reuse-values --set dynamo-operator.checkpoint.enabled=false</pre><p>deletionPolicy: Retain keeps the checkpoint files on the PVC. Delete them yourself if you don&#39;t want them.</p><h3>Resources</h3><ul><li><a href="https://developer.nvidia.com/blog/nvidia-dynamo-snapshot-fast-startup-for-inference-workloads-on-kubernetes/">NVIDIA blog: Dynamo Snapshot</a></li><li><a href="https://docs.nvidia.com/dynamo/v1.0.1/kubernetes-deployment/deployment-guide/snapshot">Dynamo 1.4.2 snapshot guide</a></li><li><a href="https://github.com/ai-dynamo/snapshot">Placeholder Dockerfile and Makefile</a></li><li>CRIU, cuda-checkpoint</li></ul><p><em>Has anyone tried this with bigger models or over network storage? I’m curious how restore time scales once the checkpoint is 80+ GiB.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c55ee2b7bff7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 16 GB/s to 189 GB/s: Debugging a 400G RoCE Fabric for NCCL]]></title>
            <link>https://medium.com/@shivank1128/from-16-gb-s-to-189-gb-s-debugging-a-400g-roce-fabric-for-nccl-e0b6ce56c4a2?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0b6ce56c4a2</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rdma]]></category>
            <category><![CDATA[gdr]]></category>
            <category><![CDATA[rocev2]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 10:21:15 GMT</pubDate>
            <atom:updated>2026-08-26T10:21:15.761Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ZiDy0faLeK8QQWdthKFzkw.png" /></figure><p><em>A systematic walk through everything that can silently throttle GPU collective performance — MTU enums, NUMA topology, PCIe ACS, Kubernetes device plugins, and rail isolation.</em></p><h3>The starting point</h3><p>Two H200 nodes. Eight 400 Gb/s RoCE NICs each. A rail-optimized fabric. On paper, roughly 3.1 Tb/s of east-west bandwidth per node.</p><p>The first NCCL benchmark returned <strong>16 GB/s</strong>.</p><p>That is about 4% of what the hardware should deliver. This post walks through every layer that turned out to be involved, in the order the problems actually surfaced — because the order matters. Several times I chased the wrong layer because I hadn’t yet established what the layer below was capable of.</p><p>Final result: <strong>189 GB/s</strong>. Not perfect, and I’ll explain what’s still on the table.</p><h3>Before anything: know what you’re measuring</h3><p>Three things caused early confusion. Getting these straight saves hours.</p><h3>algbw vs busbw</h3><p>nccl-tests reports two bandwidth columns and they mean different things:</p><ul><li><strong>algbw</strong> (algorithm bandwidth) = bytes moved by the user ÷ time. What the application sees.</li><li><strong>busbw</strong> (bus bandwidth) = the actual traffic on the wire, accounting for how many times data traverses the interconnect.</li></ul><p>For all_reduce with n ranks:</p><pre>busbw = algbw × 2(n − 1) / n</pre><p>Ranks Multiplier 2 1.0 8 1.75 16 1.875</p><p>At 2 ranks the two columns are identical — which initially looked like a bug in the tool. It isn’t; the multiplier is exactly 1.0.</p><p><strong>busbw is the number to compare against hardware capability.</strong></p><h3>“Avg bus bandwidth” is not your result</h3><p>At the bottom of every run:</p><pre># Avg bus bandwidth : 45.3857</pre><p>This is the arithmetic mean across <em>every</em> message size in the sweep — from 8 bytes to 8 GiB. Half those rows are latency-bound and score ~0.00 GB/s.</p><p>Add more small sizes to your sweep and this number drops, while your actual performance is unchanged. <strong>It is not a metric.</strong> Report the plateau at large message sizes.</p><h3>One run proves nothing</h3><p>More on this later, but: I found two rails “underperforming” by 5% and 11%, wrote up a hardware fault hypothesis, and then both came back completely normal on re-test. Run-to-run variance on this platform reached 11%.</p><p><strong>Minimum three runs. Report the median.</strong></p><h3>Part 1: Validate the fabric before touching NCCL</h3><p>The single most useful decision in this whole exercise was refusing to tune NCCL until I knew what the network could actually do.</p><p>ib_write_bw from the perftest package gives you raw RDMA bandwidth with no GPU, no collective algorithm, and no framework in the path.</p><pre># server<br>ib_write_bw -d mlx5_0 -x 3 -F --report_gbits -q 4 -s 1048576 -D 10</pre><pre># client<br>ib_write_bw -d mlx5_0 -x 3 -F --report_gbits -q 4 -s 1048576 -D 10 &lt;server-ip&gt;</pre><p>Flag Why -x 3 GID index 3 = RoCEv2 with IPv4 encapsulation -F Suppress the CPU-frequency warning --report_gbits Gb/s rather than MB/s, so it compares to line rate directly -q 4 4 queue pairs — spreads across ECMP paths -s 1048576 1 MB messages, minimising per-message overhead -D 10 Run for 10 seconds rather than a fixed iteration count</p><h3>Mapping RDMA devices to interfaces</h3><p>RDMA device names (mlx5_N) don&#39;t correspond to anything intuitive. They follow PCIe enumeration order, so mlx5_9 might be your second rail and mlx5_4 your sixth. Never assume — derive it:</p><pre>for d in /sys/class/infiniband/*; do<br>  case &quot;$(cat $d/ports/1/rate 2&gt;/dev/null)&quot; in *400*) ;; *) continue ;; esac<br>  ib=$(basename $d)<br>  nd=$(ls $d/device/net/ 2&gt;/dev/null | head -1)<br>  [ -z &quot;$nd&quot; ] &amp;&amp; continue<br>  printf &quot;%-9s %-14s %-10s mtu=%-6s rdma=%-6s %s\n&quot; &quot;$ib&quot; &quot;$nd&quot; \<br>    &quot;$(cat $d/ports/1/state)&quot; \<br>    &quot;$(cat /sys/class/net/$nd/mtu)&quot; \<br>    &quot;$(ibv_devinfo -d $ib 2&gt;/dev/null | awk &#39;/active_mtu/{print $2; exit}&#39;)&quot; \<br>    &quot;$(ip -o -4 addr show dev $nd | awk &#39;{print $4}&#39;)&quot;<br>done</pre><p>This one loop became the most-used command of the entire exercise. It answers: which RDMA device, which netdev, is the link up, what MTU, what IP — all at once.</p><h3>Part 2: The MTU trap</h3><p>The first sweep produced this:</p><p>Rail Bandwidth Efficiency Rail 1 <strong>391.4 Gb/s</strong> 97.9% Rails 2–8 ~369.7 Gb/s 92.4%</p><p>One rail fast, seven rails 5.9% slower. Same hardware, same switch, same cables.</p><p>The difference: active_mtu was <strong>4096</strong> on rail 1 and <strong>1024</strong> on the rest.</p><h3>Why RDMA MTU is not the MTU you think it is</h3><p>This is the part that trips people up. <strong>RDMA supports only five payload sizes</strong>, defined by the InfiniBand spec and inherited by RoCE:</p><p>Enum Payload 1 256 B 2 512 B 3 1024 B 4 2048 B <strong>5</strong> <strong>4096 B</strong></p><p><strong>4096 is the ceiling.</strong> There is no 9000 in RDMA, no matter what you set on the netdev.</p><p>The driver derives the RDMA MTU by taking the Ethernet MTU, subtracting a fixed 100-byte allowance for RoCEv2 encapsulation, and rounding <strong>down</strong> to the nearest legal enum:</p><p>Header Bytes GRH (global route header) 40 UDP 8 BTH (base transport header) 12 Extended XRC 8 Extended AtomicETH 28 ICRC 4 <strong>Reserved total</strong> <strong>100</strong></p><pre>rdma_mtu = largest legal enum ≤ (netdev_mtu − 100)</pre><p>Which produces:</p><p>Netdev MTU Minus 100 RDMA MTU Wire efficiency 1500 1400 <strong>1024</strong> ~92.4% 4200 4100 <strong>4096</strong> ~98.1% 9000 8900 <strong>4096</strong> (capped) ~98.1%</p><p>Three consequences that aren’t obvious:</p><ol><li><strong>The threshold is 4196, not 9000.</strong> Anything above that behaves identically.</li><li><strong>It’s a cliff, not a slope.</strong> Going 1500 → 1600 buys you <em>nothing</em> — still enum 3. You have to clear 4196.</li><li><strong>Header overhead is what costs you.</strong> At 1024 B payload, ~78 bytes of fixed overhead is 7.1% of the wire. At 4096 B it’s 1.9%. That difference is exactly the 92.4% vs 98.1% observed.</li></ol><h3>Both endpoints negotiate</h3><p>This is why only <em>one</em> rail was fast. At queue-pair setup, the two ends exchange their path MTU and take the <strong>minimum</strong>:</p><pre>effective_mtu = min(local active_mtu, remote active_mtu)</pre><p>One node had MTU 1500 on seven interfaces. The other node was configured correctly throughout. <strong>One misconfigured endpoint dragged down every rail it participated in.</strong></p><h3>The fix</h3><pre>for i in &lt;rail interfaces&gt;; do<br>  ip link set dev $i up          # link up FIRST<br>  ip link set dev $i mtu 9000    # then MTU<br>done</pre><p>Order matters — some drivers reject an MTU change while the interface is administratively down. I lost twenty minutes to that.</p><p><strong>Result: 369.7 → 392.2 Gb/s.</strong> A 6% gain across seven of eight rails, from a single misconfigured node.</p><h3>Part 3: The false positive that taught me methodology</h3><p>After the MTU fix, a full sweep showed two rails still low:</p><p>Rail Bandwidth vs. peers Rail A 371.8 −5.2% Rail B 348.6 −11.1%</p><p>I had good reasons to believe these:</p><ul><li>Another rail had been run twice and returned <em>identical</em> results (392.17 both times) — the test looked highly repeatable</li><li>Iteration counts scaled proportionally with the low bandwidth, meaning the shortfall was sustained across the full 10-second window, not a brief stall</li><li>One of the two rails had also been the lowest in the earlier sweep</li></ul><p>I wrote up a hardware fault hypothesis, complete with lane-error counter analysis.</p><p><strong>Then I re-ran them. 390.35 and 390.32. Both completely normal.</strong></p><h3>What I got wrong</h3><p>Two adjacent stable runs on <em>one</em> rail says nothing about variance across rails or across time. The real spread was ~11%.</p><p>Likely causes of the transients: CPU frequency or C-state transitions during the run, scheduler placing the test process across the interconnect from the NIC, or another workload on the node.</p><h3>The methodology that replaced it</h3><ul><li><strong>Minimum 3 runs per device</strong>, report median and range</li><li><strong>Pin to the NIC’s NUMA node</strong> to remove scheduler placement as a variable:</li><li>numactl --cpunodebind=&lt;n&gt; --membind=&lt;n&gt; ib_write_bw ...</li><li><strong>Confirm the CPU governor</strong> is in performance mode</li><li><strong>Investigate only what’s consistently low</strong> across repetitions</li></ul><p>This cost me an hour and a retracted finding. Worth it.</p><h3>Part 4: Understanding GPU ↔ NIC topology</h3><p>With the fabric proven at 392 Gb/s per rail, the network was exonerated. NCCL was still at 16 GB/s. Time to look at the host.</p><pre>nvidia-smi topo -m</pre><p>This matrix is the single most important diagnostic for GPU networking. The GPU-to-NIC cells tell you the PCIe path:</p><p>Code Meaning Impact PIX Same PCIe switch Best PXB Multiple switches, same root complex Good NODE Same NUMA node, different root complex Acceptable PHB Via host bridge Poor SYS <strong>Across sockets (UPI/xGMI)</strong> Worst</p><p>A rail-optimized node typically pairs each GPU with one NIC at PIX:</p><p>GPU NUMA PIX rail 0–3 0 Rails 1–4 4–7 1 Rails 5–8</p><h3>The NCCL_NET_GDR_LEVEL trap</h3><p>The original config had:</p><pre>NCCL_NET_GDR_LEVEL=SYS</pre><p>This tells NCCL: <em>use GPUDirect RDMA even if the GPU and NIC are on different sockets.</em></p><p>It sounds like “enable the good thing everywhere.” It is not. When a NIC DMAs into GPU memory across an inter-socket link, PCIe read performance in that direction is poor — often <strong>worse than staging through host memory</strong>.</p><p>Measured, on a cross-socket pairing:</p><p>Setting Result GDR_LEVEL=SYS (forced GDR) 15.2 GB/s Default (PXB — GDR refused) <strong>26.0 GB/s</strong></p><p><strong>+71% from deleting one environment variable.</strong></p><p>You can see it in the debug log. With GDR active:</p><pre>Connected all rings, use ring PXN 0 GDR 1<br>Channel 00 : ... via NET/IBext/0/GDRDMA</pre><p>Without:</p><pre>Connected all rings, use ring PXN 0 GDR 0<br>Channel 00 : ... via NET/IBext/0</pre><p>The /GDRDMA suffix is the tell.</p><h3>And the same trap in the other direction</h3><p>Later, on a PHB pairing, I set NCCL_NET_GDR_LEVEL=PHB to force GDR on. Result: <strong>32.6 → 13.6 GB/s.</strong> A 58% regression.</p><p><strong>The lesson: NCCL’s default threshold exists for a reason.</strong> It refuses GDR on paths where GDR is slower. Overriding it in either direction, without measuring, tends to make things worse. Fix the topology instead of overriding the guard.</p><h3>Fixing the topology</h3><p>Aligning GPU and NIC to the same NUMA node: <strong>26.0 → 32.6 GB/s</strong>.</p><p>The fully-aligned PIX pair is worth more still, but requires pinning the specific GPU rather than accepting whatever the scheduler hands you.</p><h3>Part 5: Exposing RDMA to Kubernetes</h3><p>Running these workloads under Kubernetes adds a layer. Pods need access to the RDMA devices, and ideally the scheduler should understand them as a resource.</p><h3>hostNetwork or device plugin?</h3><p>hostNetwork: true RDMA device plugin Setup Trivial One DaemonSet + ConfigMap Pod IPs Host&#39;s Normal pod networking Port conflicts Yes — one pod per node for anything with fixed ports No Scheduler awareness None Named resources, quotas NUMA alignment possible No Yes (in principle — see below)</p><p><strong>The device plugin is the right answer</strong> for anything beyond a one-off benchmark, and especially for frameworks with wide dynamic port ranges.</p><p>The key insight that makes it work: <strong>RDMA defaults to shared network-namespace mode.</strong> Verbs devices are visible in every netns, and the RDMA data path bypasses the network namespace entirely. So a pod can have an overlay IP for control traffic while its RDMA data flows over the physical rail.</p><p>Verify with:</p><pre>rdma system show    # want: netns shared</pre><h3>The configuration</h3><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: rdma-devices<br>data:<br>  config.json: |<br>    {<br>      &quot;periodicUpdateInterval&quot;: 300,<br>      &quot;configList&quot;: [<br>        {&quot;resourceName&quot;:&quot;hca_gpu0&quot;,&quot;rdmaHcaMax&quot;:16,&quot;selectors&quot;:{&quot;ifNames&quot;:[&quot;&lt;rail1-netdev&gt;&quot;]}},<br>        {&quot;resourceName&quot;:&quot;hca_gpu1&quot;,&quot;rdmaHcaMax&quot;:16,&quot;selectors&quot;:{&quot;ifNames&quot;:[&quot;&lt;rail2-netdev&gt;&quot;]}}<br>        // ... one per rail<br>      ]<br>    }</pre><p>Naming each resource after the GPU it pairs with encodes the topology so that whoever writes a pod spec doesn’t have to rediscover it.</p><p>Notes:</p><ul><li>Selectors match on <strong>netdev name</strong>, not mlx5_N</li><li>rdmaHcaMax is a scheduling cap, not a hardware partition — shared mode gives each pod full device access</li><li><strong>Exclude storage/management NICs.</strong> If NCCL picks a 100G management NIC, throughput drops to a quarter</li><li>The ConfigMap is read at plugin <strong>startup</strong> — edit it and you must restart the DaemonSet</li></ul><h3>The missing kernel module</h3><p>The plugin registered all resources but exposed <strong>zero</strong> devices:</p><pre>Updating &quot;rdma/hca_gpu0&quot; devices<br>exposing &quot;0&quot; devices<br>missing RDMA device spec for device 0000:XX:00.0, RDMA device &quot;umad&quot; not found</pre><p>The plugin requires <strong>both</strong> uverbsN and umadN character devices per HCA. umad comes from the ib_umad module, which wasn&#39;t loaded.</p><pre>modprobe ib_umad<br>echo ib_umad &gt;&gt; /etc/modules-load.d/rdma.conf</pre><p>Also worth loading (one modprobe per module — passing two names treats the second as a <em>parameter</em> to the first and silently ignores it):</p><pre>modprobe rdma_cm<br>modprobe rdma_ucm    # creates /dev/infiniband/rdma_cm</pre><p>After that: exposing &quot;16&quot; devices.</p><p><strong>Bonus finding:</strong> the plugin does real device isolation. A pod requesting one rail sees only that rail’s character devices:</p><pre>$ ls /dev/infiniband/<br>issm10 rdma_cm umad10 uverbs10</pre><p>Not all twelve. Better isolation than expected.</p><h3>Part 6: Topology Manager, and why it didn’t help</h3><p>Kubernetes has a mechanism for exactly the GPU/NIC alignment problem: <strong>Topology Manager</strong>.</p><p>The idea: each device plugin reports a NUMA affinity hint with its devices. Kubelet intersects the hints across all resources a pod requested, and either allocates from the common NUMA node or refuses to admit the pod.</p><pre># /var/lib/kubelet/config.yaml<br>topologyManagerPolicy: restricted<br>topologyManagerScope: pod<br>cpuManagerPolicy: static<br>reservedSystemCPUs: &quot;0-1,48-49&quot;</pre><p>Policy choice matters:</p><p>Policy Behaviour none No alignment (default) best-effort Prefers alignment, never rejects restricted Prefers alignment, rejects if hints conflict single-numa-node Strict — <strong>breaks any pod spanning both sockets</strong></p><p>If you run both fractional (1–4 GPU) and whole-node (8 GPU) workloads, single-numa-node will make the 8-GPU pods permanently unschedulable. Use restricted.</p><p>Two prerequisites people miss:</p><ul><li><strong>Only </strong><strong>Guaranteed QoS pods are considered.</strong> Requests must exactly equal limits on every resource, with integer CPU.</li><li><strong>Changing </strong><strong>cpuManagerPolicy requires deleting </strong><strong>/var/lib/kubelet/cpu_manager_state</strong> before restarting kubelet, or kubelet won&#39;t start.</li></ul><h3>It didn’t work — and the logs said why</h3><p>Test pod: request one arbitrary GPU plus one specific NUMA-1 rail. Expected: a NUMA-1 GPU.</p><p>Got a NUMA-0 GPU. Three times running.</p><p>The kubelet journal:</p><pre>topology_hints.go: &quot;Resource does not have a topology preference&quot; resource=&quot;rdma/hca_gpu6&quot;<br>scope_pod.go: &quot;TopologyHints&quot; hints={&quot;nvidia.com/gpu&quot;:[<br>    {&quot;NUMANodeAffinity&quot;:1,&quot;Preferred&quot;:true},<br>    {&quot;NUMANodeAffinity&quot;:2,&quot;Preferred&quot;:true}],<br>  &quot;rdma/hca_gpu6&quot;:null}<br>scope_pod.go: &quot;PodTopologyHint&quot; bestHint={&quot;NUMANodeAffinity&quot;:1,&quot;Preferred&quot;:true}</pre><p>Decoding the bitmask: 1 = NUMA 0, 2 = NUMA 1, 3 = both.</p><p><strong>The RDMA shared device plugin returns </strong><strong>null topology hints.</strong> Per the Kubernetes documentation, a nil TopologyInfo means &quot;no NUMA affinity preference.&quot; So there was nothing to intersect against — kubelet took the GPU plugin&#39;s first preferred hint, which was NUMA 0.</p><p>The GPU plugin was working perfectly. The RDMA plugin simply doesn’t implement the interface.</p><h3>The workaround</h3><p>Since the allocator consistently offers NUMA 0 first, <strong>request a NUMA-0 rail and you get alignment for free</strong>:</p><pre>GPU NUMA=0 | HCA NUMA=0  ==&gt; ALIGNED</pre><p>Caveat: this is exploiting a bias, not a guarantee. Once all NUMA-0 GPUs are allocated, new pods land on NUMA 1 and the rule inverts.</p><h3>Real fixes, ranked</h3><ol><li><strong>Patch the plugin.</strong> It already knows each device’s PCI address; it needs to read /sys/bus/pci/devices/&lt;bdf&gt;/numa_node and populate TopologyInfo. Small change, worth filing upstream.</li><li><strong>Node-pool separation.</strong> Model NUMA islands at the scheduling layer.</li><li><strong>Explicit device pinning</strong> via NVIDIA_VISIBLE_DEVICES, dropping the GPU resource request. Deterministic, but the scheduler stops tracking that GPU.</li><li><strong>Dynamic Resource Allocation (DRA).</strong> The proper answer — see below.</li></ol><h3>A note on DRA</h3><p>DRA graduated to GA in Kubernetes 1.34. It expresses exactly this constraint declaratively:</p><pre>devices:<br>  requests:<br>    - name: gpu<br>      exactly: {deviceClassName: gpu.nvidia.com, count: 1}<br>    - name: nic<br>      exactly:<br>        deviceClassName: &lt;nic-driver&gt;<br>        count: 1<br>        selectors:<br>          - cel: {expression: &#39;device.attributes[&quot;&lt;driver&gt;&quot;].rdma == true&#39;}<br>  constraints:<br>    - requests: [&quot;gpu&quot;, &quot;nic&quot;]<br>      matchAttribute: &quot;resource.kubernetes.io/pcieRoot&quot;</pre><p>That matchAttribute constraint gives you the exact <strong>PIX pair</strong>, not merely NUMA-level alignment. It&#39;s the correct long-term answer.</p><p>The catch is version. On anything below 1.34 the API is alpha and materially different. If your cluster is older, the cost isn’t DRA — it’s the control-plane upgrade. Weigh that against the incremental gain: Topology Manager already eliminates the SYS cliff (which cost 71%), while DRA closes the NODE → PIX gap (a few percent).</p><h3>Part 7: The big one — rail isolation</h3><p>Scaling to 8 GPUs per node, 16 ranks total, the result was <strong>40 GB/s</strong>. Intra-node NVLink alone measured 174 GB/s, so the network was clearly the constraint.</p><p>The debug log showed only three rings, and net device indices /1 and /3 — <strong>two rails out of eight</strong>.</p><p>I assumed a channel-count problem and started tuning. Then I checked something more basic.</p><h3>The test</h3><p>Each rail is its own subnet. What happens if a NIC on rail 4 tries to reach rail 1 on the other node?</p><pre># node A<br>ib_write_bw -d &lt;rail4-device&gt; -x 3 -F --report_gbits -q 4 -s 1048576 -D 10 &lt;nodeB-rail1-ip&gt;</pre><p>Result:</p><pre>Completion with error at client<br>Failed status 12: wr_id 2 syndrom 0x81<br>scnt=512, ccnt=0</pre><p>Status 12 is IBV_WC_RETRY_EXC_ERR — transport retry counter exhausted. The queue pairs connected (GID exchange happens over a separate TCP channel), but <strong>not a single RDMA write reached the far side.</strong> 512 sent, zero completed.</p><p>Control test, same NIC, rail 4 → rail 4:</p><pre>1048576   280505   392.18 Gb/sec</pre><p>Full line rate.</p><p><strong>Rails are isolated L2 domains. A NIC on rail N can only reach rail N.</strong></p><h3>A trap worth flagging</h3><p>An earlier ping between rails <em>succeeded</em>, which nearly sent me down the wrong path. Why?</p><pre>ping -I 192.168.104.13 192.168.101.14   # worked</pre><p>The -I flag only set the <em>source address</em>. There&#39;s no route to the rail-1 subnet via the rail-4 interface, so the packet actually egressed the rail-1 interface carrying a rail-4 source IP. Asymmetric, and it only worked because reverse-path filtering wasn&#39;t strict.</p><p><strong>ICMP through the kernel stack tells you nothing about RDMA connectivity.</strong> RoCE bypasses the kernel routing table entirely — the HCA resolves the destination GID itself. Always test RDMA reachability with an RDMA tool.</p><h3>Why this shapes everything</h3><p>In a ring across 16 ranks, rank 7 on node A (GPU7 → rail 8) needs to talk to rank 8 on node B (GPU0 → rail 1). <strong>That connection is physically impossible.</strong></p><p>This is a deliberate design in most rail-optimized deployments — one switch per rail, no inter-rail path, which keeps the fabric simple and cheap. But it constrains how collectives can be built, and two NCCL settings become mandatory rather than optional:</p><p>Setting Why <strong>PXN enabled</strong> (do <em>not</em> set NCCL_PXN_DISABLE=1) PXN routes a GPU&#39;s inter-node traffic through a local peer GPU over NVLink so the network hop is rail-aligned. At 174 GB/s intra-node, that hop is nearly free. <strong>This is what makes the topology work at all.</strong> <strong>NCCL_CROSS_NIC=0</strong> Prevents a ring using different NICs for send and receive — which would be exactly the cross-rail connection that fails</p><p>I had disabled PXN earlier on the reasoning that it “consolidates traffic onto fewer rails.” On a fully-connected fabric that reasoning holds. On an isolated one it’s backwards.</p><h3>Part 8: Scaling up — the channel count</h3><p>With rail isolation understood, one setting produced the single biggest jump of the entire exercise:</p><pre>NCCL_MIN_NCHANNELS=16</pre><p><strong>40.1 → 189.0 GB/s.</strong> A 4.7× improvement.</p><p>The reason is straightforward once you see it. NCCL had built <strong>3 rings</strong>. Each ring crosses the node boundary once, and with CROSS_NIC=0 each is pinned to a single NIC. Three rings → at most three rails → roughly one rail&#39;s worth of throughput after overhead.</p><p>Forcing more channels recruits more rails.</p><h3>The full working configuration</h3><pre>NCCL_IB_HCA=&lt;comma-separated 400G rails only&gt;<br>NCCL_IB_GID_INDEX=3<br>NCCL_IB_QPS_PER_CONNECTION=4<br>NCCL_IB_SPLIT_DATA_ON_QPS=1<br>NCCL_CROSS_NIC=0<br>NCCL_MIN_NCHANNELS=16<br>NCCL_NCHANNELS_PER_NET_PEER=8<br>NCCL_BUFFSIZE=8388608<br>NCCL_P2P_NET_CHUNKSIZE=524288<br># NCCL_NET_GDR_LEVEL deliberately unset — let NCCL decide<br># NCCL_PXN_DISABLE deliberately unset — PXN is required here</pre><h3>How to verify rails are actually in use</h3><p>Don’t trust the plateau number alone. Count the device indices in the debug output:</p><pre>kubectl logs &lt;launcher&gt; | grep -oP &#39;Plugin v\d+/\d+&#39; | sort | uniq -c</pre><p>You want to see indices /0 through /7 in roughly equal proportion. One or two dominant buckets means you&#39;re still bottlenecked on rail count.</p><p>And the ring count:</p><pre>kubectl logs &lt;launcher&gt; | grep -oP &#39;Channel \d+/\d+&#39; | head -3</pre><p>The second number is total rings.</p><h3>Part 9: Firmware and PCIe tuning</h3><p>Two lower-level knobs worth knowing about, both requiring NVIDIA’s MFT package.</p><h3>MAX_ACC_OUT_READ</h3><p>Controls how many outstanding PCIe read requests the NIC accumulates before waiting for completions. The default is conservative; on a 400G NIC doing RDMA reads from host or GPU memory, it limits read-pipeline depth.</p><pre>mstconfig -d &lt;bdf&gt; -y set ADVANCED_PCI_SETTINGS=1 MAX_ACC_OUT_READ=128</pre><p><strong>Gotcha:</strong> MAX_ACC_OUT_READ doesn&#39;t appear in mstconfig q output at all until ADVANCED_PCI_SETTINGS is enabled. If you query for it first and get nothing, that&#39;s why — it&#39;s gated, not missing.</p><p>Applying requires a reboot or mstfwreset -d &lt;bdf&gt; -y reset.</p><p><strong>mstfwreset drops the link.</strong> The netdev is destroyed and recreated. If your IP configuration is runtime-only, it vanishes. (More on this in the lessons section — it bit me repeatedly.)</p><h3>PCIe ACS</h3><p>Access Control Services is a PCIe security feature that forces peer-to-peer transactions up to the root complex for validation rather than letting them route directly through the switch.</p><p>For a GPU and NIC that are a PIX pair under a common switch, this means a GPUDirect transfer that <em>should</em> go:</p><pre>GPU → switch → NIC</pre><p>instead goes:</p><pre>GPU → switch → root complex → switch → NIC</pre><p>Longer path, higher latency, and the root complex becomes a bottleneck.</p><p>The standard disable script:</p><pre>#!/bin/bash<br>[ &quot;$EUID&quot; -ne 0 ] &amp;&amp; { echo &quot;must be root&quot; &gt;&amp;2; exit 1; }</pre><pre>for BDF in $(lspci -D | awk &#39;{print $1}&#39;); do<br>    setpci -v -s &quot;$BDF&quot; ECAP_ACS+0x6.w &gt;/dev/null 2&gt;&amp;1 || continue<br>    CUR=$(setpci -s &quot;$BDF&quot; ECAP_ACS+0x6.w)<br>    [ &quot;$CUR&quot; = &quot;0000&quot; ] &amp;&amp; continue<br>    setpci -s &quot;$BDF&quot; ECAP_ACS+0x6.w=0000 || continue<br>    NEW=$(setpci -s &quot;$BDF&quot; ECAP_ACS+0x6.w)<br>    [ &quot;$NEW&quot; = &quot;0000&quot; ] \<br>      &amp;&amp; logger -t disable-acs &quot;ACS disabled on $BDF&quot; \<br>      || logger -t disable-acs &quot;FAILED on $BDF: $NEW&quot;<br>done</pre><p>Points worth noting:</p><ul><li><strong>Use </strong><strong>lspci -D</strong>, not the short form — gives domain-qualified BDFs</li><li><strong>Include a shebang</strong> if you’ll run it from systemd</li><li><strong>It does not survive reboot</strong> — PCIe config space is reinitialised by firmware. Install as a systemd unit with Before=kubelet.service</li><li><strong>It removes IOMMU isolation between PCIe devices.</strong> Standard and NVIDIA-recommended for single-tenant bare-metal GPU nodes. Not appropriate where untrusted workloads or VM passthrough are involved. Flag it to whoever owns security posture.</li></ul><p>Interesting observation: running this on two supposedly identical nodes changed <strong>3 devices on one and 49 on the other</strong>. Worth investigating why rather than shrugging — it usually means a BIOS setting differs.</p><h3>Part 10: Things that looked like faults but weren’t</h3><p>Three separate scares, each of which cost real time.</p><h3>MIG partitioning looks like broken NVLink</h3><p>nvidia-smi topo -m showed most GPU-to-GPU pairs as NODE/SYS instead of NV18. Worse, it was <em>asymmetric</em> — GPU1&#39;s row claimed NV18 to GPU0, but GPU0&#39;s row said NODE to GPU1.</p><p>That asymmetry looked like link-state corruption. It wasn’t.</p><p>nvidia-smi -L revealed six of eight GPUs had MIG instances. <strong>MIG mode disables NVLink peer-to-peer</strong> — a documented restriction. topo -m correctly falls back to reporting the PCIe path for those GPUs.</p><p>Everything else was healthy:</p><pre>Fabric State: Completed, Status: Success<br>Discovered Link Mask: 3ffff   Enabled Link Mask: 3ffff    (18 links, all up)<br>Replay/Recovery/CRC Errors: 0</pre><p><strong>Check </strong><strong>nvidia-smi -L before diagnosing NVLink from </strong><strong>topo -m.</strong></p><p>Practical consequence: MIG-partitioned GPUs can’t participate in multi-GPU collectives. If you’re running both inference and training, keep them on separate nodes or plan for the MIG reconfiguration.</p><h3>Pre-FEC lane errors that don’t matter (yet)</h3><p>ethtool -S &lt;netdev&gt; | grep rx_err_lane showed one lane with 28 billion errors against ~500 million on its neighbours — a 50× imbalance.</p><p>These are <strong>pre-FEC</strong> counts. RS-FEC is always on for 400G NDR, correction is fixed-latency, and every post-FEC counter read zero:</p><pre>rx_crc_errors_phy: 0    rx_symbol_err_phy: 0    rx_discards_phy: 0</pre><p>The rail measured full line rate. So the errors cost nothing today.</p><p>But a lane consuming most of the FEC correction budget has <strong>no headroom</strong>. As optics age or ambient temperature rises, it crosses the threshold and you get uncorrected errors → RDMA retransmits → PFC backpressure → stalled collectives.</p><p><strong>Latent risk, not a current fault.</strong> Two caveats on interpreting these:</p><ul><li>Counters are cumulative since driver load with no timestamp. Measure a <em>delta</em> over a fixed window under load, not the absolute number.</li><li>Without a baseline from other rails you can’t tell abnormal from normal. Sample all of them.</li></ul><p>For a real assessment you want the FEC histogram (mlxlink --show_fec --show_histogram), which shows whether correction is concentrated in low bins or pushing the ceiling.</p><h3>The errno 93 warnings</h3><pre>wrap_ibv_query_port_speed failed with error Protocol not supported errno 93</pre><p>Alarming, harmless. ibv_query_port_speed is an InfiniBand call and doesn&#39;t apply to Ethernet link layer. NCCL falls back to sysfs.</p><h3>Part 11: The operational lesson that cost the most time</h3><p>Not a performance issue. A configuration-persistence issue. And it cost more debugging cycles than anything else in this exercise.</p><p><strong>The rail configuration — IPs, MTU, link state — was applied at runtime and never written to a persistent config.</strong></p><p>Over one session, one node lost its entire rail setup <strong>three times</strong>:</p><p>Trigger Result Reboot #1 All 8 rails: no IP, MTU 1500, qdisc noop, DOWN Reboot #2 Same mstfwreset Links down (IPs survived that time)</p><p>Each time, the symptom presented as something else entirely. After the firmware reset, the NCCL job failed with:</p><pre>worker-1: NET/IB : No device found.<br>Detected mixed local CollNet device counts across ranks (min 0, max 8)</pre><p>That reads like a device plugin problem, or a driver problem, or an NCCL problem. It was none of those — the ports simply weren’t ACTIVE because nothing had brought the links back up.</p><p><strong>Write your network configuration to a persistent config file before you touch anything else.</strong></p><pre># /etc/netplan/60-roce-rails.yaml<br>network:<br>  version: 2<br>  renderer: networkd<br>  ethernets:<br>    &lt;rail1-netdev&gt;: {addresses: [10.0.1.10/24], mtu: 9000}<br>    &lt;rail2-netdev&gt;: {addresses: [10.0.2.10/24], mtu: 9000}<br>    # ... one per rail</pre><pre>chmod 600 /etc/netplan/60-roce-rails.yaml<br>netplan generate &amp;&amp; netplan try &amp;&amp; netplan apply</pre><p>netplan try auto-reverts after 120 seconds if you lose connectivity — safe as long as your management interface isn&#39;t one of the rails.</p><h3>Results summary</h3><p>Stage Result Change <strong>Single rail, </strong><strong>ib_write_bw</strong> Baseline (MTU 1024) 369.7 Gb/s — After MTU fix (4096) <strong>392.2 Gb/s</strong> +6.1% <strong>2 nodes × 1 GPU, NCCL</strong> GDR_LEVEL=SYS, cross-socket 15.2 GB/s — Default GDR, cross-socket 26.0 GB/s +71% NUMA-aligned <strong>32.6 GB/s</strong> +115% <strong>2 nodes × 8 GPU, NCCL</strong> Initial (3 rings, 2 rails) 40.1 GB/s — MIN_NCHANNELS=16 <strong>189.0 GB/s</strong> <strong>+371%</strong> <strong>Reference</strong> Intra-node NVLink, 8 GPU 174 GB/s Per-node fabric capacity ~3.1 Tb/s</p><p><strong>16 GB/s → 189 GB/s overall.</strong></p><h3>What’s still on the table</h3><p>Being honest about the gap: 189 GB/s busbw at 16 ranks is ~202 GB/s of actual inter-node traffic per node — roughly <strong>51% of fabric capacity</strong>. A well-tuned 2-node 8×400G setup should reach 300–370 GB/s.</p><p>Remaining items:</p><ul><li><strong>A mid-size bandwidth valley.</strong> Performance collapses between 64 MB and 512 MB (down to ~7 GB/s at 134 MB) before recovering at 1 GB. Present in every collective. Prime suspects: workload contention on the node, or a protocol-transition artifact. This range matters — gradient buckets in real training live there.</li><li><strong>Intra-node NVLink at 174 GB/s</strong> was measured with default settings and no tuning. An H200 node with NVSwitch should reach 350–450. Worth re-measuring properly, because if NVLink caps at 174, no amount of network tuning gets past ~190.</li><li><strong>alltoall at 23 GB/s.</strong> Expected to be poor on an isolated fabric — every rank talks to every other rank, so most peers need a PXN hop. Relevant if you&#39;re running point-to-point KV-transfer frameworks rather than collectives.</li><li><strong>Exact PIX pairing</strong>, which needs either a patched device plugin or DRA.</li></ul><h3>A checklist</h3><p>If you’re bringing up a GPU cluster on RoCE, roughly this order:</p><p><strong>Physical and driver layer</strong></p><ul><li>[ ] Every rail link ACTIVE at expected rate</li><li>[ ] Netdev MTU ≥ 4196 on all rails (9000 conventional); verify active_mtu is 4096</li><li>[ ] <strong>Network config persisted</strong> to netplan or equivalent — verify by rebooting one node</li><li>[ ] ib_umad, rdma_cm, rdma_ucm loaded and in /etc/modules-load.d/</li><li>[ ] Same driver stack and firmware version on every node</li></ul><p><strong>Fabric validation</strong></p><ul><li>[ ] ib_write_bw on every rail, ≥95% of line rate, <strong>3 runs minimum</strong></li><li>[ ] Cross-rail RDMA tested explicitly — don’t assume, and don’t trust ping</li><li>[ ] Lane error counters sampled across <em>all</em> rails to establish a baseline</li></ul><p><strong>Host topology</strong></p><ul><li>[ ] nvidia-smi topo -m captured and documented</li><li>[ ] GPU ↔ NIC PIX pairs mapped</li><li>[ ] nvidia-smi -L checked for MIG before diagnosing NVLink</li><li>[ ] ACS disabled (single-tenant nodes only)</li><li>[ ] MAX_ACC_OUT_READ tuned</li></ul><p><strong>Kubernetes</strong></p><ul><li>[ ] rdma system show reports netns shared</li><li>[ ] RDMA device plugin exposing per-rail resources</li><li>[ ] Storage/management NICs excluded from the resource list</li><li>[ ] IPC_LOCK capability on workload pods</li><li>[ ] /dev/shm sized adequately</li><li>[ ] Topology Manager policy chosen deliberately (restricted if mixing workload shapes)</li></ul><p><strong>NCCL</strong></p><ul><li>[ ] NCCL_IB_HCA explicitly lists only the data-plane rails</li><li>[ ] NCCL_IB_GID_INDEX=3 for RoCEv2</li><li>[ ] NCCL_NET_GDR_LEVEL <strong>unset</strong> unless you&#39;ve measured both ways</li><li>[ ] NCCL_MIN_NCHANNELS raised — this was the single biggest lever</li><li>[ ] NCCL_CROSS_NIC=0 on an isolated-rail fabric</li><li>[ ] PXN left <strong>enabled</strong> on an isolated-rail fabric</li><li>[ ] Rail usage verified from debug output, not inferred from the bandwidth number</li></ul><h3>Closing thoughts</h3><p>Five things I’d take into the next cluster bring-up:</p><p><strong>Validate bottom-up.</strong> Every hour spent tuning NCCL before the fabric was characterised was wasted. ib_write_bw gave a ground truth that made everything above it interpretable — once I knew a rail did 392 Gb/s, &quot;NCCL gets 128 Gb/s&quot; became a precise question rather than a vague disappointment.</p><p><strong>One measurement is not a measurement.</strong> I wrote a hardware fault report based on single-run outliers and had to retract it. Three runs, median, and only chase what’s consistent.</p><p><strong>Defaults often encode real knowledge.</strong> NCCL_NET_GDR_LEVEL cost 71% when forced one way and 58% when forced the other. NCCL&#39;s default threshold was right both times. Override guards only after measuring.</p><p><strong>Persist configuration immediately.</strong> The most expensive problem of the session wasn’t performance — it was runtime-only network config disappearing on reboot, three times, each time masquerading as a different failure.</p><p><strong>Check the layer you’re assuming is fine.</strong> Rail isolation had been shaping every result from the beginning. One ib_write_bw between two subnets — thirty seconds — reframed the entire investigation. I ran it late because I&#39;d assumed a fabric that passed per-rail tests was fully connected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0b6ce56c4a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a CXI Fabric DRA Driver on Kubernetes: CDI, NRI, ResourceClaims, and Shared VNIs[learning…]]></title>
            <link>https://medium.com/@shivank1128/deploying-a-cxi-fabric-dra-driver-on-kubernetes-cdi-nri-resourceclaims-and-shared-vnis-learning-4b3b2bfa4711?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b3b2bfa4711</guid>
            <category><![CDATA[rocm]]></category>
            <category><![CDATA[mi300a]]></category>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[slingshot]]></category>
            <category><![CDATA[ai-inferencing]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 10:16:56 GMT</pubDate>
            <atom:updated>2026-08-26T10:16:56.717Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*tJfYOBUnuum8GuMY2CWHeg.jpeg" /></figure><h3>Deploying a CXI Fabric DRA Driver on Kubernetes: CDI, NRI, ResourceClaims, and Shared VNIs[learning resource]</h3><p>An end-to-end guide to exposing high-performance HPC NICs to pods with Dynamic Resource Allocation — the concepts, the commands, the actual output, and the failure modes that cost me the most time</p><p>If you work anywhere near AI infrastructure on bare metal, you have probably hit this wall: your cluster has beautiful, expensive fabric NICs — 200/400 Gbps, RDMA-capable, purpose-built for collective communication — and Kubernetes has absolutely no idea they exist.</p><p>You can hack around it. Everybody does, at first. hostNetwork: true, a privileged pod, a hostPath mount of /dev, and a prayer. It works right up until you need two tenants on the same node, or per-job isolation, or a scheduler that understands the NIC is <em>out of queue budget</em> rather than merely <em>present</em>.</p><p>Dynamic Resource Allocation (DRA) is the answer to that, and as of Kubernetes 1.32+ (resource.k8s.io/v1 stable in the 1.34+ line) it is genuinely usable. This post is the notes I wish I&#39;d had the day my driver image finished building.</p><p>I’m using a CXI-class fabric (the Slingshot NIC family, libcxi, libfabric&#39;s cxi provider) as the running example, because that&#39;s what I had on the bench. Nearly all of it generalizes — the shape of a DRA driver is the same whether the device is a NIC, a GPU, or an FPGA.</p><h3>Part 1: The vocabulary</h3><p>DRA introduces about six new nouns, and two of them (CDI, NRI) aren’t Kubernetes objects at all — they’re container-runtime plumbing. Mixing these up is the root cause of most confusion, so let’s be precise before we touch a cluster.</p><h3>The Kubernetes side</h3><p><strong>DeviceClass</strong> — a cluster-scoped object that says “here is a <em>category</em> of device, and here is the CEL expression that decides whether a given device belongs to it.” Think of it as the type system. You’ll have one for physical NICs (cxi-device) and one for fabric network identifiers (cxi-vni). Admins create these; users reference them.</p><p><strong>ResourceSlice</strong> — published <em>by the driver</em>, one or more per node. This is the driver’s inventory report: “node worker-01 has devices cxi0 through cxi3, and here is each one&#39;s attributes and <em>currently available</em> capacity.&quot; You never write these by hand. They&#39;re the reason the scheduler can make an informed decision instead of a coin flip.</p><p><strong>ResourceClaim</strong> — a request for a device, made by (or on behalf of) a workload. It has a lifecycle: pending → allocated → allocated,reserved (a specific pod has it) → deleted. The state field is the single most useful debugging signal in the whole system.</p><p><strong>ResourceClaimTemplate</strong> — a stamp for making per-pod ResourceClaims. Reference a template in a pod spec and each pod gets its own fresh claim. Reference a <em>claim</em> directly and pods share one. That distinction is not cosmetic; Part 5 explains why it silently breaks jobs.</p><p><strong>MutatingAdmissionPolicy</strong> — a 1.32+ feature that lets you rewrite objects at admission time using CEL, no webhook server required. Here it turns human-friendly capacity parameters into the selector expressions the scheduler needs.</p><p>The flow, in one line: <em>driver publishes ResourceSlices → user creates a ResourceClaim → scheduler matches claim against slices and picks a node → kubelet calls the driver’s </em><em>NodePrepareResources → driver programs hardware and emits a CDI spec → runtime injects the device.</em></p><h3>The runtime side</h3><p><strong>CDI (Container Device Interface)</strong> — a vendor-neutral spec for “how do I get a device into a container.” A CDI spec is a small YAML file dropped in /etc/cdi or /var/run/cdi that names device nodes, mounts, and env vars. The container runtime reads it and applies it. It exists so that runtimes don&#39;t need per-vendor code.</p><p>Here’s what our driver writes for a single claim — worth seeing once, because it demystifies the whole thing:</p><pre># /var/run/cdi/cxi.example.com-claim-9v2xq.yaml   (written by the driver)<br>cdiVersion: &quot;0.6.0&quot;<br>kind: cxi.example.com/device<br>devices:<br>  - name: claim-9v2xq-cxi0<br>    containerEdits:<br>      deviceNodes:<br>        - path: /dev/cxi0<br>          type: c<br>          major: 234<br>          minor: 0<br>          permissions: rw</pre><p><strong>NRI (Node Resource Interface)</strong> — a containerd/CRI-O plugin interface that lets a long-running process hook container lifecycle events (CreateContainer, StartContainer, …) and mutate the spec. CDI is declarative and static; NRI is dynamic and imperative. We need NRI because creating a VNI-scoped fabric service and learning its ID can only happen <em>at container creation time</em> — it isn&#39;t knowable when the CDI spec is written.</p><p><strong>Rule of thumb: CDI gets you the device node. NRI gets you the environment variable.</strong> Almost every “half-working” symptom in this post traces back to one of those two being off.</p><h3>The fabric side</h3><p><strong>VNI (Virtual Network Identifier)</strong> — a fabric-level partition. Two endpoints can only communicate if they’re on the same VNI. This is the tenancy boundary: it’s what stops job A’s traffic from reaching job B’s NICs. Treat it as <em>a network</em>, not <em>a device</em>.</p><p><strong>CXI service</strong> — a pre-programmed allocation of NIC resources bound to a VNI. The driver creates one per claim and hands you its ID.</p><p><strong>FI_CXI_SVC_ID</strong> — the environment variable that tells libfabric &quot;use this service, don&#39;t allocate your own.&quot; This is load-bearing for isolation. Without it, libfabric cheerfully allocates its own service and walks straight past the budget the scheduler reserved for you.</p><p><strong>Capacity units</strong> — a CXI NIC isn’t one indivisible thing. It exposes:</p><p>Unit What it is txqs Transmit queues tgqs Target queues (receive side) eqs Event queues cts Counters ptes Portal table entries — addressable receive endpoints acs Addressing contexts le_pools List-entry pools; 1 pool = 1024 LEs (posted receive buffers) tle_pools Triggered list-entry pools; 1 pool = 512 TLEs (for triggered ops / offloaded collectives)</p><p>Four pods can share one NIC happily if their combined budgets fit. They cannot if they don’t. <strong>That sentence is the entire reason DRA exists.</strong></p><h3>Part 2: Why not just a device plugin?</h3><p>The classic device plugin API treats a device as a countable, opaque integer. You advertise vendor.com/nic: 4, a pod asks for 1, and that&#39;s the whole conversation.</p><p>Given the capacity table above, you can see the problem. An integer count can express “there are four NICs.” It cannot express “cxi0 has 763 transmit queues and 13 LE pools left, which fits your request but not your neighbour&#39;s.&quot;</p><p>DRA replaces the integer with a structured negotiation, and adds three things device plugins never had:</p><ol><li><strong>Typed capacity</strong>, so bin-packing is real rather than nominal.</li><li><strong>Per-claim setup hooks</strong>, so the driver can <em>program</em> hardware, not just expose it.</li><li><strong>Sharing semantics</strong>, so two pods can deliberately be placed on the same fabric partition.</li></ol><p>The cost is more moving parts. That’s the trade.</p><h3>Part 3: Where you are</h3><p>Assume the driver image is built and sitting in your registry.</p><p>Step What it proves 1. Build driver image (CGO + libcxi) Image exists 2. Push to your registry Workers can pull it 3. Enable CDI + NRI on every worker Pods can receive /dev/cxi* 4. Label nodes, deploy DaemonSet + DeviceClasses Driver publishes slices 5. Smoke test with a ResourceClaim + Pod Device + VNI + FI_CXI_SVC_ID 6. Point real workloads at claims Actual value</p><p>Steps 3–5 are where people get stuck.</p><h3>Part 4: Prerequisites</h3><h3>4.1 The host actually has working devices</h3><p>Obvious, but check it before blaming Kubernetes:</p><pre>$ ls /dev/cxi*<br>/dev/cxi0  /dev/cxi1  /dev/cxi2  /dev/cxi3</pre><pre>$ ls /sys/class/cxi/<br>cxi0  cxi1  cxi2  cxi3</pre><pre>$ ldconfig -p | grep libcxi<br>	libcxi.so.1 (libc6,x86-64) =&gt; /usr/lib64/libcxi.so.1<br>	libcxi.so (libc6,x86-64) =&gt; /usr/lib64/libcxi.so</pre><p>You need real device nodes and libcxi.so.1 on the host. The driver container bundles the libraries it was linked against, but it programs <em>live hardware</em> through /dev and /sys. No devices on the host, no slices in the cluster.</p><p>While you’re here, confirm the NICs are actually up and have fabric addresses:</p><pre>$ cat /sys/class/cxi/cxi0/device/properties/nic_addr<br>0x1a2b<br>$ cat /sys/class/cxi/cxi0/device/properties/link<br>1</pre><p>A link of 0 means the port is down. The driver will still publish a slice; your RDMA test will still fail. Catch it now.</p><h3>4.2 containerd: CDI and NRI both on</h3><p>This is the step everyone skips.</p><pre># /etc/containerd/config.toml  (containerd 2.x)<br>[plugins.&#39;io.containerd.cri.v1.runtime&#39;]<br>  enable_cdi = true<br>  cdi_spec_dirs = [&#39;/etc/cdi&#39;, &#39;/var/run/cdi&#39;]</pre><pre>[plugins.&#39;io.containerd.nri.v1.nri&#39;]<br>  disable = false<br>  disable_connections = false<br>  socket_path = &#39;/var/run/nri/nri.sock&#39;<br>  plugin_registration_timeout = &#39;5s&#39;<br>  plugin_request_timeout = &#39;2s&#39;</pre><p>Restart and verify — don’t assume:</p><pre>$ systemctl restart containerd</pre><pre>$ ls -l /var/run/nri/nri.sock<br>srw-rw---- 1 root root 0 Aug  1 09:12 /var/run/nri/nri.sock</pre><pre>$ containerd config dump | grep -A3 &#39;io.containerd.nri&#39;<br>    [plugins.&quot;io.containerd.nri.v1.nri&quot;]<br>      disable = false<br>      disable_connections = false<br>      socket_path = &quot;/var/run/nri/nri.sock&quot;</pre><pre>$ containerd config dump | grep -E &#39;enable_cdi|cdi_spec_dirs&#39;<br>      enable_cdi = true<br>      cdi_spec_dirs = [&quot;/etc/cdi&quot;, &quot;/var/run/cdi&quot;]</pre><p>containerd config dump shows the <em>effective</em> config after defaults and drop-ins are merged — which is not always what&#39;s in your file. If enable_cdi reads false here despite your edit, you have a config precedence problem, not a DRA problem.</p><h3>4.3 The DRA APIs are present</h3><pre>$ kubectl api-resources | grep -E &#39;deviceclasses|resourceclaims|resourceslices|mutatingadmissionpolicies&#39;<br>deviceclasses                     resource.k8s.io/v1        false   DeviceClass<br>resourceclaims                    resource.k8s.io/v1        true    ResourceClaim<br>resourceclaimtemplates            resource.k8s.io/v1        true    ResourceClaimTemplate<br>resourceslices                    resource.k8s.io/v1        false   ResourceSlice<br>mutatingadmissionpolicies         admissionregistration.k8s.io/v1beta1   false   MutatingAdmissionPolicy</pre><p>You want resource.k8s.io/v1. If you only see v1beta1, the manifests below need version edits and some field names differ.</p><h3>4.4 The registry pull path is clean</h3><p>Bare-metal clusters behind a corporate proxy produce a very specific, very annoying failure: 502 Bad Gateway on image pull, because the proxy intercepts the internal registry.</p><ul><li>Add the registry VIP/hostname to /etc/hosts if DNS is unreliable</li><li>Drop the registry CA into containerd’s certs.d/&lt;registry&gt;/ca.crt for private TLS</li><li>Put the registry hostname <strong>and</strong> IP in NO_PROXY</li><li>Restart containerd</li></ul><p>Verify from the node, not from your laptop:</p><pre>$ crictl pull &lt;YOUR_REGISTRY&gt;/cxi-dra-driver:v0.4.2<br>Image is up to date for sha256:3f8c1e...</pre><h3>Part 5: Building the image (if you need to)</h3><p>The build is the one place you don’t get to be clever: the driver uses <strong>CGO to link against the host’s </strong><strong>libcxi</strong>, so there is no cross-compilation. Build on a native node of the target architecture. One source tree covers both x86_64 and aarch64; you just run the build twice, on two machines.</p><pre>$ ls /usr/include/libcxi/libcxi.h<br>/usr/include/libcxi/libcxi.h<br>$ go version<br>go version go1.24.4 linux/amd64<br>$ uname -m<br>x86_64</pre><pre>./scripts/build.sh                # detect arch, stage dist/rootfs/<br>docker build -t &lt;YOUR_REGISTRY&gt;/cxi-dra-driver:&lt;TAG&gt; .<br>docker push &lt;YOUR_REGISTRY&gt;/cxi-dra-driver:&lt;TAG&gt;</pre><p>A design note worth stealing: make the container ENTRYPOINT arch-agnostic by invoking the bundled loader explicitly (/usr/lib/cxi-dra/ld.so …) rather than relying on the image&#39;s default dynamic linker path. It saves you from maintaining two Dockerfiles.</p><h3>Part 6: Deploy the driver</h3><p>Four objects plus a node label.</p><pre># 1) Label every fabric-equipped worker<br>kubectl label node worker-01 feature.node.kubernetes.io/cxi=true --overwrite</pre><pre># 2) RBAC + DeviceClasses + admission policy + DaemonSet<br>kubectl apply -f deploy/kubernetes/clusterrole.yaml<br>kubectl apply -f deploy/kubernetes/deviceclass.yaml<br>kubectl apply -f deploy/kubernetes/deviceclass-vni.yaml<br>kubectl apply -f deploy/kubernetes/admission-policy.yaml<br>kubectl apply -f deploy/kubernetes/daemonset.yaml</pre><h3>What healthy looks like</h3><pre>$ kubectl -n kube-system get ds -l app=cxi-dra-driver<br>NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE<br>cxi-dra-driver   4         4         4       4            4           feature.node.kubernetes.io/cxi=true      3m</pre><pre>$ kubectl -n kube-system get pods -l app=cxi-dra-driver -o wide<br>NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE<br>cxi-dra-driver-4kzn9   1/1     Running   0          3m    10.42.1.17    worker-01<br>cxi-dra-driver-8mqpt   1/1     Running   0          3m    10.42.2.23    worker-02<br>cxi-dra-driver-hv2xl   1/1     Running   0          3m    10.42.3.11    worker-03<br>cxi-dra-driver-w9dfc   1/1     Running   0          3m    10.42.4.08    worker-04</pre><pre>$ kubectl get deviceclass<br>NAME         AGE<br>cxi-device   3m<br>cxi-vni      3m</pre><p>Now the payoff — the driver’s inventory report:</p><pre>$ kubectl get resourceslices -o wide<br>NAME                                  NODE        DRIVER               POOL              DEVICES   AGE<br>worker-01-cxi.example.com-h7k2p       worker-01   cxi.example.com      worker-01         4         3m<br>worker-02-cxi.example.com-p3n8v       worker-02   cxi.example.com      worker-02         4         3m<br>worker-03-cxi.example.com-x9c4d       worker-03   cxi.example.com      worker-03         4         3m<br>worker-04-cxi.example.com-m2s6b       worker-04   cxi.example.com      worker-04         4         3m<br>cxi.example.com-vni-pool-q8w1z        &lt;none&gt;      cxi.example.com      vni-pool          201       3m</pre><p>Note the last line: the VNI pool is <strong>cluster-scoped</strong> (NODE is &lt;none&gt;), because a VNI isn&#39;t attached to any one machine. 201 devices = VNIs 100–300. That asymmetry between the node-local device pool and the cluster-wide VNI pool is the structural thing to internalize.</p><h3>Reading a ResourceSlice</h3><p>This is the single most informative command in the system, and almost nobody runs it:</p><pre>$ kubectl describe resourceslice worker-01-cxi.example.com-h7k2p<br>Name:         worker-01-cxi.example.com-h7k2p<br>API Version:  resource.k8s.io/v1<br>Kind:         ResourceSlice<br>Spec:<br>  Driver:     cxi.example.com<br>  Node Name:  worker-01<br>  Pool:<br>    Generation:           7<br>    Name:                 worker-01<br>    Resource Slice Count: 1<br>  Devices:<br>    Name:  cxi0<br>    Attributes:<br>      pci_address:   {string: &quot;0000:c1:00.0&quot;}<br>      nic_addr:      {string: &quot;0x1a2b&quot;}<br>      link_state:    {string: &quot;up&quot;}<br>      fw_version:    {version: &quot;1.6.0&quot;}<br>      numa_node:     {int: 0}<br>      driver_version:{version: &quot;0.4.2&quot;}<br>    Capacity:<br>      avail_txqs:       763<br>      avail_tgqs:       379<br>      avail_eqs:        1525<br>      avail_cts:        1789<br>      avail_ptes:       1524<br>      avail_acs:        1012<br>      avail_le_pools:   13<br>      avail_tle_pools:  3<br>      total_txqs:       1024<br>      total_tgqs:       512<br>      total_eqs:        2047<br>      total_ptes:       2048<br>    Name:  cxi1<br>    Attributes:<br>      pci_address:   {string: &quot;0000:c2:00.0&quot;}<br>      numa_node:     {int: 0}<br>      link_state:    {string: &quot;up&quot;}<br>    ...<br>Events:  &lt;none&gt;</pre><p>Two things to read off this:</p><ul><li><strong>avail_* vs </strong><strong>total_*.</strong> The gap is consumed capacity — claims already allocated, plus the host reservation (Part 9). If avail_txqs is 763 out of 1024 on an idle node, ~261 went to reservations and existing claims. That&#39;s the number the scheduler filters on.</li><li><strong>numa_node.</strong> Present so you can co-schedule a NIC with a GPU on the same socket. Crossing NUMA domains on a 400 Gbps link is a measurable, avoidable tax.</li></ul><h3>Driver logs at startup</h3><pre>$ kubectl -n kube-system logs cxi-dra-driver-4kzn9<br>I0801 09:14:02.118934  1 main.go:142] starting cxi-dra-driver v0.4.2 (go1.24.4, linux/amd64)<br>I0801 09:14:02.119512  1 discover.go:88] discovered 4 CXI device(s) via /sys/class/cxi<br>I0801 09:14:02.119887  1 discover.go:103] cxi0: pci=0000:c1:00.0 link=up txqs=1024 tgqs=512 eqs=2047 ptes=2048 le_pools=15 tle_pools=4<br>I0801 09:14:02.120114  1 discover.go:103] cxi1: pci=0000:c2:00.0 link=up txqs=1024 tgqs=512 eqs=2047 ptes=2048 le_pools=15 tle_pools=4<br>I0801 09:14:02.121003  1 reserve.go:41] host reservation: txqs=5 tgqs=5 eqs=10 ptes=12 acs=4 le_pools=2 tle_pools=1<br>I0801 09:14:02.245100  1 slice.go:210] published ResourceSlice worker-01-cxi.example.com-h7k2p (4 devices, generation 7)<br>I0801 09:14:02.246003  1 nri.go:64] registered NRI plugin &quot;cxi-dra&quot; (index 10) on /var/run/nri/nri.sock<br>I0801 09:14:02.246880  1 kubelet.go:71] registered DRA plugin with kubelet at /var/lib/kubelet/plugins_registry/cxi.example.com.sock<br>I0801 09:14:02.247109  1 main.go:198] ready</pre><p>Those last three lines are your checklist. <strong>Slice published, NRI registered, kubelet registered.</strong> If any one is missing you have a specific, findable problem rather than a vague one.</p><h3>Two things that will bite you</h3><p><strong>Control-plane taints.</strong> Nodes with a NoSchedule taint won&#39;t run the DaemonSet unless you add a toleration. Usually that&#39;s what you want. But if you expected slices from a control-plane node that also has a NIC, this is why they&#39;re missing.</p><p><strong>The label is load-bearing.</strong> No label → no DaemonSet pod → no slice → claims sit Pending forever with a message that doesn&#39;t obviously point at a missing label. Sanity check:</p><pre>$ kubectl get nodes -l feature.node.kubernetes.io/cxi=true<br>NAME        STATUS   ROLES    AGE   VERSION<br>worker-01   Ready    &lt;none&gt;   88d   v1.36.2<br>worker-02   Ready    &lt;none&gt;   88d   v1.36.2<br>worker-03   Ready    &lt;none&gt;   88d   v1.36.2<br>worker-04   Ready    &lt;none&gt;   88d   v1.36.2</pre><p>Four labelled nodes, four DaemonSet pods, four node-local slices. If those three numbers disagree, stop and reconcile before going further.</p><h3>Part 7: The smoke test</h3><p>Resist the urge to go straight to your inference stack. A five-line pod tells you in thirty seconds what a full training job will tell you in forty minutes of log archaeology.</p><h3>7.1 Claim templates</h3><p>A well-designed driver ships <strong>no defaults</strong> for capacity parameters — a pod that doesn’t state its budget shouldn’t silently get an arbitrary one. Apply the packaged templates first:</p><pre>$ kubectl apply -f deploy/kubernetes/test/resourceclaim.yaml<br>resourceclaimtemplate.resource.k8s.io/cxi-quarter created<br>resourceclaimtemplate.resource.k8s.io/cxi-half created<br>resourceclaimtemplate.resource.k8s.io/cxi-vni created</pre><p>What “quarter” actually means:</p><pre>$ kubectl get resourceclaimtemplate cxi-quarter -o yaml<br>spec:<br>  spec:<br>    devices:<br>      requests:<br>        - name: device<br>          exactly:<br>            deviceClassName: cxi-device<br>            allocationMode: ExactCount<br>            count: 1<br>      config:<br>        - requests: [&quot;device&quot;]<br>          opaque:<br>            driver: cxi.example.com<br>            parameters:<br>              txqs: 256<br>              tgqs: 128<br>              eqs: 512<br>              cts: 256<br>              ptes: 512<br>              acs: 8<br>              le_pools: 1<br>              tle_pools: 1</pre><p>Roughly a quarter of a NIC’s budget — hence the name. cxi-half doubles it.</p><h3>7.2 The minimal pod</h3><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: cxi-dra-smoke<br>spec:<br>  restartPolicy: Never<br>  resourceClaims:<br>    - name: cxi<br>      resourceClaimTemplateName: cxi-quarter<br>    - name: vni<br>      resourceClaimTemplateName: cxi-vni<br>  containers:<br>    - name: workload<br>      image: &lt;any-pullable-image&gt;<br>      command: [&quot;sh&quot;, &quot;-c&quot;]<br>      args:<br>        - |<br>          echo &quot;=== CXI devices ===&quot;<br>          ls -la /dev/cxi*<br>          echo &quot;=== fabric env ===&quot;<br>          env | grep -E &#39;FI_|CXI&#39;<br>          sleep 300<br>      resources:<br>        claims:<br>          - name: cxi<br>          - name: vni</pre><h3>7.3 Pass criteria — with output</h3><pre>$ kubectl apply -f cxi-dra-smoke.yaml<br>pod/cxi-dra-smoke created</pre><pre>$ kubectl get resourceclaim<br>NAME                       STATE                AGE<br>cxi-dra-smoke-cxi-9v2xq    allocated,reserved   12s<br>cxi-dra-smoke-vni-t4blm    allocated,reserved   12s</pre><pre>$ kubectl get pod cxi-dra-smoke -o wide<br>NAME            READY   STATUS    RESTARTS   AGE   IP           NODE<br>cxi-dra-smoke   1/1     Running   0          14s   10.42.1.88   worker-01</pre><pre>$ kubectl logs cxi-dra-smoke<br>=== CXI devices ===<br>crw-rw-rw-    1 root     root      234,   0 Aug  1 09:31 /dev/cxi0<br>=== fabric env ===<br>FI_CXI_SVC_ID_cxi0=17<br>FI_CXI_SVC_ID=17</pre><p>allocated,reserved on both claims, 1/1 Running, a device node, and a service ID. That&#39;s the pass.</p><h3>7.4 Reading an allocated claim</h3><p>When it works, describe tells you <em>which</em> device you got — essential once you&#39;re debugging placement:</p><pre>$ kubectl describe resourceclaim cxi-dra-smoke-cxi-9v2xq<br>Name:         cxi-dra-smoke-cxi-9v2xq<br>Namespace:    default<br>Owner:        Pod/cxi-dra-smoke<br>API Version:  resource.k8s.io/v1<br>Kind:         ResourceClaim<br>Spec:<br>  Devices:<br>    Requests:<br>      Name:               device<br>      Exactly:<br>        Allocation Mode:    ExactCount<br>        Count:              1<br>        Device Class Name:  cxi-device<br>    Config:<br>      Opaque:<br>        Driver:  cxi.example.com<br>        Parameters:<br>          acs:        8<br>          cts:        256<br>          eqs:        512<br>          le_pools:   1<br>          ptes:       512<br>          tgqs:       128<br>          tle_pools:  1<br>          txqs:       256<br>Status:<br>  Allocation:<br>    Devices:<br>      Results:<br>        Device:   cxi0<br>        Driver:   cxi.example.com<br>        Pool:     worker-01<br>        Request:  device<br>    Node Selector:<br>      Node Selector Terms:<br>        Match Fields:<br>          Key:       metadata.name<br>          Operator:  In<br>          Values:    worker-01<br>  Reserved For:<br>    Name:      cxi-dra-smoke<br>    Resource:  pods<br>    UID:       6f2a...<br>Events:<br>  Type    Reason     Age   From                   Message<br>  ----    ------     ----  ----                   -------<br>  Normal  Allocated  14s   resource-scheduler     Allocated device cxi0 on node worker-01<br>  Normal  Prepared   13s   cxi.example.com        Created service id=17 vni=142 on cxi0</pre><p>Status.Allocation.Devices.Results is the answer to &quot;which NIC did this pod actually get.&quot; Reserved For is the answer to &quot;who&#39;s holding it.&quot;</p><h3>7.5 The corresponding driver log</h3><pre>$ kubectl -n kube-system logs cxi-dra-driver-4kzn9 --since=1m<br>I0801 09:31:44.902118  1 prepare.go:126] NodePrepareResources: claim default/cxi-dra-smoke-cxi-9v2xq<br>I0801 09:31:44.903442  1 prepare.go:151]   request &quot;device&quot;: device=cxi0 txqs=256 tgqs=128 eqs=512 cts=256 ptes=512 acs=8 le_pools=1 tle_pools=1<br>I0801 09:31:44.905201  1 fitcheck.go:78]   fit ok on cxi0: avail_txqs 1019-&gt;763, avail_le_pools 14-&gt;13<br>I0801 09:31:44.951877  1 cdi.go:94]   wrote /var/run/cdi/cxi.example.com-claim-9v2xq.yaml (1 device node)<br>I0801 09:31:45.010433  1 svc.go:181] created CXI service id=17 vni=142 on cxi0 for claim 9v2xq<br>I0801 09:31:45.011002  1 nri.go:212] CreateContainer(cxi-dra-smoke/workload): injecting FI_CXI_SVC_ID=17 FI_CXI_SVC_ID_cxi0=17<br>I0801 09:31:45.011884  1 slice.go:210] republished ResourceSlice worker-01-... (generation 8)</pre><p>Trace it: fit-check → CDI spec written (that’s the device node) → service created → NRI injects the env var (that’s FI_CXI_SVC_ID) → slice republished with reduced availability. <strong>Every stage of the architecture, visible in seven lines.</strong> When something breaks, the missing line tells you the layer.</p><p><strong>Do not set </strong><strong>privileged: true for this test.</strong> Unprivileged device access is the entire point. If it only works privileged, you have a permissions problem you&#39;ll carry into production — find it now.</p><h3>7.6 Optional fabric check</h3><p>Once the workload image carries libfabric and the CXI tools:</p><pre>$ kubectl exec -it cxi-dra-smoke -- fi_info -p cxi<br>provider: cxi<br>    fabric: cxi<br>    domain: cxi0<br>    version: 1.22<br>    type: FI_EP_RDM<br>    protocol: FI_PROTO_CXI<br>provider: cxi;ofi_rxm<br>    fabric: cxi<br>    domain: cxi0<br>    ...</pre><p>If FI_CXI_SVC_ID were missing, this is where you&#39;d see fi_getinfo: -61 (No data available) or a service-allocation failure instead.</p><p>Then the test that actually matters — two pods, two nodes:</p><pre>$ kubectl exec -it cxi-rdma-a -- cxi_write_bw -d cxi0 &amp;<br>$ kubectl exec -it cxi-rdma-b -- cxi_write_bw -d cxi0 &lt;peer-nic-addr&gt;<br>------------------------------------------------------------------<br>    Size[B]     Writes   BW[Gb/s]   MRate[pkt/s]<br>       1024      10000      24.31        2.97M<br>       8192      10000     162.44        2.48M<br>      65536      10000     191.02        0.36M<br>    1048576       1000     193.87        0.02M<br>------------------------------------------------------------------</pre><p>Single-pod fi_info proves the provider initializes. Only the two-node bandwidth test proves the fabric works.</p><h3>Part 8: How applications should request the fabric</h3><h3>Pattern A — one NIC + one VNI per pod</h3><p>The default. Each pod gets its own device slice and its own isolated fabric service.</p><pre>resourceClaims:<br>  - name: cxi<br>    resourceClaimTemplateName: cxi-quarter<br>  - name: vni<br>    resourceClaimTemplateName: cxi-vni<br>containers:<br>  - resources:<br>      claims:<br>        - name: cxi<br>        - name: vni</pre><h3>Pattern B — multiple pods sharing one VNI</h3><p>This is the one that trips people up, and it matters: <strong>a VNI is a communication domain.</strong></p><p>Using the VNI <em>template</em> for every pod gives each pod its <strong>own</strong> VNI. They will be mutually unreachable, and the failure looks like a hang, not an error — your collective just never completes.</p><p>Instead, create <strong>one persistent ResourceClaim</strong> for the VNI and have every pod reference it by name:</p><pre>apiVersion: resource.k8s.io/v1<br>kind: ResourceClaim<br>metadata:<br>  name: job-alpha-vni<br>spec:<br>  devices:<br>    requests:<br>      - name: vni<br>        exactly:<br>          deviceClassName: cxi-vni<br>          allocationMode: ExactCount<br>          count: 1</pre><pre># in every pod of the job:<br>resourceClaims:<br>  - name: cxi<br>    resourceClaimTemplateName: cxi-quarter   # per-pod: own NIC slice<br>  - name: vni<br>    resourceClaimName: job-alpha-vni         # shared: same fabric partition</pre><p>Verify the sharing actually happened — a shared claim shows <em>multiple</em> pods under Reserved For:</p><pre>$ kubectl describe resourceclaim job-alpha-vni | grep -A8 &#39;Reserved For&#39;<br>Reserved For:<br>  Name:      alpha-worker-0<br>  Resource:  pods<br>  Name:      alpha-worker-1<br>  Resource:  pods<br>  Name:      alpha-worker-2<br>  Resource:  pods<br>  Name:      alpha-worker-3<br>  Resource:  pods</pre><p>Four pods, one VNI claim. If you see four <em>separate</em> claims each with one pod, you’ve made the mistake and your job will hang.</p><p>You can also confirm from inside the pods — same VNI, different service IDs:</p><pre>$ kubectl exec alpha-worker-0 -- sh -c &#39;cat /sys/class/cxi/cxi0/device/svc/17/vni&#39;<br>142<br>$ kubectl exec alpha-worker-3 -- sh -c &#39;cat /sys/class/cxi/cxi0/device/svc/23/vni&#39;<br>142</pre><p>Mentally: <strong>device claim = “give me a NIC slice” (per-pod). VNI claim = “put me on this network” (per-job).</strong></p><h3>How the admission policy fits in</h3><p>The MutatingAdmissionPolicy translates your capacity parameters into the CEL selector the scheduler filters on. You can see the result on a mutated claim:</p><pre>$ kubectl get resourceclaim cxi-dra-smoke-cxi-9v2xq -o jsonpath=&#39;{.spec.devices.requests[0].exactly.selectors}&#39; | jq<br>[<br>  {<br>    &quot;cel&quot;: {<br>      &quot;expression&quot;: &quot;device.capacity[&#39;cxi.example.com&#39;].avail_txqs.compareTo(quantity(&#39;256&#39;)) &gt;= 0 &amp;&amp; device.capacity[&#39;cxi.example.com&#39;].avail_le_pools.compareTo(quantity(&#39;1&#39;)) &gt;= 0 &amp;&amp; device.attributes[&#39;cxi.example.com&#39;].link_state == &#39;up&#39;&quot;<br>    }<br>  }<br>]</pre><p>You wrote txqs: 256. The policy wrote that expression. The scheduler evaluates it against every device in every slice, and only schedules onto a node with a NIC that fits.</p><p>The driver then <strong>fit-checks again</strong> at prepare time. Belt and braces, and worth it: scheduler state can be stale, hardware state can’t.</p><h3>Environment variables your app must honor</h3><p>Variable Set by Purpose FI_CXI_SVC_ID NRI plugin Use the pre-programmed service instead of allocating one FI_CXI_SVC_ID_cxiN NRI plugin Per-device variant FI_PROVIDER=cxi You (app/chart) Select the CXI provider FI_HMEM=rocr You (ROCm path) GPU memory registration for AMD</p><p><strong>GPUs are a separate driver.</strong> GPU allocation goes through its own DRA driver/operator. Claim both in the same pod spec — they compose fine, they just don’t know about each other:</p><pre>resourceClaims:<br>  - name: gpu<br>    resourceClaimTemplateName: amd-gpu-single<br>  - name: cxi<br>    resourceClaimTemplateName: cxi-quarter<br>  - name: vni<br>    resourceClaimName: job-alpha-vni</pre><p>If you care about NUMA locality, that’s what the numa_node attribute on the slice is for — a CEL selector can require the NIC be on the same socket as the GPU.</p><h3>Part 9: Troubleshooting, with real signatures</h3><p>Symptom Cause Fix ErrImagePull / 502 Registry through HTTP proxy NO_PROXY + hosts + restart containerd x509: unknown authority Missing registry CA Install CA in certs.d; restart containerd Pods Running, no ResourceSlices Driver crash, or no /dev/cxi* Check driver logs Claim stuck pending No matching device / insufficient avail_* describe resourceclaim; check slice Pod Running, no /dev/cxi* <strong>CDI disabled</strong> Fix containerd CDI; restart Device present, libfabric fails <strong>NRI off</strong> → no FI_CXI_SVC_ID Enable NRI; check socket will never fit on this hardware Request + reserved &gt; capacity Lower claim params or reserved flags</p><p>Now the parts you can actually pattern-match against.</p><h3>Claim stuck pending</h3><pre>$ kubectl get resourceclaim<br>NAME                       STATE     AGE<br>big-job-cxi-2xk4p          pending   4m</pre><pre>$ kubectl describe resourceclaim big-job-cxi-2xk4p<br>...<br>Status:<br>  Allocation:  &lt;none&gt;<br>Events:<br>  Type     Reason             Age                From                Message<br>  ----     ------             ----               ----                -------<br>  Warning  FailedAllocation   4m (x12 over 4m)   resource-scheduler  no devices matched request &quot;device&quot;: <br>           4 devices on 4 nodes evaluated, 0 satisfied selectors <br>           (avail_txqs &lt; 900 on cxi0..cxi3)</pre><p>The event names the failing predicate. Cross-reference against a slice:</p><pre>$ kubectl get resourceslices -o json | jq -r &#39;.items[] | select(.spec.nodeName) | .spec.nodeName as $n | .spec.devices[] | &quot;\($n)/\(.name)  txqs=\(.capacity.avail_txqs)  le_pools=\(.capacity.avail_le_pools)&quot;&#39;<br>worker-01/cxi0  txqs=763  le_pools=13<br>worker-01/cxi1  txqs=1019  le_pools=14<br>worker-02/cxi0  txqs=1019  le_pools=14<br>worker-02/cxi1  txqs=507  le_pools=11<br>...</pre><p>You asked for 900, nothing has 900 free. Either shrink the claim or free capacity. <strong>This is the query to keep in your notes</strong> — it’s the fastest way to see cluster-wide fabric availability.</p><h3>Pod running, no device node (CDI off)</h3><pre>$ kubectl logs cxi-dra-smoke<br>=== CXI devices ===<br>ls: /dev/cxi*: No such file or directory<br>=== fabric env ===<br>FI_CXI_SVC_ID=17</pre><p>Note the tell: <strong>the env var is present, the device is not.</strong> NRI is working, CDI is not. Confirm the driver <em>wrote</em> the spec:</p><pre>$ kubectl -n kube-system exec cxi-dra-driver-4kzn9 -- ls /var/run/cdi/<br>cxi.example.com-claim-9v2xq.yaml</pre><p>Spec exists but device didn’t appear → the runtime isn’t reading it. Back to enable_cdi and cdi_spec_dirs.</p><h3>Device node present, no env var (NRI off)</h3><p>The mirror image:</p><pre>$ kubectl logs cxi-dra-smoke<br>=== CXI devices ===<br>crw-rw-rw-    1 root     root      234,   0 /dev/cxi0<br>=== fabric env ===<br>(nothing)</pre><p>And in the driver log, the registration line is missing:</p><pre>$ kubectl -n kube-system logs cxi-dra-driver-4kzn9 | grep -i nri<br>E0801 09:14:02.246003  1 nri.go:58] NRI registration failed: dial unix /var/run/nri/nri.sock: connect: no such file or directory<br>W0801 09:14:02.246101  1 main.go:181] continuing without NRI; FI_CXI_SVC_ID will NOT be injected</pre><p>The driver warns rather than crashing — which is friendly but means you must read the log. Downstream, libfabric fails like this:</p><pre>libfabric:cxi:core:cxip_alloc_svc():482&lt;warn&gt; failed to allocate service: -12 (Cannot allocate memory)<br>fi_getinfo: -61 (No data available)</pre><p>That misleading -12 is a resource-allocation failure caused by no reserved service to attach to. Chasing it as a memory problem costs about two hours. Ask me how I know.</p><h3>The “will never fit” error</h3><pre>$ kubectl describe resourceclaim oversized-cxi-8k2mq | tail -5<br>Events:<br>  Warning  FailedAllocation  10s  resource-scheduler  request &quot;device&quot; will never fit on this <br>           hardware: txqs=1200 exceeds device total 1024 minus reserved 5 (max allocatable 1019)</pre><p>Note “<strong>will never fit</strong>” versus “no devices matched.” The first is permanent — no amount of waiting helps. The second is transient — capacity may free up. Different words, deliberately, because they call for different responses.</p><p>The reservation that produces this:</p><pre>--reserved-txqs=5  --reserved-tgqs=5  --reserved-eqs=10  --reserved-ptes=12<br>--reserved-acs=4   --reserved-le-pools=2  --reserved-tle-pools=1</pre><p>That headroom is for host-side fabric services. Raise it if your host services are heavier; lower it only if you’re certain nothing on the host needs the NIC.</p><h3>Part 10: A sane rollout order</h3><ol><li>Confirm CDI + NRI on <strong>all</strong> fabric workers. Restart containerd if anything changed.</li><li>Build and push the driver image if it isn’t in your registry.</li><li>Set image: in daemonset.yaml; apply the five manifests.</li><li>Verify: labelled nodes == DaemonSet pods == node-local ResourceSlices.</li><li>Pass the smoke test: allocated,reserved + device node + FI_CXI_SVC_ID.</li><li>Install the GPU DRA driver/operator; run one pod claiming GPU + NIC + VNI.</li><li>Wire transport: ROCm-enabled libfabric in the image, FI_PROVIDER=cxi, FI_HMEM=rocr.</li><li>Two-pod cxi_write_bw across nodes. Don&#39;t skip this.</li><li>Migrate real workloads off privileged hostNetwork hacks. One shared VNI claim per group whose pods must peer.</li><li>Operationalize: pin the image by digest, alert on ResourceSlice count and driver restarts, document the node label and registry CA in your cluster bootstrap.</li></ol><p>Step 10 gets skipped and then costs you a Tuesday six months later, when someone reimages a node and nobody remembers the label.</p><h3>What I’d tell my past self</h3><p><strong>Learn the CDI/NRI split first.</strong> Device node missing = CDI. Env var missing = NRI. Half of all “DRA is broken” reports are one of those two, and the diagnosis takes seconds once the mapping is in your head.</p><p><strong>kubectl describe resourceslice is the most underused command here.</strong> It shows you exactly what the scheduler sees. Most placement mysteries dissolve on contact with it.</p><p><strong>Smoke-test with </strong><strong>alpine, not your real image.</strong> A 5 MB image and an ls finds your problem faster than a 12 GB inference container.</p><p><strong>Never run the smoke test privileged.</strong> It hides exactly the class of bug you’re trying to find.</p><p><strong>The VNI is a network, not a device.</strong> Per-pod VNI templates for pods that must talk to each other is a silent, hang-shaped failure. Shared persistent claim, every time — and check Reserved For to prove it.</p><p><strong>Read the error verb.</strong> “Will never fit” and “no devices matched” mean different things. One is your spec; the other is your capacity.</p><p>DRA is more setup than a device plugin. It also gives you something a device plugin fundamentally cannot: a scheduler that understands your hardware has <em>internal capacity</em>, and tenants that can share a NIC without trusting each other. On a bare-metal AI cluster, that trade is worth making.</p><p><em>Deployed DRA drivers for other device classes? I’m curious whether the CDI/NRI split shows up the same way elsewhere — drop a note in the responses.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b3b2bfa4711" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serving a Multi-Node LLM on AMD MI300A over HPE Slingshot and Kubernetes [learning resource]]]></title>
            <link>https://medium.com/@shivank1128/serving-a-multi-node-llm-on-amd-mi300a-over-hpe-slingshot-and-kubernetes-learning-resource-8fbe6dda1282?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fbe6dda1282</guid>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[slingshot]]></category>
            <category><![CDATA[ai-inferencing]]></category>
            <category><![CDATA[rays]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 07:48:33 GMT</pubDate>
            <atom:updated>2026-08-12T07:48:33.309Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ucx_Oqyl71rG_YZ7.png" /></figure><p>Most “deploy an LLM” tutorials assume one box with a handful of NVIDIA GPUs, a CUDA image, and a single docker run. The moment your model outgrows a single machine — and modern models do — almost every assumption in those tutorials breaks. You now care about which GPUs talk over which wire, whether your collectives are secretly crawling over TCP, and how Kubernetes even <em>hands a specialized network card</em> to a pod.</p><p>This post walks the entire stack for a setup that is increasingly common in HPC-flavored AI clusters but barely documented end to end: <strong>AMD MI300A accelerators, connected by HPE Slingshot, orchestrated by Kubernetes</strong>. By the end you’ll understand every layer from the silicon to the scheduler, and — I hope — be able to reason about <em>why</em> each piece is shaped the way it is, not just copy a manifest.</p><p>We’ll use a concrete target throughout: <strong>16 GPUs across 4 nodes</strong>, serving one large model. Real measurements from a working cluster are included so the numbers feel tangible.</p><h3>1. Why multi-node is a different sport</h3><p>When a model fits on one GPU, “parallelism” is a footnote. When it doesn’t, the central fact of your life becomes this: <strong>split models talk constantly</strong>, and the speed of that talking determines whether your expensive accelerators do work or sit idle.</p><p>Two things follow. First, you must understand your interconnect <em>tiers</em> — not all GPU-to-GPU links are equal. Second, your orchestration layer has to place work with those tiers in mind, or it will scatter tightly-coupled computation across slow links and you’ll wonder why 16 GPUs feel like 6.</p><p>So we start at the bottom.</p><h3>2. The hardware: what an APU actually is</h3><p>The AMD Instinct <strong>MI300A</strong> is not a GPU in the sense you’re used to. It’s an <strong>APU — an Accelerated Processing Unit</strong> — which means CPU cores and GPU compute engines are fused into a <em>single package</em> and, crucially, <strong>share one pool of memory</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*ysCZFPvp3J_8YezD" /></figure><p>Each MI300A APU contains:</p><ul><li><strong>24 “Zen 4” CPU cores</strong> (three 8-core chiplets), presented to the OS as 48 threads.</li><li><strong>6 CDNA 3 GPU compute dies (XCDs)</strong> — the matrix and vector engines that do the actual model math.</li><li><strong>128 GB of unified HBM3</strong> — high-bandwidth memory addressed by <em>both</em> the CPU and the GPU.</li></ul><p>A single compute <strong>node</strong> (an HPE Cray EX255a-class blade) packs <strong>four</strong> of these APUs. That “4 GPUs per node” number turns out to be the single most consequential fact in the entire design — nearly every layout decision falls out of it.</p><h3>Unified memory, and why it matters to you</h3><p>On a conventional discrete GPU, there are two separate memories: host RAM and device VRAM. Data shuttles between them across PCIe, and that copy is often a bottleneck. The MI300A abolishes the distinction — CPU and GPU read and write the <em>same</em> HBM3, with no copy across a bus.</p><p>That’s elegant, but it has a practical consequence for anyone serving models: the 128 GB on each APU is <strong>shared</strong> between whatever the CPU is doing and everything the GPU needs — model weights, activations, and the KV cache. You are not the sole tenant of that memory. The lesson, which we’ll return to, is to leave headroom: don’t crank your inference engine’s memory-utilization knob to its maximum and assume the whole 128 GB is yours.</p><blockquote><strong><em>Takeaway to memorize:</em></strong><em> 4 GPUs per node, 512 GB of HBM per node. Hold that number; the rest of the architecture is a response to it.</em></blockquote><h3>3. Reading the machine: what Kubernetes actually sees</h3><p>Before any theory, look at how a node advertises itself. Running kubectl describe node &lt;node&gt; on a compute node produces something like this (identifiers genericized):</p><pre>Name:               gpu-node-01<br>Taints:             example.com/compute=true:NoSchedule<br>...<br>Capacity:<br>  amd.com/gpu:        4<br>  cpu:                192<br>  ephemeral-storage:  256962Mi<br>  hugepages-1Gi:      0<br>  hugepages-2Mi:      8Mi<br>  memory:             526258180Ki<br>  pods:               110<br>Allocatable:<br>  amd.com/gpu:        4<br>  cpu:                192<br>  memory:             526249988Ki<br>  pods:               110<br>Node-Local ResourceSlices:<br>  Driver                  Pool          Slices   Devices<br>  ----------------------  ------------  -------  -------<br>  cxi.slingshot.hpe.com   gpu-node-01   1        4<br>System Info:<br>  Kernel Version:            5.14.0-...el9.x86_64<br>  OS Image:                  Red Hat Enterprise Linux 9.6<br>  Container Runtime Version: containerd://2.x<br>  Kubelet Version:           v1.36.x<br>  Architecture:              amd64</pre><p>Every number here maps to §2:</p><ul><li>amd.com/gpu: 4 — the four MI300A APUs, exposed to Kubernetes as four GPUs.</li><li>cpu: 192 — four APUs × 24 cores × 2 threads.</li><li>memory: ~502Gi — four × 128 GB of HBM3.</li><li><strong>Node-Local ResourceSlices</strong> with cxi.slingshot.hpe.com … 4 devices — this is the <em>network</em> side of the house, and it&#39;s a newer Kubernetes mechanism we&#39;ll unpack in §8. For now, notice the node is advertising <strong>two entirely different kinds of hardware through two different systems</strong>: GPUs the classic way, and Slingshot NICs a new way.</li></ul><p>That single node dump is a map of the whole post.</p><h3>4. The interconnect: two tiers, and a note on NIC generations</h3><p>There are exactly two ways two GPUs can exchange data here, and they differ enormously in speed.</p><h3>Tier 1 — inside a node: Infinity Fabric / XGMI</h3><p>The four APUs in a node are wired directly to each other with AMD’s <strong>Infinity Fabric</strong>, often referred to as <strong>XGMI</strong>. This is huge bandwidth and very low latency. A single-node, 4-GPU all-reduce on this fabric measured <strong>~242 GB/s</strong> of bus bandwidth in our testing. That number <em>is</em> XGMI.</p><h3>Tier 2 — between nodes: HPE Slingshot</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*INfmV6o3n4j6VMsm" /></figure><p>To leave a node, traffic crosses <strong>HPE Slingshot</strong>, a high-performance fabric built on an enhanced Ethernet foundation. Three names matter:</p><ul><li><strong>Cassini</strong> — the Slingshot NIC in each node.</li><li><strong>Rosetta</strong> — the Slingshot switch ASIC, with adaptive routing and congestion control designed for the bursty, all-to-all traffic that collective operations generate.</li><li><strong>CXI (“Cassini eXascale Interface”)</strong> — the low-level driver/API for talking to the NIC. This is the cxi you&#39;ll see everywhere: the /dev/cxi0 device, the FI_CXI_* tuning variables, and the cxi.slingshot.hpe.com driver name in that node dump.</li></ul><h3>A word on NIC generations (and why “Slingshot 11” keeps coming up)</h3><p>If you research this hardware you’ll trip over version numbers, so here’s the orientation:</p><ul><li><strong>Slingshot 10</strong> was an earlier incarnation that ran the Slingshot protocol enhancements over relatively standard 100/200G Ethernet NICs. It got you the switch benefits without the full custom NIC.</li><li><strong>Slingshot 11</strong> is the one people mean today: the purpose-built <strong>Cassini</strong> NIC paired with <strong>Rosetta</strong> switches, running at <strong>200 Gbps</strong> per port (~25 GB/s each direction). Cassini is what enables hardware features like fine-grained adaptive routing and efficient RDMA for GPU traffic.</li></ul><p>For contrast, the InfiniBand world you may know from NVIDIA clusters has its own generation ladder — EDR (100G), HDR (200G), NDR (400G) — with Mellanox/NVIDIA ConnectX and BlueField adapters. Slingshot 11 sits comparably to HDR-class InfiniBand on raw per-port bandwidth, but its distinguishing bet is the Rosetta switch’s congestion handling at exascale node counts rather than peak point-to-point numbers. The practical upshot for us: <strong>Slingshot is Ethernet-derived, not InfiniBand</strong>, which is exactly why the software plumbing (§5) looks different from the NCCL_IB_HCA incantations in most GPU tutorials.</p><p>A container-to-container point-to-point test over Slingshot measured <strong>~23.9 GB/s at 64 KB messages</strong> with <strong>~2.5 µs send latency</strong> — essentially line rate for a single 200G NIC, from inside a pod. That “from inside a pod” part is the whole ballgame, and we’ll see why.</p><h3>The rule the whole design bends around</h3><p>Inter-node bandwidth is a fraction of intra-node bandwidth. So:</p><blockquote><strong><em>Keep the heaviest, chattiest communication inside a node; spend the scarce inter-node bandwidth only on traffic that tolerates it.</em></strong></blockquote><p>Everything in §6 is an application of that one sentence.</p><h3>5. How split models communicate: collectives, RCCL, and a silent trap</h3><h3>Collectives, briefly</h3><p>When a model is spread across GPUs, they periodically combine partial results using standardized group operations called <strong>collectives</strong>:</p><ul><li><strong>All-Reduce</strong> — everyone contributes a tensor, it’s summed, and everyone gets the total back. The workhorse of tensor parallelism, and the standard benchmark (it stresses both directions of every link).</li><li><strong>All-Gather</strong> — everyone shares a shard until all hold the full concatenation.</li><li><strong>Reduce-Scatter</strong> — sum, then hand each GPU one slice of the result.</li><li><strong>All-to-All</strong> — every GPU sends a distinct chunk to every other; the signature of Mixture-of-Experts models.</li></ul><p>An LLM’s parallel execution is, underneath, a schedule of these operations. Slow collectives mean idle GPUs.</p><h3>RCCL — the AMD collectives library</h3><p><strong>RCCL</strong> (ROCm Communication Collectives Library) is AMD’s implementation, the counterpart to NVIDIA’s NCCL. A key detail that saves a lot of confusion: RCCL is <strong>NCCL-ABI-compatible</strong>. It honors the same NCCL_* environment variables and loads the same style of network plugin. That&#39;s why guides written for NVIDIA still apply — and why an HPE plugin literally named aws-ofi-nccl happily serves RCCL.</p><h3>The plugin chain — and the number-one trap in the whole project</h3><p>Here’s the part that quietly wrecks deployments. <strong>RCCL does not know how to drive a Slingshot NIC by itself.</strong> Left to its own devices, it falls back to ordinary <strong>TCP sockets</strong> over whatever Ethernet interface it can find. Everything <em>appears</em> to work — the model loads, requests get answered — you just get a small fraction of the bandwidth. There’s no error. You find out only by reading logs or being disappointed by a benchmark.</p><p>The fix is a chain of libraries, each handing off to the next:</p><pre>vLLM / PyTorch                  (the model requests an all-reduce)<br>      │<br>  RCCL                            (AMD collectives; chooses a network transport)<br>      │   ← plugin: aws-ofi-nccl / aws-ofi-rccl   (HPE-tuned fork)<br>  libfabric (OFI)                 (vendor-neutral high-performance networking API)<br>      │   ← provider: cxi<br>  CXI / Cassini driver            (the physical Slingshot NIC)<br>      │<br>  Slingshot fabric                (Rosetta switches → other nodes)</pre><ul><li><strong>aws-ofi-nccl / aws-ofi-rccl</strong> is the shim that lets RCCL speak <strong>libfabric</strong> instead of TCP. It registers as a network plugin (a libnccl-net.so). HPE ships a tuned fork for Slingshot.</li><li><strong>libfabric</strong> (a.k.a. OFI, “OpenFabrics Interfaces”) is a vendor-neutral networking API with swappable <strong>providers</strong>. The provider that drives Slingshot is called <strong>cxi</strong>. Setting FI_PROVIDER=cxi forces it.</li><li><strong>GPUDirect RDMA</strong> is the optimization that lets the NIC read and write GPU memory <em>directly</em>, without bouncing through CPU RAM. You enable it with NCCL_NET_GDR_LEVEL=3. The line-rate, ~2.5 µs numbers from §4 are the evidence it&#39;s actually engaging — which, on Kubernetes, was the single biggest risk of the whole exercise.</li></ul><h3>How to prove you’re on the fabric</h3><p>Commit this to muscle memory. Run any RCCL job with NCCL_DEBUG=INFO and read the startup log:</p><ul><li><strong>Good:</strong> lines showing the OFI plugin loaded, the <strong>cxi</strong> provider selected, and <strong>GDRDMA</strong> references. Bandwidth in the tens-to-hundreds of GB/s.</li><li><strong>Bad:</strong> a <strong>“socket”</strong> transport is chosen. Bandwidth in the single digits of GB/s. The plugin didn’t load — check LD_LIBRARY_PATH and the plugin path.</li></ul><p>Never trust an RCCL bandwidth number without first confirming the transport.</p><h3>MI300-specific knobs worth knowing</h3><pre>NCCL_MIN_NCHANNELS=42          # MI300 sweet spot; the max is 64 but more isn&#39;t always faster<br>HSA_NO_SCRATCH_RECLAIM=1       # needed on ROCm &lt; 6.4 for good MI300 performance<br>FI_CXI_DEFAULT_CQ_SIZE=131072  # larger completion queue avoids stalls at scale</pre><p>(Treat the ROCm version threshold as “verify on your build” rather than gospel.)</p><h3>6. Parallelism: splitting one model across 16 GPUs</h3><p>There are several independent axes for splitting a model; real systems combine them.</p><h3>Tensor Parallelism (TP) — keep it inside a node</h3><p>Each layer’s weight matrices are sliced across GPUs. Every GPU computes part of every layer, and they <strong>all-reduce</strong> after each split operation — many times per token. This is extremely communication-heavy, so <strong>TP must ride the fast link</strong>. With four GPUs per node on XGMI, the natural choice is <strong>TP = 4, contained within a node</strong>.</p><h3>Pipeline Parallelism (PP) — spread it across nodes</h3><p>The model’s layers are cut into consecutive <strong>stages</strong>; each node holds one stage and passes activations forward to the next. The communication is far lighter than TP — one handoff per stage boundary — so PP tolerates the slower link. Hence <strong>PP across nodes</strong>, giving <strong>PP = 4</strong> for four nodes.</p><p>The 16-GPU layout is therefore <strong>TP=4 × PP=4</strong>:</p><pre>       PP stage 0     PP stage 1     PP stage 2     PP stage 3<br>        Node A         Node B         Node C         Node D<br>        [4 GPU TP]  →  [4 GPU TP]  →  [4 GPU TP]  →  [4 GPU TP]<br>         (XGMI)         (XGMI)         (XGMI)         (XGMI)<br>             └──────── Slingshot carries the PP handoffs ───────┘</pre><h3>The other two axes, for context</h3><ul><li><strong>Data Parallelism (DP)</strong> — replicate the whole model; each replica serves different requests. This is how you scale <em>throughput</em> once the model fits.</li><li><strong>Expert Parallelism (EP)</strong> — for Mixture-of-Experts models, experts are spread across GPUs and stitched with <strong>all-to-all</strong>. On MI300, enabling AMD’s AITER path (VLLM_ROCM_USE_AITER=1) is recommended when combining PP with EP.</li></ul><h3>The judgement call people skip</h3><p>Here’s the counterintuitive part: <strong>multi-node buys you memory, not speed.</strong> Splitting across nodes adds communication and introduces pipeline “bubbles” (stages idling while they wait for work upstream). So the decision tree is:</p><ul><li>If the model <strong>fits in one node’s 512 GB</strong> → run <strong>TP=4 on single nodes</strong> and scale out with <strong>replicas (DP)</strong>. This serves far more throughput than stretching one model across nodes.</li><li>Only reach for <strong>PP across nodes</strong> when the model genuinely <strong>exceeds 512 GB</strong>.</li></ul><p>Do the memory math first. A rough estimate: memory ≈ params × bytes-per-param + KV-cache + overhead. A 70B model in bf16 (~140 GB) fits one node comfortably; a 405B model in bf16 (~810 GB) genuinely needs two or more nodes. Choosing PP=4 for a model that fits in one node is a common and expensive mistake.</p><h3>7. The serving engine: vLLM</h3><p><strong>vLLM</strong> runs the model and exposes an OpenAI-compatible HTTP API. Two ideas make it fast:</p><ul><li><strong>PagedAttention</strong> manages the KV cache like an operating system manages virtual memory — in fixed-size pages — so memory isn’t wasted on padding and many concurrent requests pack efficiently. At serving time the KV cache is often the real memory hog, which is why this matters.</li><li><strong>Continuous batching</strong> lets requests join and leave the running batch token-by-token, instead of waiting for a fixed batch to drain, keeping the GPUs saturated.</li></ul><p>For multi-node, vLLM uses a distributed executor. Pointed at Ray, it places all TP×PP workers across a Ray cluster spanning your pods:</p><pre>vllm serve &lt;model&gt; \<br>  --tensor-parallel-size 4 \<br>  --pipeline-parallel-size 4 \<br>  --distributed-executor-backend ray \<br>  --gpu-memory-utilization 0.85    # leave headroom on unified memory (see §2)</pre><p>On MI300 use AMD’s ROCm build of vLLM, match the image’s ROCm version to the host driver, and remember the unified-memory caveat when setting the memory-utilization knob.</p><h3>8. How Kubernetes hands hardware to a pod: device plugins vs. DRA</h3><p>This is the section that surprises people, and it’s the reason that node dump in §3 advertised hardware two different ways.</p><h3>The classic Device Plugin (how your GPUs arrive)</h3><p>A per-node DaemonSet advertises an <strong>extended resource</strong>: an opaque integer count like amd.com/gpu: 4. Pods ask for it under resources.limits, and the scheduler does simple arithmetic — are four free on this node? It works, but it can only ever express <em>how many</em>. It cannot express <em>which kind</em> or <em>configured how</em>.</p><pre>resources:<br>  limits:<br>    amd.com/gpu: &quot;4&quot;     # &quot;give me any 4 GPUs&quot; — pure counting</pre><h3>Dynamic Resource Allocation — DRA (how your Slingshot NICs arrive)</h3><p>Some devices need more than counting. A Slingshot NIC has to be <strong>selected</strong>, <strong>parameterized</strong>, and <strong>prepared by its driver</strong> (assigned a CXI service, a /dev/cxi0 node injected). For that class of device, Kubernetes introduced <strong>Dynamic Resource Allocation (DRA)</strong>. It reached general availability in the Kubernetes <strong>1.34–1.35</strong> timeframe; on a <strong>1.36</strong> cluster it&#39;s the stable resource.k8s.io/v1 API.</p><p>DRA introduces a whole new API group with four object types:</p><p>Object Scope Created by What it is <strong>DeviceClass</strong> cluster driver / admin a category of devices plus a selector (e.g. device.driver == &quot;cxi.slingshot.hpe.com&quot;) <strong>ResourceSlice</strong> cluster driver DaemonSet the <strong>inventory</strong> — &quot;this node has N devices.&quot; This is the Node-Local ResourceSlices line from §3 <strong>ResourceClaim</strong> namespace user / template a <strong>request</strong> for device(s); carries allocated / reserved status <strong>ResourceClaimTemplate</strong> namespace user a stamp that mints a <strong>fresh, per-pod</strong> claim for each pod</p><p>And two new fields inside the Pod spec — the “extra section next to resources&quot; that catches everyone&#39;s eye the first time:</p><pre>spec:<br>  resourceClaims:                          # pod level: &quot;I need a claim named cxi&quot;<br>    - name: cxi<br>      resourceClaimTemplateName: cxi-nic-template<br>  containers:<br>    - name: worker<br>      resources:<br>        limits:<br>          amd.com/gpu: &quot;4&quot;                 # ← device plugin (old, countable)<br>        claims:<br>          - name: cxi                      # ← DRA (new): bind the NIC into THIS container</pre><p>You can watch the machinery directly:</p><pre># The inventory the driver publishes, per node:<br>$ kubectl get resourceslices<br>NAME                       NODE          DRIVER                  POOL          AGE<br>gpu-node-01-cxi-abc123     gpu-node-01   cxi.slingshot.hpe.com   gpu-node-01   5d</pre><pre># The cluster-scoped category your claims point at:<br>$ kubectl get deviceclass<br>NAME                    AGE<br>cxi.slingshot.hpe.com   6d</pre><pre># A live claim and its state:<br>$ kubectl get resourceclaim<br>NAME             STATE                AGE<br>worker-0-cxi     allocated,reserved   2m</pre><p><strong>Lifecycle</strong>, so the states make sense: a claim starts unallocated → the scheduler picks devices from a matching ResourceSlice and marks it <strong>allocated</strong> → it’s <strong>reserved</strong> for a specific pod → the driver prepares the device on the node (via the Container Device Interface, which injects /dev/cxi0 and sets FI_CXI_SVC_ID) → the pod runs → on deletion, the claim is released. If a pod dies uncleanly, a claim can get stuck reserved; since CXI services per node are finite, kubectl get resourceclaim is worth watching.</p><p><strong>Why a <em>template</em> rather than one shared claim:</strong> each worker pod needs its <em>own</em> NIC and CXI service. A ResourceClaimTemplate mints a separate claim per pod automatically. A single shared ResourceClaim would try to hand the same device to everyone.</p><p>The one-line contrast to remember:</p><blockquote><em>requests/limits says </em>&quot;give me N of these countable things.&quot;<em> </em><em>resourceClaims / </em><em>claims says </em>&quot;allocate me a real, configured device from this driver and wire it into my container.&quot;</blockquote><h3>9. Scheduling and quota: KubeRay and Kueue</h3><p>Two control-plane systems sit above the pods.</p><h3>KubeRay — running Ray on Kubernetes</h3><p>The <strong>KubeRay operator</strong> watches custom resources and turns them into pods:</p><ul><li><strong>RayCluster</strong> — a persistent cluster: one <strong>head</strong> group plus one or more <strong>worker</strong> groups. The head runs Ray’s global control store, dashboard, and scheduler; workers run the raylets that host tasks. Your vLLM workers live here.</li><li><strong>RayJob</strong> — runs a workload to completion (great for a one-shot benchmark).</li><li><strong>RayService</strong> — a managed, self-healing serving deployment; the production-grade way to run vLLM once you’ve validated it as a RayCluster.</li></ul><p>A Ray cluster needs <strong>all its pods together</strong> — a lone head with no workers just hangs. That’s why quota-aware, all-or-nothing scheduling matters.</p><h3>Kueue — admission and quota</h3><p><strong>Kueue</strong> doesn’t place pods itself; it decides <strong>whether and when</strong> a workload is admitted, then lets the normal scheduler take over. Its objects:</p><ul><li><strong>ResourceFlavor</strong> — describes a <em>flavor</em> of capacity (e.g. “the MI300A compute nodes”). Critically, a flavor <strong>can inject a </strong><strong>nodeSelector and </strong><strong>tolerations into every admitted workload</strong>. This is the detail that saves you from a very common failure mode (below).</li><li><strong>ClusterQueue</strong> — a cluster-scoped <strong>quota pool</strong> (e.g. 48 GPU / 2160 CPU / a few TB of memory), often set deliberately <em>below</em> physical capacity so platform DaemonSets always have room.</li><li><strong>LocalQueue</strong> — a namespaced pointer to a ClusterQueue. Workloads join it by label.</li><li><strong>Workload</strong> — the internal object Kueue creates to represent your job while it’s queued or admitted. ADMITTED: True means quota was granted.</li></ul><p>You submit by adding one label:</p><pre>metadata:<br>  labels:<br>    kueue.x-k8s.io/queue-name: gpu</pre><p>Kueue then creates the workload <strong>suspended</strong>, checks quota, and releases it when it fits. A queueing strategy like <strong>BestEffortFIFO</strong> means a job that doesn’t currently fit <strong>waits in the queue</strong> rather than being rejected — so when something looks stuck, your first command is:</p><pre>kubectl get workload -n &lt;namespace&gt;    # check admission BEFORE looking at pods</pre><p>The pods won’t even exist until Kueue admits the workload.</p><h3>The single most valuable operational rule</h3><p>If your cluster uses a ResourceFlavor that injects the compute-node toleration, then:</p><blockquote><strong><em>Do NOT put tolerations in your manifests, and DO add the Kueue queue-name label.</em></strong></blockquote><p>Hand-writing the toleration while also routing around Kueue is, on clusters like this, the number-one cause of pods stuck Pending forever. Let the flavor own the toleration; you own the queue label. (The corollary: if you ever bypass Kueue deliberately, you must add the toleration back by hand.)</p><h3>10. Benchmarking the fabric with rccl-tests</h3><p>Before serving anything, prove the interconnect. rccl-tests is the standard tool.</p><pre>all_reduce_perf -b 8 -e 8G -f 2 -g 1</pre><ul><li>-b 8 -e 8G sweeps message sizes from 8 bytes to 8 GB; -f 2 doubles each step; -g 1 uses one GPU per process (the recommended 1:1 mapping).</li><li><strong>Report busbw, not algbw.</strong> <em>algbw</em> is raw bytes ÷ time. <em>busbw</em> (“bus bandwidth”) normalizes for the collective’s traffic pattern so the figure reflects real link utilization and is comparable across GPU counts. For all-reduce, busbw = algbw × 2(n−1)/n. Read it <strong>out-of-place, at the largest message sizes.</strong></li><li><strong>Launching across nodes needs MPI</strong>, since rccl-tests spawns one rank per GPU and MPI coordinates them. On bare metal that’s srun/mpirun; on Kubernetes it&#39;s the MPI Operator (an MPIJob) or a submission pod.</li><li><strong>Isolate the fabric.</strong> To measure a clean single Slingshot hop, run <strong>two GPUs on two different nodes</strong> (one each). Mixing GPUs on the same node folds in XGMI and inflates the number.</li></ul><p>Example results from a healthy cluster, read correctly:</p><p>Test Result What it means all-reduce, 1 node, 4 GPU ~242 GB/s peak the <strong>XGMI</strong> ceiling — intra-node all-reduce, 2 nodes, 8 GPU ~146 GB/s peak, ~136 GB/s at 1 GB the <strong>Slingshot</strong> number — inter-node p2p, container-to-container ~23.9 GB/s at 64 KB one 200G NIC at line rate, from inside a pod send latency ~2.5 µs at 8 B GPUDirect RDMA is really engaging</p><p>The 2-node figure being lower than the 1-node figure is not a regression — it’s the expected step down when traffic leaves XGMI for Slingshot. The fact that it <em>holds</em> down to realistic 1 GB messages is the quality signal.</p><h3>11. Putting it together: the life of a token</h3><pre>1. A request hits the vLLM OpenAI endpoint (a Service in front of the Ray head pod).<br>2. vLLM slots it into the continuously-batched running set (PagedAttention KV cache).<br>3. Compute runs across the 4 GPUs of a node (TP); each split op triggers an<br>   all-reduce carried over XGMI — fast, intra-node.<br>4. As the token flows through the model&#39;s layer-stages, activations hand off<br>   node → node (PP) over Slingshot: RCCL → OFI plugin → libfabric(cxi) → Cassini → Rosetta.<br>5. The final stage emits the token; repeat until the response is complete.</pre><p>And how the deployment came to exist:</p><pre>KubeRay sees your RayCluster (labelled for the LocalQueue)<br>   → Kueue suspends it, checks quota, injects the node toleration via the flavor, admits it<br>   → the scheduler places one pod per node (anti-affinity)<br>   → each pod gets 4 GPUs (device plugin) + a CXI NIC (DRA claim)<br>   → the driver preps /dev/cxi0 and FI_CXI_SVC_ID<br>   → Ray head + workers form a cluster<br>   → vLLM spreads its TP×PP workers across all 16 GPUs</pre><h3>12. Lessons worth carrying to your own cluster</h3><ul><li><strong>Understand your interconnect tiers before you choose parallelism.</strong> TP inside the node, PP across nodes. Get this backwards and you’ll saturate the slow link with the chattiest traffic.</li><li><strong>The silent TCP fallback is the enemy.</strong> Always confirm the cxi provider and GDRDMA in NCCL_DEBUG=INFO before trusting any bandwidth number.</li><li><strong>Multi-node buys memory, not speed.</strong> If the model fits in one node, replicate instead of stretching it across the fabric.</li><li><strong>Know which mechanism owns which device.</strong> GPUs via the device plugin (limits), specialized NICs via DRA (claims). They coexist in the same pod.</li><li><strong>Let the platform own tolerations; you own the queue label.</strong> On a Kueue-managed cluster, hand-written tolerations plus bypassing the queue is the classic Pending-forever trap.</li><li><strong>Benchmark the fabric before you deploy the model.</strong> rccl-tests first, vLLM second. Every serving symptom otherwise traces back to a question you should have answered up front.</li></ul><h3>13. Glossary + further reading</h3><p><strong>Hardware &amp; fabric</strong></p><ul><li><strong>APU</strong> — CPU + GPU + memory fused in one package (MI300A: 4 per node, 128 GB HBM3 each).</li><li><strong>Unified memory</strong> — CPU and GPU share the same HBM3; no PCIe copy, but a shared budget.</li><li><strong>XGMI / Infinity Fabric</strong> — the fast intra-node GPU link.</li><li><strong>Slingshot</strong> — HPE’s inter-node fabric. <strong>Cassini</strong> = NIC, <strong>Rosetta</strong> = switch, <strong>Slingshot 11</strong> = the Cassini/Rosetta 200G generation.</li><li><strong>CXI</strong> — the Cassini interface/driver. <strong>GPUDirect RDMA</strong> — NIC reads/writes GPU memory directly.</li></ul><p><strong>Collectives &amp; libraries</strong></p><ul><li><strong>RCCL</strong> — AMD collectives, NCCL-compatible. <strong>Collective</strong> — all-reduce, all-gather, reduce-scatter, all-to-all.</li><li><strong>libfabric (OFI)</strong> — vendor-neutral networking API; the <strong>cxi provider</strong> drives Slingshot.</li><li><strong>aws-ofi-nccl/rccl</strong> — the shim letting RCCL use libfabric instead of TCP.</li></ul><p><strong>Parallelism</strong></p><ul><li><strong>TP</strong> — split layers, all-reduce heavy → keep intra-node. <strong>PP</strong> — split into stages → across nodes.</li><li><strong>DP</strong> — replicas for throughput. <strong>EP</strong> — MoE experts, all-to-all.</li></ul><p><strong>Kubernetes</strong></p><ul><li><strong>Device plugin</strong> — countable extended resource (amd.com/gpu).</li><li><strong>DRA</strong> — rich device allocation: DeviceClass, ResourceSlice, ResourceClaim(+Template); pod resourceClaims, container resources.claims.</li><li><strong>KubeRay</strong> — the RayCluster / RayJob / RayService operator.</li><li><strong>Kueue</strong> — ResourceFlavor, ClusterQueue, LocalQueue, Workload; admission and quota.</li></ul><p><strong>Key environment variables</strong></p><pre>NCCL_DEBUG=INFO              # verify the transport (look for cxi + GDRDMA)<br>FI_PROVIDER=cxi              # force libfabric onto Slingshot<br>NCCL_NET_GDR_LEVEL=3         # enable GPUDirect RDMA<br>NCCL_MIN_NCHANNELS=42        # MI300 sweet spot<br>FI_CXI_DEFAULT_CQ_SIZE=131072<br>NCCL_SOCKET_IFNAME=&lt;hsn-iface&gt;   # bootstrap interface on the high-speed net<br>HSA_NO_SCRATCH_RECLAIM=1     # only on ROCm &lt; 6.4<br>VLLM_ROCM_USE_AITER=1        # MoE + PP/EP on MI300</pre><p><strong>Further reading</strong></p><ul><li>RCCL &amp; rccl-tests: the ROCm rccl-tests repository (build, flags, busbw).</li><li>ROCm documentation: “prerequisite system validation” (RCCL bandwidth test, MI300 NUMA notes).</li><li>AMD Instinct system-acceptance guide: RCCL benchmarking thresholds.</li><li>Kubernetes docs: Dynamic Resource Allocation (resource.k8s.io).</li><li>Kueue docs: ResourceFlavor, ClusterQueue, admission, queueing strategies.</li><li>KubeRay docs and vLLM’s distributed-serving guide (tensor/pipeline parallel).</li></ul><p><em>If this saved you a week of debugging silent TCP fallbacks and Pending pods, that was the goal. The stack is young and under-documented; corrections and war stories welcome.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fbe6dda1282" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Squeezing a 550B Model onto a Single Node: Our Nemotron-3-Ultra Journey on 8× H200]]></title>
            <link>https://medium.com/@shivank1128/squeezing-a-550b-model-onto-a-single-node-our-nemotron-3-ultra-journey-on-8-h200-70fdb6d983e2?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/70fdb6d983e2</guid>
            <category><![CDATA[nemotron]]></category>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[ultra]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 17:47:32 GMT</pubDate>
            <atom:updated>2026-06-10T17:47:32.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZHCVJfPryoFL1LLn.png" /></figure><p>We wanted to serve <strong>NVIDIA Nemotron-3-Ultra-550B-A55B</strong> in production. It’s a beast:</p><ul><li><strong>550B total parameters / 55B active</strong> — a Mixture-of-Experts model</li><li>A hybrid <strong>Mamba-2 + Attention + MoE</strong> architecture with <strong>Multi-Token Prediction (MTP)</strong> speculative decoding</li><li>A native <strong>256K context window</strong></li></ul><p>Our hardware was a single GPU node: <strong>8× NVIDIA H200</strong>, ~140 GiB each, for <strong>~1,123 GiB of total HBM</strong>.</p><p>The official recipe for the BF16 checkpoint uses tensor-parallel-size 16 across 2 nodes — because BF16 weights alone are ~1.1 TB (550B × 2 bytes), which leaves no room for caches and activations on one node. We didn’t have a second node. So: <strong>can we make this fit on one 8-GPU box?</strong></p><p>Ref: <a href="https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16">https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1Hb-3mT3w0LM2ADTKQ2iQ.png" /></figure><p>Before the memory fight, it helps to understand what this model actually is — because its architecture is exactly why the memory story plays out the way it does.</p><h3>A quick primer: Transformers, Mamba, and MoE</h3><p>If you’ve only ever heard “it’s a transformer,” this model has two more tricks worth knowing.</p><h4>Transformers and the KV cache problem</h4><p>A Transformer processes a sequence using self-attention: every token looks at every other token. That’s what makes them so good at recall and in-context reasoning — but it has a cost. Attention is quadratic in sequence length, and at inference time the model caches the keys/values of every past token (the “KV cache”) so it doesn’t recompute them. The longer your context, the bigger that KV cache grows — and at a 256K context window, that’s a lot of memory.</p><h4>Mamba: constant-size state instead of a growing cache</h4><p>Mamba is a State Space Model (SSM). Instead of attending to all past tokens, it carries a fixed-size recurrent state that it updates token by token — closer to how an RNN works, but designed to be parallelizable and hardware-efficient. The payoff:</p><ul><li>Linear scaling with sequence length (not quadratic)</li><li>A constant-size state per sequence, no matter how long the context — no ever-growing KV cache</li></ul><p>The downside: a pure SSM can be weaker at precise long-range recall (the thing attention is great at).</p><h4>The hybrid: best of both</h4><p>Nemotron-3-Ultra is a hybrid (the “Nemotron-H” lineage): it’s mostly Mamba-2 layers with a few attention layers sprinkled in. You get Mamba’s cheap, long-context efficiency for the bulk of the network, plus attention’s sharp recall where it matters most. This is why the model has both a “Mamba cache” (the SSM state) and a small KV cache (for the attention layers) — and why both show up as memory knobs later.</p><h4>MoE: 550B of knowledge, 55B of compute</h4><p>Finally, the Mixture-of-Experts (MoE) part. Instead of one giant dense feed-forward block per layer, an MoE layer has many smaller “expert” blocks and a router that sends each token to just a few of them. That’s the “550B total / 55B active” split: the model holds 550B parameters, but any single token only activates ~55B of them.</p><p>Here’s the crucial infra consequence: <strong>MoE saves compute, not memory.</strong> Every expert’s weights must still live in HBM, because any token might need any expert. So the full 550B of weights occupy GPU memory regardless — which makes the weight footprint the dominant cost, and weight quantization the single biggest lever for fitting on fewer/smaller GPUs.</p><p>And MTP (Multi-Token Prediction): the model can predict several future tokens at once, which vLLM uses for speculative decoding to speed up generation.</p><p>Keep three things in mind as we go:</p><ol><li>All 550B weights sit in HBM → quantize the weights to fit.</li><li>Mamba layers keep per-sequence state small → long context is cheap-ish.</li><li>A few attention layers still need a KV cache → shrinking it (FP8) buys headroom.</li></ol><p>The manifests below are plain Kubernetes — no operator, no Helm. They assume the NVIDIA device plugin is installed and a storage class that supports ReadWriteMany (we use nfs-csi).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iq68w7KMAdRdglfO6P5HSQ.png" /></figure><h3>Phase 1: Brute-force it onto one node with runtime FP8 + VRAM tricks</h3><p>If BF16 is too big, quantize the weights. vLLM can do on-the-fly FP8 weight quantization — you still hand it the BF16 checkpoint, but it converts weights to FP8 (1 byte/param) at load. That’s ~1.1 TB → ~550 GB, which fits on 8× H200… if you claw back memory everywhere else.</p><h4>The VRAM-saving tricks</h4><ul><li>--quantization fp8 — on-the-fly FP8 weight quantization. The biggest win: ~1.1 TB → ~550 GB.</li><li>--no-enable-prefix-caching — on this hybrid, prefix caching forces the Mamba cache into a memory-heavy &quot;all&quot; mode that OOM&#39;d us. Disabling it shrinks the persistent cache.</li><li>--max-num-batched-tokens 8192 — reduced from 32768. This was the allocation that tipped us into OOM; the activation buffer scales with it. Smaller buffer fits, at the cost of long-prompt prefill throughput.</li><li>--mamba-ssm-cache-dtype float16 — keep the Mamba SSM state in fp16, not fp32, to halve it.</li><li>--gpu-memory-utilization 0.90 — use 90% of each card.</li><li>--tensor-parallel-size 8 + --enable-expert-parallel — shard across all 8 GPUs, with expert parallelism for the MoE layers.</li></ul><h4>Phase 1 manifest (complete)</h4><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: hf-token<br>  namespace: vllm<br>type: Opaque<br>stringData:<br>  token: &quot;hf_REPLACE_ME&quot;          # HF token;<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: nemotron-550b-bf16-cache<br>  namespace: vllm<br>spec:<br>  accessModes: [&quot;ReadWriteMany&quot;]<br>  storageClassName: nfs-csi<br>  resources:<br>    requests:<br>      storage: 1200Gi<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nemotron-550b-bf16<br>  namespace: vllm<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate                 # single-node, whole-node GPUs: never surge a 2nd pod<br>  selector:<br>    matchLabels:<br>      app: nemotron-550b<br>  template:<br>    metadata:<br>      labels:<br>        app: nemotron-550b<br>    spec:<br>      affinity:<br>        nodeAffinity:<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            nodeSelectorTerms:<br>              - matchExpressions:<br>                  - key: kubernetes.io/hostname<br>                    operator: In<br>                    values: [&quot;ihc-gpu-compute-05&quot;]<br>      containers:<br>        - name: vllm<br>          image: vllm/vllm-openai:v0.22.0<br>          command: [&quot;vllm&quot;, &quot;serve&quot;, &quot;nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16&quot;]<br>          args:<br>            - &quot;--served-model-name&quot;<br>            - &quot;nvidia/NVIDIA-Nemotron-3-Ultra&quot;<br>            - &quot;--trust-remote-code&quot;<br>            - &quot;--tensor-parallel-size&quot;<br>            - &quot;8&quot;<br>            - &quot;--enable-expert-parallel&quot;<br>            - &quot;--dtype&quot;<br>            - &quot;bfloat16&quot;<br>            - &quot;--quantization&quot;<br>            - &quot;fp8&quot;                         # on-the-fly weight quantization to fit<br>            - &quot;--max-model-len&quot;<br>            - &quot;262144&quot;<br>            - &quot;--max-num-seqs&quot;<br>            - &quot;16&quot;<br>            - &quot;--max-num-batched-tokens&quot;<br>            - &quot;8192&quot;                        # reduced from 32768 to dodge the OOM allocation<br>            - &quot;--gpu-memory-utilization&quot;<br>            - &quot;0.90&quot;<br>            - &quot;--enable-chunked-prefill&quot;<br>            - &quot;--no-enable-prefix-caching&quot;  # avoid Mamba &quot;all&quot;-mode cache OOM<br>            - &quot;--mamba-ssm-cache-dtype&quot;<br>            - &quot;float16&quot;<br>            - &quot;--mamba-backend&quot;<br>            - &quot;flashinfer&quot;<br>            - &quot;--reasoning-parser&quot;<br>            - &quot;nemotron_v3&quot;<br>            - &quot;--enable-auto-tool-choice&quot;<br>            - &quot;--tool-call-parser&quot;<br>            - &quot;qwen3_coder&quot;                 # yes — the official tool parser for Nemotron-3<br>            - &quot;--speculative-config&quot;<br>            - &#39;{&quot;method&quot;: &quot;nemotron_h_mtp&quot;, &quot;num_speculative_tokens&quot;: 5}&#39;<br>          ports:<br>            - containerPort: 8000<br>          env:<br>            - name: HF_HOME<br>              value: /data<br>            - name: HF_TOKEN<br>              valueFrom:<br>                secretKeyRef:<br>                  name: hf-token<br>                  key: token<br>          resources:<br>            limits:<br>              nvidia.com/gpu: 8<br>            requests:<br>              cpu: &quot;64&quot;<br>              memory: 512Gi<br>              nvidia.com/gpu: 8<br>          volumeMounts:<br>            - name: cache<br>              mountPath: /data<br>            - name: shm<br>              mountPath: /dev/shm<br>          startupProbe:                     # ~3h budget: 1.1TB download + quantize + load<br>            httpGet: { path: /health, port: 8000 }<br>            initialDelaySeconds: 120<br>            periodSeconds: 60<br>            failureThreshold: 180<br>          readinessProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 10<br>          livenessProbe:                    # generous: a restart reloads 550B<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 30<br>            timeoutSeconds: 10<br>            failureThreshold: 6<br>      volumes:<br>        - name: cache<br>          persistentVolumeClaim:<br>            claimName: nemotron-550b-bf16-cache<br>        - name: shm<br>          emptyDir:<br>            medium: Memory<br>            sizeLimit: 16Gi<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: nemotron-550b<br>  namespace: vllm<br>spec:<br>  selector:<br>    app: nemotron-550b<br>  ports:<br>    - name: http<br>      port: 80<br>      targetPort: 8000</pre><p>It worked — but Phase 1 had an ugly underbelly: we downloaded the full ~1.1 TB BF16 checkpoint, quantized it at load time (a load-time memory spike as BF16 weights are materialized then converted), suffered slow cold starts, and gave up prefix caching and prefill batch size just to fit. A working hack, not a clean answer.</p><h3>Phase 2: Stop quantizing at runtime — use pre-quantized FP8 weights</h3><p>On the Hugging Face Hub, RedHatAI publishes pre-quantized variants:</p><ul><li>RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic</li><li>RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16</li></ul><p>The FP8-dynamic checkpoint changed the game. vLLM loads weights that are already FP8 (auto-detected — you don’t even pass --quantization). That fixes every Phase 1 pain point:</p><ul><li>Smaller download (~525 GB vs ~1.1 TB)</li><li>No load-time quantization spike — weights land in HBM already FP8</li><li>Faster cold starts</li><li>H200-native FP8 tensor cores do the math directly — Hopper’s fast path</li><li>Near-BF16 quality, via dynamic per-token activation scaling</li></ul><p>And the parsers are quantization-independent: qwen3_coder + nemotron_v3 are unchanged, because quantization touches weights, not the chat template.</p><h4>The zero-downtime trick: pre-warm the weights with a 0-GPU Job</h4><p>The old pod holds all 8 GPUs on our only GPU node. If a new 8-GPU deployment starts before the old one dies, it sits Pending forever — and then suffers a slow 525 GB cold download. So we pull the weights ahead of time, in parallel, with a Job that requests zero GPUs. It schedules anywhere, touches no GPU, and just downloads into the new PVC while production keeps serving.</p><h4>Phase 2a manifest — 0-GPU pre-warm Job (complete)</h4><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: nemotron-550b-fp8-cache<br>  namespace: vllm<br>spec:<br>  accessModes: [&quot;ReadWriteMany&quot;]<br>  storageClassName: nfs-csi<br>  resources:<br>    requests:<br>      storage: 1200Gi<br>---<br>apiVersion: batch/v1<br>kind: Job<br>metadata:<br>  name: nemotron-550b-fp8-prewarm<br>  namespace: vllm<br>spec:<br>  backoffLimit: 4<br>  template:<br>    spec:<br>      restartPolicy: OnFailure<br>      # No GPU requested -&gt; schedules off the GPU node; the PVC is RWX/NFS so the<br>      # GPU node can read these weights later. This pod only downloads.<br>      containers:<br>        - name: weight-puller<br>          image: vllm/vllm-openai:v0.22.0<br>          command: [&quot;/bin/sh&quot;, &quot;-c&quot;]<br>          args:<br>            - &gt;-<br>              set -eu;<br>              pip install -q hf_transfer 2&gt;/dev/null || true;<br>              export HF_HUB_ENABLE_HF_TRANSFER=1;<br>              echo &quot;Downloading ${MODEL_ID} into ${HF_HOME} ...&quot;;<br>              huggingface-cli download &quot;${MODEL_ID}&quot; --repo-type model;<br>              echo &quot;DOWNLOAD COMPLETE: ${MODEL_ID}&quot;;<br>          env:<br>            - name: HF_HOME<br>              value: /data<br>            - name: MODEL_ID<br>              value: &quot;RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic&quot;<br>            - name: HF_TOKEN<br>              valueFrom:<br>                secretKeyRef:<br>                  name: hf-token<br>                  key: token<br>          resources:<br>            requests:<br>              cpu: &quot;16&quot;<br>              memory: 32Gi<br>          volumeMounts:<br>            - name: cache<br>              mountPath: /data<br>      volumes:<br>        - name: cache<br>          persistentVolumeClaim:<br>            claimName: nemotron-550b-fp8-cache</pre><p>When the Job logs DOWNLOAD COMPLETE, the weights are warm on the PVC.</p><h4>Phase 2b manifest — promote to serving, then sunset the old one (complete)</h4><blockquote><em>⚠️ </em><strong><em>Sequencing:</em></strong><em> both deployments pin the same node and want 8 GPUs. Delete/scale the old deployment first to free the GPUs, then apply this. The warm PVC means it starts fast.</em></blockquote><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nemotron-550b-fp8<br>  namespace: vllm<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: nemotron-550b<br>  template:<br>    metadata:<br>      labels:<br>        app: nemotron-550b              # same label -&gt; reuses the existing Service<br>    spec:<br>      affinity:<br>        nodeAffinity:<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            nodeSelectorTerms:<br>              - matchExpressions:<br>                  - key: kubernetes.io/hostname<br>                    operator: In<br>                    values: [&quot;ihc-gpu-compute-05&quot;]<br>      containers:<br>        - name: vllm<br>          image: vllm/vllm-openai:v0.22.0<br>          command: [&quot;vllm&quot;, &quot;serve&quot;, &quot;RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic&quot;]<br>          args:<br>            - &quot;--served-model-name&quot;<br>            - &quot;nvidia/NVIDIA-Nemotron-3-Ultra&quot;   # keep old name -&gt; transparent cutover<br>            - &quot;--trust-remote-code&quot;<br>            - &quot;--tensor-parallel-size&quot;<br>            - &quot;8&quot;<br>            - &quot;--kv-cache-dtype&quot;<br>            - &quot;fp8&quot;                               # FP8 KV cache -&gt; more headroom<br>            - &quot;--max-model-len&quot;<br>            - &quot;262144&quot;<br>            - &quot;--max-num-seqs&quot;<br>            - &quot;16&quot;<br>            - &quot;--max-num-batched-tokens&quot;<br>            - &quot;32768&quot;                             # back to recipe value; we have room now<br>            - &quot;--gpu-memory-utilization&quot;<br>            - &quot;0.90&quot;<br>            - &quot;--enable-chunked-prefill&quot;<br>            - &quot;--no-enable-prefix-caching&quot;<br>            - &quot;--enable-flashinfer-autotune&quot;<br>            - &quot;--async-scheduling&quot;<br>            - &quot;--mamba-backend&quot;<br>            - &quot;triton&quot;<br>            - &quot;--mamba-ssm-cache-dtype&quot;<br>            - &quot;float32&quot;<br>            - &quot;--reasoning-parser&quot;<br>            - &quot;nemotron_v3&quot;<br>            - &quot;--enable-auto-tool-choice&quot;<br>            - &quot;--tool-call-parser&quot;<br>            - &quot;qwen3_coder&quot;<br>            - &quot;--speculative_config.method&quot;<br>            - &quot;mtp&quot;<br>            - &quot;--speculative_config.num_speculative_tokens&quot;<br>            - &quot;5&quot;<br>          # NOTE vs Phase 1: no --quantization (auto-detected), no --dtype, and the<br>          # flashinfer/float16 Mamba survival-hacks are gone — the FP8 checkpoint<br>          # freed enough memory that we don&#39;t need them.<br>          ports:<br>            - containerPort: 8000<br>          env:<br>            - name: HF_HOME<br>              value: /data<br>            - name: HF_TOKEN<br>              valueFrom:<br>                secretKeyRef:<br>                  name: hf-token<br>                  key: token<br>          resources:<br>            limits:<br>              nvidia.com/gpu: 8<br>            requests:<br>              cpu: &quot;64&quot;<br>              memory: 512Gi<br>              nvidia.com/gpu: 8<br>          volumeMounts:<br>            - name: cache<br>              mountPath: /data<br>            - name: shm<br>              mountPath: /dev/shm<br>          startupProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            initialDelaySeconds: 60<br>            periodSeconds: 30<br>            failureThreshold: 60            # warm PVC -&gt; much faster than Phase 1<br>          readinessProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 10<br>          livenessProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 30<br>            timeoutSeconds: 10<br>            failureThreshold: 6<br>      volumes:<br>        - name: cache<br>          persistentVolumeClaim:<br>            claimName: nemotron-550b-fp8-cache<br>        - name: shm<br>          emptyDir:<br>            medium: Memory<br>            sizeLimit: 16Gi</pre><h3>Phase 3 (planned): FP4 weights to run on 4× H200</h3><p>Phase 2 is clean, but we want to free up GPUs — run on 4× H200 and use the other four elsewhere. That means 4-bit weights:</p><ul><li>RedHatAI/...quantized.w4a16 — INT4 weight-only</li><li>nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 — NVIDIA&#39;s FP4 format</li></ul><p>The math is compelling: 4-bit weights are ~280 GB, which fits on 4× H200 (~561 GiB total, ~505 GiB usable at 0.9), leaving ~225 GiB for caches. FP8 (~525 GB) can’t fit on 4 cards; FP4 can.</p><h4>The catch: NVFP4 is not hardware-accelerated on H200</h4><p>NVFP4’s native tensor-core acceleration requires Blackwell (B200 / GB200). Our H200s are Hopper — native FP8, but no native FP4 math units. So on H200, NVFP4 (or w4a16) runs via dequantization kernels: the 4-bit weights are unpacked to higher precision for the matmul. You get the memory savings (the whole reason to drop to 4 GPUs) but not the FP4 compute speedup Blackwell would give you. In compute-bound regimes, FP4-on-Hopper can even be slower than FP8-on-Hopper, because FP8 hits the silicon directly while FP4 pays a dequant tax.</p><p>So Phase 3 is a deliberate trade: fit on 4 GPUs (FP4, no native accel) vs. peak throughput on 8 GPUs (FP8, native Hopper accel).</p><h4>Phase 3 manifest (complete)</h4><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: nemotron-550b-fp4-cache<br>  namespace: vllm<br>spec:<br>  accessModes: [&quot;ReadWriteMany&quot;]<br>  storageClassName: nfs-csi<br>  resources:<br>    requests:<br>      storage: 600Gi                  # ~280 GB of 4-bit weights<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nemotron-550b-fp4<br>  namespace: vllm<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: nemotron-550b-fp4<br>  template:<br>    metadata:<br>      labels:<br>        app: nemotron-550b-fp4<br>    spec:<br>      affinity:<br>        nodeAffinity:<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            nodeSelectorTerms:<br>              - matchExpressions:<br>                  - key: kubernetes.io/hostname<br>                    operator: In<br>                    values: [&quot;ihc-gpu-compute-05&quot;]<br>      containers:<br>        - name: vllm<br>          image: vllm/vllm-openai:v0.22.0<br>          command: [&quot;vllm&quot;, &quot;serve&quot;, &quot;RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16&quot;]<br>          args:<br>            - &quot;--served-model-name&quot;<br>            - &quot;nvidia/NVIDIA-Nemotron-3-Ultra&quot;<br>            - &quot;--trust-remote-code&quot;<br>            - &quot;--tensor-parallel-size&quot;<br>            - &quot;4&quot;                         # half the node now<br>            - &quot;--kv-cache-dtype&quot;<br>            - &quot;fp8&quot;<br>            - &quot;--max-model-len&quot;<br>            - &quot;262144&quot;<br>            - &quot;--max-num-seqs&quot;<br>            - &quot;16&quot;<br>            - &quot;--gpu-memory-utilization&quot;<br>            - &quot;0.90&quot;<br>            - &quot;--enable-chunked-prefill&quot;<br>            - &quot;--no-enable-prefix-caching&quot;<br>            - &quot;--mamba-backend&quot;<br>            - &quot;triton&quot;<br>            - &quot;--reasoning-parser&quot;<br>            - &quot;nemotron_v3&quot;<br>            - &quot;--enable-auto-tool-choice&quot;<br>            - &quot;--tool-call-parser&quot;<br>            - &quot;qwen3_coder&quot;<br>            - &quot;--speculative_config.method&quot;<br>            - &quot;mtp&quot;<br>            - &quot;--speculative_config.num_speculative_tokens&quot;<br>            - &quot;5&quot;<br>          ports:<br>            - containerPort: 8000<br>          env:<br>            - name: HF_HOME<br>              value: /data<br>            - name: HF_TOKEN<br>              valueFrom:<br>                secretKeyRef:<br>                  name: hf-token<br>                  key: token<br>          resources:<br>            limits:<br>              nvidia.com/gpu: 4<br>            requests:<br>              cpu: &quot;48&quot;<br>              memory: 256Gi<br>              nvidia.com/gpu: 4<br>          volumeMounts:<br>            - name: cache<br>              mountPath: /data<br>            - name: shm<br>              mountPath: /dev/shm<br>          startupProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            initialDelaySeconds: 60<br>            periodSeconds: 30<br>            failureThreshold: 60<br>          readinessProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 10<br>          livenessProbe:<br>            httpGet: { path: /health, port: 8000 }<br>            periodSeconds: 30<br>            timeoutSeconds: 10<br>            failureThreshold: 6<br>      volumes:<br>        - name: cache<br>          persistentVolumeClaim:<br>            claimName: nemotron-550b-fp4-cache<br>        - name: shm<br>          emptyDir:<br>            medium: Memory<br>            sizeLimit: 16Gi</pre><p><em>(Use the same 0-GPU pre-warm Job from Phase 2 — just point </em><em>MODEL_ID at the w4a16 or NVFP4 repo and a fresh PVC.)</em></p><h3>Lessons learned</h3><ul><li><strong>Architecture drives the memory story.</strong> MoE keeps all 550B weights in HBM (it saves compute, not memory), so weight quantization is the master lever. The Mamba layers keep per-sequence state tiny; the few attention layers still need a KV cache you can shrink with FP8.</li><li><strong>Runtime quantization is a survival hack, not a strategy.</strong> It fits the model, but you pay in download size, a load-time spike, and slow cold starts. Pre-quantized checkpoints are strictly better when they exist.</li><li><strong>Match the quantization to the silicon.</strong> H200 = Hopper = native FP8. FP4 buys you memory (fewer GPUs), not speed on Hopper — native FP4 is a Blackwell feature.</li><li><strong>Quantization is orthogonal to parsers.</strong> Tool-calling and reasoning parsers key off the chat template, not the weights.</li><li><strong>Pre-warm with a 0-GPU Job.</strong> Downloading into a PVC while the old model keeps serving turns a terabyte-scale migration into a near-zero-downtime cutover.</li></ul><p>PhaseWeightsGPUsFootprintNotes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p9ZsdtUjMq1oJCKLc16HbA.png" /></figure><p>From “this needs two nodes” to “this runs on half of one” — that’s the power of picking the right quantization for your hardware.</p><p>Running large hybrid MoE models on constrained GPU fleets? The checkpoint format is as much an infrastructure decision as a model decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70fdb6d983e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running LLMs on AMD MI210 GPUs with vLLM and Kubernetes: A Deployment Guide]]></title>
            <link>https://medium.com/@shivank1128/running-llms-on-amd-mi210-gpus-with-vllm-and-kubernetes-a-deployment-guide-5a0533e9a1ca?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a0533e9a1ca</guid>
            <category><![CDATA[amd]]></category>
            <category><![CDATA[mi210]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt-oss]]></category>
            <category><![CDATA[inferencing]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 09:19:24 GMT</pubDate>
            <atom:updated>2026-06-09T09:19:24.410Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*Er41L5ro8dd1l1B8rt9zQg.png" /></figure><p><em>Chat, embedding, and speech models on AMD Instinct MI210 (gfx90a) — with the full manifests and the version/quantization decisions that actually matter.</em></p><p>If you’ve got a cluster of <strong>AMD Instinct MI210</strong> cards and you want to serve open LLMs on them, here’s the honest summary: it works well, but nearly every image, guide, and “day 0 support” headline out there is written for the newer <strong>MI300-class cards</strong> — and a few of those assumptions quietly break on the MI210.</p><p>This guide gives you the working manifests for six models, and — just as importantly — explains <em>why</em> each choice was made: which vLLM image, why not the newest version, and the FP8/MXFP4 story that decides what you can even run. If you’re on MI210 (or its dual-die cousin the MI250, same chip), this should get you serving quickly.</p><h3>TL;DR</h3><ul><li>The MI210 is <strong>gfx90a</strong> (CDNA2). Fully supported by ROCm and vLLM, but <strong>not</strong> part of the MI300-era “day 0” optimized story.</li><li><strong>It has no native FP8 or MXFP4.</strong> This single fact decides which models and checkpoints you can run.</li><li>Use a <strong>pinned</strong> AMD rocm/vllm image — not the latest vLLM — and set VLLM_USE_AITER=0.</li><li>For MXFP4-native models like gpt-oss, use <strong>BF16 upcasts</strong> (e.g. unsloth’s) to route around the missing format.</li><li>Full copy-paste manifests for all six models are below.</li></ul><h3>Part 1: The “day 0 support” reality check</h3><p>AMD’s same-day support announcements for new models are real — but the fine print targets <strong>MI355X, MI350X, MI325X, and MI300X</strong>. The MI210 isn’t on that list.</p><p>That doesn’t mean MI210 is unsupported. gfx90a is a first-class CI target for vLLM’s ROCm backend. It means:</p><ul><li>No AMD-branded “optimized” prebuilt images tuned for your card.</li><li>No newest FP8/FP4 kernels or AITER fast paths — those target gfx942+ (MI300).</li><li>“Day 0 supported in vLLM” means a model’s <em>architecture</em> merged on launch day. That’s architecture-agnostic Python — it’ll generally <strong>load</strong> on gfx90a, but only if it doesn’t need a numeric format or kernel your card lacks.</li></ul><p>Which brings us to the one constraint that shapes everything.</p><h3>Part 2: The MXFP4 and FP8 story — and why we used unsloth weights</h3><p>This is the most important section in the guide, so let me be concrete.</p><p>Modern frontier models are increasingly shipped in <strong>low-precision formats</strong> to fit on fewer GPUs. Two show up constantly:</p><ul><li><strong>FP8</strong> (8-bit float) — many DeepSeek checkpoints.</li><li><strong>MXFP4</strong> (4-bit microscaling) — this is how <strong>gpt-oss</strong> fits 120B parameters onto a single 80 GB H100.</li></ul><p>The MI210 (gfx90a) supports <strong>neither</strong> in the way these checkpoints assume. If you try to serve a model whose weights are published <em>only</em> in FP8, vLLM fails with a beautifully specific error:</p><pre>ValueError: type fp8e4b8 not supported in this architecture.<br>The supported fp8 dtypes are (&#39;fp8e5&#39;,)</pre><p>That’s the GPU telling you it doesn’t have the e4m3 FP8 datatype the checkpoint needs. MXFP4 is the same story — gfx90a has no kernels for it.</p><p>So your model-selection rule is simple:</p><blockquote><strong><em>Run BF16/FP16 weights, or INT4 (AWQ/GPTQ) quants. Avoid FP8/MXFP4-native checkpoints.</em></strong></blockquote><p>That sounds like it rules out gpt-oss entirely. Here’s the trick that saved it:</p><h3>Using unsloth’s BF16 weights</h3><p>The community (unsloth, among others) publishes <strong>BF16 upcasts</strong> of MXFP4 models. These take every 4-bit MXFP4 weight and expand it back to full BF16. The model is numerically the same architecture — it just no longer requires the MXFP4 kernels your card doesn’t have.</p><p>The trade-off is <strong>size</strong>. MXFP4 is 4 bits/weight; BF16 is 16. So a BF16 upcast is roughly 4× larger:</p><ul><li>unsloth/gpt-oss-20b-BF16 → ~42 GB (fits one card)</li><li>unsloth/gpt-oss-120b-BF16 → ~230 GB (needs ~4 cards via tensor-parallel)</li></ul><p>But it <em>runs</em>. I had gpt-oss-20b-BF16 generating coherent text on MI210 — a model I’d assumed was off-limits because of MXFP4. The lesson generalizes:</p><blockquote><em>When a model “doesn’t support” your GPU, the blocker is usually the </em><strong><em>numeric format</em></strong><em>, not the architecture. Find a BF16 or INT4 repackaging and you’re often back in business — you just pay for it in VRAM.</em></blockquote><p>The models in this guide, all BF16 and all gfx90a-friendly: <strong>Qwen3–32B</strong>, <strong>Mixtral-8x7B</strong>, <strong>gpt-oss-20b/120b (BF16 upcasts)</strong>, <strong>Qwen3-Embedding-8B</strong>, and <strong>Whisper-large-v3</strong>.</p><h3>Part 3: The image story — what’s inside, and why not the latest vLLM</h3><p>I used:</p><pre>rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909</pre><p>Decode the tag: <strong>ROCm 6.4.1</strong>, <strong>vLLM 0.10.1</strong>, built <strong>2025–09–09</strong>. These AMD-published images are <em>curated combinations</em> — a ROCm userspace, a PyTorch built against that exact ROCm, vLLM, Triton, the OpenAI-compatible server, and CK flash-attention, all known to work together.</p><h3>What comes with it (and what to watch on gfx90a)</h3><ul><li>A <strong>PyTorch built for a multi-arch list that includes gfx90a</strong> — so your MI210 kernels are present.</li><li><strong>AITER</strong> is in there, but built for <strong>gfx942/gfx950 only</strong> (MI300-class). On gfx90a it has no kernels, so you must disable it (VLLM_USE_AITER=0) or you&#39;ll hit runtime dispatch errors.</li><li>A full ROCm 6.4.1 runtime — which means your <strong>node’s ROCm driver should be ≥ 6.4.1</strong>. A node older than the image is the most common “it loads but crashes” cause. Match them.</li></ul><h3>Why not the latest vLLM?</h3><p>It’s tempting to grab the newest vLLM (0.2x at time of writing) for the freshest model support. Don’t, at least not blindly, and here’s the reasoning:</p><ol><li><strong>gfx90a isn’t in vLLM’s hot CI path.</strong> The project’s momentum is firmly toward MI300+ and CUDA. Newer releases regularly deprecate or refactor older code paths, and a regression on gfx90a can ship unnoticed because nobody’s testing MI210 on every PR. A release that’s a little older has had time to bake.</li><li><strong>The curated combo matters more than the version number.</strong> Pulling the latest vLLM via pip into a base with a mismatched ROCm/PyTorch is a great way to spend an afternoon debugging ABI errors. The AMD prebuilt image hands you a ROCm+torch+vLLM trio that was validated together.</li><li><strong>0.10.1 was new enough.</strong> It’s a September 2025 build — after gpt-oss’s release — so it already knew about the architectures I cared about. The cost of pinning an older version is missing the very newest model types; for everything in this guide, it wasn’t a problem.</li><li><strong>Reproducibility.</strong> Pin the image (ideally by digest) so what you validated is exactly what you ship.</li></ol><p>The rule of thumb: <strong>a known-good pinned image that already contains gfx90a beats chasing </strong><strong>latest every time</strong> — for an older architecture especially.</p><h3>Two env vars, every pod</h3><pre>env:<br>  - { name: VLLM_USE_AITER, value: &quot;0&quot; }              # AITER&#39;s FP8 kernels don&#39;t exist on gfx90a<br>  - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }  # Triton flash-attn = the safe gfx90a path</pre><h3>One entrypoint quirk</h3><p>The rocm/vllm:* images don&#39;t auto-launch the server, so set an explicit command: [&quot;vllm&quot;, &quot;serve&quot;, &quot;&lt;model&gt;&quot;, ...]. (The upstream vllm/vllm-openai-rocm image is the opposite — its entrypoint <em>is</em> the server, so you&#39;d pass --model ... directly. Mixing these up means the container starts and does nothing.)</p><h3>Part 4: Deployment fundamentals</h3><p>Install the <strong>AMD GPU Operator</strong> first — it advertises the amd.com/gpu schedulable resource.</p><p><strong>Sizing.</strong> Each MI210 presents as <strong>one 64 GB device</strong>, so amd.com/gpu: N = N cards, and --tensor-parallel-size N spans them. Rough BF16 math (≈2 bytes/param):</p><p>Model Weights Cards (TP)</p><p>Qwen3-Embedding-8B ~16 GB 1</p><p>Whisper-large-v3 ~3 GB 1</p><p>Qwen3–32B ~64 GB 2</p><p>Mixtral-8x7B ~93 GB 2</p><p>gpt-oss-20b-BF16 ~42 GB 2</p><p>gpt-oss-120b-BF16 ~230 GB 8</p><p>--enforce-eager is recommended on gfx90a (skips HIP graph capture — saves memory, dodges a class of issues), and give each pod a real /dev/shm for tensor-parallel comms.</p><h3>Part 5: The manifests</h3><p>All six assume namespace llm and a Hugging Face token secret (gated models like Qwen3 and Mixtral need it):</p><pre>kubectl create namespace llm<br>kubectl create secret generic hf-token-secret -n llm --from-literal=token=hf_xxxxxxxx</pre><p>PVCs below use storageClassName: local-path — swap in whatever StorageClass your cluster has.</p><h3>1. Qwen3–32B (chat, TP=2)</h3><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-qwen3-32b, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 120Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: qwen3-32b, namespace: llm, labels: { app: qwen3-32b } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: qwen3-32b } }<br>  template:<br>    metadata: { labels: { app: qwen3-32b } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }   # video/render gids; verify per node<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-qwen3-32b } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;Qwen/Qwen3-32B&quot;,<br>                    &quot;--tensor-parallel-size=2&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=32768&quot;,&quot;--gpu-memory-utilization=0.92&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;2&quot; } }<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 80 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: qwen3-32b, namespace: llm }<br>spec:<br>  selector: { app: qwen3-32b }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><p>Every other manifest follows this exact shape — only the model, --tensor-parallel-size, amd.com/gpu, and a couple of flags change. Test any chat model with:</p><pre>kubectl exec -n llm deploy/qwen3-32b -- curl -s http://localhost:8000/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;model&quot;:&quot;Qwen/Qwen3-32B&quot;,&quot;messages&quot;:[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:&quot;Say hi&quot;}],&quot;max_tokens&quot;:32}&#39;</pre><h3>2. Mixtral-8x7B-Instruct (chat, TP=2)</h3><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-mixtral-8x7b, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 200Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: mixtral-8x7b, namespace: llm, labels: { app: mixtral-8x7b } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: mixtral-8x7b } }<br>  template:<br>    metadata: { labels: { app: mixtral-8x7b } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-mixtral-8x7b } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;mistralai/Mixtral-8x7B-Instruct-v0.1&quot;,<br>                    &quot;--tensor-parallel-size=2&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=8192&quot;,&quot;--gpu-memory-utilization=0.90&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;2&quot; } }<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 80 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: mixtral-8x7b, namespace: llm }<br>spec:<br>  selector: { app: mixtral-8x7b }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><blockquote><em>Mixtral is tight at TP=2 across two cards. If it OOMs or restart-loops on load, that’s the first lever — drop </em><em>--max-model-len to 8192 (done above) and </em><em>--gpu-memory-utilization to 0.88.</em></blockquote><h3>3. gpt-oss-20b-BF16 (chat, TP=2)</h3><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-gpt-oss-20b, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 120Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: gpt-oss-20b, namespace: llm, labels: { app: gpt-oss-20b } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: gpt-oss-20b } }<br>  template:<br>    metadata: { labels: { app: gpt-oss-20b } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-gpt-oss-20b } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 16Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;unsloth/gpt-oss-20b-BF16&quot;,<br>                    &quot;--tensor-parallel-size=2&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=8192&quot;,&quot;--gpu-memory-utilization=0.90&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;2&quot; } }<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 80 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: gpt-oss-20b, namespace: llm }<br>spec:<br>  selector: { app: gpt-oss-20b }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><blockquote><em>This is the MXFP4-upcast model from Part 2. Tip: gpt-oss puts its reasoning in a </em><em>reasoning_content field — if </em><em>content looks short, check both. A coherent reply here is your proof that the architecture runs on gfx90a.</em></blockquote><h3>4. gpt-oss-120b-BF16 (chat, TP=8 — needs 8 cards on one node)</h3><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-gpt-oss-120b, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 320Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: gpt-oss-120b, namespace: llm, labels: { app: gpt-oss-120b } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: gpt-oss-120b } }<br>  template:<br>    metadata: { labels: { app: gpt-oss-120b } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-gpt-oss-120b } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 32Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;unsloth/gpt-oss-120b-BF16&quot;,<br>                    &quot;--tensor-parallel-size=8&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=8192&quot;,&quot;--gpu-memory-utilization=0.92&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;8&quot; } }   # all 8 cards on ONE node<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 20, failureThreshold: 120 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 15 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: gpt-oss-120b, namespace: llm }<br>spec:<br>  selector: { app: gpt-oss-120b }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><blockquote><em>The ~230 GB BF16 upcast is the price of running an MXFP4 model on a card without MXFP4. It needs eight MI210s in a single node — if you don’t have that, this is the one model here that won’t fit as-is.</em></blockquote><h3>5. Qwen3-Embedding-8B (embeddings, TP=1)</h3><p>Embedding models are a different task — note --task embed and the /v1/embeddings endpoint.</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-qwen3-embed-8b, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 60Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: qwen3-embed-8b, namespace: llm, labels: { app: qwen3-embed-8b } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: qwen3-embed-8b } }<br>  template:<br>    metadata: { labels: { app: qwen3-embed-8b } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-qwen3-embed-8b } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 8Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;Qwen/Qwen3-Embedding-8B&quot;,<br>                    &quot;--task=embed&quot;,            # newer vLLM: --runner pooling<br>                    &quot;--tensor-parallel-size=1&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=8192&quot;,&quot;--gpu-memory-utilization=0.85&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;1&quot; } }<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 60 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: qwen3-embed-8b, namespace: llm }<br>spec:<br>  selector: { app: qwen3-embed-8b }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><p>Test it:</p><pre>kubectl exec -n llm deploy/qwen3-embed-8b -- curl -s http://localhost:8000/v1/embeddings \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;model&quot;:&quot;Qwen/Qwen3-Embedding-8B&quot;,&quot;input&quot;:&quot;hello world&quot;}&#39;</pre><p>A healthy response has a data[0].embedding array (4096 floats for this model).</p><h3>6. Whisper-large-v3 (speech-to-text, TP=1)</h3><p>Whisper auto-detects as transcription, uses --max-model-len 448, and serves /v1/audio/transcriptions. <strong>Do not</strong> copy vLLM&#39;s example kv_cache_dtype=fp8 — that&#39;s the FP8 wall again.</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata: { name: cache-whisper-v3, namespace: llm }<br>spec:<br>  accessModes: [&quot;ReadWriteOnce&quot;]<br>  storageClassName: local-path<br>  resources: { requests: { storage: 40Gi } }<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata: { name: whisper-v3, namespace: llm, labels: { app: whisper-v3 } }<br>spec:<br>  replicas: 1<br>  selector: { matchLabels: { app: whisper-v3 } }<br>  template:<br>    metadata: { labels: { app: whisper-v3 } }<br>    spec:<br>      securityContext: { supplementalGroups: [44, 109] }<br>      volumes:<br>        - { name: cache, persistentVolumeClaim: { claimName: cache-whisper-v3 } }<br>        - { name: shm, emptyDir: { medium: Memory, sizeLimit: 8Gi } }<br>      containers:<br>        - name: vllm<br>          image: rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909<br>          command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;openai/whisper-large-v3&quot;,<br>                    &quot;--tensor-parallel-size=1&quot;,&quot;--dtype=bfloat16&quot;,<br>                    &quot;--max-model-len=448&quot;,&quot;--gpu-memory-utilization=0.85&quot;,<br>                    &quot;--enforce-eager&quot;,&quot;--port=8000&quot;]<br>          securityContext:<br>            capabilities: { add: [&quot;SYS_PTRACE&quot;] }<br>            seccompProfile: { type: Unconfined }<br>          env:<br>            - { name: VLLM_USE_AITER, value: &quot;0&quot; }<br>            - { name: VLLM_USE_TRITON_FLASH_ATTN, value: &quot;1&quot; }<br>            - { name: HUGGING_FACE_HUB_TOKEN, valueFrom: { secretKeyRef: { name: hf-token-secret, key: token } } }<br>          ports: [ { containerPort: 8000 } ]<br>          resources: { limits: { amd.com/gpu: &quot;1&quot; } }<br>          volumeMounts:<br>            - { name: cache, mountPath: /root/.cache/huggingface }<br>            - { name: shm, mountPath: /dev/shm }<br>          startupProbe:   { httpGet: { path: /health, port: 8000 }, periodSeconds: 15, failureThreshold: 60 }<br>          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }<br>---<br>apiVersion: v1<br>kind: Service<br>metadata: { name: whisper-v3, namespace: llm }<br>spec:<br>  selector: { app: whisper-v3 }<br>  ports: [ { port: 8000, targetPort: 8000 } ]</pre><p><strong>The Whisper gotcha:</strong> a base vLLM image often lacks audio libraries. You’ll get a 500, and — confusingly — the text field may literally read &quot;Please install vllm[audio] for audio support&quot;. That&#39;s the error, not a transcription. Fix it by adding the audio extras to your image:</p><pre>RUN pip install --no-cache-dir librosa soundfile<br>RUN apt-get update &amp;&amp; apt-get install -y ffmpeg &amp;&amp; rm -rf /var/lib/apt/lists/*</pre><p>(You can pip install into the running pod to confirm the fix, but it&#39;s ephemeral — bake it into the image to make it stick.)</p><h3>The checklist I’d tape to my monitor</h3><ol><li><strong>MI210 = gfx90a. No FP8/MXFP4.</strong> Choose BF16/FP16 or INT4 checkpoints.</li><li>For MXFP4 models (gpt-oss), use a <strong>BF16 upcast</strong> and budget ~4× the VRAM.</li><li>Use a <strong>pinned</strong> rocm/vllm image; match the <strong>node ROCm driver</strong> to (or above) the image&#39;s ROCm.</li><li>Set VLLM_USE_AITER=0 and the Triton attention backend on every pod.</li><li>Match the image’s <strong>entrypoint</strong> to its arg style (vllm serve … vs --model …).</li><li>amd.com/gpu: N = N cards; size TP to fit weights in 64 GB chunks.</li><li>Embeddings → --task embed + /v1/embeddings. Whisper → /v1/audio/transcriptions + <strong>vllm[audio]</strong>.</li><li><strong>Validate before you trust</strong> — one real request per model.</li></ol><h3>A closing thought</h3><p>The biggest mindset shift was treating “it deployed” and “it works” as two different facts. A pod going 1/1 Running only means the process started. The real proof is one successful request that exercises the actual kernels on the actual silicon — because gfx90a isn&#39;t in most CI matrices, <em>you</em> are the validation step. Send the request, read the response, then trust it.</p><p>If you’re staring at an MI210 box wondering whether modern LLMs will run on it: they will. Route around FP8/MXFP4, pin a good image, and the rest is ordinary Kubernetes.</p><p><em>Happy serving.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a0533e9a1ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Kimi K2.5 on H200 GPUs: The Real Story Nobody Tells You]]></title>
            <link>https://medium.com/@shivank1128/deploying-kimi-k2-5-on-h200-gpus-the-real-story-nobody-tells-you-7a18a6ca905a?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a18a6ca905a</guid>
            <category><![CDATA[kimi-k2]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[ai-inferencing]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Thu, 19 Feb 2026 10:08:35 GMT</pubDate>
            <atom:updated>2026-02-19T10:08:35.642Z</atom:updated>
            <content:encoded><![CDATA[<p>I spent the last few days deploying Moonshot AI’s <strong>Kimi K2.5</strong> — a 1 trillion parameter Mixture-of-Experts model — on NVIDIA H200 GPUs using vLLM and llama.cpp. What was supposed to be a straightforward deployment turned into a deep dive through CUDA driver bugs, quantization rabbit holes, and some surprisingly tricky GPU memory math.</p><p>If you’re planning to self-host Kimi K2.5 (or any large MoE model), this is the guide I wish I had before I started.</p><h3>What is Kimi K2.5 and Why Should You Care?</h3><p>Kimi K2.5 is Moonshot AI’s latest multimodal model. On paper, it’s massive — 1 trillion parameters. But it’s a Mixture-of-Experts (MoE) model, which means not all parameters are active at once. It has 384 routed experts per layer with only 8 selected per token, plus a shared expert. So while it has 1T total parameters, the active parameter count per token is much smaller.</p><p>Here’s what makes it interesting for deployment:</p><ul><li><strong>256K context window</strong> — great for long document processing</li><li><strong>Built-in tool calling</strong> — natively supports function calling with kimi_k2 parser</li><li><strong>Reasoning capabilities</strong> — has a dedicated reasoning parser</li><li><strong>Multimodal</strong> — supports vision tasks via MoonViT (400M vision encoder)</li><li><strong>Pre-quantized</strong> — the official weights are already INT4 quantized (more on this later, because it’s not what you think)</li></ul><h3>The Quantization Lie: “Itss INT4” is Only Half the Story</h3><p>Before we get into deployment, let me save you the headache I went through.</p><p>When you look at the <a href="https://huggingface.co/moonshotai/Kimi-K2.5">official Kimi K2.5 model card</a>, it says the model uses compressed-tensors with INT4 quantization. You&#39;d naturally think: &quot;Great, INT4 means roughly 0.5 bytes per parameter, so a 1T model should be around 500GB, maybe less.&quot;</p><p><strong>Wrong.</strong></p><p>If you dig into the config.json, you&#39;ll find this in the quantization_config:</p><pre>{<br>  &quot;ignore&quot;: [<br>    &quot;lm_head&quot;,<br>    &quot;re:.*self_attn.*&quot;,<br>    &quot;re:.*shared_experts.*&quot;,<br>    &quot;re:.*mlp\\.(gate|up|gate_up|down)_proj.*&quot;<br>  ]<br>}</pre><p>That ignore list is doing a lot of heavy lifting. It means the following layers are <strong>not quantized at all</strong> and remain in full BF16 (2 bytes per parameter):</p><ul><li><strong>All attention layers</strong> (the entire MLA mechanism across all 61 layers)</li><li><strong>All shared expert layers</strong> (one shared expert per MoE layer)</li><li><strong>The dense MLP in layer 0</strong> (the first layer is fully dense, not MoE)</li><li><strong>The language model head</strong> (lm_head)</li><li><strong>Embeddings</strong></li></ul><p>Only the <strong>MoE routed expert weights</strong> are INT4 quantized. Everything else is BF16.</p><p>This matters because the attention layers alone account for roughly <strong>120GB</strong> of VRAM. Add shared experts, the dense layer, embeddings, and lm_head, and you’re looking at <strong>130–140GB of BF16 weights</strong> on top of the INT4 expert weights.</p><p>The actual memory breakdown looks something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*_so4zOKICH1C0p64x70aiQ.png" /></figure><p>That’s 549GB just for weights. No KV cache, no activations, no CUDA overhead. Just weights.</p><h3>First Hurdle: The CUDA 13.0 Driver Incompatibility (Error 803)</h3><p>Before I even got to worry about model size, I hit something unexpected. My cluster nodes run CUDA 13.0 drivers (fairly recent). When the vLLM container (vllm/vllm-openai:latest, which was v0.15.1 at the time) tried to start, I got:</p><pre>RuntimeError: CUDA error: system not yet initialized<br>CUDA kernel errors might be asynchronously reported at some other API call,<br>so the stacktrace below might be incorrect.</pre><p>The actual error code was <strong>803</strong> — cudaErrorSystemNotYet. Cryptic, right?</p><p>after some digging through GitHub issues before finding the root cause. vLLM 0.15.x’s Dockerfile includes this line:</p><pre>RUN ldconfig /usr/local/cuda-12.9/compat/</pre><p>This bakes CUDA 12.9 compatibility libraries into the container’s dynamic linker cache. When the container runs on a host with CUDA 13.0 drivers, these stale compatibility libraries conflict with the host driver. The container tries to use CUDA 12.9 compat libs instead of the host’s 13.0 driver, and everything falls apart.</p><h3>The Fix</h3><p>I tried the obvious thing first — setting LD_LIBRARY_PATH to prioritize the host driver:</p><pre>env:<br>  - name: LD_LIBRARY_PATH<br>    value: &quot;/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib64&quot;</pre><p><strong>Didn’t work.</strong> vLLM forks worker processes, and those workers use the ldconfig cache, not LD_LIBRARY_PATH. I was stuck for a while on this one.</p><p>The actual solution ended up being dumb but effective: <strong>mount an empty directory over the problematic path</strong> to hide the broken compat libs entirely.</p><pre>volumes:<br>  - name: cuda-compat-fix<br>    emptyDir: {}</pre><pre>volumeMounts:<br>  - name: cuda-compat-fix<br>    mountPath: /usr/local/cuda-12.9/compat</pre><p>This creates an empty overlay at /usr/local/cuda-12.9/compat, so when ldconfig looks there, it finds nothing and falls back to the host&#39;s CUDA 13.0 driver. Problem solved.</p><p>This is a <a href="https://github.com/vllm-project/vllm/issues/32373">known issue</a> and should be fixed in future vLLM releases, but if you’re running vLLM 0.15.x on CUDA 13.0 hosts today, this is the workaround you need.</p><h3>Attempt 1: “Let’s Try 4x H200, That Should Be Enough”</h3><p>With the CUDA issue fixed, it was time to actually load the model. Each H200 has 141GB of HBM3e. Four of them give you 564GB total. The model weighs ~549GB. Simple math says it should fit, right?</p><p>I set up a Kubernetes deployment with vLLM, tensor parallelism of 4, and hit deploy.</p><pre># Deployment config (simplified)<br>containers:<br>  - name: vllm<br>    image: vllm/vllm-openai:latest<br>    args:<br>      - &quot;--model&quot;<br>      - &quot;moonshotai/Kimi-K2.5&quot;<br>      - &quot;--tensor-parallel-size&quot;<br>      - &quot;4&quot;<br>      - &quot;--trust-remote-code&quot;<br>    resources:<br>      requests:<br>        nvidia.com/gpu: 4</pre><p>The model started loading. Weights were distributing across the 4 GPUs. Then:</p><pre>torch.OutOfMemoryError: CUDA out of memory.<br>Tried to allocate 2.62 GiB. GPU 0 has a total capacity of 140.65 GiB,<br>of which 137.54 GiB is already allocated.</pre><p>137GB allocated out of 140GB — and it still needed more. There was essentially <strong>zero room left for KV cache</strong>. With tensor parallelism of 4, each GPU was holding ~137GB of weights, leaving about 3GB free. You need KV cache for even a single request, and with MLA attention and 256K context, the KV cache isn’t small.</p><p>The lesson: <strong>549GB of weights / 4 GPUs = ~137GB per GPU</strong>, which leaves just 3GB per GPU for everything else. That’s not going to work.</p><h3>Attempt 2: 8x H200 with vLLM (The Production Setup)</h3><p>With 8x H200 GPUs, you get 1,128GB of total VRAM. The model weighs ~549GB, leaving around <strong>579GB free for KV cache, activations, and CUDA overhead</strong>. That’s plenty.</p><p>Here’s the final working configuration:</p><pre>containers:<br>  - name: vllm<br>    image: vllm/vllm-openai:latest<br>    args:<br>      - &quot;--model&quot;<br>      - &quot;moonshotai/Kimi-K2.5&quot;<br>      - &quot;--tensor-parallel-size&quot;<br>      - &quot;8&quot;<br>      - &quot;--trust-remote-code&quot;<br>      - &quot;--tool-call-parser&quot;<br>      - &quot;kimi_k2&quot;<br>      - &quot;--reasoning-parser&quot;<br>      - &quot;kimi_k2&quot;<br>      - &quot;--mm-encoder-tp-mode&quot;<br>      - &quot;data&quot;<br>      - &quot;--enable-prefix-caching&quot;<br>      - &quot;--kv-cache-dtype&quot;<br>      - &quot;fp8&quot;<br>    resources:<br>      requests:<br>        cpu: &quot;48&quot;<br>        memory: &quot;256Gi&quot;<br>        nvidia.com/gpu: 8</pre><pre>    # The CUDA 13.0 compat fix<br>    volumeMounts:<br>      - name: cuda-compat-fix<br>        mountPath: /usr/local/cuda-12.9/compat</pre><pre>  volumes:<br>    - name: cuda-compat-fix<br>      emptyDir: {}</pre><p>A few notes on the flags:</p><ul><li><strong>--kv-cache-dtype fp8</strong>: This quantizes the KV cache to FP8, cutting cache memory usage in half. With 579GB free, we have headroom, but FP8 KV cache means more concurrent requests and longer context windows.</li><li><strong>--enable-prefix-caching</strong>: Caches common prompt prefixes across requests. Huge win if you&#39;re serving system prompts repeatedly.</li><li><strong>--tool-call-parser kimi_k2</strong> and <strong>--reasoning-parser kimi_k2</strong>: Enable native tool calling and chain-of-thought reasoning parsing. Kimi K2.5 has built-in support for both.</li><li><strong>--mm-encoder-tp-mode data</strong>: Distributes the vision encoder&#39;s data across tensor parallel ranks instead of replicating it.</li></ul><p>The model loaded successfully, and we had a working OpenAI-compatible API:</p><pre>curl http://&lt;service-ip&gt;:8000/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;moonshotai/Kimi-K2.5&quot;,<br>    &quot;messages&quot;: [<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Explain quantum computing in simple terms&quot;}<br>    ],<br>    &quot;max_tokens&quot;: 512<br>  }&#39;</pre><p>Response times were solid — production-grade latency with vLLM’s PagedAttention and continuous batching doing their thing. It honestly felt like a weight off my shoulders after all the debugging.</p><h3>Storage</h3><p>The model weights are around 380GB on disk. I used a PersistentVolumeClaim to cache them so subsequent pod restarts don’t re-download:</p><pre>volumeClaimTemplates:<br>  - metadata:<br>      name: model-storage<br>    spec:<br>      accessModes: [&quot;ReadWriteOnce&quot;]<br>      storageClassName: local-path<br>      resources:<br>        requests:<br>          storage: 400Gi</pre><p>The HuggingFace token was stored in a Kubernetes Secret and injected as an environment variable:</p><pre>env:<br>  - name: HF_TOKEN<br>    valueFrom:<br>      secretKeyRef:<br>        name: model-secrets<br>        key: hf_token</pre><h3>The llama.cpp Detour: Deploying the GGUF Version</h3><p>While waiting for the 8-GPU node to become available, I tried a different approach — running the <a href="https://huggingface.co/unsloth/Kimi-K2.5-GGUF">unsloth/Kimi-K2.5-GGUF</a> variant via llama.cpp. Specifically, the UD-Q2_K_XL quantization, which is a dynamic 2-bit quant that should fit on fewer GPUs.</p><p>The setup used ghcr.io/ggml-org/llama.cpp:server-cuda as the container image, with an init container to download the GGUF files:</p><pre>initContainers:<br>  - name: download-model<br>    image: python:3.11-slim<br>    command: [&quot;sh&quot;, &quot;-c&quot;]<br>    args:<br>      - |<br>        pip install huggingface-hub &amp;&amp;<br>        python -c &quot;<br>        from huggingface_hub import snapshot_download<br>        snapshot_download(<br>          &#39;unsloth/Kimi-K2.5-GGUF&#39;,<br>          allow_patterns=[&#39;*UD-Q2_K_XL*&#39;],<br>          local_dir=&#39;/data&#39;<br>        )<br>        &quot;</pre><p>The main container ran llama-server with offloading all layers to GPU:</p><pre>containers:<br>  - name: llamacpp<br>    image: ghcr.io/ggml-org/llama.cpp:server-cuda<br>    command: [&quot;/app/llama-server&quot;]<br>    args:<br>      - &quot;--model&quot;<br>      - &quot;/data/UD-Q2_K_XL/Kimi-K2.5-UD-Q2_K_XL-00001-of-00008.gguf&quot;<br>      - &quot;--n-gpu-layers&quot;<br>      - &quot;-1&quot;<br>      - &quot;--ctx-size&quot;<br>      - &quot;98304&quot;<br>      - &quot;--parallel&quot;<br>      - &quot;8&quot;<br>      - &quot;--flash-attn&quot;<br>      - &quot;on&quot;<br>      - &quot;--cont-batching&quot;<br>      - &quot;--metrics&quot;<br>      - &quot;--jinja&quot;<br>      - &quot;--override-tensor&quot;<br>      - &quot;.ffn_.*_exps.=CPU&quot;<br>    resources:<br>      requests:<br>        nvidia.com/gpu: 4</pre><p>This worked. The model loaded on 4x H200, with expert FFN layers offloaded to CPU via --override-tensor .ffn_.*_exps.=CPU. The server started, accepted requests on port 8080 (llama.cpp&#39;s default), and responded correctly.</p><p><strong>But it was slow.</strong> Painfully slow.</p><p>The Q2_K_XL quantization is aggressive — you’re cramming a 1T model into ~2 bits per weight. The quality takes a noticeable hit, and with expert offloading to CPU, every token generation involves shuffling data between GPU and system memory. I’m talking response times measured in minutes for anything beyond a couple sentences. For a model with 384 experts per layer, that’s a LOT of CPU-GPU traffic on every single token.</p><p><strong>Verdict: useful for testing and validation, not for production.</strong> If you just need to verify the model works or do some lightweight experiments, llama.cpp with GGUF is a valid path. But for serving real traffic, you need vLLM with enough GPUs.</p><h3>“Can We Squeeze It Into Fewer GPUs?” — The Quantization Deep Dive</h3><p>After seeing the OOM on 4 GPUs, my first thought was: can we quantize the BF16 layers to INT8? Keep the expert weights at INT4 (they’re already there), but compress the attention, shared experts, and other BF16 layers to INT8. That would roughly look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*iqhg71WadyMe8MYSLm4hTQ.png" /></figure><p>475GB / 4 = ~119GB per GPU, leaving ~22GB per GPU for KV cache. Tight, but potentially workable.</p><h3>The Bad News: You Can’t Do This at Runtime</h3><p>vLLM doesn’t support mixed or layered quantization at inference time. The --quantization flag selects a single method, and since the model already declares compressed-tensors as its quant method, you can&#39;t stack bitsandbytes or fp8 on top. There is no --quantize-bf16-layers int8 flag.</p><p>The compressed-tensors format itself <strong>does</strong> support per-layer quantization configs (multiple config_groups with different bit-widths), but that requires creating a new checkpoint offline using <a href="https://github.com/vllm-project/llm-compressor">llm-compressor</a>. Nobody has published a mixed INT4/INT8 variant for Kimi K2.5 yet.</p><h3>What About NVIDIA’s NVFP4?</h3><p>NVIDIA has published <a href="https://huggingface.co/nvidia/Kimi-K2.5-NVFP4">nvidia/Kimi-K2.5-NVFP4</a> — a variant that uses their NVFP4 (4-bit floating point) format. NVFP4 is a microscaled FP4 data type that uses per-block scaling factors, and it achieves much better accuracy than naive INT4 at the same bit width.</p><p>The NVFP4 variant runs on just <strong>4 GPUs with tensor parallelism of 4</strong>, which is exactly what we wanted. It’s deployed with vLLM:</p><pre>python3 -m vllm.entrypoints.openai.api_server \<br>  --model nvidia/Kimi-K2.5-NVFP4 \<br>  --tensor-parallel-size 4 \<br>  --tool-call-parser kimi_k2 \<br>  --reasoning-parser kimi_k2 \<br>  --trust-remote-code</pre><p><strong>The catch?</strong> NVFP4 requires <strong>NVIDIA Blackwell GPUs</strong> (compute capability 10.0 and above). That means B200, GB200, or newer. The Blackwell architecture’s 5th-generation Tensor Cores have native FP4 support — they can execute mixed-precision operations with FP4 weights and FP16 accumulation directly in hardware.</p><p>H200 GPUs are Hopper architecture (compute capability 9.0). They simply don’t have the FP4 tensor core hardware. So if you’re on H200s like me, NVFP4 is not an option today — which was frustrating to discover after getting excited about the 4-GPU possibility. But if you’re planning new infrastructure or have access to Blackwell nodes, the NVFP4 variant is the most GPU-efficient way to deploy Kimi K2.5 — half the GPUs, same model quality.</p><h3>Other Pre-Quantized Variants Worth Knowing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*7PX9p6AQ5Z8qQux0Ku8GHQ.png" /></figure><h3>Probes and Health Checks: Don’t Skip These</h3><p>One thing I want to call out — if you’re deploying any large model on Kubernetes, your probe configuration matters a lot. A 1T model doesn’t load in 30 seconds. Here’s what worked:</p><pre>startupProbe:<br>  httpGet:<br>    path: /health<br>    port: 8000<br>  initialDelaySeconds: 60<br>  periodSeconds: 30<br>  failureThreshold: 60    # 60 x 30s = 30 minutes to start</pre><pre>livenessProbe:<br>  httpGet:<br>    path: /health<br>    port: 8000<br>  initialDelaySeconds: 30<br>  periodSeconds: 30<br>  failureThreshold: 3</pre><pre>readinessProbe:<br>  httpGet:<br>    path: /health<br>    port: 8000<br>  initialDelaySeconds: 30<br>  periodSeconds: 10<br>  failureThreshold: 3</pre><p>The key is the <strong>startup probe</strong> with a high failureThreshold. The model download alone can take 20+ minutes on a decent connection, and then loading 380GB of weights into GPU memory takes another several minutes. I set failureThreshold: 60 with periodSeconds: 30, giving the pod up to 30 minutes to become healthy before Kubernetes kills it.</p><h3>Lessons Learned</h3><ol><li><strong>“INT4 quantized” doesn’t mean the whole model is INT4.</strong> Always check the ignore list in the quantization config. For Kimi K2.5, the attention layers, shared experts, and several other components are full BF16.</li><li><strong>Do the memory math before requesting GPUs.</strong> The formula is: total_weight_memory / num_gpus + kv_cache + cuda_overhead. If that exceeds your per-GPU VRAM, you&#39;ll OOM.</li><li><strong>vLLM + CUDA driver version mismatches are real.</strong> If you’re on CUDA 13.0 hosts with vLLM 0.15.x containers, you’ll hit Error 803. The emptyDir volume mount trick works today; check if newer vLLM releases have fixed this.</li><li><strong>llama.cpp is great for experimentation, not production.</strong> The GGUF format and CPU offloading give you flexibility, but the performance cost of aggressive quantization and CPU-GPU data movement is significant.</li><li><strong>NVFP4 is the future for dense deployment</strong>, but you need Blackwell GPUs. If you’re budgeting for new hardware and want to run Kimi K2.5 on 4 GPUs, B200s with NVFP4 is the play.</li><li><strong>Use </strong><strong>--kv-cache-dtype fp8</strong> when you can. It halves KV cache memory usage with minimal quality impact, and the extra headroom lets you serve longer contexts or more concurrent requests.</li></ol><h3>Final Architecture</h3><pre>                    ┌─────────────────────┐<br>                    │   Kubernetes Service │<br>                    │    (ClusterIP:8000)  │<br>                    └──────────┬──────────┘<br>                               │<br>                    ┌──────────▼──────────┐<br>                    │   vLLM Deployment    │<br>                    │                      │<br>                    │  Image: vllm-openai  │<br>                    │  Model: Kimi-K2.5    │<br>                    │  TP: 8               │<br>                    │  GPUs: 8x H200       │<br>                    │                      │<br>                    │  Volumes:            │<br>                    │  - PVC (400Gi)       │<br>                    │  - emptyDir (compat) │<br>                    │                      │<br>                    │  Env:                │<br>                    │  - HF_TOKEN (Secret) │<br>                    └──────────┬──────────┘<br>                               │<br>                    ┌──────────▼──────────┐<br>                    │   8x NVIDIA H200     │<br>                    │   141GB HBM3e each   │<br>                    │   Total: 1,128GB     │<br>                    │   Used: ~549GB       │<br>                    │   Free: ~579GB       │<br>                    └─────────────────────┘</pre><p>The model is live, serving requests, and handling tool calls and reasoning natively. Was it straightforward? No. Was it worth it? Absolutely. Kimi K2.5 is genuinely impressive — the quality you get from a self-hosted 1T MoE model with native tool calling, reasoning, and 256K context is hard to beat.</p><p>If you’re running into similar issues or have found better solutions, I’d love to hear about it. Drop a comment or reach out.</p><p><em>Deployed on bare-metal Kubernetes with NVIDIA H200 GPUs. All configurations tested on Ubuntu with CUDA 13.0 drivers and vLLM v0.15.1.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a18a6ca905a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking LLModels with vLLM: A Comprehensive Guide]]></title>
            <link>https://medium.com/@shivank1128/benchmarking-llmodels-with-vllm-a-comprehensive-guide-dd51e7d1b9cd?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd51e7d1b9cd</guid>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[benchmarking]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Tue, 09 Dec 2025 17:50:20 GMT</pubDate>
            <atom:updated>2025-12-09T17:50:20.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding LLM Inference Performance at Scale</h3><p>As large language models (LLMs) become increasingly central to production applications, understanding their inference performance characteristics is crucial. Whether you’re building a chatbot, a code assistant, or any AI-powered service, knowing how your model performs under different loads can make or break your user experience.</p><p>In this comprehensive guide, I’ll walk you through setting up a professional benchmarking environment using vLLM and share real-world performance data from testing four different models ranging from 20B to 120B parameters.</p><h3>Why Benchmark LLM Inference?</h3><p>Before diving into the technical details, let’s understand what we’re measuring and why it matters:</p><h3>Key Performance Metrics</h3><p><strong>1. Time to First Token (TTFT)</strong></p><ul><li>How quickly users see the first response</li><li>Critical for perceived responsiveness</li><li>Typically ranges from 500ms to 2000ms in production</li></ul><p><strong>2. Time Per Output Token (TPOT)</strong></p><ul><li>Average time to generate each subsequent token</li><li>Affects the overall streaming experience</li><li>Lower is better for smooth text generation</li></ul><p><strong>3. Inter-Token Latency (ITL)</strong></p><ul><li>Time between consecutive tokens</li><li>Determines the “smoothness” of streaming</li><li>Inconsistent ITL creates a choppy user experience</li></ul><p><strong>4. End-to-End Latency (E2EL)</strong></p><ul><li>Total time from request to complete response</li><li>Includes TTFT + total generation time</li><li>The ultimate user-facing metric</li></ul><p><strong>5. Throughput</strong></p><ul><li>Tokens generated per second</li><li>Indicates system capacity under load</li><li>Essential for cost and scaling decisions</li></ul><h3>Setting Up Your Benchmarking Environment</h3><h3>Prerequisites</h3><pre># System Requirements<br>- Linux/macOS environment<br>- Python 3.8+<br>- GPU with sufficient VRAM (recommended for large models)<br>- Running vLLM server instance</pre><h3>Step 1: Clone the Repository</h3><pre>git clone https://github.com/Alt-Shivam/vllm-benchmarking-studio.git<br>cd vllm-benchmarking-studio</pre><h3>Step 2: Install Dependencies</h3><pre>pip install -r requirements.txt</pre><p>The requirements.txt includes:</p><ul><li>vllm - High-performance inference engine</li><li>transformers - Model handling</li><li>aiohttp - Async HTTP requests</li><li>pandas - Data analysis</li><li>datasets - Dataset management</li></ul><h3>Step 3: Configure Your Benchmark</h3><p>Edit combos.yaml to define your test parameters:</p><pre># Model Configuration<br>model: &quot;your-model-name&quot;<br>base_url: &quot;http://your-server:port&quot;<br>tokenizer: &quot;/path/to/tokenizer&quot;</pre><pre># Input/Output Token Combinations<br>input_output:<br>  - [256, 256]    # Short prompts<br>  - [2048, 2048]  # Long prompts</pre><pre># Concurrency Levels<br>concurrency_prompts:<br>  - [1, 10]       # Serial processing<br>  - [4, 40]       # Low concurrency<br>  - [8, 80]       # Medium concurrency<br>  - [16, 160]     # High concurrency<br>  - [32, 320]     # Maximum concurrency</pre><h3>Step 4: Run the Benchmark Suite</h3><pre># Execute all test combinations<br>python3 run_sweep.py</pre><pre># This will generate results in the results/ directory</pre><h3>Step 5: Aggregate and Visualize</h3><pre># Consolidate all results into a single CSV<br>python3 aggregate_result.py</pre><pre># Generate performance charts<br>python3 visualize.py</pre><h3>Real-World Benchmark Results</h3><p>I tested four models under identical conditions to understand how model size affects performance. Here’s what I discovered:</p><h3>Test Configuration</h3><ul><li><strong>Dataset</strong>: Random synthetic prompts</li><li><strong>Input Lengths</strong>: 256 and 2048 tokens</li><li><strong>Output Lengths</strong>: 256 and 2048 tokens</li><li><strong>Concurrency Levels</strong>: 1, 4, 8, 16, 32</li><li><strong>Request Pattern</strong>: Burst mode (all requests sent simultaneously)</li></ul><h3>Model Performance Analysis</h3><h3>1. Qwen/Qwen3-Omni-30B-A3B-Instruct</h3><p><strong>Model Size</strong>: 30 billion parameters<br> <strong>Architecture</strong>: Multimodal (Omni) transformer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*FLdf6TCRHF1_1O8c5o-hDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*QlIlRioaW3-Uq7uYwPCqtQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*pwnBmOIGV_YLoOHAAX9EFg.png" /></figure><h4>Key Findings</h4><p><strong>Short Prompts (256×256 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 893ms (median: 535ms)</li><li>TPOT: 7.3ms</li><li>Throughput: 182 tokens/s</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 785ms (median: 705ms)</li><li>TPOT: 53ms</li><li>Throughput: 1,870 tokens/s</li><li><strong>10× throughput improvement</strong> with acceptable latency trade-off</li></ul><p><strong>Long Prompts (2048×2048 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 810ms (median: 828ms)</li><li>TPOT: 6.8ms</li><li>Throughput: 278 tokens/s</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 1,101ms (median: 798ms)</li><li>TPOT: 49ms</li><li>Throughput: 2,632 tokens/s</li><li>P99 latency: 76s (note: some requests experience significant queuing)</li></ul><h4>Verdict</h4><p>✅ <strong>Solid performance</strong> for a 30B parameter model<br> ✅ <strong>Linear throughput scaling</strong> with concurrency<br> ⚠️ <strong>P99 latencies spike</strong> at maximum concurrency (acceptable for batch workloads)</p><h3>2. OpenAI/GPT-OSS-20B</h3><p><strong>Model Size</strong>: 20 billion parameters<br> <strong>Architecture</strong>: GPT-style decoder-only transformer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bFXZtMRlg4rBXBhnPt2CaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*cL5duf5nKLfV4HC4FYNFZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*coygyjocWfoOjz1sZCSKhg.png" /></figure><h4>Key Findings</h4><p><strong>Short Prompts (256×256 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 562ms (median: 447ms)</li><li>TPOT: 5.2ms</li><li>Throughput: 264 tokens/s</li></ul><p><strong>30% faster TTFT</strong> than Qwen 30B</p><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 716ms (median: 654ms)</li><li>TPOT: 19ms</li><li>Throughput: 2,751 tokens/s</li></ul><p><strong>Long Prompts (2048×2048 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 563ms (median: 584ms)</li><li>TPOT: 4.5ms</li><li>Throughput: 235 tokens/s</li><li><strong>Fastest TPOT</strong> among all tested models</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 1,148ms (median: 878ms)</li><li>TPOT: 16ms</li><li>Throughput: 3,605 tokens/s</li></ul><h4>Verdict</h4><p>✅ <strong>Excellent low-latency performance</strong><br> ✅ <strong>Best-in-class throughput</strong> at high concurrency<br> ✅ <strong>Smaller model size</strong> = lower operational costs<br> ⚠️ <strong>P99 latencies</strong> still high at extreme concurrency (103s)</p><h3>3. OpenAI/GPT-OSS-120B</h3><p><strong>Model Size</strong>: 120 billion parameters<br> <strong>Architecture</strong>: GPT-style decoder-only transformer (larger variant)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*WCFUSUeqlP63k4nSejkm4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bd-wfis2HUVlUGuOuKBcPg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Bwu5iAI3BgwXIxEkeGMw5g.png" /></figure><h4>Key Findings</h4><p><strong>Short Prompts (256×256 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 642ms (median: 474ms)</li><li>TPOT: 5.7ms</li><li>Throughput: 234 tokens/s</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 680ms (median: 606ms)</li><li>TPOT: 23ms</li><li>Throughput: 2,403 tokens/s</li></ul><p><strong>Long Prompts (2048×2048 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 999ms (median: 1,113ms)</li><li>TPOT: 4.6ms</li><li>Throughput: 329 tokens/s</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 925ms (median: 683ms)</li><li>TPOT: 14ms</li><li>Throughput: 4,076 tokens/s</li></ul><h4>Verdict</h4><p>✅ <strong>Competitive performance</strong> despite 6× model size vs 20B<br> ✅ <strong>Strong throughput</strong> at high concurrency<br> ⚠️ <strong>Higher TTFT variance</strong> under load<br> ⚠️ <strong>P99 latencies</strong> reach 88s at maximum concurrency</p><h3>4. Google/Gemma-3–27B-IT</h3><p><strong>Model Size</strong>: 27 billion parameters<br> <strong>Architecture</strong>: Gemma instruction-tuned transformer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ax2MTuIBREwrjz65VEtSUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*j3BJQXt7nEVL5Hc2TeuQoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*4gTm6Sijm4xRv8wqmI4ouQ.png" /></figure><h4>Key Findings</h4><p><strong>Short Prompts (256×256 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 707ms (median: 438ms)</li><li>TPOT: 16.7ms</li><li>Throughput: 103 tokens/s</li></ul><p><strong>Higher TPOT</strong> than other models</p><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 769ms (median: 702ms)</li><li>TPOT: 25ms</li><li>Throughput: 2,173 tokens/s</li></ul><p><strong>Long Prompts (2048×2048 tokens)</strong></p><p><strong>Best Case (Concurrency=1)</strong></p><ul><li>TTFT: 874ms (median: 733ms)</li><li>TPOT: 17.2ms</li><li>Throughput: 114 tokens/s</li></ul><p><strong>High Load (Concurrency=32)</strong></p><ul><li>TTFT: 1,803ms (median: 1,442ms)</li><li>TPOT: 26ms</li><li>Throughput: 2,323 tokens/s</li></ul><h4>Verdict</h4><p>✅ <strong>Most consistent latency distribution</strong> (lowest std deviation)<br> ✅ <strong>Stable performance</strong> across concurrency levels<br> ⚠️ <strong>Lower raw throughput</strong> than GPT models<br> ✅ <strong>Excellent for latency-sensitive applications</strong></p><h3>📊 Comparative Analysis</h3><h3>TTFT Comparison (Short Prompts, Concurrency=1)</h3><p>Model Mean TTFT Median TTFT Winner GPT-OSS-20B 562ms 447ms 🥇 Gemma-3–27B 707ms 438ms 🥈 GPT-OSS-120B 642ms 474ms 🥉 Qwen-30B 893ms 535ms -</p><p><strong>Insight</strong>: Smaller models (20B) provide the fastest initial response time.</p><h3>Throughput Comparison (Long Prompts, Concurrency=32)</h3><p>Model Throughput (tok/s) Winner GPT-OSS-120B 4,076 🥇 GPT-OSS-20B 3,605 🥈 Qwen-30B 2,632 🥉 Gemma-3–27B 2,323 -</p><p><strong>Insight</strong>: The 120B model achieves highest throughput despite its size, showcasing excellent optimization.</p><h3>Latency Consistency (Standard Deviation)</h3><p>Model TTFT Std Dev Winner Gemma-3–27B Lowest 🥇 GPT-OSS-20B Low-Medium 🥈 Qwen-30B Medium 🥉 GPT-OSS-120B Highest -</p><p><strong>Insight</strong>: Gemma-3–27B offers the most predictable latency for production workloads.</p><h3>🎯 Key Takeaways &amp; Recommendations</h3><h3>1. Choose Based on Your Use Case</h3><p><strong>For Real-Time Chat Applications</strong></p><ul><li>Use: <strong>GPT-OSS-20B</strong> or <strong>Gemma-3–27B</strong></li><li>Why: Low TTFT (&lt; 600ms) provides responsive user experience</li><li>Concurrency: Keep at 4–8 for optimal latency</li></ul><p><strong>For Batch Processing</strong></p><ul><li>Use: <strong>GPT-OSS-120B</strong> or <strong>Qwen-30B</strong></li><li>Why: Highest throughput at scale</li><li>Concurrency: Maximize at 32 for best token/s</li></ul><p><strong>For Latency-Critical Systems</strong></p><ul><li>Use: <strong>Gemma-3–27B</strong></li><li>Why: Most consistent performance (lowest variance)</li><li>Trade-off: Slightly lower throughput</li></ul><h3>2. Concurrency vs. Latency Trade-off</h3><pre>Concurrency 1-4:   Best latency, lower throughput<br>Concurrency 8-16:  Balanced performance<br>Concurrency 32+:   Maximum throughput, higher tail latencies</pre><h3>3. The P99 Problem</h3><p>All models show significant P99 latency spikes at high concurrency:</p><ul><li>Qwen-30B: 76s</li><li>GPT-OSS-20B: 103s</li><li>GPT-OSS-120B: 88s</li><li>Gemma-3–27B: 73s</li></ul><p><strong>Solution</strong>: Implement request queuing and load balancing at the application layer.</p><h3>4. Token Length Matters</h3><p>Prompt Length Impact on TTFT Impact on Throughput 256 tokens ✅ Lower ❌ Higher per-token overhead 2048 tokens ❌ Higher ✅ Better GPU utilization</p><p><strong>Recommendation</strong>: Batch similar-length requests together for optimal performance.</p><h3>Conclusion</h3><p>Benchmarking LLM inference is not just about finding the “fastest” model — it’s about understanding the trade-offs and choosing the right tool for your specific use case.</p><p><strong>Key Insights from Our Testing</strong>:</p><ol><li><strong>GPT-OSS-20B</strong> wins on raw speed and throughput</li><li><strong>Gemma-3–27B</strong> wins on consistency and predictability</li><li><strong>GPT-OSS-120B</strong> demonstrates impressive optimization despite size</li><li><strong>Qwen-30B</strong> offers balanced performance with multimodal capabilities</li></ol><p>The benchmarking framework we’ve built is open-source and extensible. Use it to test your own models and configurations, and share your findings with the community!</p><h3>🔗 Resources</h3><ul><li><strong>GitHub Repository</strong>: <a href="https://github.com/Alt-Shivam/vllm-benchmarking-studio">vllm-benchmarking-studio</a></li><li><strong>vLLM Documentation</strong>: <a href="https://docs.vllm.ai/">docs.vllm.ai</a></li><li><strong>Benchmark Datasets</strong>: ShareGPT, MT-Bench, AIMO</li></ul><p><strong>Tags</strong>: #LLM #MachineLearning #vLLM #Benchmarking #AI #Performance #GPT #Gemma #Qwen #InferenceOptimization</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd51e7d1b9cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Production-Ready VLLM Stack on Kubernetes with HPA Autoscaling]]></title>
            <link>https://medium.com/@shivank1128/deploying-a-production-ready-vllm-stack-on-kubernetes-with-hpa-autoscaling-107501b8b687?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/107501b8b687</guid>
            <category><![CDATA[production]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Sat, 01 Nov 2025 12:38:39 GMT</pubDate>
            <atom:updated>2025-11-01T12:38:39.406Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TPaeg4ZeGxF7lZUW.png" /></figure><p>Running large language models (LLMs) in production requires careful orchestration of resources, efficient scaling mechanisms, and robust infrastructure. In this guide, I’ll walk you through deploying a multi-model VLLM (Very Large Language Model) stack on Kubernetes with Horizontal Pod Autoscaling (HPA), based on our battle-tested production configuration.</p><p>This setup has been successfully managing multiple LLMs from OpenAI, Meta, Qwen, and Google, serving production traffic with automatic scaling based on demand.</p><h3>Why VLLM?</h3><p>VLLM is a fast and memory-efficient inference engine designed specifically for LLMs. It provides:</p><ul><li>High throughput serving with PagedAttention</li><li>Continuous batching of incoming requests</li><li>Optimized CUDA kernels for better GPU utilization</li><li>OpenAI-compatible API endpoints</li></ul><h3>Architecture Overview</h3><p>Our production stack consists of three key components:</p><p><strong>Multiple Model Deployments</strong>: Each LLM runs in its own deployment with dedicated resources and configuration tailored to the model’s requirements.</p><p><strong>Router Service</strong>: A lightweight router that distributes incoming requests across model instances and handles load balancing.</p><p><strong>Autoscaling Infrastructure</strong>: HPA configuration that scales deployments based on custom Prometheus metrics, specifically monitoring the number of waiting requests.</p><h3>Prerequisites</h3><p>Before diving into the deployment, ensure you have:</p><ul><li>A Kubernetes cluster with GPU nodes</li><li>Helm 3.x installed</li><li>Prometheus operator for custom metrics (required for HPA)</li><li>A storage class that supports ReadWriteMany (RWX) access mode</li><li>Hugging Face tokens for model downloads</li></ul><h3>Installation Guide</h3><h3>Step 1: Add the Repository</h3><pre>helm repo add vllm https://vllm-project.github.io/production-stack</pre><h3>Step 2: Understanding the Configuration</h3><p>Let me break down the key configuration parameters for each model:</p><pre>- name: &quot;gptoss-20b&quot;<br>    repository: &quot;vllm/vllm-openai&quot;     # VLLM image supporting your model<br>    tag: &quot;latest&quot;                       <br>    modelURL: &quot;openai/gpt-oss-20b&quot;     # Hugging Face model identifier<br>    replicaCount: 1                     # Minimum replicas<br>    requestCPU: 10<br>    requestMemory: &quot;64Gi&quot;<br>    requestGPU: 1<br>    pvcStorage: &quot;250Gi&quot;<br>    pvcAccessMode:<br>      - ReadWriteMany                   # Critical for fast scaling!</pre><p><strong>Important Note</strong>: Using ReadWriteMany (RWX) access mode is crucial when HPA is enabled. This allows multiple pods to mount the same persistent volume simultaneously, dramatically reducing scaling time since new pods don&#39;t need to re-download the model weights.</p><h3>Step 3: VLLM Configuration Deep Dive</h3><p>Each model has specific VLLM runtime configurations:</p><pre>vllmConfig:<br>  enableChunkedPrefill: false<br>  enablePrefixCaching: false<br>  dtype: &quot;bfloat16&quot;<br>  extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.8&quot;]</pre><p><strong>Key Parameters</strong>:</p><ul><li>dtype: Using bfloat16 provides a good balance between performance and memory usage</li><li>gpu-memory-utilization: Set to 0.8-0.85 to leave headroom for CUDA operations</li><li>tensorParallelSize: For larger models requiring multiple GPUs (e.g., 120B model uses 4 GPUs)</li><li>maxModelLen: Context window size, adjust based on your use case</li></ul><h3>Step 4: Autoscaling Configuration</h3><p>Please refer to keda autoscaling guide for HPA</p><p><a href="https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html">https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html</a></p><h3>Production Model Examples</h3><h3>Small Model: Llama 3.1 8B Instruct</h3><p>Ideal for cost-effective inference with good performance:</p><pre>- name: &quot;meta-llama-31-8b-instruct&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;latest&quot;<br>    modelURL: &quot;meta-llama/Llama-3.1-8B-Instruct&quot;<br>    replicaCount: 1<br>    requestCPU: 8<br>    requestMemory: &quot;32Gi&quot;<br>    requestGPU: 1<br>    pvcStorage: &quot;100Gi&quot;<br>    vllmConfig:<br>      maxModelLen: 80000<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.85&quot;]</pre><h3>Large Model: GPT-OSS 120B</h3><p>For high-quality outputs requiring significant compute:</p><pre>- name: &quot;gptoss-120b&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;latest&quot;<br>    modelURL: &quot;openai/gpt-oss-120b&quot;<br>    replicaCount: 1<br>    requestCPU: 16<br>    requestMemory: &quot;128Gi&quot;<br>    requestGPU: 4                       # Multi-GPU setup<br>    pvcStorage: &quot;600Gi&quot;<br>    vllmConfig:<br>      tensorParallelSize: 4             # Parallel across 4 GPUs<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.85&quot;]</pre><h3>Embedding Model: Qwen3 Embedding 8B</h3><p>Perfect for semantic search and RAG applications:</p><pre>- name: &quot;qwen3-embedding-8b&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;latest&quot;<br>    modelURL: &quot;Qwen/Qwen3-Embedding-8B&quot;<br>    replicaCount: 1<br>    requestCPU: 16<br>    requestMemory: &quot;64Gi&quot;<br>    requestGPU: 1<br>    pvcStorage: &quot;100Gi&quot;<br>    vllmConfig:<br>      maxModelLen: 32768<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.85&quot;]</pre><h3>Multimodal Model: Qwen3 Omni</h3><p>Audio-visual-text multimodal model with custom CUDA image:</p><pre>- name: &quot;qwen3-omni&quot;<br>    repository: &quot;qwenllm/qwen3-omni&quot;    # Custom repository<br>    tag: &quot;3-cu124&quot;                       # CUDA 12.4 optimized<br>    modelURL: &quot;Qwen/Qwen3-Omni-30B-A3B-Instruct&quot;<br>    replicaCount: 1<br>    requestCPU: 24<br>    requestMemory: &quot;128Gi&quot;<br>    requestGPU: 1<br>    pvcStorage: &quot;100Gi&quot;<br>    vllmConfig:<br>      tensorParallelSize: 1<br>      maxModelLen: 32768                 # 32768 for 1 GPU, 65536 for 4 GPUs<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.90&quot;]</pre><h3>Reasoning Model: LLM360 K2-Think</h3><p>Specialized for complex reasoning tasks:</p><pre>- name: &quot;llm360-k2-think&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;v0.10.1.1&quot;                     # Specific version for compatibility<br>    modelURL: &quot;LLM360/K2-Think&quot;<br>    replicaCount: 1<br>    requestCPU: 16<br>    requestMemory: &quot;64Gi&quot;<br>    requestGPU: 4<br>    pvcStorage: &quot;100Gi&quot;<br>    vllmConfig:<br>      tensorParallelSize: 4<br>      maxModelLen: 65536<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.85&quot;]</pre><h3>Vision-Language Model: Qwen3 Thinking</h3><p>Large-scale multimodal reasoning model:</p><pre>- name: &quot;qwen3-thinking&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;latest&quot;<br>    modelURL: &quot;Qwen/Qwen3-VL-235B-A22B-Thinking&quot;<br>    replicaCount: 1<br>    requestCPU: 32<br>    requestMemory: &quot;256Gi&quot;<br>    requestGPU: 4                        # Requires 4 GPUs<br>    pvcStorage: &quot;900Gi&quot;                  # Large storage for weights<br>    vllmConfig:<br>      tensorParallelSize: 4<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.95&quot;]</pre><h3>Vision-Language Model: Qwen3 Instruct</h3><p>High-performance multimodal instruction-following model:</p><pre>- name: &quot;qwen3-instruct&quot;<br>    repository: &quot;vllm/vllm-openai&quot;<br>    tag: &quot;latest&quot;<br>    modelURL: &quot;Qwen/Qwen3-VL-235B-A22B-Instruct&quot;<br>    replicaCount: 1<br>    requestCPU: 32<br>    requestMemory: &quot;256Gi&quot;<br>    requestGPU: 4<br>    pvcStorage: &quot;800Gi&quot;<br>    vllmConfig:<br>      tensorParallelSize: 4<br>      dtype: &quot;bfloat16&quot;<br>      extraArgs: [&quot;--disable-log-requests&quot;, &quot;--gpu-memory-utilization&quot;, &quot;0.9</pre><h3>Router Configuration</h3><p>The router handles incoming traffic and distributes it across model instances:</p><pre>routerSpec:<br>  replicaCount: 1<br>  autoscaling:<br>    enabled: true<br>    minReplicas: 1<br>    maxReplicas: 10<br>    targetCPUUtilizationPercentage: 80<br>  resources:<br>    requests:<br>      cpu: 400m<br>      memory: 700Mi<br>    limits:<br>      memory: 700Mi</pre><p>The router is lightweight and scales independently based on CPU utilization.</p><h3>Deployment</h3><h3>Deploy the Stack</h3><pre>helm install vllm vllm -f prod-values.yaml -n vllm --install --create-namespace</pre><h3>Verify Deployment</h3><p>Check your pods:</p><pre>kubectl get pods -n vllm</pre><p>Expected output:</p><pre>NAME                                                   READY   STATUS    RESTARTS   AGE<br>vllm-deployment-router-5bc8f96685-284m4                1/1     Running   0          8d<br>vllm-gemma-3-27b-it-deployment-vllm-9d9f8b554-cdlvj    1/1     Running   2          8d<br>vllm-gptoss-120b-deployment-vllm-d75bdcc7f-zprkb       1/1     Running   6          7d<br>vllm-qwen3-omni-deployment-vllm-85bfc6dfc7-jb59f       1/1     Running   0          26h</pre><p>Check HPA status:</p><pre>kubectl get hpa -n vllm</pre><p>Expected output:</p><pre>NAME                   REFERENCE                                     TARGETS        MINPODS   MAXPODS   REPLICAS<br>vllm-gptoss-120b-hpa   Deployment/vllm-gptoss-120b-deployment-vllm   0/20           1         2         1<br>vllm-router-hpa        Deployment/vllm-deployment-router             cpu: 2%/80%    3         10        3</pre><h3>Best Practices and Lessons Learned</h3><h3>Storage Strategy</h3><p><strong>Always use ReadWriteMany (RWX) for model weights</strong>: This is perhaps the most critical optimization. When a new pod spins up during scaling, it can immediately access the pre-downloaded model weights. Without RWX, each pod would need to download 100GB+ of weights, adding 5–10 minutes to scaling time.</p><h3>GPU Memory Utilization</h3><p>Start with 0.8 (80%) GPU memory utilization and adjust based on OOM errors. Larger models can often push to 0.85, but leave some headroom.</p><h3>Scaling Behavior</h3><p>Our aggressive scale-up (0 second stabilization) combined with conservative scale-down (10 minutes) prevents request queuing during traffic spikes while avoiding unnecessary pod churn during temporary dips.</p><h3>Model Selection</h3><p>Choose your models wisely:</p><ul><li>8B models for cost-effective, high-throughput workloads</li><li>27B models for balanced quality and performance</li><li>120B+ models only when output quality justifies the cost</li></ul><h3>Resource Requests</h3><p>Set realistic CPU and memory requests. Underprovisioning leads to OOMKills; overprovisioning wastes cluster resources.</p><h3>Monitoring and Observability</h3><p>Key metrics to watch:</p><ul><li>vllm_num_requests_waiting: Primary scaling trigger</li><li>vllm_num_requests_running: Current load</li><li>GPU utilization: Should stay high but not maxed out</li><li>Time to scale: Monitor how long new pods take to become ready</li></ul><h3>Cost Optimization Tips</h3><ol><li><strong>Right-size your models</strong>: Don’t use a 120B model when an 8B model suffices</li><li><strong>Use spot instances</strong>: For non-critical workloads, GPU spot instances can save 60–80%</li><li><strong>Aggressive scale-to-zero</strong>: Consider scaling to zero replicas during off-peak hours</li><li><strong>Batch requests</strong>: VLLM’s continuous batching is most efficient with multiple concurrent requests</li></ol><h3>Conclusion</h3><p>Running a production VLLM stack requires careful attention to resource allocation, storage configuration, and autoscaling policies. The configuration shared here has proven stable and cost-effective for serving multiple models under varying load conditions.</p><p>Key takeaways:</p><ul><li>Use ReadWriteMany storage for fast scaling</li><li>Configure aggressive scale-up, conservative scale-down</li><li>Monitor the right metrics (waiting requests, not just CPU/memory)</li><li>Choose appropriate models for your use case</li><li>Leave GPU memory headroom for stability</li></ul><p>This setup has been serving production traffic reliably for weeks, automatically handling traffic spikes and optimizing resource usage during quiet periods.</p><h3>Next Steps</h3><ul><li>Implement request queuing and prioritization</li><li>Integrate with litellm for router optimization and token metering.</li><li>Add model caching for frequently accessed weights</li><li>Set up comprehensive monitoring dashboards</li><li>Implement blue-green deployments for model updates</li><li>Explore multi-cluster deployments for high availability</li></ul><p><em>Have questions or improvements? I’d love to hear about your VLLM deployment experiences in the comments!</em></p><p>Thanks for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=107501b8b687" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Model Factory]]></title>
            <link>https://medium.com/@shivank1128/llm-model-factory-c9b6c56a7b8b?source=rss-51548f5ef95------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9b6c56a7b8b</guid>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[vllm]]></category>
            <dc:creator><![CDATA[Shivank Chaudhary]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 09:45:00 GMT</pubDate>
            <atom:updated>2025-11-01T11:27:04.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKl_i62vUcw-QvYPqI3WpQ.avif" /></figure><p>This blog contains all the yaml files for vllm deployment of various models.</p><p>#NOTE: This deployment doc is mostly focused on making an understanding, not for production, for production, refer to the new published blog</p><p><strong>GPT Oss 20B</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: openai-gpt-oss-20b<br>  name: openai-gpt-oss-20b<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: openai-gpt-oss-20b<br>  template:<br>    metadata:<br>      labels:<br>        app: openai-gpt-oss-20b<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: openai-gpt-oss-20b-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;openai/gpt-oss-20b&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;80000&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: openai-gpt-oss-20b-svc<br>spec:<br>  selector:<br>    app: openai-gpt-oss-20b<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: openai-gpt-oss-20b-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 150Gi<br>  volumeMode: Filesystem</pre><p><strong>GPT Oss 120B</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: openai-gpt-oss-120b<br>  name: openai-gpt-oss-120b<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: openai-gpt-oss-120b<br>  template:<br>    metadata:<br>      labels:<br>        app: openai-gpt-oss-120b<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: openai-gpt-oss-120b-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;openai/gpt-oss-120b&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;80000&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: openai-gpt-oss-120b-svc<br>spec:<br>  selector:<br>    app: openai-gpt-oss-120b<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: openai-gpt-oss-120b-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 250Gi<br>  volumeMode: Filesystem</pre><p><strong>Gemma 3 27b</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: google-gemma-3-27b-it<br>  name: google-gemma-3-27b-it<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: google-gemma-3-27b-it<br>  template:<br>    metadata:<br>      labels:<br>        app: google-gemma-3-27b-it<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: google-gemma-3-27b-it-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;google/gemma-3-27b-it&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;80000&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        - name: HUGGING_FACE_HUB_TOKEN<br>          valueFrom:<br>              secretKeyRef:<br>                 name: hf-token<br>                 key: HUGGING_FACE_HUB_TOKEN    <br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: google-gemma-3-27b-it-svc<br>spec:<br>  selector:<br>    app: google-gemma-3-27b-it<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: google-gemma-3-27b-it-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 150Gi<br>  volumeMode: Filesystem</pre><p><strong>K2 Think llm360</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: llm360-k2-think<br>  name: llm360-k2-think<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: llm360-k2-think<br>  template:<br>    metadata:<br>      labels:<br>        app: llm360-k2-think<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: llm360-k2-think-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:v0.10.1.1<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;LLM360/K2-Think&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;65536&quot;,&quot;--tensor-parallel-size&quot;,&quot;4&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 4<br>          limits:<br>            nvidia.com/gpu: 4<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: llm360-k2-think-svc<br>spec:<br>  selector:<br>    app: llm360-k2-think<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: llm360-k2-think-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  volumeMode: Filesystem</pre><p><strong>Meta llama 3.1 8b</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: meta-llama-31-8b-instruct<br>  name: meta-llama-31-8b-instruct<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: meta-llama-31-8b-instruct<br>  template:<br>    metadata:<br>      labels:<br>        app: meta-llama-31-8b-instruct<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: meta-llama-31-8b-instruct-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;meta-llama/Llama-3.1-8B-Instruct&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;80000&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        - name: HUGGING_FACE_HUB_TOKEN<br>          valueFrom:<br>              secretKeyRef:<br>                 name: hf-token<br>                 key: HUGGING_FACE_HUB_TOKEN            <br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: meta-llama-31-8b-instruct-svc<br>spec:<br>  selector:<br>    app: meta-llama-31-8b-instruct<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: meta-llama-31-8b-instruct-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  volumeMode: Filesystem</pre><p><strong>Qwen 3 Embedding 8b</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: qwen3-embedding-8b<br>  name: qwen3-embedding-8b<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: qwen3-embedding-8b<br>  template:<br>    metadata:<br>      labels:<br>        app: qwen3-embedding-8b<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: qwen3-embedding-8b-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;Qwen/Qwen3-Embedding-8B&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;32768&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: qwen3-embedding-8b-svc<br>spec:<br>  selector:<br>    app: qwen3-embedding-8b<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: qwen3-embedding-8b-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 100Gi<br>  volumeMode: Filesystem</pre><p><strong>Qwen3 Omni 30B a3b Instruct</strong></p><p>Deployment</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: qwen3-omni-30b-a3b-instruct<br>  name: qwen3-omni-30b-a3b-instruct<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: qwen3-omni-30b-a3b-instruct<br>  template:<br>    metadata:<br>      labels:<br>        app: qwen3-omni-30b-a3b-instruct<br>    spec:<br>      hostIPC: true<br>      volumes:<br>      - name: cache-volume<br>        persistentVolumeClaim:<br>          claimName: qwen3-omni-30b-a3b-instruct-pvc<br>      - name: shm<br>        emptyDir:<br>          medium: Memory<br>          sizeLimit: &quot;32Gi&quot;<br>      containers:<br>      - name: vllm-gptoss<br>        image: vllm/vllm-openai:gptoss<br>        command: [&quot;vllm&quot;,&quot;serve&quot;,&quot;Qwen/Qwen3-Omni-30B-A3B-Instruct&quot;,&quot;--host&quot;,&quot;0.0.0.0&quot;,&quot;--port&quot;,&quot;8000&quot;,&quot;--gpu-memory-utilization&quot;,&quot;0.90&quot;,&quot;--max-model-len&quot;,&quot;80000&quot;]<br>        ports:<br>        - containerPort: 8000<br>        resources:<br>          requests:<br>            nvidia.com/gpu: 1<br>          limits:<br>            nvidia.com/gpu: 1<br>        volumeMounts:<br>        - mountPath: /root/.cache/huggingface<br>          name: cache-volume<br>        - mountPath: /dev/shm<br>          name: shm<br>        env:<br>        - name: LOG_LEVEL<br>          value: &quot;DEBUG&quot;<br>        - name: TIKTOKEN_RS_CACHE_DIR<br>          value: &quot;/vllm-workspace&quot;<br>        - name: HUGGING_FACE_HUB_TOKEN<br>          valueFrom:<br>              secretKeyRef:<br>                 name: hf-token<br>                 key: HUGGING_FACE_HUB_TOKEN<br>        securityContext:<br>          privileged: true</pre><p>Service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: qwen3-omni-30b-a3b-instruct-svc<br>spec:<br>  selector:<br>    app: qwen3-omni-30b-a3b-instruct<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 8000<br>  type: NodePort</pre><p>PVC</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: qwen3-omni-30b-a3b-instruct-pvc<br>spec:<br>  accessModes:<br>  - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 250Gi<br>  volumeMode: Filesystem</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9b6c56a7b8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>