<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srini Machavarapu on Medium]]></title>
        <description><![CDATA[Stories by Srini Machavarapu on Medium]]></description>
        <link>https://medium.com/@srini.work?source=rss-b37ecbb1632c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Srini Machavarapu on Medium</title>
            <link>https://medium.com/@srini.work?source=rss-b37ecbb1632c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:59:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srini.work/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fast Decisions, Thoughtful Fallbacks: Evaluating System One Models]]></title>
            <link>https://medium.com/@srini.work/fast-decisions-thoughtful-fallbacks-evaluating-system-one-models-3b691c6dce33?source=rss-b37ecbb1632c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b691c6dce33</guid>
            <dc:creator><![CDATA[Srini Machavarapu]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 19:07:07 GMT</pubDate>
            <atom:updated>2026-10-11T19:07:07.172Z</atom:updated>
            <content:encoded><![CDATA[<p>What Decisions API, Jev and Strands reveal about the work left for fallback.</p><p>Speed makes System One models interesting for latency-sensitive applications.</p><p>A high score alone does not guarantee a correct answer; but measured accuracy and coverage tell us how to use them.</p><p>Our <a href="https://medium.com/@srini.work/evaluating-system-one-decision-models-serving-options-for-latency-sensitive-applications-594be81e4fb3">earlier serving study</a> examined latency and throughput. This companion asks how much work can be relied upon the System in path when using a high-score rule to select which answers to use and which ones to leave for a fallback path.</p><h3>The setup, briefly</h3><p>We compared OpenAI’s hosted Decisions API (gpt-6-luna), TypeSafe AI’s hosted Jev 1.13.0, and Strands Decider 2B Hobson v21 on a RunPod RTX 4090 using two datasets — 3,200 ANLI and 805 ConTRoL test cases.</p><p>Given a passage and a statement, the task was to decide whether the statement follows the passage, contradicts the passage, or cannot be determined. We used full text, one fixed prompt and option order, and one pass with no retries. Invalid answers count as wrong.</p><h3>What does a high score buy?</h3><p>We tried two rules separately for selecting which answers to keep from this oath</p><p><strong>1. confidence ≥0.90</strong></p><p><strong>2. probability assigned to the predicted label ≥0.90</strong>.</p><p>Both include all three answer labels. Then we define <strong>Coverage</strong> as the share of all cases a rule selects and <strong>Accuracy</strong> as the share of answers that are correct, separately for selected and remaining cases. The remaining group contains the original model answers.</p><p>Note: 1. We did not test a fallback system.2. These score fields are provider-specific. The same 0.90 cutoff does not mean the same reliability across systems.</p><figure><img alt="ConTRoL, 805 cases. Values are coverage, selected accuracy, remaining accuracy. Decisions: confidence 62.24%, 77.84%, 51.64%; probability 70.19%, 77.17%, 46.25%; overall 67.95%. Jev: confidence 55.03%, 89.39%, 59.39%; probability 61.37%, 88.46%, 55.95%; overall 75.90%. Strands: confidence 0%, undefined, 59.25%; probability 1.61%, 76.92%, 58.96%; overall 59.25%. Rules applied separately at 0.90. Remaining accuracy measures original answers, not fallback performance." src="https://cdn-images-1.medium.com/max/1024/1*w5KOgc4h8F0FDeWNy6m6Mg.png" /></figure><p><em>ConTRoL · 805 cases. Each rule is applied separately. Overall accuracy includes every case.</em></p><p>On ConTRoL, Decisions illustrates the tradeoff. Its probability rule covers more cases than its confidence rule, with selected accuracy near 77% for both. <strong>A 0.90 score did not produce 90% accuracy.</strong> Roughly three to four requests in ten still fall outside the selected path.</p><p>Jev selects fewer cases than Decisions, with higher observed accuracy among them. Strands selects very few by probability and none by confidence. That leaves almost its entire workload for fallback at this cutoff.</p><figure><img alt="ANLI, 3,200 cases. Values are coverage, selected accuracy, remaining accuracy. Decisions: confidence 53.56%, 68.55%, 50.40%; probability 60.88%, 67.04%, 49.36%; overall 60.12%. Jev: confidence 41.06%, 89.42%, 63.26%; probability 49.31%, 86.95%, 61.41%; overall 74.00%. Strands: confidence 0%, undefined, 48.09%; probability 0.81%, 65.38%, 47.95%; overall 48.09%. Rules applied separately at 0.90. Remaining accuracy measures original answers, not fallback performance." src="https://cdn-images-1.medium.com/max/1024/1*pG25HqfykA0XdAjIKNjUJQ.png" /></figure><p><em>ANLI · 3,200 cases. The same cutoff selects a different share of this benchmark.</em></p><p>ANLI reinforces the lesson: if every unselected answer takes another path, its load depends on the task as well as the model. Strands’ probability-selected groups contain just 26 ANLI cases and 13 ConTRoL cases. Their accuracy is too fragile to treat as a deployment estimate. No selected cases means undefined accuracy, not perfect accuracy.</p><h3>As the case for designing the fallback becomes clear,</h3><p>there are 2 key takeaways!</p><p><strong>1. Give low-score answers another path.</strong> When the chosen confidence or probability rule is not met, route the request to a separately validated model, request more evidence, defer to a person, or abstain. Measure how much traffic takes this path. Its capacity and latency are part of the application’s design.</p><p><strong>2. Validate the selected path too.</strong> About 77% accuracy in the Decisions example may still be insufficient. Choose a rule against your own task and error costs. Then test accuracy and latency across the complete system, including fallback. This study establishes neither a production cutoff nor the performance of a proposed fallback.</p><h3>Scope and sources</h3><p>These are single-pass public-benchmark results, not a universal ranking or a new latency test. Training exposure is unknown; hosted model identities do not pin immutable weights. The cutoffs were examined after the runs and are not calibrated guarantees. Jev’s 12 invalid answers remain wrong in the overall and remaining-group scores.</p><p>Sources: <a href="https://arxiv.org/abs/1910.14599">ANLI, Nie et al.</a> (<a href="https://github.com/facebookresearch/anli">data</a>; <a href="https://github.com/facebookresearch/anli/blob/main/LICENSE">CC BY-NC 4.0</a>), and <a href="https://doi.org/10.1609/aaai.v35i15.17580">ConTRoL, Liu, Cui, Liu and Zhang</a> (<a href="https://github.com/csitfun/ConTRoL-dataset">data</a>; CC BY-NC-SA 4.0). Both restrict commercial use; ConTRoL adds ShareAlike for adaptations. Only original analysis and text-free aggregates appear here.</p><p><strong>AI disclosure:</strong> Codex assisted with drafting, aggregate verification and visual preparation. I reviewed and edited the article before publication.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b691c6dce33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating System One Decision Models Serving options for latency sensitive applications]]></title>
            <link>https://medium.com/@srini.work/evaluating-system-one-decision-models-serving-options-for-latency-sensitive-applications-594be81e4fb3?source=rss-b37ecbb1632c------2</link>
            <guid isPermaLink="false">https://medium.com/p/594be81e4fb3</guid>
            <dc:creator><![CDATA[Srini Machavarapu]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 13:54:42 GMT</pubDate>
            <atom:updated>2026-10-11T13:10:08.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Can these models serve latency-sensitive workflows?</p><p>We evaluated two options with closed-loop tests. The results are encouraging. <a href="https://typesafe.ai/">TypeSafe AI</a>’s hosted Jev and Strands on one RTX 4090 each kept pooled p95 request latency below one second across all five tested concurrency levels.</p><h3>Setup</h3><p><strong>TypeSafe AI Jev:</strong> Remote hosted service accessed over HTTPS.</p><p><strong>Strands RTX 4090:</strong> Used the <a href="https://huggingface.co/StrandsAgents/strands-decider-2B-hobson-v21">Strands Decider 2B Hobson v21 model</a> (<a href="https://github.com/strands-labs/strands-decider">code</a>), built on Qwen3.5 2B Base, with isolated serving on one RTX 4090 GPU on <a href="https://www.runpod.io/">RunPod</a>, accessed over same-host HTTP.</p><p>For each option, we sent one decision question per request at concurrency 1, 2, 4, 8 and 16 across three runs. In this closed-loop test, a worker sent its next request after the previous response. We measured request latency, successful throughput and errors.</p><h3>Latency and throughput</h3><p>p50 and p95 are request latency; throughput is successful requests/s. Graph rows use separate vertical scales.</p><figure><img alt="Four plots show request latency and successful requests per second by concurrency. TypeSafe AI Jev latency changes modestly while throughput rises through 16; Strands RTX 4090 latency rises and throughput peaks at 2. Each workload has its own vertical scale." src="https://cdn-images-1.medium.com/max/1024/1*e4NGZcHurhxjzywzLMedrg.png" /></figure><h3>TypeSafe AI Jev hosted HTTPS</h3><figure><img alt="TypeSafe AI Jev results at concurrency 1, 2, 4, 8, 16. p50 ms: 196.1, 200.4, 197.8, 194.9, 209.7. p95 ms: 242.7, 253.1, 248.1, 242.1, 270.0. Successful requests/s: 4.89, 9.55, 19.39, 38.19, 67.62. Each level had 300 successes and 0 errors." src="https://cdn-images-1.medium.com/max/1024/1*H-V3GBw5LKJnRpcFXWzM3g.png" /></figure><h3>Strands RTX 4090 with isolated serving</h3><figure><img alt="Strands RTX 4090 isolated serving at concurrency 1, 2, 4, 8, 16. p50 ms: 39.8, 57.4, 127.0, 300.8, 790.8. p95 ms: 54.9, 78.4, 164.5, 371.1, 852.0. Successful requests/s: 20.51, 30.41, 29.47, 25.45, 20.28. Successes: 300, 600, 1200, 2400, 4800; zero errors." src="https://cdn-images-1.medium.com/max/1024/1*GSqzh9xRVxqxt5zZ1kVs0w.png" /></figure><h3>What we found</h3><p>All measured requests succeeded. The highest-throughput tested setting differed between these models and serving environments.</p><p>TypeSafe AI Jev: throughput kept rising through concurrency 16, the highest tested setting, reaching 67.62 successful requests/s at p95 270.0 ms.</p><p>Strands on one RTX 4090: throughput peaked at concurrency 2, with 30.41 successful requests/s and p95 78.4 ms. At concurrency 4, throughput was close at 29.47 requests/s while p95 rose to 164.5 ms.</p><p>Yes: each is a viable serving option under the tested conditions. Before rollout, check its latency, output quality, reliability and cost against your own workload.</p><p><strong>Limits.</strong></p><ol><li>These runs used different questions, input lengths, transport and timing windows. Each result describes its own serving setup.</li><li>The tables pool three repetitions or sessions. Jev reused synthetic states, with provider caching and queue time unknown; Strands ran on one host.</li><li>The numbers were internally audited, but not independently evaluated or externally replicated. Zero measured errors does not prove reliability; application quality and cost need separate evaluation.</li></ol><p>Disclosure: We have no financial, employment, sponsorship, or other material relationship with TypeSafe AI or Strands.</p><p>AI use: OpenAI Codex assisted with drafting and editing; the reported measurements come from recorded test runs and were checked against their source artifacts.</p><p><strong>Update — October 11, 2026: OpenAI Decisions API</strong></p><h3>Setup</h3><p>OpenAI Decisions API: hosted gpt-6-luna accessed over HTTPS from an Apple M1 Pro. We tested predicate, choice and score questions individually, plus all three packed into one request, at concurrency 1, 2, 4, 8 and 16 across three repetitions.</p><h3>Results</h3><figure><img alt="Four plots show mean repetition p50, p95, valid requests/s and decisions/s at concurrency 1, 2, 4, 8 and 16 for predicate, choice, score and packed workloads. Request throughput rises through 16; packed requests return three decisions each." src="https://cdn-images-1.medium.com/max/1024/1*E79Aj2tujiQphdXCkrvFmQ.png" /></figure><figure><img alt="Concurrency 16. Predicate: p50 267.97 ms, p95 364.80 ms, 43.51 requests/s, 43.51 decisions/s. Choice: 261.24 ms, 352.03 ms, 46.66 requests/s, 46.66 decisions/s. Score: 245.63 ms, 355.22 ms, 46.78 requests/s, 46.78 decisions/s. Packed: 242.05 ms, 338.72 ms, 48.41 requests/s, 145.24 decisions/s. Each workload: 288 valid requests, zero errors. Latency and rates average three repetitions." src="https://cdn-images-1.medium.com/max/1024/1*NyhW8513G2_JyNQx3izz2A.png" /></figure><p>Throughput rose through concurrency 16. Packing three questions reached 145.24 decisions/s at 48.41 requests/s; each measured request passed protocol validation.</p><p>Figures average three repetition statistics; the earlier tables use pooled percentiles. Client overhead affects throughput, and different workloads prevent provider ranking. Protocol validity does not establish answer accuracy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=594be81e4fb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where LLM Inference for Qwen2.5–7B Hits its Limits on RTX PRO 4500]]></title>
            <link>https://medium.com/@srini.work/where-llm-inference-for-qwen2-5-7b-hits-its-limits-on-rtx-pro-4500-5f9956233199?source=rss-b37ecbb1632c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f9956233199</guid>
            <dc:creator><![CDATA[Srini Machavarapu]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 23:34:09 GMT</pubDate>
            <atom:updated>2026-09-28T23:34:09.295Z</atom:updated>
            <content:encoded><![CDATA[<p>A throughput measurement tells us how fast a configuration ran. However, to understand why it ran at that speed, we also need to understand the workload the model requires and the rate at which the hardware can supply it.</p><p>This study connects these two kinds of evidence: a <strong>roofline analysis</strong>, which estimates hardware performance ceilings from the specifications, and <strong>controlled benchmark sweeps</strong>, which show what happened as the number of simultaneous requests changed.</p><h3>The setup</h3><p>The tests used <strong>Qwen2.5–7B-Instruct</strong>, a model with approximately 7.6 billion parameters, on one <strong>RTX PRO 4500 Blackwell with 32 GB of memory</strong>, rented through RunPod.</p><p><em>Model</em>: bf16, a 16-bit numerical format, with no quantization. <a href="https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json">[4]</a></p><p><em>Server</em>: vLLM</p><p><em>Request shape</em>: Each request contained <strong>1,024 input tokens and generated 256 output tokens</strong>. This long-input, short-output shape approximates retrieval-augmented generation, where a prompt includes retrieved context. The prompts were random benchmark inputs, so this tested performance rather than answer quality.</p><p><em>Load generation:</em> This is a closed-loop test. A load generator on the same host kept a fixed number of requests in flight, replacing each completed request with another.</p><p><em>Measurements:</em> The tests measured output throughput, time to first token, gaps between streamed tokens, GPU power and memory use. External network delay was excluded. <a href="https://docs.vllm.ai/en/v0.29.0/cli/bench/serve/">[6]</a><a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/measurements.json">[7]</a><a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/provenance.json">[8]</a></p><h3>What limits are driven by the GPU, and whats shaped by the workload</h3><p>The GPU supplies three distinct resources.</p><ol><li><strong>Compute capacity</strong> limits how much arithmetic it can perform per second.</li><li><strong>Memory bandwidth</strong> limits how quickly it can move data from device memory.</li><li><strong>Memory capacity</strong> limits how much data it can keep resident.</li></ol><p>The model and workload determine the demand on those resources.</p><ol><li>Model weights must be read; input tokens must be processed; and each active request needs a <strong>key-value (KV) cache</strong> that stores attention state for its context.</li><li>Model architecture, numerical precision, context length and batch size determine the amount of arithmetic and data movement per step.</li></ol><p>The serving engine connects the two. It batches requests and schedules prompt processing alongside token generation.</p><p>Hardware specifications alone therefore cannot give a universal tokens-per-second limit for a GPU: that limit depends on the model, workload and execution.</p><h4>Two phases with two different constraints</h4><p><strong>Prefill</strong> processes the prompt tokens, builds their key-value (KV) cache and produces the first output token. Many input tokens can be processed together, reusing the same weights for substantial arithmetic. Time to first token (TTFT) includes this work plus queueing and other overhead.</p><p><strong>Decode</strong> uses and extends the KV cache to generate subsequent tokens one at a time. Batching several sequences together lets them share a weight read, but each sequence also brings its own KV-cache traffic. Time to last token (TTLT) includes TTFT plus the time to generate all remaining tokens, including any stalls.</p><h3>Comparing measured performance with estimated limits</h3><p>The roofline chart shows how the estimated performance ceiling changes with arithmetic intensity — the work performed per byte read from memory. The horizontal axis measures that intensity in FLOP/byte; the vertical axis measures the rate of computation in TFLOP/s. At each intensity, the ceiling is whichever limit is lower: the rate memory bandwidth can support or the GPU’s compute capacity.</p><ul><li>At low intensity, bandwidth sets the ceiling.</li><li>At high intensity, compute sets it. The roofline analysis estimates the performance ceiling imposed by memory bandwidth and compute capacity. The measured operating points show how closely this workload approaches those limits.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgFo73KQlBCTZyRwmbYLBw.png" /></figure><p>For the two plotted operating points, the analysis places decode in the bandwidth-limited region and prefill in the compute-limited region. The decode estimate uses the concurrency-64 run and reaches about 82% of its estimated bandwidth ceiling (38.9 / 47.5 TFLOP/s). The prefill estimate uses the concurrency-256 run and reaches about 76% of the clock-adjusted compute ceiling (131.1 / 173 TFLOP/s). These are phase-specific estimates from different runs, not performance achieved together at one concurrency. [7][8]</p><h4>What drove the roofline and the ridge</h4><p>The sloped part comes from memory bandwidth. Lets say the workload performs “I” floating-point operations for every byte fetched, with a memory system supplying 896 billion bytes per second can support at most I × 896 billion operations per second. In the chart’s units, that is 0.896 × I TFLOP/s.</p><p>At 53 FLOP/byte, for example, the bandwidth ceiling is about 47.5 TFLOP/s. Reusing each byte for more arithmetic moves the workload rightward and raises this ceiling.</p><p>The flat part comes from compute capacity. This study estimates a nominal dense bf16 peak of 203 TFLOP/s from NVIDIA’s published tensor figures and the dense/sparse relationship. That peak assumes the boost clock. Under the sampled load, the GPU sustained 2,067 MHz rather than the implied 2,429 MHz boost clock, so we scale the estimate: 203 × (2,067 ÷ 2,429) ≈ 173 TFLOP/s. This clock-adjusted estimate is the horizontal roof; it is not a separately measured compute benchmark. The hardware sources and assumptions are detailed in the appendix.</p><p>The roofline takes the lower of those two limits: min(0.896 × I, 173) TFLOP/s. The ridge occurs at about 193 FLOP/byte, calculated as 173 ÷ 0.896. Below that ridge, memory bandwidth sets the estimated ceiling; above it, compute does.</p><p>See the technical appendix below for all assumptions, sources and step-by-step calculations.</p><h4>What drove the operating points</h4><p>The model and workload determine where a point falls along the horizontal axis, while execution determines how close its achieved rate gets to the roof. Note: The dashed FP8 point is a separate projection of reduced memory traffic, not a measured improvement in execution efficiency.</p><p>Each operating point combines an estimate of the work per byte with measurements collected during concurrency sweeps. We varied the number of requests in flight and recorded token timings and server throughput at each load. The decode point uses the median token gap from the concurrency-64 run; the prefill point uses the logged prompt-processing rate from the concurrency-256 run. These are two phase-specific estimates drawn from different runs, not two measurements of a single configuration. To place each on the roofline, we calculate the horizontal coordinate from the estimated arithmetic and memory traffic, then convert the observed token rate into the vertical coordinate, in approximate FLOP/s. The calculations below explain that conversion; afterward, we return to the sweep results to see what those limits mean for the service as a whole.</p><p><strong>Decode: estimate the work and traffic for one generation step.</strong> At a batch size of 64, the calculation assumes that a step produces one token for each of 64 sequences. Using approximately two operations per model parameter per token gives about 972.8 billion operations. The traffic estimate includes 14.14 GB of streamed weights plus 4.23 GB of KV-cache reads, based on the mean resident context of 1,152 tokens. Dividing the work by the 18.37 GB of traffic gives about 53 FLOP/byte: the point’s horizontal position, well below the ridge at 193.</p><p>For its vertical position, we use the measured median token gap of 25.01 ms as an estimate of decode step time. Under that approximation, 64 ÷ 0.02501 gives about 2,559 decode tokens per second. Multiplying by the approximate operations per token gives 38.9 TFLOP/s. The bandwidth roof at this intensity is 47.5 TFLOP/s, so the point reaches about 82% of that estimated limit. This calculation assumes the token gap represents a shared generation step; batching and mixed prefill/decode scheduling make it an estimate rather than an isolated decode measurement.</p><p><strong>Prefill: more tokens reuse the same weights.</strong> We place the prefill point in two steps: first calculate its horizontal position to identify the relevant roof, then calculate its vertical position to see how close the measured rate comes to that roof. For the horizontal position, use a representative 2,048-token chunk. At approximately 2 × 7.6 billion operations per token, that chunk requires about 31 trillion operations. Dividing by 14.14 billion bytes of streamed weights gives roughly 2,200 FLOP/byte. This is far beyond the ridge at 193, so the relevant ceiling is the flat 173 TFLOP/s compute roof. The chunk is an upper-bound assumption based on the scheduling budget; actual mixed steps also contain decode work, and this simplified traffic estimate omits attention and activations.</p><p>The prefill rate supplies the vertical coordinate. The concurrency-256 run logged 8,600.5 prompt tokens per second. We multiply that rate by the same approximate work per token used above: 8,600.5 × 2 × 7.62 billion parameters ≈ 131.1 trillion operations per second, or 131.1 TFLOP/s. Here 7.62 billion is the more precise parameter estimate rounded to 7.6 billion elsewhere. The plotted point is therefore approximately (2,200 FLOP/byte, 131.1 TFLOP/s). Its horizontal position tells us to compare against the compute roof; its vertical position gives the achieved fraction: 131.1 ÷ 173 ≈ 76%. The representative chunk determines the estimated work per byte, while the logged token rate determines the achieved work per second. They describe the two axes of the same phase estimate; we did not directly time an isolated 2,048-token prefill chunk.</p><p>The two points now connect the roofline analysis to the experiment: the sweep measurements supplied their achieved rates, and the analysis supplied the ceilings against which we compared them.</p><h3>Concurrency sweeps</h3><p>We can now look at those same sweeps from the service perspective. As concurrency increases, how much output does the server deliver, how long do requests wait, and what does each output token cost? These end-to-end results include prefill, decode and scheduling together, so their output throughput differs from the phase-rate estimate used for the decode point.</p><p>Concurrency sweeps covered two ranges. The first tested 16, 32, 64, 128 and 256 requests in flight. A second sweep examined the range from 32 to 64 more closely, in increments of eight. The model and request lengths stayed fixed; the coarse sweep used 12 host vCPUs and the fine sweep used 48, both with vLLM 0.29.0. This tested how batching affected throughput and latency as load increased. <a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/measurements.json">[7]</a><a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/provenance.json">[8]</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cu82LdQdhQ64Dp5A9P7z3w.png" /></figure><ul><li>Output throughput rises with batching, then reverses. It peaked at 1,298.8 output tokens/s at concurrency 128. At 256, it fell to 1,120.1 tokens/s, a 13.8% drop, while KV-cache usage reached 100%. Each active request needs KV state, so memory capacity constrains how much work can stay resident. At the same time, median time to first token jumped from 1.1 to 14.2 seconds, and median completion time rose from 25.5 to 55.4 seconds for the same 256-token output. The reversal is consistent with memory pressure and scheduling overhead at overload; the sweep does not isolate their individual contributions.</li><li>The finer sweep shows that streaming quality degrades before that capacity limit. Between concurrency 40 and 48, median token spacing barely changed, from 22.46 to 23.72 ms, but P90 rose from 24.43 to 130.19 ms. The fraction of gaps above 50 ms increased from 8.71% to 10.27%. Once more than 10% of gaps were slow, P90 began to include those stalls. This was a percentile threshold crossing, not a new hardware limit at concurrency 48.</li><li><strong>Time</strong> between streamed tokens includes occasional long stalls. P99 gaps were already about 230 ms at concurrency 40 and stayed near that level at 48. A separate chunk-budget experiment supported prefill sharing a scheduling step with decode as the stall mechanism. That follow-up used vLLM 0.30.0 rather than the sweeps’ 0.29.0, so it supplies supporting evidence rather than a controlled cross-version comparison.</li><li>Beyond maximum throughput, the practical question is usable capacity under a latency target. Concurrency 40 was the highest tested point in the original fine sweep meeting P90 token gaps ≤ 50 ms: 1,019.9 output tokens/s, or 78.5% of peak. The streaming target became limiting well before the high-load KV-cache saturation.</li><li>Cost reflects that trade-off: about $0.198 per million output tokens at concurrency 40, versus $0.156 at concurrency 128. These use the historical $0.728/hour pod rate and exclude production overhead and idle capacity. Peak throughput and the tested smooth-streaming target call for different operating points.</li></ul><p><a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/concurrency.csv">Download the full measurement table (CSV).</a></p><h3>Next steps</h3><p>The measurements place prefill and decode fairly close to their respective estimated ceilings in the tested configurations. This bounds expectations for that lever without ruling out other changes to the workload or implementation.</p><p><strong>Lower precision changes the bytes that must move.</strong> FP8 uses an 8-bit floating-point representation. In the batch-64 calculation, FP8 weights alone would raise the decode bandwidth ceiling by about 1.6×. Using FP8 for both weights and the KV cache would raise it by about 2×, because both sources of traffic shrink.</p><p>Those are predictions for a decode ceiling, not measured end-to-end speedups. FP8 was not tested here. A follow-up experiment would need to measure throughput, latency and answer quality; the current random-prompt benchmarks cannot establish the quality trade-off.</p><p>Stay tuned for the results from the follow up experiment with FP8 :-)</p><h3>Technical appendix: assumptions and derivations</h3><p>The estimates below combine published hardware specifications, model dimensions and benchmark measurements. They describe the tested workload and separate measured performance from projected improvements. A <a href="https://srini-research.github.io/ml-research/where-llm-inference-hits-its-limits/">text version of the derivations</a> is available in the original article’s technical appendix.</p><h4>1 — Constants and sources</h4><figure><img alt="Constants table: bandwidth 896 GB/s; dense bf16 estimate 203 TFLOP/s; clocks 2,429 MHz implied and 2,067 MHz sustained; about 7.6 billion parameters; streamed weights 14.14 GB; KV cache 57,344 bytes/token; mean context 1,152 tokens; decode median gap 25.01 ms; prefill rate estimate 131.1 TFLOP/s. Sources and formulas are shown in each row. The linked text appendix provides the full derivations." src="https://cdn-images-1.medium.com/max/1024/1*Is8L8DGpIg7hftObs_q-uA.png" /></figure><p>The input embedding table is excluded from streamed weight traffic: looking up 64 token vectors gathers 64 rows rather than reading the entire matrix. Charging all 15.23 GB to every step understates the estimated bandwidth ceiling by about 7.7%.</p><h4>2 — Two roofs, and why the compute roof is 173</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMOoZkrs6iNDFhE3yW5u2w.png" /></figure><p>Doubling the arithmetic performed per fetched byte doubles the bandwidth ceiling until compute becomes limiting. The 203 and 173 TFLOP/s values are estimates, not independently measured compute benchmarks.</p><h4>3 — The ridge, where the roofs cross</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaiRSwIkwzFlbhiSFXHJWA.png" /></figure><p>The ridge reflects the hardware specifications and sustained-clock assumption used in this analysis. It separates the two estimated constraints; it does not account for every kernel, scheduling or utilization overhead.</p><h4>4 — Decode at 64 sequences in flight</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgiRcdN1VImsKCcbwoknEg.png" /></figure><p>The achieved-rate calculation approximates the median token gap as a shared generation-step time, with one token produced per sequence. Actual batching and mixed prefill/decode scheduling mean this is an estimate, not an isolated decode measurement. The batch-48 timing comparison is consistent with bandwidth limiting decode performance.</p><h4>5 — Prefill: a representative 2,048-token chunk</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKuEkV7WZTYymLISJwmNxw.png" /></figure><p>In this simplified estimate of matrix operations, prefill reuses the same weights across N input tokens, much as batched decode does across B sequences. The 2,048-token scheduling budget is an upper-bound representative chunk: decode work also consumes part of mixed steps. The traffic estimate omits attention and activations; prefill writes new KV state rather than reading each sequence’s full existing cache.</p><p>We did not directly time an isolated 2,048-token chunk. The 76% result compares the logged prompt-processing rate with the estimated compute roof; it is not a universal efficiency ceiling.</p><h4>6 — Projected effect of FP8 weights and KV cache</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5-f6xf8PQXyjxX7OQAAoAg.png" /></figure><p>These projections hold the approximate matrix arithmetic per token constant while reducing memory traffic. They neglect quantization overhead and kernel changes, and predict decode bandwidth ceilings rather than measured end-to-end speedups. FP8 was not tested.</p><h3>Study limitations and disclosures</h3><p>Results apply to one GPU, model and synthetic workload (1,024 input / 256 output tokens); answer quality and external network latency were not tested. Costs use the historical pod rate and exclude idle capacity and production overhead.</p><p>The study has not been independently reproduced or verified. Roofline ceilings are estimates; FP8 gains are untested projections. Public data includes selected measurements and provenance, not complete request-level records. Validate against your own workload before purchasing or deploying.</p><p>Claude Code and OpenAI Codex assisted with harness development, benchmark execution, analysis, charts and writing. The author directed the research, reviewed findings, challenged assumptions and redirected the tools throughout; this oversight does not replace independent verification.</p><h3>References and supporting data</h3><p>External sources accessed September 28, 2026. Measurements collected September 18–22, 2026.</p><p>[1] NVIDIA. <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/data-center/rtx-pro-4500-blackwell/workstation-datasheet-blackwell-rtx-pro-4500-gtc25-spring-nvidia-3662540.pdf">RTX PRO 4500 Blackwell workstation datasheet</a>. The analysis uses the specification table’s 896 GB/s bandwidth and 200 W board power; the narrative contains a conflicting 784 GB/s figure.</p><p>[2] NVIDIA. <a href="https://www.nvidia.com/en-us/data-center/rtx-pro-4500-blackwell-server-edition/">RTX PRO 4500 Blackwell Server Edition specifications</a>. Source for the 406 TFLOP/s tensor and 51 TFLOP/s FP32 figures used to estimate compute; workstation memory and power values remain from [1].</p><p>[3] NVIDIA. <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-RTX-Blackwell-PRO-GPU-Architecture-v1.0.pdf">RTX Blackwell GPU Architecture, v1.0, Appendix A</a>. Dense/sparse context supporting the inferred 203 TFLOP/s nominal and 173 TFLOP/s clock-adjusted ceilings, neither a measured compute benchmark.</p><p>[4] Qwen. <a href="https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/blob/main/config.json">Qwen2.5–7B-Instruct model configuration</a>. Geometry for parameter and KV-cache calculations.</p><p>[5] vLLM contributors. <a href="https://docs.vllm.ai/en/v0.30.0/configuration/optimization/#chunked-prefill">Optimization and Tuning, v0.30.0: Chunked Prefill</a>. Scheduling and token-budget context for the follow-up experiment.</p><p>[6] vLLM contributors. <a href="https://docs.vllm.ai/en/v0.29.0/cli/bench/serve/">bench serve CLI reference, v0.29.0</a>. Concurrency controls and benchmark latency metrics.</p><p>[7] This study. <a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/measurements.json">Selected measurements (JSON)</a>, <a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/concurrency.csv">concurrency data (CSV)</a> and <a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/chunks.csv">chunk-budget data (CSV)</a>. Eight selected concurrency points and four follow-up chunk settings, exported from saved benchmark results.</p><p>[8] This study. <a href="https://srini-research.github.io/ml-research/assets/qwen-baseline/provenance.json">Conditions, formulas and source hashes (JSON)</a>. Experiment provenance and calculation assumptions; hashes identify private source records.</p><p>Originally published at <a href="https://srini-research.github.io/ml-research/where-llm-inference-hits-its-limits/">Srini Research</a>. The original includes interactive charts and downloadable evidence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f9956233199" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>