<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sean Fraser on Medium]]></title>
        <description><![CDATA[Stories by Sean Fraser on Medium]]></description>
        <link>https://medium.com/@cloudhacks_?source=rss-a3e37d7ff868------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XTm7TwGdPjP_Q0lcguDTMQ.jpeg</url>
            <title>Stories by Sean Fraser on Medium</title>
            <link>https://medium.com/@cloudhacks_?source=rss-a3e37d7ff868------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:53:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cloudhacks_/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What decades of networking and distributed systems engineering can teach us about AI inference]]></title>
            <link>https://medium.com/@cloudhacks_/ai-inference-systems-engineering-b995bdf1c77d?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/b995bdf1c77d</guid>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 00:00:41 GMT</pubDate>
            <atom:updated>2026-11-02T12:09:44.444Z</atom:updated>
            <content:encoded><![CDATA[<p>The models are new. Many of the constraints are familiar.</p><p>A GPU can stay busy while the service around it gets slower. A smaller weight file can leave too little memory for the requests you need to serve. Separating a workload across machines can save compute and add enough communication to erase the gain.</p><p>Choosing a better model will not, on its own, solve these problems.</p><p>Scheduling, numerical representation, caching and distributed execution are familiar territory for systems engineers. Adapting that work to autoregressive models takes substantial engineering, but it gives teams a useful place to start. Find the resource that limits the service, then check whether an optimization helps requests finish sooner or at a lower cost.</p><h3>Continuous batching is a scheduling problem</h3><p>A serving batch might contain a short reply alongside a long explanation. If the system holds that batch together until every request finishes, completed requests leave capacity that new arrivals cannot immediately use. Orca’s 2022 paper addressed this with iteration-level scheduling: the scheduler chooses the requests for each model iteration rather than fixing the batch for their entire lifetimes.<a href="https://www.usenix.org/conference/osdi22/presentation/yu">[1]</a></p><p>This is the scheduling idea behind continuous batching. The scheduler can retire finished work and admit new work between iterations.<a href="https://www.usenix.org/conference/osdi22/presentation/yu">[1]</a></p><p>Earlier server designs faced similar scheduling constraints. SEDA organized internet services into stages connected by explicit queues, with resource controllers to manage changing load.<a href="https://www.sosp.org/2001/papers/welsh.pdf">[10]</a> Both designs give the system explicit control over pending work and resource allocation. The comparison helps explain the design problem; it does not establish that Orca descended from SEDA.</p><p>A decoding sequence also has requirements that a packet or event does not. An active LLM request retains attention state across iterations, so admission decisions must account for memory as well as execution capacity.<a href="https://www.usenix.org/conference/osdi22/presentation/yu">[1]</a></p><p>Keeping the GPU occupied is only part of the job. A batch policy must also protect latency as prompt and output lengths change. Admitting more work can improve aggregate throughput while making each user wait longer. Orca explicitly treats maximum batch size as a throughput-versus-latency control.<a href="https://www.usenix.org/conference/osdi22/presentation/yu">[1]</a></p><p>Test the scheduler with the request mix you expect to serve. A uniform set of short prompts will tell you little about how the system behaves when long requests arrive in a burst.</p><h3>Quantization spends an error budget</h3><p>Telecommunications engineers were reducing numerical precision long before language models needed to fit in accelerator memory. Stuart Lloyd’s <em>Least Squares Quantization in PCM</em>, published in 1982 with material presented in 1957, examined how to choose quantization levels to minimize average quantization noise power.<a href="https://www.stat.cmu.edu/~rnugent/PCMI2016/papers/LloydKMeans.pdf">[2]</a></p><p>Fewer bits reduce storage, but the approximation introduces error. For a language model, the test is whether that error changes the answers enough to matter for the task. Methods such as GPTQ address the numerical work needed to make low-bit weight quantization practical for large transformers.<a href="https://arxiv.org/abs/2210.17323">[3]</a></p><p>Consider an illustrative model with exactly 70 billion parameters. Raw weight storage is parameter count multiplied by bits per weight, divided by eight. Using decimal gigabytes, the calculation gives:</p><p>Weight precision | Theoretical raw weight storage</p><p>16 bits | 140 GB</p><p>8 bits | 70 GB</p><p>4 bits | 35 GB</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*IlMttG8cNz2o0uM0.png" /></figure><p><em>Raw weight storage for an illustrative model with exactly 70 billion parameters. Original calculation: parameters × bits ÷ 8 ÷ 10⁹. These are theoretical sizes, not measured GPU memory requirements or speedups. Scales, metadata, activations, KV cache and runtime overhead are excluded.</em></p><p>Four-bit weights occupy one quarter of the raw storage of sixteen-bit weights. That arithmetic does not establish a fourfold speedup. GPTQ’s paper distinguishes faster memory loading from faster multiplication and describes the limits of its hardware support.<a href="https://arxiv.org/abs/2210.17323">[3]</a></p><p>Check capacity and performance separately. The model needs to fit with the required context and concurrency, and it needs to serve requests fast enough on the intended hardware. Evaluate output quality too, including cases where a small error has a high cost. Fitting the weights into memory is useful only if the model still handles the work you need it to do.</p><h3>Speculative decoding buys concurrency with extra work</h3><p>In the draft-model form of speculative decoding, a cheaper model proposes several tokens. The target model evaluates those proposals in parallel, and the algorithm accepts a prefix before correcting a rejection or producing an additional token.<a href="https://arxiv.org/abs/2211.17192">[4]</a></p><p>Leviathan, Kalman and Matias explicitly connect their method to processor speculation, describing it as a generalization of speculative execution to a stochastic setting.<a href="https://arxiv.org/abs/2211.17192">[4]</a></p><p>The correctness condition is different from following the right CPU branch. Their exact sampling algorithm preserves the target model’s output distribution. It does not promise that two runs will emit the same text, and a plausible-looking draft is not sufficient for acceptance.<a href="https://arxiv.org/abs/2211.17192">[4]</a></p><p>The extra work can pay off if it reduces expensive sequential target-model passes. The paper’s performance model depends on how often proposals are accepted and how costly the approximation model is relative to the target.<a href="https://arxiv.org/abs/2211.17192">[4]</a></p><p>A draft model that predicts well on one type of request may contribute less on another. Include verification, draft execution and additional state when measuring the result, rather than treating the draft’s work as free.</p><p>Benchmark the full serving path with speculation on and off. Count accepted draft tokens, but do not confuse a high acceptance rate with a service-level improvement. The objective is lower latency or lower cost at the required load, with the required correctness guarantees.</p><h3>Distillation changes the model you operate</h3><p>Distillation belongs primarily to machine learning’s own history. The 2006 <em>Model Compression</em> paper demonstrated how smaller models could learn to mimic larger ensembles.<a href="https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf">[5]</a> Hinton, Vinyals and Dean’s 2015 paper developed that approach further as knowledge distillation.<a href="https://arxiv.org/abs/1503.02531">[6]</a></p><p>The appeal is familiar: use a cheaper model when it can do the job. A student learns behavior from a teacher, rather than retrieving a cached answer. Unlike exact speculative decoding, distillation does not guarantee the teacher’s output distribution.<a href="https://arxiv.org/abs/2211.17192">[4]</a><a href="https://arxiv.org/abs/1503.02531">[6]</a></p><p>A smaller student needs its own quality evaluation. If you route requests between small and large models, evaluate the router too. Even a capable large model cannot help with a difficult request that never reaches it.</p><p>Include routing delay, fallback calls and retries in cost and completion time. Test difficult inputs separately so a large volume of easy requests does not hide failures. A specialized model may be worth the additional work, but someone still has to maintain the models, routing rules and evaluation sets.</p><h3>KV caching makes memory management part of serving</h3><p>For the autoregressive transformers discussed here, a KV cache retains attention keys and values from earlier tokens so decoding can reuse them. The state grows as sequences grow, and active requests compete for the memory that stores it.<a href="https://arxiv.org/abs/2309.06180">[7]</a></p><p>The comparison with memoization is useful, but it does not explain the allocation problem. A serving system must accommodate sequences of different lengths whose final sizes are not known in advance. Reserving a large contiguous region for each request can waste memory and restrict the batch size.<a href="https://arxiv.org/abs/2309.06180">[7]</a></p><p>PagedAttention makes the historical connection explicit. Its authors describe an attention algorithm inspired by operating-system virtual memory and paging. Logical KV blocks map to physical blocks that need not be contiguous, with allocation and sharing managed at block granularity.<a href="https://arxiv.org/abs/2309.06180">[7]</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*V0N7YxWK3xegehgi.png" /></figure><p><em>OS paging and PagedAttention both separate a logical sequence from its physical placement. The labels are illustrative; this is an abstraction comparison, not an implementation trace. Original illustration based on the PagedAttention paper.</em><a href="https://arxiv.org/abs/2309.06180"><em>[7]</em></a></p><p>Memory efficiency affects how many requests can remain active. Sharing can avoid duplication where the model inputs and implementation permit it. Preemption introduces a choice about preserving state or paying to recreate it.<a href="https://arxiv.org/abs/2309.06180">[7]</a></p><p>Track cache occupancy alongside the request mix, including long contexts and periods of overload. Check that abandoned requests release their state after cancellation. Verify how prefix reuse works in the implementation you operate; similar-looking prompts do not, by themselves, establish that the cached state can be shared.</p><h3>Disaggregation puts the network on the critical path</h3><p>Prefill processes the prompt and establishes the initial attention state. Decode extends the sequence through successive output tokens. The phases place different demands on the serving system, and colocating them can create interference.<a href="https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin">[8]</a></p><p>DistServe separates prefill and decode onto different GPUs and chooses resource allocation and placement around latency requirements and available bandwidth.<a href="https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin">[8]</a></p><p>Separating the phases requires moving the relevant KV state, and DistServe’s placement strategy accounts for that transfer.<a href="https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin">[8]</a> Whether the split helps your service depends on the workload and the infrastructure connecting the pools.</p><p>Test a colocated baseline before adding separate pools. Compare prompt lengths, concurrency and latency under equivalent resource budgets. Include transfer time and queuing. A faster prefill stage does not justify an architecture that delays decode enough to make the request slower.</p><p>Engineers who have worked with disaggregated infrastructure will recognize the tradeoff: separate pools can scale independently, but they add coordination and data movement.</p><h3>Measure the service, then choose the optimization</h3><p>In <em>The Tail at Scale</em>, Jeffrey Dean and Luiz André Barroso describe how latency variability becomes harder to contain as systems grow and utilization rises.<a href="https://research.google/pubs/the-tail-at-scale">[9]</a> Inference gives operators another reason to take that lesson seriously: each additional queue, model pool or state transfer can affect request completion.</p><p>Use a fixed evaluation procedure before combining optimizations:</p><ol><li>Define the workload. Record prompt lengths, expected output lengths, arrival patterns, context limits and the task-quality tests. Include difficult requests and bursts.</li><li>Set service targets. Measure latency from the client’s initial submission, including queues, routing and retries. Specify acceptable time to first delivered token, response completion time and output quality. For streaming, report gaps between output-bearing events separately from average time per generated token, and state how you calculate that average. Speculative decoding can produce several tokens per target-model pass; delivery cadence need not match token generation.<a href="https://arxiv.org/abs/2211.17192">[4]</a></li><li>Establish a baseline. Measure the complete request path under realistic concurrency. Retain latency distributions rather than reporting only averages.</li><li>Change one mechanism. Compare equivalent workloads and resource budgets. Include draft-model work, cache transfers, routing, retries and failures where applicable.</li><li>Test overload and cancellation. Observe queue growth, rejected work, resource release and recovery when demand falls.</li></ol><p>For this evaluation, define <strong>goodput</strong> as qualifying logical requests completed per second. Each must pass the task-quality checks and meet the latency targets. Count each original request once, regardless of retries or fallback calls. Report offered load, failures and rejections alongside goodput.</p><p>Declare a measurement window with a consistent warm-up and drain policy. Report outstanding requests at the boundary and account for the work needed to resolve them. Separate explicit user cancellations from deadline expirations and service failures. Cancelled requests do not count as qualifying completions, but their consumed resources remain in the cost calculation. Keep that policy consistent across comparisons.</p><p>For cost per qualifying request, divide the declared serving cost by qualifying logical requests. Include provisioned resources across all serving stages, including idle capacity, routing, draft models and transfers. State whether model preparation and ongoing evaluation costs are excluded or amortized separately. A lower bill achieved by silently dropping difficult work is not an improvement.</p><p>This is a suggested evaluation procedure. No serving benchmark was conducted for this article.</p><p>Start with the measured constraint. If weight storage prevents deployment, evaluate quantization. If active sequence state limits concurrency, inspect KV allocation. If mixed request lengths cause avoidable waiting, examine scheduling. Use speculation, distillation or phase separation only when their additional work and operating complexity earn a measurable return.</p><p>Experience with networks and distributed systems is useful here. It helps teams recognize when a promising component-level improvement has moved the bottleneck somewhere else. The models have changed, but requests still queue, memory still fills and data still takes time to move. There is no reason to learn all of those lessons again from scratch.</p><h3>Sources</h3><p>[1] <a href="https://www.usenix.org/conference/osdi22/presentation/yu">Orca: A Distributed Serving System for Transformer-Based Generative Models</a></p><p>[2] <a href="https://www.stat.cmu.edu/~rnugent/PCMI2016/papers/LloydKMeans.pdf">Least Squares Quantization in PCM</a></p><p>[3] <a href="https://arxiv.org/abs/2210.17323">GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers</a></p><p>[4] <a href="https://arxiv.org/abs/2211.17192">Fast Inference from Transformers via Speculative Decoding</a></p><p>[5] <a href="https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf">Model Compression</a></p><p>[6] <a href="https://arxiv.org/abs/1503.02531">Distilling the Knowledge in a Neural Network</a></p><p>[7] <a href="https://arxiv.org/abs/2309.06180">Efficient Memory Management for Large Language Model Serving with PagedAttention</a></p><p>[8] <a href="https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin">DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving</a></p><p>[9] <a href="https://research.google/pubs/the-tail-at-scale">The Tail at Scale</a></p><p>[10] <a href="https://www.sosp.org/2001/papers/welsh.pdf">SEDA: An Architecture for Well-Conditioned, Scalable Internet Services</a></p><p><strong>Until next time, protect yourselves and safeguard each other</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b995bdf1c77d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inference Serving: Optimize the Request Path Before Buying More GPUs]]></title>
            <link>https://medium.com/@cloudhacks_/inference-serving-optimize-the-request-path-before-buying-more-gpus-efd15d2cd79b?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/efd15d2cd79b</guid>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[gpu-optimization]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 00:00:35 GMT</pubDate>
            <atom:updated>2026-10-16T16:06:38.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjiVebe7lAESs0X7TxrSzQ.png" /></figure><p>A serving endpoint can have requests waiting while its GPU still has room for useful work. Continuous batching addresses one cause: requests finish at different iterations, and a static batch cannot immediately replace the finished ones.[2] Buying another accelerator before understanding that queue risks paying for the same scheduling problem twice.</p><p>The useful networking analogy is resource admission, not a utilization target. Ask which work can enter, what state it must retain, and which requests wait behind it. Then optimize for completed requests within a latency objective. A busier device is evidence to investigate, not proof of a better service.</p><p>The decision rule is to trace the request path before changing capacity. Separate admission, cache pressure, prefill interference, and host-side overhead. This article uses published implementation details and historical benchmarks, not a new measurement of my single-node machine. The recommendations below are a test framework, not promised speedups.</p><p>The implementation guidance concerns vLLM V1 serving autoregressive text-generation requests.[1] Check feature support and defaults against the installed release: the linked stable and latest documentation are moving references, not a compatibility guarantee. Distributed topology, multimodal preprocessing and availability sizing are outside this framework. No engine release was benchmarked for this article.</p><h3>Continuous batching changes when work can enter</h3><p>Autoregressive generation proceeds iteratively.[2] In a static batch, shorter completions leave unused opportunities while longer requests finish.[2] Continuous batching makes admission decisions between iterations, allowing a new request to join without waiting for every existing request to complete.[2]</p><p>Think of statistical multiplexing, with an important limitation: a request is not a packet that disappears after transmission. Its attention state remains resident across subsequent generation steps.[4] Scheduling and memory allocation must cooperate, which is why a batch-size setting alone does not describe serving capacity.[2][4]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3s3tV_lKR3B5v2rjtXUvAA.png" /></figure><p><em>Figure 1. Continuous batching can reuse a finished request’s place at an iteration boundary. This schematic omits prefill cost and does not predict throughput. Original illustration based on Anyscale’s scheduling explanation.[2]</em></p><p>Anyscale’s June 2023 comparison reported up to 8× throughput over naive batching from continuous batching, and up to 23× when using vLLM with additional memory optimizations.[2] Those compare different serving implementations against the same naive-static baseline.[2] Calling the larger figure a pure scheduler improvement erases part of the mechanism.</p><p>The experiment varied output-length distributions and used a burst submission for its throughput test; its latency experiments used separately controlled arrival patterns.[2] These details matter more than the headline multiplier. A workload with uniform short responses offers a different scheduling opportunity from a mixture of brief answers and long generations.</p><p>For a local evaluation, preserve the prompt and output distributions, including their relationship. Do not compare a serial client against a concurrent client and attribute the entire difference to the engine. Record offered request rate independently of completed throughput. When the server falls behind, keep counting rejected, cancelled, and timed-out requests rather than quietly removing them from the result.</p><p>The original vLLM engineering report identified 60–80% memory waste in the systems it examined, caused by fragmentation and over-reservation.[4] That is a memory-management finding, not evidence that every GPU wastes that percentage of compute. The distinction changes the diagnosis: recovering cache space helps admission only when cache capacity is the limiting resource.</p><p>PagedAttention divides each sequence’s key-value cache into fixed-size blocks.[4] A block table maps logical sequence positions to physical blocks that need not be adjacent, and the engine allocates physical blocks as the sequence grows.[4] This avoids reserving one contiguous region for the entire possible sequence at arrival.[4]</p><p>The original report described residual waste below 4% in its evaluated setting, concentrated in a sequence’s final block.[4] Treat that as an attributed historical result, not a guarantee for all memory consumed by a modern serving process. Model weights, execution buffers, and other allocations are outside that particular claim.</p><p>Current vLLM guidance documents the next failure mode: when the KV cache cannot hold the active work, the engine can preempt requests and recompute their state later, increasing end-to-end latency.[1] A server can therefore remain busy while spending effort recovering work instead of finishing more useful requests.</p><p>Before increasing concurrency, watch cache occupancy and preemption alongside queue delay. If preemptions rise, first test a lower active-sequence limit. Compare that with increasing the cache memory allocation only where measured headroom permits. vLLM also documents reducing max_num_seqs or max_num_batched_tokens as ways to reduce cache pressure.[1]</p><p>Do not turn a memory-utilization fraction into a universal recommendation. Reserve room according to the exact model, execution mode, co-tenancy, and worst supported request. A configuration that boots is not a capacity test.</p><h3>Chunked prefill negotiates between two latency budgets</h3><p>Prefill processes the prompt; decode generates subsequent tokens.[2] vLLM’s optimization guide describes prefill as compute-bound and decode as memory-bound, and explains how chunked prefill mixes smaller prefill pieces with decode work.[1] Those descriptions explain the scheduling opportunity, but profiling should still establish the bottleneck for the workload being operated.</p><p>In V1, chunked prefill is enabled by default whenever possible.[1] The documented policy schedules pending decodes first, then uses the remaining max_num_batched_tokens budget for prefills, splitting a prefill when it does not fit.[1] This is a token budget, not a promise that every prefill chunk has one permanent size.</p><p>The tradeoff is explicit. Smaller budgets can improve inter-token latency because less prefill work delays decode; larger budgets can improve time to first token by processing more prompt tokens per batch.[1] A recommendation that claims both tails always improve conceals which workload paid for the gain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1DiJOHVEdINaQMtPRGdJ9A.png" /></figure><p><em>Figure 2. Decode-first scheduling allocates remaining token budget to prefill. Smaller budgets favor inter-token latency; larger budgets can favor time to first token. Measure both under the same offered load. Original conceptual illustration based on vLLM’s tuning guidance.[1]</em></p><p>For interactive traffic, define the acceptable gap between streamed outputs separately from the wait for the first output. An output event can contain multiple tokens under speculative decoding, so its gap is not necessarily a per-token delay.[7] For document analysis, retain the longest supported prompts in the test. A configuration selected from short prompts alone has not passed the workload that creates prefill interference.</p><p>Sweep the budget while holding the request trace constant. Inspect time to first token and inter-token latency by prompt-length bucket, including tails. Reject a result that makes established streams smoother by leaving new requests waiting beyond their objective. Equally, reject an apparent first-token win that creates visibly stalled streams.</p><h3>Prefix reuse and speculation solve different problems</h3><p>Automatic prefix caching reuses cached attention state when a new query shares a prefix with an earlier query.[5] It skips the shared prefill computation; it does not accelerate generation of new tokens.[5] Repeated system prompts are therefore a reason to test reuse, not a reason to promise faster decode.</p><p>Use representative rendered prompts when testing cache benefit. Include cold-cache runs and requests that do not share prefixes. Record the engine’s reported reuse and the actual first-token improvement. Otherwise a warm benchmark can become a capacity assumption that disappears after restart or a prompt change.</p><p>Speculative decoding deserves its own experiment. vLLM’s documentation frames it as a way to reduce inter-token latency for memory-bound workloads at medium-to-low request rates, and says gains depend on the model family, hardware, traffic, and sampling settings.[6] Its method-selection guidance also includes potential high-load benefits, so “latency only, never throughput” is too broad a rule.[6]</p><p>Test speculation against the same service objective as the baseline. Measure accepted draft progress, resource use, and end-to-end results together. Do not set a universal acceptance-rate threshold and call the decision finished: the cost of proposing and verifying work belongs in the comparison.</p><p>Keep weight or cache precision changes in separate experiments. If a candidate uses a different representation, require a task-quality evaluation as well as the performance run. A serving optimization should not quietly change what counts as an acceptable answer.</p><h3>The host can leave the accelerator waiting</h3><p>The V1 architecture report explains why host work matters: request state, scheduling, input preparation, detokenization, and response streaming all consume CPU time around model execution.[3] Current optimization documentation warns that CPU underprovisioning can degrade throughput and latency even when the GPU looks underused.[1]</p><p>That breaks the tempting diagnosis that low GPU utilization necessarily means bad batching. Check the path before the device: incoming requests, tokenization, scheduler activity, and output handling. Compare server-side timings with the client’s view so transport or client bottlenecks do not become imaginary GPU constraints.</p><p>The V1 alpha report claimed up to 1.7× throughput relative to V0 without multi-step scheduling, attributing improvements largely to reduced CPU overhead.[3] That is a historical architecture comparison. It is not an additional multiplier to stack onto Anyscale’s earlier results, and its alpha-era feature limitations should not be read as today’s support matrix.</p><p>For each run, save the engine version and full configuration with the model revision, tokenizer, driver, and hardware identity. Save the benchmark client version and documentation retrieval date too. Verify that the chosen speculation method and feature combination are supported by the installed release. Record which optimizations actually activated in startup logs. Documentation defaults are a starting hypothesis; the running process is the thing being measured.</p><h3>Use goodput as the purchase gate</h3><p>Define goodput for the evaluation as completed logical requests per second that satisfy the agreed latency and quality requirements, counting each logical request once. Report ordinary throughput beside it, in requests per second and output tokens per second. This is a proposed operating definition, not a benchmark result.</p><p>Use a fixed arrival window and fixed drain period for every candidate. Follow that arrival cohort until completion or its deadline; count requests still unfinished when the drain ends as failures. Divide qualifying completions by the combined arrival-window and drain duration. Rejected, timed-out, cancelled and unfinished requests contribute no goodput. Keep them in the offered-request count when reporting the qualifying fraction and failure rate. Count logical requests and retry attempts separately, and report client abandonment even if the server later finishes the work.</p><p>Build the test in a controlled sequence:</p><ul><li><strong>Establish the baseline.</strong> Use the intended model and a representative request trace, including arrival timestamps. Replay arrivals independently of response completion and verify that the generator sustains them. Record scheduled arrivals, actual dispatches, client-side queueing, concurrency caps, deadlines and retry policy; label response-paced tests separately. Keep sampling settings, stop conditions, chat templates and output-token caps fixed, and report actual output lengths. Separate warm-up from measurement. Include quiet periods and bursts rather than measuring only an indefinitely saturated queue.</li><li><strong>Identify the constraint.</strong> Correlate queue delay with host load, cache pressure, preemptions, and device activity. Select one hypothesis before changing a setting.</li><li><strong>Change one mechanism.</strong> Evaluate token budget, active-sequence limit, prefix reuse, or speculation independently where possible. Keep any precision change paired with its quality gate.</li><li><strong>Find the service boundary.</strong> Increase offered load until the agreed objectives fail. Record failures, cancellation behavior, and recovery after the burst subsides.</li><li><strong>Repeat the candidate.</strong> Retest after restart and on the difficult request buckets. Keep the old configuration available for rollback.</li></ul><p>Report client-observed time to first streamed output (TTFT, milliseconds), gaps between successive streamed outputs (ITL, milliseconds), per-request time per output token excluding the first token (TPOT, milliseconds per token), and end-to-end latency (seconds).[7] For requests with more than one output token, TPOT = (end-to-end latency − TTFT) / (output tokens − 1).[8] Report both ITL and TPOT when speculative decoding can bundle tokens into one output event.[7]</p><p>State the measurement boundaries and percentile populations: TTFT, TPOT and end-to-end latency are request-level distributions; ITL is an output-event-gap distribution.[7][8] For zero- or one-output-token requests, document how the measurement tool handles TPOT; current vLLM server histograms record zero while its benchmark descriptive statistics exclude those requests.[8] Report sample counts by prompt-length bucket and do not average percentile summaries across buckets or runs. Add offered and completed request rates, output token throughput, error rate and preemption counts. Label the observation window and cache state.</p><p>Choose acceptance criteria before inspecting the winning run. Report the fraction of offered logical requests meeting all request-level latency requirements together. State whether quality is checked for each request or used as a separate gate on the candidate configuration. For candidate-level quality testing, report latency-qualified goodput and the separate quality result rather than implying every request was checked. A candidate should increase goodput at the intended traffic mix without breaching that quality gate or consuming the headroom required for recovery. If a longer queue merely postpones errors until clients time out, reject it as a capacity improvement.</p><p>Consider additional GPUs when repeatable tests identify an accelerator-capacity or model-memory constraint and show that the proposed replica or sharding topology addresses the service objective. Replicas and shards solve different problems, and parallelism can add communication costs.[1] Keep model-fit and availability provisioning separate from this tuning sequence. A CPU, transport or single-request latency limit needs its own diagnosis; additional replicas are not a universal remedy.</p><p>A scheduler earns its place by moving useful requests through the service, not by making a utilization display look expensive. The purchase decision belongs at the end of that measurement, after the request path has explained what is actually missing.</p><h3>Sources</h3><p>[1] <a href="https://docs.vllm.ai/en/stable/configuration/optimization">https://docs.vllm.ai/en/stable/configuration/optimization</a></p><p>[2] <a href="https://www.anyscale.com/blog/continuous-batching-llm-inference">https://www.anyscale.com/blog/continuous-batching-llm-inference</a></p><p>[3] <a href="https://vllm.ai/blog/2025-01-27-v1-alpha-release">https://vllm.ai/blog/2025-01-27-v1-alpha-release</a></p><p>[4] <a href="https://blog.vllm.ai/2023/06/20/vllm.html">https://blog.vllm.ai/2023/06/20/vllm.html</a></p><p>[5] <a href="https://docs.vllm.ai/en/stable/features/automatic_prefix_caching">https://docs.vllm.ai/en/stable/features/automatic_prefix_caching</a></p><p>[6] <a href="https://docs.vllm.ai/en/latest/features/speculative_decoding">https://docs.vllm.ai/en/latest/features/speculative_decoding</a></p><p>[7] <a href="https://docs.vllm.ai/en/stable/benchmarking/cli">https://docs.vllm.ai/en/stable/benchmarking/cli</a></p><p>[8] <a href="https://docs.vllm.ai/en/stable/design/metrics">https://docs.vllm.ai/en/stable/design/metrics</a></p><p><strong>Until next time, protect yourselves and safeguard each other</strong></p><p><em>Originally published at </em><a href="https://seanfraser.io/blog/inference-serving-optimize-the-request-path/"><em>https://seanfraser.io</em></a><em> on October 16, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efd15d2cd79b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[InfiniBand vs RoCEv2 for Inference: Start With the Traffic, Not the Logo]]></title>
            <link>https://medium.com/@cloudhacks_/infiniband-vs-rocev2-for-inference-start-with-the-traffic-not-the-logo-51a468bc91e1?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/51a468bc91e1</guid>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[infiniband]]></category>
            <category><![CDATA[rocev2]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 00:00:06 GMT</pubDate>
            <atom:updated>2026-10-02T14:57:44.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bWzCzZxZD_6OXJkT-DPXw.png" /></figure><p>My current inference machine is one Threadripper PRO 7960X with one RTX PRO 6000. It is not a cluster, and I have not run an InfiniBand-versus-RoCE bake-off. That limitation exposes the first mistake in many fabric debates: arguing about the scale-out network before proving that the workload must leave the box.</p><p>I come to this from designing SASE and RoCEv2 networks, with experience around InfiniBand-adjacent infrastructure. The lesson that carries over is unglamorous. A protocol name does not rescue a poor traffic model. Map the flows first. Then decide what deserves measurement.</p><p>The practical thesis is simple: keep communication inside the fastest scale-up domain available, and choose a scale-out fabric only after the workload, latency objective, congestion pattern, failure model, and operator capacity are explicit. InfiniBand and RoCEv2 can both carry serious AI workloads. They impose different operating costs, and neither fixes an architecture that crosses nodes unnecessarily.</p><h3>Find the Real Network Boundary</h3><p>A serving system may contain several networks, even when a diagram labels all of them “the fabric.” A single GPU does not perform a distributed collective. Multiple GPUs inside one server may communicate over PCIe or NVLink. Only traffic that crosses a host reaches the InfiniBand or Ethernet scale-out network.</p><p>That distinction matters because decode can be sensitive to communication that never touches an external switch. NVIDIA tested Gemma 2 inference with eight-way tensor parallelism on one node containing eight H100 GPUs connected by NVLink. An all-reduce of roughly 30 KB accounted for about 23% of end-to-end decode latency because dependencies prevented overlap with nearby compute. A custom one-shot collective, fused with adjacent operations, improved end-to-end decode latency by about 27%.[4]</p><p>The result demonstrates the cost of small, synchronous communication. It does not demonstrate an InfiniBand or RoCEv2 advantage. NVIDIA specifies 900 GB/s of GPU-to-GPU bandwidth for the eight-GPU DGX H100/H200 NVSwitch domain used for this class of scale-up communication.[5] Buying a 400 Gb/s external fabric would not improve a request path that remains inside that domain.</p><p>Start the design review with a boundary map:</p><ul><li>Which parallelism mode creates each flow: tensor, pipeline, expert, data, or request routing?</li><li>Does the flow stay within a GPU, cross GPUs inside a host, or cross hosts?</li><li>What is the message-size distribution, not merely its average?</li><li>Which transfers sit on the token-generation critical path?</li><li>Which flows can overlap with compute, and which force every rank to wait?</li></ul><p>Cross a node boundary because model capacity, throughput, fault-domain design, expert parallelism, or prefill/decode separation requires it. Do not cross it because the network purchase came first.</p><h3>Inference Is More Than One Packet Profile</h3><p>Training guidance often assumes large synchronized collectives where sustained bandwidth dominates. Inference produces a less uniform mix. Tensor-parallel decode can generate frequent small collectives with synchronization in the critical path. Expert parallelism can create all-to-all traffic with uneven destinations. Disaggregated serving moves KV cache between prefill and decode workers.</p><p>vLLM’s disaggregated-prefill implementation illustrates the last case. It runs separate prefill and decode instances, then uses a connector to transfer KV cache and results between them. The feature remains experimental, and its documentation explicitly states that disaggregated prefill does not improve throughput by itself.[8] Its value is isolation: teams can tune time to first token and inter-token latency separately, prevent long prefills from blocking decode, or use different parallel strategies for each stage.</p><p>That design also adds another network dependency. KV transfer has a different size and burst pattern from a small all-reduce. It may create incast at decode workers, compete with collectives, or turn a slow connector into the request bottleneck. A network selected from one bandwidth chart cannot represent all of these paths.</p><p>For each traffic class, record four things before testing a fabric: payload distribution, concurrency, synchronization behavior, and acceptable tail latency. Add background traffic and placement constraints. A clean-room benchmark with one flow can validate wiring, but it cannot predict a shared serving cluster.</p><h3>What the Large Clusters Actually Prove</h3><p>Meta built two clusters with 24,576 H100 GPUs each. One used a RoCE fabric and the other NVIDIA Quantum-2 InfiniBand; both connected 400 Gb/s endpoints.[1] Meta later reported that, after tuning, the two clusters delivered equivalent performance for its large generative-AI training workloads, and that it used the RoCE cluster to train the largest Llama 3 model.[2]</p><p>This is strong evidence against the claim that Ethernet cannot support large AI systems. It is not evidence that default RoCE settings equal a tuned InfiniBand deployment, nor is it an inference benchmark.</p><p>The tuning history carries the useful lesson. Meta reported initial communication utilization ranging from 10% to 90% on unoptimized large clusters, compared with more than 90% after changes across scheduling, routing, collective software, and the network.[1] In its RoCE engineering report, fragmented job placement and flow collisions degraded training performance by more than 30% in one case. Enhanced ECMP combined with queue-pair scaling improved AllReduce performance by up to 40% over baseline ECMP in that environment.[3]</p><p>Meta also found that default DCQCN settings performed poorly in its 400G deployment. Firmware changes created bugs and reduced visibility into congestion-notification counting, so the team proceeded without DCQCN and coordinated traffic admission through the collective library and receiver behavior instead.[3] That is not a recipe to copy blindly. It shows that congestion control, collective implementation, placement, and routing form one system.</p><p>The narrow conclusion is operational: RoCE performance is produced, not inherited. A successful deployment depends on path entropy, queue design, PFC and ECN policy, buffers, NIC firmware, collective behavior, topology-aware scheduling, and telemetry. A team that owns those layers may value Ethernet’s flexibility. A team that does not must price the engineering gap.</p><h3>Failure Surfaces Differ, but Neither Fabric Is Automatic</h3><p>RoCEv2 carries RDMA over routed Ethernet. That allows reuse of familiar switching, cabling, automation, and routing practices. It also exposes Ethernet-specific failure modes that can preserve link-up status while damaging latency.</p><p>NVIDIA’s Cumulus Linux documentation describes RoCE configuration as complex and documents lossless operation using PFC and ECN. The same interface exposes pause packets, pause duration, ECN-marked packets, queue use, and discards because those signals are part of operating the service, not optional switch trivia.[6] PFC can protect a lossless traffic class, but a bad threshold or blocked receiver can spread pauses. ECN may mark congestion without preventing a queueing tail if endpoints react too slowly or inconsistently.</p><p>NVIDIA’s NCCL troubleshooting guide separates bandwidth from latency for the same reason: bandwidth can look healthy while tail latency remains poor. It recommends point-to-point latency tests and inspection of PFC, ECN, CNP, queue drops, retransmission symptoms, and RDMA errors when stalls or long tails appear.[7] An operator needs correlated timestamps from the request layer, collective library, NIC, and switches to determine whether a token stall came from compute, synchronization, congestion, loss recovery, or a degraded link.</p><p>InfiniBand narrows the Ethernet tuning surface and offers an integrated RDMA fabric. It does not eliminate operations. The subnet manager, routing, cables, rails, firmware, PCIe topology, GPU Direct configuration, and collective settings remain possible failure points. NVIDIA’s guidance calls for checking the subnet manager, link state and rate, port-error counters, topology, and routing, then comparing bandwidth and latency rather than trusting one test.[7]</p><p>The real comparison is therefore not “complex versus automatic.” It is one failure surface versus another, measured against the skills, tooling, vendor support, and change process already present in the organization.</p><h3>Run a Serving Bake-Off, Not a Link Contest</h3><p>A useful proof of concept holds the serving workload constant and changes only the fabric-specific stack. Match GPU model and count, host topology, NIC placement, transceivers, switch oversubscription, model, quantization, parallelism, software versions, request trace, and background load. Record every deviation.</p><p>Begin with component tests to catch basic faults:</p><ul><li>Verify link state, negotiated rate, PCIe width, GPU Direct behavior, and rail mapping.</li><li>Measure point-to-point bandwidth and latency across every path class, including cross-rack paths.</li><li>Run collective tests across the real message-size distribution, not only large buffers that approach line rate.</li><li>Inject contention, a failed link, a degraded link, and an unavailable switch path.</li></ul><p>Then replay the serving workload. Report time to first token, inter-token latency, end-to-end request latency, and goodput at the target concurrency. Show P50, P95, and P99 rather than one average. Segment results by prompt and output length so changes in traffic mix do not masquerade as fabric effects.</p><p>Correlate application results with collective duration by message size, NIC throughput, queue depth, PFC pause duration, ECN marks, CNPs, drops, retransmission or retry counters, and GPU idle time. Also measure recovery: requests lost during a fault, time to restore capacity, and performance after rerouting. A fabric that wins an empty-network throughput run but produces unstable P99 under contention has not won the serving decision.</p><p>Set acceptance criteria before the test. For example, require both candidates to stay within the latency service-level objective at expected concurrency, recover from a single-link failure without manual repair, and expose enough telemetry to identify the cause of a tail regression. Compare operator hours and unresolved incidents alongside hardware cost. Otherwise the cheaper bill of materials can become the more expensive service.</p><h3>A Conditional Decision Rule</h3><p>For a single node, buy no scale-out fabric for inference. Measure the PCIe or NVLink path and optimize the collective implementation first.</p><p>For latency-sensitive tensor parallelism that must cross nodes, give InfiniBand the first proof of concept when the team wants a more integrated fabric and lacks deep lossless-Ethernet operating experience. Make RoCEv2 prove equivalent P99 inter-token latency under contention, not merely similar peak bandwidth.</p><p>For throughput-oriented serving, disaggregated prefill/decode, or an organization with strong Ethernet automation and telemetry, give RoCEv2 a controlled trial. Meta’s published work establishes that the performance ceiling can be high. The same work documents how much full-stack engineering may be required to reach it.[1][2][3]</p><p>Ultra Ethernet should influence the next test plan, not overwrite current evidence. The UEC 1.0 specification defines transport profiles with multiple ordering modes, congestion-management mechanisms, multipath path selection, and packet trimming.[9] Those mechanisms target real limitations in conventional Ethernet transports, but a specification is not an implementation benchmark. Validate the shipping NIC, switch, software, and observability stack against the workload before assigning production value.</p><p>The purchase decision follows the request path. Keep communication within the scale-up domain when the model permits. When traffic must cross nodes, evaluate the actual messages, contention, tails, failure recovery, and operator burden. The logo on the switch is an input to that work, not the answer.</p><h3>Sources</h3><p>[1] <a href="https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure">https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure</a></p><p>[2] <a href="https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta">https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta</a></p><p>[3] <a href="https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale">https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale</a></p><p>[4] <a href="https://developer.nvidia.com/blog/optimizing-for-low-latency-communication-in-inference-workloads-with-jax-and-xla">https://developer.nvidia.com/blog/optimizing-for-low-latency-communication-in-inference-workloads-with-jax-and-xla</a></p><p>[5] <a href="https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html">https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html</a></p><p>[6] <a href="https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-510/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE">https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-510/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE</a></p><p>[7] <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html">https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html</a></p><p>[8] <a href="https://docs.vllm.ai/en/stable/features/disagg_prefill">https://docs.vllm.ai/en/stable/features/disagg_prefill</a></p><p>[9] <a href="https://ultraethernet.org/wp-content/uploads/sites/20/2025/06/UE-Specification-6.11.25.pdf">https://ultraethernet.org/wp-content/uploads/sites/20/2025/06/UE-Specification-6.11.25.pdf</a></p><p><em>Originally published at </em><a href="https://seanfraser.io/blog/infiniband-vs-rocev2-for-inference"><em>https://seanfraser.io</em></a><em> on October 1, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51a468bc91e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Your Cloud Assets: CDR Cloud Detection Response]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/secure-your-cloud-assets-cdr-cloud-detection-response-5833e0249160?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/976/0*eHHH4C4TD4ZOXtQb.jpeg" width="976"></a></p><p class="medium-feed-snippet">Enhance cloud security with CDR cloud detection response. Discover how to protect your cloud assets effectively on our blog.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/secure-your-cloud-assets-cdr-cloud-detection-response-5833e0249160?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/secure-your-cloud-assets-cdr-cloud-detection-response-5833e0249160?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/5833e0249160</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cdr]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[cloud-detection]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 14:55:15 GMT</pubDate>
            <atom:updated>2024-06-22T14:55:15.838Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Log Management Best Practices: Boost Efficiency & Security]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/log-management-best-practices-boost-efficiency-security-49e1a7bf362d?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/1200/0*Rdd7cPewvXgeNi_H.jpeg" width="1200"></a></p><p class="medium-feed-snippet">Improve efficiency and security with expert insights on log management best practices. Visit our blog for more details.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/log-management-best-practices-boost-efficiency-security-49e1a7bf362d?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/log-management-best-practices-boost-efficiency-security-49e1a7bf362d?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/49e1a7bf362d</guid>
            <category><![CDATA[data-storage]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[log-management]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[siem]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Sat, 22 Jun 2024 14:28:36 GMT</pubDate>
            <atom:updated>2024-06-22T14:28:36.965Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Monitoring vs Observability: A Comparative Analysis]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/monitoring-vs-observability-a-comparative-analysis-023228ef9373?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/650/0*Ygg5SwOVs_Zsdrda.png" width="650"></a></p><p class="medium-feed-snippet">Explore the differences between monitoring vs observability in our latest blog post. Learn more about the key distinctions.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/monitoring-vs-observability-a-comparative-analysis-023228ef9373?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/monitoring-vs-observability-a-comparative-analysis-023228ef9373?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/023228ef9373</guid>
            <category><![CDATA[siem]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Sun, 19 May 2024 14:41:41 GMT</pubDate>
            <atom:updated>2024-05-19T14:41:41.416Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Simplifying Connectivity: Software-Defined WAN Solutions]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/simplifying-connectivity-software-defined-wan-solutions-1a7fff5540b7?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/600/1*xVhSj4_0MuCRo4FgtGFLpw.jpeg" width="600"></a></p><p class="medium-feed-snippet">Explore the fundamentals of software defined WAN (SD-WAN) and its benefits. Find all the information you need on our blog.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/simplifying-connectivity-software-defined-wan-solutions-1a7fff5540b7?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/simplifying-connectivity-software-defined-wan-solutions-1a7fff5540b7?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a7fff5540b7</guid>
            <category><![CDATA[ztna]]></category>
            <category><![CDATA[sase]]></category>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[sd-wan]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Tue, 07 May 2024 16:22:59 GMT</pubDate>
            <atom:updated>2024-05-07T16:28:11.038Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Master SASE Solutions: Expert Techniques]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/master-sase-solutions-expert-techniques-d9e2cc1850cb?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/1200/1*Dw_4T1QWQD2WgU0ilMoE9A.jpeg" width="1200"></a></p><p class="medium-feed-snippet">Master the art of SASE solutions with our expert guide. Elevate your knowledge and skills to optimize your network security.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/master-sase-solutions-expert-techniques-d9e2cc1850cb?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/master-sase-solutions-expert-techniques-d9e2cc1850cb?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9e2cc1850cb</guid>
            <category><![CDATA[sase]]></category>
            <category><![CDATA[casb]]></category>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[sd-wan]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Tue, 07 May 2024 15:41:40 GMT</pubDate>
            <atom:updated>2024-05-08T16:21:13.978Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Adopting Zero Trust Network Access Solutions (ZTNA)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/adopting-zero-trust-network-access-solutions-ztna-0a3cf7b46b24?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/1626/1*qrXgcoytId5rHg71W9fwRA.jpeg" width="1626"></a></p><p class="medium-feed-snippet">Discover the power of zero trust network access solutions (ZTNA) for enhanced security. Explore ZTNA technology in our blog post.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/adopting-zero-trust-network-access-solutions-ztna-0a3cf7b46b24?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/adopting-zero-trust-network-access-solutions-ztna-0a3cf7b46b24?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a3cf7b46b24</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[multi-cloud]]></category>
            <category><![CDATA[firewall]]></category>
            <category><![CDATA[ztna]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Tue, 07 May 2024 15:11:18 GMT</pubDate>
            <atom:updated>2024-05-07T15:11:18.037Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Enhance Your Cloud Security with CASB Solutions]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@cloudhacks_/enhance-your-cloud-security-with-casb-solutions-9603c8156aa6?source=rss-a3e37d7ff868------2"><img src="https://cdn-images-1.medium.com/max/768/0*r1atahaGt5R-gn1b.jpg" width="768"></a></p><p class="medium-feed-snippet">Safeguard your cloud environment with CASB security measures. Learn how CASB can help secure your organization&#x2019;s data effectively.</p><p class="medium-feed-link"><a href="https://medium.com/@cloudhacks_/enhance-your-cloud-security-with-casb-solutions-9603c8156aa6?source=rss-a3e37d7ff868------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@cloudhacks_/enhance-your-cloud-security-with-casb-solutions-9603c8156aa6?source=rss-a3e37d7ff868------2</link>
            <guid isPermaLink="false">https://medium.com/p/9603c8156aa6</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[siem]]></category>
            <category><![CDATA[casb]]></category>
            <dc:creator><![CDATA[Sean Fraser]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 15:32:09 GMT</pubDate>
            <atom:updated>2024-04-30T15:32:09.222Z</atom:updated>
        </item>
    </channel>
</rss>