<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chirawat Chitpakdee on Medium]]></title>
        <description><![CDATA[Stories by Chirawat Chitpakdee on Medium]]></description>
        <link>https://medium.com/@occlubssk?source=rss-9d1b8cd85acf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V9tDDj6aXLMZ4H_78eCTCA.jpeg</url>
            <title>Stories by Chirawat Chitpakdee on Medium</title>
            <link>https://medium.com/@occlubssk?source=rss-9d1b8cd85acf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:55:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@occlubssk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM inference engines performance testing: SGLang VS. vLLM]]></title>
            <link>https://medium.com/@occlubssk/llm-inference-engines-performance-testing-sglang-vs-vllm-cfd2a597852a?source=rss-9d1b8cd85acf------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfd2a597852a</guid>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[inference-engine]]></category>
            <category><![CDATA[sglang]]></category>
            <category><![CDATA[localai]]></category>
            <dc:creator><![CDATA[Chirawat Chitpakdee]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 05:05:53 GMT</pubDate>
            <atom:updated>2025-01-11T01:40:01.191Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlEeDn7n70ndiPd1--GrvA.png" /></figure><p>AI has reached a point where its power is undeniable. A couple of years ago, OpenAI amazed everyone with ChatGPT’s capabilities, from coding to summarizing and analyzing sentences. Today, AI is a staple tool, much like Google search has been for years. Companies are increasingly integrating AI with automation to boost efficiency. However, concerns about data privacy and security have grown, especially when sensitive customer information is handled by AI providers like OpenAI, Google, and Anthropic. People worry that sending customer data to these AI APIs might lead to it being used to train future models or, worse, being leaked.</p><blockquote>To address these concerns, the idea of “local AI” — an AI that operates entirely within an organization — has gained traction. It’s an excellent solution for mitigating privacy and security issues, but it raises important questions: How do we start with Local AI? The first step is hosting — setting up our AI/LLM model. Then, we need to decide which inference engine to use, assess its performance, and optimize our resources, like GPUs.</blockquote><p>In this post, part of the “Organization AI” series, we’ll compare the performance of two inference engines: SGLang, a new contender, and vLLM, a widely recognized leader. If you’re familiar with open-source LLM models (like those on huggingface.com), you’ve probably heard of vLLM, known for its impressive performance in tokens per second and latency. But here’s the twist: “SGLang, launched just last month, claims to consistently outperform vLLM, boasting up to 3.1x higher throughput on Llama-70B” It even competes with, and sometimes surpasses, TensorRT-LLM, as shown in the images below. Take a look, or dive into the full blog post for more details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6swWSanKcg__shSjXomf9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Awa1a4A5p0TedP00ufcIHA.png" /><figcaption><em>SGLang out outperforms vLLM in both higher throughput and lower latency, even comparable or better with TensorRT-LLM(C++).</em></figcaption></figure><p>That said, if we decide to host LLM models ourselves, it’s essential to test them with our own resources to gauge real-world performance and determine how many requests our local AI can handle. To do this, we’ll be hosting the Llama3–70B-FP8 and Llama3.1–8B models on two inference engines, SGLang and vLLM, using 2x Nvidia H100 GPUs. We’ll compare their performance in handling both small batches (sequential requests) and large batches (concurrent requests).</p><h3>Methodology:</h3><p>First, let’s break down the scenario we tested: we looked at both sequential and concurrent requests. We also limited the output tokens to 500 and set the temperature at 0.01 across all the tests. The concept for this test was inspired by the <a href="https://www.youtube.com/@TrelisResearch">@TrelisResearch</a> chanel.</p><blockquote><strong><em>Sequential Requests</em> </strong>In the sequential approach, requests are sent one after another, with a delay between each request. This method ensures that only one request is processed at a time, which can be useful for understanding the performance of individual requests without interference from others.</blockquote><p><strong><em>Example of Calculated Delay Time<br></em></strong>If you have num_requests = 5, the delays between each request will be as follows:</p><p>· Request 1: sent at time t = 0s<br>· Request 2: sent at time t = 2s<br>· Request 3: sent at time t = 4s<br>· Request 4: sent at time t = 6s<br>· Request 5: sent at time t = 8s</p><p>Here is the function for sequential requests:</p><pre>def send_request_sequentially(num_requests):<br>    for request_number in range(1, num_requests + 1):<br>        messages = [<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Write a long essay on the topic of Thailand.&quot;}<br>        ]<br>        chat_completion_request_groq(messages, client, request_number)<br>        time.sleep(2)  # Wait for 2 seconds before sending the next request</pre><blockquote><strong><em>Concurrent Requests </em></strong>In the concurrent approach, multiple requests are sent almost simultaneously, with a slight delay between the start times of each request to avoid overwhelming the system all at once. This method uses threading to handle multiple requests in parallel, which can help test the system’s performance under load and its ability to handle concurrent processing.</blockquote><p><strong><em>Example of Concurrent Requests<br></em></strong>If you have num_requests = 5 with a delay of 0.06125 seconds between each request, the delays will be as follows:</p><p>· Request 1: sent at time t = 0s<br>· Request 2: sent at time t = 0.06125s<br>· Request 3: sent at time t = 0.1225s<br>· Request 4: sent at time t = 0.18375s<br>· Request 5: sent at time t = 0.245s</p><p>Here is the function for concurrent requests:</p><pre>def send_request_every_x_seconds(num_requests):<br>    for i in range(num_requests):<br>        threading.Timer(0.06125 * i, send_request, args=(i + 1,)).start()</pre><p><strong>Here is our AI server spec:</strong></p><p>· <strong>GPU</strong>: Nvidia H100 80G × 2 (PCIe) with NV Bridge<br>· <strong>CPU</strong>: Intel 6442Y Xeon 2.6GHz, 24 Cores, 60MB Cache, 225W × 2<br>· <strong>Memory</strong>: 64GB DDR5–4800MHz ECC-RDIMM × 8</p><h3>Performance Test Results:</h3><p>We have tested in two LLM models with <strong>Llama3–70B-FP8 and Llama3.1–8B</strong></p><ol><li><strong>Llama3-70B-FP8 with 2x Nvidia H100 GPUs with (Sequential request)</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zt3WFhJXAf8WuSvsAR0Dyg.png" /><figcaption><strong>SGLang — Llama3–70B-FP8 on 50% vRAM of H100 (Sequential Request)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNioW94KhpsJRxz9fZCj6Q.png" /><figcaption><strong>vLLM — Llama3–70B-FP8 on 50% vRAM of H100 (Sequential Request)</strong></figcaption></figure><p><strong><em>For sequential requests of Llama3-70B-F8</em></strong><em>, SGLang shows slightly higher performance for sequential requests, achieving 38 tokens per second, compared to vLLM’s 35 tokens per second. This suggests that SGLang is more efficient in processing requests individually. Both engines demonstrate consistent performance across multiple requests, but SGLang has a clear edge in throughput.</em></p><p><strong>2. Llama3–70B-FP8 with 2x Nvidia H100 GPUs with (Concurrent request)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zE4qzw8pc5UMe7XU4iMYmQ.png" /><figcaption><strong>SGLang — Llama3–70B-FP8 on 50% vRAM of H100 (Concurrent Request)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbAZzH_5yXfETzBgHmqjMQ.png" /><figcaption><strong>vLLM — Llama3–70B-FP8 on 50% vRAM of H100 (Concurrent Request)</strong></figcaption></figure><p><strong><em>For concurrent requests of Llama3-70B-F8</em></strong><em>,</em><strong><em> </em></strong><em>SGLang maintains a stable performance, consistently processing 30–31 tokens per second across all requests, demonstrating its efficiency and resilience under load. In contrast, vLLM shows a noticeable decline in performance as the number of requests increases, starting at 22 tokens per second and dropping to 16 tokens per second. This suggests that SGLang is better suited for handling high concurrency, providing more reliable throughput under heavy workloads.</em></p><p><strong>3. Llama3.1–8B with 1x Nvidia H100 GPUs with (Sequential request)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EvTUF31_iOQCgYGMzCevWg.png" /><figcaption><strong>SGLang- Llama3.1–8B on 25% vRAM of H100 (Sequential Request)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hQ-fAOMrT5n7o16WpBWYSQ.png" /><figcaption><strong>vLLM- Llama3.1–8B on 25% vRAM of H100 (Sequential Request)</strong></figcaption></figure><p><strong><em>For sequential requests of Llama3.1–8B</em></strong><em>, Both engines demonstrate high and consistent performance. SGLang processes requests at around 91 tokens per second, while vLLM handles them at 80 tokens per second. SGLang exhibits slightly better throughput, suggesting it can handle individual requests more efficiently compared to vLLM under these conditions.</em></p><p><strong>4. Llama3.1–8B with 1x Nvidia H100 GPUs with (Concurrent request)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrbU22sYh0QeMfJIJ7KKuA.png" /><figcaption><strong>SGLang- Llama3.1–8B on 25% vRAM of H100 (Concurrent Request)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZIzqn8VRCJ2Qjnp0LmMpQ.png" /><figcaption><strong>vLLM- Llama3.1–8B on 25% vRAM of H100 (Concurrent Request)</strong></figcaption></figure><p><em>SGLang maintains a steady performance, consistently processing around 75–78 tokens per second, showcasing its robust handling of high-concurrency workloads. In contrast, vLLM starts at 37 tokens per second but gradually decreases to around 35 tokens per second, highlighting a significant performance gap. This suggests that SGLang delivers nearly double the throughput compared to vLLM under concurrent request conditions, making it a more efficient choice for scenarios requiring high concurrency.</em></p><h3><strong>Conclusions:</strong></h3><blockquote>Overall, SGLang consistently outperforms vLLM in both sequential and concurrent request scenarios. The differences are especially stark under concurrent loads, where SGLang’s ability to maintain stable throughput highlights its superior scalability and robustness. These findings suggest that SGLang is the better choice for applications requiring high concurrency and efficient processing of large volumes of requests.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfd2a597852a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>