<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by duongmn89 on Medium]]></title>
        <description><![CDATA[Stories by duongmn89 on Medium]]></description>
        <link>https://medium.com/@duongmn89?source=rss-60a6ca07749c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*klQjlQXyyQvIBrUc</url>
            <title>Stories by duongmn89 on Medium</title>
            <link>https://medium.com/@duongmn89?source=rss-60a6ca07749c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:51:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@duongmn89/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving multiple AI models serving challenges]]></title>
            <link>https://duongmn89.medium.com/solving-multiple-ai-models-serving-challenges-f25c74a970ad?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f25c74a970ad</guid>
            <category><![CDATA[model-serving-at-scale]]></category>
            <category><![CDATA[modelmesh]]></category>
            <category><![CDATA[ai-for-kubernetes]]></category>
            <category><![CDATA[gpu-resource-management]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 09:56:33 GMT</pubDate>
            <atom:updated>2026-01-23T09:56:33.723Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Your company has hundreds of AI models and you don’t have enough GPU resources to load all of them?</em></p><p><em>Among those hundreds of models, you don’t know which model to load, which model to unload to save GPU resources?</em></p><p><em>Some models are receiving a lot of requests and others only have a few requests per day but you don’t know how to scale them independently?</em></p><p><em>Sometimes when you want to move your models to different GPU nodes (like stronger ones), you have to configure your clients manually to point to a new endpoint?</em></p><p><em>You already tried some multiple models inference solutions like Nvidia Triton but they can’t solve your issues completely?</em></p><p>If you guys are having those questions &amp; issues, you’re more likely in the same situation as me and I hope you guys can find some useful information in this article to address your challenges.</p><p>Let’s come back to the first question: I have the model now. How do I use it in a production environment to serve inference requests?</p><p>There are different ways to do this, but basically I will divide the method into 2 main categories: <strong><em>Single Model Serving</em></strong> &amp; <strong><em>Multiple Models Serving</em></strong></p><h3><strong>Single Model Serving</strong></h3><p>So serving a model is quite easy, if you already have the model (model weight) after the training process, you can easily wrap it in some API endpoints using some framework like Fast API and use it for inference requests with only few line of codes, for example:</p><pre>from fastapi import FastAPI, UploadFile, File<br>from PIL import Image<br>import torch<br>import torchvision<br><br>app = FastAPI()<br><br>model = torchvision.models.detection.ssd300_vgg16(weights=&quot;DEFAULT&quot;)<br>model.eval()<br><br>@app.post(&quot;/predict&quot;)<br>async def predict(image: UploadFile = File(...)):<br>    img = Image.open(image.file).convert(&quot;RGB&quot;)<br>    tensor = torchvision.transforms.functional.to_tensor(img)<br><br>    with torch.no_grad():<br>        output = model([tensor])[0]<br><br>    return {<br>        &quot;boxes&quot;: output[&quot;boxes&quot;].tolist(),<br>        &quot;scores&quot;: output[&quot;scores&quot;].tolist(),<br>        &quot;labels&quot;: output[&quot;labels&quot;].tolist(),<br>    }</pre><p>It’s the fastest &amp; simplest way to bring a model to production. You can even put them in containers and using docker &amp; kubernetes to load &amp; schedule them in GPU nodes, sounds easy right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*CbiG-q2My9veRELBlS15wA.png" /></figure><p>But the issues will appear when you have more and more models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*ncfyo3S51hFgpiUk4Gat8w.png" /></figure><p>If you have less than a dozen models then I think you can go with this strategy, it’s simple and the number of models is not too much so you can manage them manually. You can remember which models are using which GPUs, which models have more replicas than others, which models need stronger GPUs, …</p><p>But let’s imagine if you have a couple of dozen models, can you continue manage all of them in this way?</p><p>I believe that you will gradually get the issues when operating serving infrastructure as I already mentioned in the beginning of this article. And you’ll need another strategy for serving all of your models.</p><h3><strong>Multiple models serving</strong></h3><p>What’s multiple models serving?</p><p>This is a solution where you’ll use a Model Runtime to serve multiple models at the same time instead of serving each model independently.</p><p>There are a lot of solutions/products that can help you use multiple models like: Nvidia Triton, Seldon Core, … and they usually can support common model frameworks like Pytorch, Tensorflow, Onnx, … so you can easily use them with your current models without changing anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*BgaFMUMrn-Tm8DOOC1aDzg.png" /></figure><p>Instead of running a lot of single model serving in dedicated or shared GPU resources, you can group them and run them in Model Runtime instances. And you only need to schedule GPU resources for Model Runtime instances only, not for all the models like it was before.</p><p>So what’s the difference between single vs multiple models serving?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*3GeUj-94exHZhYDJX-3krA.png" /></figure><p>There are a lot of advantages of using Multiple Models Serving when compared with Single Model Serving:</p><ul><li>Reduce resource overhead: each API wrapper component will consume a small number of compute resources (let say 100MB GPU), so if you have 100 models -&gt; you will need 10GB GPU only for API wrapper components. But if you use Multiple Model Serving, you only need extra resource overhead for Model Runtime once.</li><li>Advanced batching mechanism: Model Runtime Solutions usually come with some batching solutions, so you can increase inference throughput and use your GPU resources more effectively. If you want batching with your single model serving, you’ll need to implement it manually for your models.</li><li>Support load/unload a model, increase number of replicas for a specific model: Most of Model Runtime Solutions can help you choose load/unload models, so you can unload the models you’re not using, only load them when necessary. For the models that are receiving a lot of requests, you can scale them independently so you can serve more requests with better performance.</li><li>Advanced acceleration engine: Model Runtime Solutions often offer some acceleration engines like lower precision, tensor/layer fusing so they can help your models become faster without changing anything.</li></ul><p>Those are only some of the advanced features that Multiple Models Serving solutions can offer. So in my opinion, if you have to manage more than 20 models, Multiple Models Serving can definitely make your life become easier.</p><p>So if this strategy has a lot of advanced features, does it solve all of our issues? Is it a perfect solution for us? Sadly :( the answer is No.</p><h3><strong>So, what are the problems with Multiple Model Serving?</strong></h3><p>The issues still come back when you have more and more models in your infrastructures, when you don’t have enough resources to load all of them.</p><p>Of course I already mentioned that Model Runtime allows you to load/unload but you have to do it manually. So if you have ~ 100 models and several Model Runtime instances, you’ll need to:</p><ul><li>Remember mapping which models are running on which Instances.</li><li>Load/Unload manually models based on current GPU resource usage, for example: when you get an alert about high GPU mem usage, you’ll need to choose which model needs to be unloaded to reduce GPU mem usage ! But how can you identify them, you’ll ask your boss/co-workers or something like that?</li><li>And again, when you want to add/remove a Model Runtime instance or when you want to move some models to other Runtime instances because they have stronger GPU cards, you’ll need to modify client endpoints manually.</li></ul><p>So What do we need to do to get away from those issues?</p><p>The answer is we need a <strong>Dynamic Multiple Models Serving</strong> solution instead of Using Static Models Serving</p><h3><strong>What is Dynamic Multiple Models Serving ?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*dad9r4vQZ27yIUsfeP0UNQ.png" /></figure><p>Dynamic Multiple Models Serving is basically still a Multiple Models Serving solution, but it has some other components and features:</p><ul><li>It usually has a centralized router component for all the models, so no matter how much Model Runtime instances do you have, it is still under only one Router endpoint. You can add/remove Model Runtime Instances, you also can change Models between Model Runtime Instances and you don’t have to worry about changing client endpoints.</li><li>Automatic load/unload the models based on combination of current GPU resources usage and number of requests for each model. Let say if one of your GPU cards is going to run out of memory soon, it will choose some specific models to unload based on their request history using some common methods like LRU or LFU.</li><li>Auto scale the hot models: If some models are receiving more requests than others, Dynamic Multiple Models Serving should be smart enough to scale those models to higher number of replicas on other Model Runtime instances. It will make sure about the performance &amp; reliability of those important models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmJQ5tA5g3r6cydnRIAtzw.png" /></figure><p>Luckily, Dynamic Multiple Models Serving is not a theoretical solution, currently <strong><em>Kserve Modelmesh</em></strong> is a Dynamic Multiple Models Serving open source product. We’ve been applying it in our infrastructure for about a year and it works really well.</p><p>Moving from Static Multiple Models Serving to Dynamic Multiple Models Serving is really a big step, it really can help address most of our current issues quite thoroughly.</p><p>Of course, there are some challenges when applying Kserve Modelmesh in our infrastructure like how to let Modelmesh calculate resource usage correctly because Modelmesh doesn’t monitor GPU utilization directly, it needs to guess (yes, it needs to guess !), and we need to help Modelmesh guess properly, close to the real GPU utilization.</p><p>After applying and turning Kserve Modelmesh in our infrastructure, I don’t even have to care about my GPU resources anymore because Modelmesh has been doing it so well.</p><p>So Is it a final solution? Sadly :( the answer is still No.</p><p>Because Kserve Modelmesh is quite inactive for a while, so if Kserve Modelmesh becomes a dead project, I think I need to find other solutions or we have to maintain it by ourselves. But I think it will be an issue in the next 1 or 2 years, for right now I can say that I’m satisfied with my Kserve Modelmesh infrastructure :) .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f25c74a970ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Monitor GPU Compute Utilization]]></title>
            <link>https://duongmn89.medium.com/how-to-monitor-gpu-compute-utilization-a43f07179684?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a43f07179684</guid>
            <category><![CDATA[sm-activities]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[nvidia-nsight-systems]]></category>
            <category><![CDATA[gpu-utilization]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 15:47:24 GMT</pubDate>
            <atom:updated>2025-12-20T09:35:58.286Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why do we need to Monitor GPU Compute Utilization?</strong></p><p>Nowadays, GPU devices play a very important role in modern application infrastructure because they’re optimized for parallel tasks, so GPU devices now are becoming the default choices for a lot of AI/ML workloads, from hosting simple Computer Vision (CV) models, to Large Language Models (LLM), from training to inference.</p><p>So using GPU devices effectively is crucial because everyone knows that GPU devices are very expensive, and not every company can afford to buy and use GPU resources comfortably and effortlessly, without any worries.</p><p>So we all want to make sure we can make the most use of GPU resources, not wasting any of them. In order to do that, the development team needs to make sure they optimize their code to utilize GPU resources efficiently. We also need to monitor the GPU resources usage so we can verify our current usage, find bottlenecks and optimize infrastructure &amp; application again.</p><p>From my point (as an SRE team member) of view, a GPU device is quite similar to a combination of a traditional CPU + physical memory. So what I want to monitor about GPU resources usage are also based on what I usually monitor CPU and memory resource usage.</p><p>The main metrics I expect will be % utilization and saturation level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhX5uvIMVaAI6k8qaIwAsA.png" /></figure><p>It’s obvious that there are a lot of tools like nvidia-smi, nvtop, dcgm exporter seem like can provide these information. While I think we can trust GPU RAM usage metric from these tools, but after a while maintaining GPU infrastructure, I have my doubt about the metric GPU Utilization provided by these tools because of many reasons, for example:</p><ul><li>When I fraction a physical GPU to multiple smaller GPU devices for sharing purposes and putting some workloads on a small set of them, I expect to get &lt; 100% of GPU utilization but I always get 100% GPU utilization report from the physical GPU device.</li><li>When I benchmark some solutions like vLLM and reach 100% of GPU utilization, I thought I hit the limit and vLLM can’t generate more tokens/s. But I’m wrong because I still can put more input, vLLM can generate more tokens/s and GPU utilization keeps reporting the same 100%. So I ask myself: When will I reach the limit? and What the difference between those 100% GPU utilization?</li></ul><p><strong>What happens with existing GPU utilization metric?</strong></p><p>After getting confused about GPU Utilization metric for a while, I decided that I need to understand:</p><ul><li>Why it happened?</li><li>How can I get a more accurate metric about GPU compute resources?</li></ul><p>So for the first question, after researching a little bit, I realize that GPU utilization is the metric for % time busy of the whole GPU if there is any kernel that is running inside GPU.</p><p>Let’s have another simpler example, if you are an elevator manager, and you want to know about utilization of elevator usage. Let’s say if that elevator stays idle for 50% of the day, so basically elevator utilization ~ 50%. Sounds reasonable right?</p><p>But Can you know if the elevator is being used effectively or not? For example: if the elevator can handle 5 people max and it always serves only 1 person at a time, so What is the elevator utilization? Should it be 20% or 100%?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2LoxGJdW4FLny2-NxYpuhQ.png" /></figure><p>So that’s what happens with the GPU utilization metric, it only reports % time of usage, not % capacity. If there is a very small workload (let’s say 1% capacity of GPU compute) but it’s always running inside the GPU, the GPU utilization will report 100%, and we don’t know about the 99% left.</p><p>So basically, when you get 100% GPU utilization, it doesn’t mean your application is using GPU resources effectively. In fact, with this metric, we can’t tell.</p><p>So, let’s move to the 2nd question, How can I monitor the GPU compute capacity utilization? To answer that question, let’s dive in to understand GPU architecture a little bit.</p><p><strong>The GPU architecture</strong></p><p>As you might have heard the term “CUDA cores”, for example A100 has 6912 cuda cores, h100 has 14592 cuda cores. But from a system perspective, we don’t work directly with that core, we usually work with a higher abstraction name “Streaming Multiprocessors” (SM).</p><p>Actually SM is the one that closest to the definition of “core” in the CPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*GmfhdOFkQI-4EzxuIlXh3g.png" /></figure><p>Each GPU device contains multiple SMs, such as a100 contains 108 SMs, h100 contains 132 SMs. In each SM, it contains different processor units like FP32 cores, INT32 cores, Tensor cores, …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNTxZLnEfAdXxXjV0MCPug.png" /></figure><p>OK, that’s how GPU organizes its compute resources to SM and smaller units. Let’s see how GPU assign tasks (threads) to them.</p><p>GPU is known for its capacity for handling hundreds of thousands of threads in parallel. But GPU doesn’t schedule for each individual thread, it also groups them in some higher abstractions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hSmzTwQlOXNfX1njEE9yw.png" /></figure><p>Basically:</p><ul><li>A thread will be executed by a core (FP32/INT32 or Tensor Core)</li><li>Threads will be grouped into WARP that contains fixed 32 threads. WARP is scheduled by WARP scheduler in each SM (Each SM has ~ 4 WARP schedulers), so 4 WARPS can run at a clock cycle.</li><li>WARP belongs to Thread Group, a Thread Group can contain multiple WARPS, usually 16,32 or 64 WARPS. A thread Group will be executed in a SM. =&gt; 1 SM can handle multiple WARPS at the same time (4 WARPS can run at a clock cycle, other WARPS need to be wait)</li><li>The kernel grid contains multiple Thread Groups and will be executed by the entire GPU device.</li></ul><p><strong>How we can monitor GPU compute capacity utilization?</strong></p><p>After understanding GPU architecture &amp; how it assigns tasks to cores, we can answer by ourselves about the original GPU utilization:</p><ul><li>If GPU has &gt;= 1 WARP (¼ SM) busy at a time =&gt; GPU utilization will be 100% no matter what if others remain WARPS and SM are busy or free.</li></ul><p>So Do we have other metrics that can reflect better for GPU compute capacity?</p><p>Fortunately, yes we have. We have some other metrics that can help us understand better about GPU compute capacity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2YR5DY1k4cd9cxmccpV7KQ.png" /></figure><p>Ref: <a href="https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#metrics">https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#metrics</a></p><p>In short:</p><ul><li>SM activity: % SM that have at least 1 warp active (1 SM can have maximum multiple warps active, usually ~ 16, 32 or 64 depending on each GPU card).</li><li>SM occupancy: % warps handled by SM / maximum warps can handle by SM.</li></ul><p>So with these 2 metrics, they’re helpful for us more than original GPU utilization. Normally we’ll need these numbers in order to make the most out of GPU capacity:</p><ul><li>SM activity &gt;= 80%</li><li>SM occupancy &gt;= 50%</li></ul><p>But these metrics are only helpful if our application is compute bound. If it’s memory bound, these metrics don’t reflect the real GPU compute capacity because 1 warp is waiting for data movement still be considered as active =&gt; you can have SM activity and SM occupancy ~ 100% but no warp is doing compute because all of them are waiting for the data.</p><p>But at least these are still better than original GPU utilization. So how do we can get them?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOPLxNoqXSd2ntkrhqmhQQ.png" /></figure><p>Because these metrics are profile metrics, not normal metrics, so we can’t get them via NVML api. Tools like nvidia-smi, nvtop only get metrics via NVML api, so basically they can’t get these metrics.</p><p>In order to get them, we’ll need two components:</p><ul><li>Profiling interface at GPU device.</li><li>Profiling module at monitor tool.</li></ul><p>If you’re using datacenter GPU (Like A100, H100, A40, A6000, T4, …) you’re lucky because dcgm tool already has profiling tool for these GPU, you just need to enable profiling module &amp; these metrics easily.</p><p>For example, if you’re using dcgm exporter to get metrics, you just need to enable these 2 metrics in csv file</p><ul><li>DCGM_FI_PROF_SM_ACTIVE for SM Activity</li><li>DCGM_FI_PROF_SM_OCCUPANCY for SM Occupancy</li></ul><p>But if you’re using consumer GPU like RTX3090, 4090 or TITAN, these cards has profiling interface at GPU devices but dcgm tool don’t support profiling modules for them.</p><p><strong>How do we monitor SM Active and SM Occupancy on consumer GPU?</strong></p><p>As I mentioned earlier, consumer GPU also has profiling interface, so we just need to find other tools that support profiling module to get these metrics.</p><p>One of those tool is NVIDIA Nsight system, it is the tool to get metrics of application running on GPU, but we can use that tool to get GPU metrics globally with — gpu-metrics-devices option.</p><p>For example, this is an output of NVIDIA Nsight system report about SM Active of a consumer GPU card.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZcnA1ym44WSL_9Iod2aqA.png" /></figure><p>But SRE team doesn’t like this method, we can’t use GUI like that to monitor GPU metrics, so I write a simple code to get these metrics and display as prometheus format, so everyone can get metrics and write some alerts.</p><p><a href="https://github.com/duongmn89/gpu_metrics">https://github.com/duongmn89/gpu_metrics</a></p><p>This tool also uses Nsight System to get traces and then store results in a sqlite file, after that I only need to query sql and expose them as prometheus metrics.</p><p><strong>Conclution</strong></p><p>So Finally, we can have a better understanding about our GPU resources, for example this is the different results report by GPU Utilization ~ 100% and also SM Active ~ 25% and 50% for the same GPU card at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgVY2tFm8lCFkDrEVINymA.png" /></figure><p>But keep in mind these metrics are still not perfect, like I discussed earlier, they’re only useful if we can make sure that our applications are compute bound. If they’re memory bound, we can’t rely totally on them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a43f07179684" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Reliable & Fast Embedding System for AI Agent Workloads]]></title>
            <link>https://duongmn89.medium.com/building-a-reliable-fast-embedding-system-for-ai-agent-workloads-cc348b888e08?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc348b888e08</guid>
            <category><![CDATA[triton-inference-server]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tensorrt]]></category>
            <category><![CDATA[performace]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Thu, 20 Nov 2025 13:50:46 GMT</pubDate>
            <atom:updated>2025-11-24T08:50:43.544Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>TL;DR:</em></strong> This post details our journey building a reliable, in-house embedding system using NVIDIA Triton and the Jina-embeddings-v3 model. We faced GPU Out-Of-Memory (OOM) issues caused by variable document lengths. We solved this by implementing an Ensemble model with a Python preprocessing step to truncate tokens, followed by optimizing the system using TensorRT with specific profile shapes. The result: doubled inference speed, halved GPU memory usage (~3.5GB), and eliminated OOM crashes.</p><h4>What is the embedding process and why is it important?</h4><p>In common AI Agent workloads like customer support, personal assistants, or information querying, a framework named “RAG” (Retrieval Augmented Generation) is usually used so AI agents can query knowledge from user-uploaded documents. In these use cases, the AI Agent can search for information about sales programs, company policies, or internal knowledge and combine it with an LLM to return correct answers to users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*i5PZv5AK7dMeu7WyJb8igQ.png" /></figure><p>The core of the RAG framework is the <strong>Knowledge Base (KB)</strong> system, where users can upload documents and turn them into databases for the AI Agent to query. The RAG framework is important in most AI Agent workloads nowadays because it provides accurate information and reduces the chance of hallucination from the LLM.</p><p>Let’s zoom into the Knowledge Base (KB) system; this usually has 2 main components:</p><ul><li><strong>Embedding model:</strong> Provides a mechanism to convert complex data (like text, images, or audio) into numerical representations called vectors that capture the semantic meaning and relationships between data points.</li><li><strong>Vector databases:</strong> After getting numerical representations of data from the embedding model, the KB stores that information in a type of database named a vector database.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swCYlaD96e10Qui10L14AA.png" /></figure><h4>Why Build In-House?</h4><p>While using embedding models from cloud service providers like Google or OpenAI is convenient and easy to deploy, many companies are concerned about data privacy. They don’t want their documents — especially important ones — to leave their infrastructure.</p><p>That’s why deploying an in-house embedding infrastructure is a common solution for organizations that care about data privacy.</p><p>It’s not too hard to implement this solution. In my company, we are using the open-source <strong>NVIDIA Triton</strong> (Triton) and a model named <strong>jina-embeddings-v3 </strong>(Jina). They are a good combination; NVIDIA Triton is a multi-model serving solution that is reliable and high-performing, while jina-embeddings-v3 is an excellent embedding model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jo81cb5MqIQQPLAeRewbzA.png" /></figure><p>The setup is quite simple. We just need to download the model from Hugging Face (ONNX version) and create a simple config file. After that, Triton will take care of the rest.</p><pre>name: &quot;jinaai_jina-embeddings-v3-origin&quot;<br>backend: &quot;onnxruntime&quot;<br>max_batch_size : 16<br>input [<br>{<br>name: &quot;input_ids&quot;<br>data_type: TYPE_INT64<br>dims: [-1]<br>},<br>{<br>name: &quot;attention_mask&quot;<br>data_type: TYPE_INT64<br>dims: [-1]<br>},<br>{<br>name: &quot;task_id&quot;<br>data_type: TYPE_INT64<br>dims: [1],<br>}<br>]<br>output [<br>{<br>name: &quot;text_embeds&quot;<br>data_type: TYPE_FP32<br>dims: [-1, 1024]<br>}<br>]</pre><h4>What’s wrong with our Embedding system? (The Problem)</h4><p>This embedding system worked really well until the load increased. When users needed to upload multiple large documents at the same time, we faced a serious performance issue that disrupted the whole RAG framework.</p><p>Currently, we have an embedding system consisting of multiple NVIDIA Triton nodes. Each node has a separate NVIDIA L4 24GB GPU dedicated to embedding models. It looks good because the model weight is quite small (~1GB), leaving the rest (~23GB) for GPU memory allocation during the embedding process.</p><p>However, when users upload multiple large files (PDF, txt, xlsx, docx) at the same time (~10k pages), we noticed the GPU memory usage increasing constantly, quickly ending up in a <strong>GPU Out-Of-Memory (OOM)</strong> error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4UQV2jfjbAfSf5BbZ-SkNA.png" /></figure><p>When this happens, Triton can’t process any further embedding requests, and we have to restart it to make it work again. This issue made our embedding system unreliable, so we decided to find the root cause and a solution.</p><h4>Why did this happen?</h4><p>OOM usually comes from large request sizes, and GPU memory is no exception. We tried to reduce the batch size of the Jina model from 16 to smaller numbers like 8 or 4, but OOM still occurred under high load.</p><p>After digging a bit deeper into how Triton organizes the batch and <strong>dims</strong> (dimensions) size of the model, we understood the workload characteristics of embedding requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LuOF8dSb5zLsXSUtbFsaPw.png" /></figure><p>We can fix the number of requests in a batch, but the number of tokens in one request is effectively unlimited (represented as -1 in input dims). If a request contains a large number of tokens, it will consume a lot of Triton’s GPU memory.</p><p>We could set a fixed token length per request, but that would require changing all application behaviors to send a fixed length for every request. That’s doable but complicated, and it makes our application less flexible because the token length depends on the input document length, which is naturally variable and unpredictable.</p><h4>How can we make it reliable?</h4><p>After finding the root cause, we worked closely with the developer team and decided to take action on both sides:</p><ul><li><strong>Application side:</strong> Developers will set an upper limit for each request at 1024 tokens. If a document needs more than 1024 tokens, the application will split it into multiple requests.</li><li><strong>Triton side:</strong> Just in case an application doesn’t follow the upper limit, we also need to ensure we limit the maximum token length for each request on the Triton side.</li></ul><p>For the second part, there is no simple configuration solution. Fortunately, Triton has a feature named <strong>“Ensemble”</strong> to pipeline multiple models into a single model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDKKAG0SKqG30uXR_tKNvw.png" /></figure><ul><li><strong>Analogy:</strong> If we want to count cats in a picture, we usually have 2 or 3 models (resize, object separation, animal detection). The client application usually needs to handle the output of each model separately, sending requests and receiving responses multiple times.</li><li><strong>Solution:</strong> Using an Ensemble Model, Triton pipelines all models to act as one. The client sends 1 inference request and gets the final response. Every step from preprocessing to postprocessing is handled by Triton, increasing performance by avoiding unnecessary information exchange.</li></ul><p>How does this help our issue? We created a <strong>preprocessing model</strong> for the Jina model that truncates the first 1024 tokens of every request. If a request has a token length &gt; 1024, it will process only the first 1024 tokens. This prevents crashes, and since the client side also handles splitting, this acts as a safety net.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-oEib8X6f9PADwwXJcq2MQ.png" /></figure><h4>Implementation</h4><p>It’s quite easy to create a Jina preprocess model with a few lines of Python code.</p><pre>import triton_python_backend_utils as pb_utils<br>class TritonPythonModel:<br>def initialize(self, args):<br>self.max_length = 1024<br>def execute(self, requests):<br>responses = []<br>for request in requests:<br># Get input tensors<br>input_ids = pb_utils.get_input_tensor_by_name(request, &quot;input_ids&quot;).as_numpy()<br>attn_mask = pb_utils.get_input_tensor_by_name(request, &quot;attention_mask&quot;).as_numpy()<br>task_id = pb_utils.get_input_tensor_by_name(request, &quot;task_id&quot;).as_numpy()<br>truncated_ids = input_ids[:self.max_length]<br>truncated_attn = attn_mask[:self.max_length]<br># Create output tensors<br>out_tensor_ids = pb_utils.Tensor(&quot;truncated_input_ids&quot;, truncated_ids)<br>out_tensor_attn = pb_utils.Tensor(&quot;truncated_attention_mask&quot;, truncated_attn)<br>out_tensor_task = pb_utils.Tensor(&quot;task_id&quot;, task_id)<br># Build response<br>response = pb_utils.InferenceResponse(<br>output_tensors=[out_tensor_ids, out_tensor_attn, out_tensor_task]<br>)<br>responses.append(response)<br>return responses</pre><p>After deploying the ensemble model, the result was impressive. There were no OOM errors even under the same high-load scenario. We benchmarked it, and the GPU memory usage is fixed (about ~7GB) no matter how many requests are coming or how large they are.</p><p>Of course, there is a small overhead (about &lt;5% as observed), but it is a big win because we don’t need to restart Triton anymore.</p><h4>Double performance &amp; Halve GPU Memory</h4><p>Can we do even better? Yes.</p><p>One of the key advantages of Triton compared to other serving solutions is <strong>TensorRT</strong>. TensorRT is an optimizer and runtime library designed to maximize the performance of deep learning models for inference on NVIDIA GPUs.</p><p>We are using the ONNX version of the Jina model, which is usually a good format for performance compared to PyTorch or TensorFlow. But TensorRT can push it further; research suggests TensorRT can boost performance 2x-8x compared to the original.</p><p>Basically, TensorRT reduces CUDA kernel launches by combining multiple basic operations into a single, larger kernel launch. This improves performance and reduces GPU memory consumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*-YMm-Iul5XpL2EOlUZUjkA.png" /></figure><p>According to Triton documentation, for ONNX models, we can use TensorRT without changing the model simply by adding configuration to the config file.</p><pre>optimization { execution_accelerators {<br>gpu_execution_accelerator : [ {<br>name : &quot;tensorrt&quot;<br>parameters { key: &quot;precision_mode&quot; value: &quot;FP16&quot; }<br>parameters { key: &quot;max_workspace_size_bytes&quot; value: &quot;1073741824&quot; }<br>}]<br>}}</pre><h4>The “Freezing” Issue</h4><p>Surprisingly, after enabling the “accelerate engine,” when we put load on Triton, the behavior was <strong>super weird</strong>:</p><ol><li>It looked frozen for ~1 minute after booting.</li><li>Then it served requests at very high speed.</li><li>But then it froze again later before serving other requests.</li></ol><p>After reading more documentation about how graph optimization works, we realized this weird behavior came from the <strong>dynamic input token size</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bMbR_Aw8PfyKR2jfLYHYig.png" /></figure><p>TensorRT needs to calculate the optimization path for each token size range. Every time it meets a new token size outside of what it has previously computed, it needs to calculate the optimization path again on the fly.</p><h4>Solving with Optimization Profiles</h4><p>Theoretically, after meeting all ranges of token sizes, TensorRT works smoothly. We could verify this using Triton’s <strong>warmup mode</strong>, but that can take a very long time to startup (in our case, ~10 minutes).</p><pre>model_warmup [<br>{<br>name: &quot;warmup_50&quot;<br>batch_size: 16<br>inputs: {<br>key: &quot;input_ids&quot;<br>value: { dims: [50], data_type: TYPE_INT64, zero_data: false }<br>}<br>inputs: {<br>key: &quot;attention_mask&quot;<br>value: { dims: [50], data_type: TYPE_INT64, zero_data: true }<br>}<br>inputs: {<br>key: &quot;task_id&quot;<br>value: { dims: [1], data_type: TYPE_INT64, zero_data: true }<br>}<br>},<br>{<br>name: &quot;warmup_100&quot;<br>batch_size: 16<br>inputs: {<br>key: &quot;input_ids&quot;<br>value: { dims: [100], data_type: TYPE_INT64, zero_data: false }<br>}<br>inputs: {<br>key: &quot;attention_mask&quot;<br>value: { dims: [100], data_type: TYPE_INT64, zero_data: true }<br>}<br>inputs: {<br>key: &quot;task_id&quot;<br>value: { dims: [1], data_type: TYPE_INT64, zero_data: true }<br>}<br>},<br>]<br>…</pre><p>We can reduce this by caching the optimization result on disk. However, Triton has a setting where we can tell it to calculate only <strong>one optimization graph</strong> for a specific range. Since we already limited the input to 1024 tokens, we can set up Triton to calculate the optimization path once for all input lengths from 0 to 1024.</p><pre>optimization {<br>execution_accelerators {<br>gpu_execution_accelerator : [ {<br>name : &quot;tensorrt&quot;<br>parameters { key: &quot;precision_mode&quot; value: &quot;FP32&quot; }<br>parameters { key: &quot;max_workspace_size_bytes&quot; value: &quot;10737418240&quot; }<br>parameters { key: &quot;trt_engine_cache_enable&quot; value: &quot;1&quot; }<br>parameters { key: &quot;trt_profile_min_shapes&quot; value: &quot;input_ids:1x1,attention_mask:1x1,task_id:1x1&quot; }<br>parameters { key: &quot;trt_profile_max_shapes&quot; value: &quot;input_ids:16x1024,attention_mask:16x1024,task_id:16x1&quot; }<br>parameters { key: &quot;trt_profile_opt_shapes&quot; value: &quot;input_ids:16x500,attention_mask:16x500,task_id:16x1&quot; }<br>}]<br>}<br>}</pre><p>After setting the optimization config for only one range, we got an even better result: we doubled the inference speed with half the GPU memory consumption (~3.5GB). And of course, it remains reliable regardless of the data load.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVResq2OGFdne3BbjMIS-A.png" /></figure><p>So Finally, we’re having an reliable &amp; fast embedding system for our AI agent workloads now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc348b888e08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Python Performance Tip: Why String Concatenation in Loops Slows You Down]]></title>
            <link>https://duongmn89.medium.com/python-performance-tip-why-string-concatenation-in-loops-slows-you-down-4031f8f32d65?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4031f8f32d65</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 12:02:24 GMT</pubDate>
            <atom:updated>2025-04-17T12:02:24.363Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I was working on n performance issue of a project, there was a function that manipulated a string line by line and then return a new processed string.</p><p>It has been working fine with most files &lt; 10 MB, but when it processed a large file (8000 lines with 90MB size), it was very slow, it took ~ 8 minutes to finish.</p><p>The original code is:</p><pre>#python<br>newcontent = &#39;&#39;<br>with open(&#39;oldfile.txt&#39;, &#39;r&#39;) as infile:<br>    for line in infile:<br>        # Replace &#39;foo&#39; with &#39;bar&#39; and trim trailing whitespace<br>        newline = line.replace(&#39;foo&#39;, &#39;bar&#39;).rstrip()<br>        newcontent = newcontent + newline + &#39;\n&#39;</pre><p>I tracked the process time for each line, and I saw that it started quite fast for first 2000 lines (0.01 second/each), but then it was slower gradually, for last 1000 lines, it took ~ 0.2 second for each line.</p><p>I did a little research (3 minutes with chatgpt) and found the problem:</p><h3>🧠 The Problem: Strings in Python Are Immutable</h3><p>The strings in python are immutable, it can&#39;t be modified in place, then every time we add a new line, it will create a new string, copy all previous content into it, and append new line.</p><pre>newcontent = newcontent + newline + &#39;\n&#39;</pre><p>So with 8000 lines =&gt; this code will need to create 8000 increasingly large copies =&gt; <strong>very expensive</strong></p><p>This makes the time complexity <strong>O(n²)</strong> for n lines. That’s why this script starts fast… and ends up crawling.</p><h3>✅ The Fix: Use a List and Join at the End</h3><p>The simple solution I use is using list and join at the end, instead of string concatenation in loop:</p><p>My new code:</p><pre>#python<br>lines = []<br>with open(&#39;oldfile.txt&#39;, &#39;r&#39;) as infile:<br>    for line in infile:<br>        newline = line.replace(&#39;foo&#39;, &#39;bar&#39;).rstrip()<br>        lines.append(newline)<br><br>newcontent = &#39;\n&#39;.join(lines) + &#39;\n&#39;</pre><p>This is way faster because:</p><ul><li>list.append() is <strong>O(1)</strong></li><li>Python’s str.join() is highly optimized and builds the final string <strong>in one pass</strong>.</li><li>So with n lines =&gt; time complexity will be come <strong>O(n)</strong></li></ul><p>And the result is quite impressive, the new run now need only ~ 10 seconds to run on the same file instead of 8 minutes like before (~ <strong>50x faster</strong>).</p><p>In short, you go from <strong>O(n²)</strong> to <strong>O(n)</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4031f8f32d65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My performance tracing journey: Don’t fall asleep when you’re working]]></title>
            <link>https://duongmn89.medium.com/my-performance-tracing-journey-dont-fall-asleep-when-you-re-working-af95dc62c4df?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/af95dc62c4df</guid>
            <category><![CDATA[java-thread]]></category>
            <category><![CDATA[performance-tracing]]></category>
            <category><![CDATA[sleep]]></category>
            <category><![CDATA[blocking-io]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 10:33:29 GMT</pubDate>
            <atom:updated>2024-01-15T01:31:16.639Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>My performance tracing journey: Don’t fall asleep when you’re working</strong></p><p>Recently, I’ve been analyzing a high latency problem of an application in my company. Some requests from clients got a “very high” latency, about ~ 9–10 seconds, which caused a very bad user experience. The accepted value is around “3–4” seconds.</p><p>I tested with some different types of clients and realized that the latency of requests depended on the response size.</p><ul><li>Response size ~ 18 KB took ~ 450 ms</li><li>Response size ~ 650 KB took ~ 9.5 s.</li></ul><p>To identify the primary source of these latencies, I employed the Latency Analysis method for a deeper investigation to find the biggest part of those latencies.</p><p>I analyzed the first request (450 ms) and found that there was a step between two components (let’s say A send request to B and receive response from B) took about 400 ms =&gt; let’s zoom closer into this step.</p><p>By using network tracing tool, I figured out that:</p><ul><li>A send request to B and B response full content to A: ~ 50ms</li><li><strong>A does nothing and finish the connection: ~ 350 ms</strong></li></ul><p>So the most interesting part was why A needed to wait ~ 350 ms after receiving a full response from B to close the connection? Did A have a wait or sleep step?</p><p>I read the code and found the reason for that. There are a lot of sleep steps in A :D.</p><p>The following code snippet illustrates the logic of A:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*RO1DrklgwhMTb0EBhiFaZQ.png" /></figure><p>Basically, A will do a loop until can read all the content of response from B:</p><ul><li>Create a buffer 4KB</li><li>Read data from network socket to buffer</li><li><strong>Put the thread to sleep for 50ms</strong></li></ul><p>So to read a 18 KB response, A needs to do 5 loop times =&gt; sleep time ~ 250ms. For a 650 KB response, A needs to sleep 163 times ~ 8.2 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*21gzxP8vulN9R9eEbVa4bg.png" /></figure><p>And there is another thing about Thread.sleep(): it will put a thread from Running state in Blocked state. After 50ms, that thread will return to Runable state, and it need to wait OS CPU Scheduler to run, so the real sleep time can &gt; 50ms.</p><p>After I can explain the problem, the solutions are quite obviously, I can choose:</p><ul><li>Increase buffer size (4K -&gt; 8K), reduce sleep time (50ms -&gt; 5ms), the response times reduce from 450ms -&gt; 140ms for 18KB response, 9.5s to 1.6s for 650KB response.</li><li>remove the sleep step completely, I even get better results: 120ms for 18KB response, 1.3s for 650KB response.</li></ul><p>Finally, I can solve the problem and we shouldn’t sleep when doing our jobs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af95dc62c4df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My performance tracing journey: Be Careful with your Application Performance Monitor tool]]></title>
            <link>https://duongmn89.medium.com/my-performance-tracing-story-1-be-careful-with-your-application-performance-monitor-tool-fae1318a0f02?source=rss-60a6ca07749c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fae1318a0f02</guid>
            <category><![CDATA[performance-tracing]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[new-relic]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[duongmn89]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 06:17:10 GMT</pubDate>
            <atom:updated>2024-01-10T02:27:38.468Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>My performance tracing journey: Be Careful with your Application Performance Monitor tool</strong></p><p><strong><em>The problem</em></strong></p><p>Recently I’ve assigned a task to optimize performance of a specific API of an application. My manager told me that the application is not fully utilizing the server’s CPU, resulting in low RPS.</p><p>First, I need to verify about the problem by myself. The techstack of that App is quite old (RHEL 6, kernel 2.6, Java 1.6, Jboss: 4), but the hardware of server is quite good (8vCPU Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz, 24GB of memory). The app is also using newrelic agent for application performance monitor.</p><p>After few tests and benchmarks, I can confirm that there is a component in the app cannot fully utilize its resources, it’s using only 190% CPU / 8 cores (max 800%). And in that 190%, the amount for system also quite high (90% for system time, 100% for user time). If I ran two separate App instances on the same server, I got double rps result with double CPU utilization. I also pay attention to RAM and Disk I/O but I couldn’t see any bottlenecks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*Qn_tXmCve_mNJou4KWVuDQ.png" /></figure><p><strong><em>Let’s figure out</em></strong></p><p>First, I tried some common optimizations for it but they didn’t help :</p><ul><li>Increase some limit of OS: nofile, tcp settings, …</li><li>Increase Thread max of jboss, connection pool max of db</li><li>Edit nice of process to -19 to get highest priority of using CPU</li><li>Using huge page 2MB instead of 4K for java</li><li>Try some GC options</li><li>Change log console, file to ASYNC type</li><li>Increasing Heapsize</li></ul><p>Next, I tried to look at newrelic portal to find if there is anything interesting but no luck.</p><p>Then, I tried to using perf to do some performance tracing, I was not familiar with perf and I also know that tracing with java app is quite difficult because app is running inside jvm. But who know, let’s try.</p><p>After doing perf record while doing benchmark, I got some clue about somethings that do JVM Write and Java IO FileOutputStream</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*BgfNDWdlwnEnqeEdb1nfEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*UcNW53XUDJR8Jpc24hlXCA.png" /></figure><p>After doing some google searches, I know that may relate to logging of java.</p><p>So first I tried to disable logging of Java app completely in log4j configuration. But I still got the same performance result ☹</p><p>Next, I tried to look if that application could write log to other files or not, checking in /prod/pid/fd I can see that it’s still writing to log file of newrelic agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*4UaL0b2MCyDCyRweu44mTg.png" /></figure><p>After completely disabling log of newrelic agent, I got a surprise results, App can use fully CPU utilization and I got triple RPS result from client benchmark.</p><p>After some further tests, I found that current log level of newrelic agent is finest (similar to debug), if I change it to info, the problem goes aways. Debug level of newrelic agent is not necessary, so I can safely disable log or set to info default level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*gPt7UZ_UH2WUpeqiEf04Dg.png" /></figure><p>So finally, I can analyze and fix the problem completely.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fae1318a0f02" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>