<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nadeesha Gamage on Medium]]></title>
        <description><![CDATA[Stories by Nadeesha Gamage on Medium]]></description>
        <link>https://medium.com/@nadeesha.g?source=rss-ab90b3c91bad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nDErahdSI8BO_0YTiQuwBQ.jpeg</url>
            <title>Stories by Nadeesha Gamage on Medium</title>
            <link>https://medium.com/@nadeesha.g?source=rss-ab90b3c91bad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:49:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nadeesha.g/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RAG from First Principles: Retrieval, Chunking, and Generation with Open Source Tools]]></title>
            <link>https://fwddeployed.dev/rag-from-first-principles-retrieval-chunking-and-generation-with-open-source-tools-248ec784b0cf?source=rss-ab90b3c91bad------2</link>
            <guid isPermaLink="false">https://medium.com/p/248ec784b0cf</guid>
            <dc:creator><![CDATA[Nadeesha Gamage]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 06:57:37 GMT</pubDate>
            <atom:updated>2026-08-07T16:34:14.323Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9sFxWDEsuhzhXDP77T5-DQ.png" /></figure><p>Ask an open-source model who won the 2026 FIFA World Cup and one of two things happens. If the tournament finished after its training cut-off and the model has no tool access, it either says it doesn’t know, or it invents a winner and states it with total confidence.</p><p>The same failure shows up with your own documents. An LLM has never seen your leave policy, your incident run books, or your customer contracts. Ask it about them and you get a plausible sounding answer assembled from nothing.</p><p>Retrieval Augmented Generation (RAG) fixes this by retrieving the chunks most relevant to a user’s question from an external knowledge base, then inserting them into the prompt sent to the LLM. The model answers from text you supplied, not from memory or guesswork which also means the answer is traceable back to a source document.</p><p>This blog covers the concepts and the open source options at each stage. The working implementation lands in part 2, so there’s no code here; part 3 covers the advanced topics related to RAG and few of it is briefly discussed at the end.</p><h3>How RAG solves this</h3><p>A RAG system has two processes.</p><p><strong>Data ingestion</strong> runs ahead of time. Documents are parsed, broken into chunks, and run through an embedding model to produce vector embeddings. Those embeddings are stored in a vector store. I won’t explain embeddings from scratch here, Datacamp has a good primer on this and definitely worth reading if you new to this [1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCf4kkToF96ncMPh0z3QFQ.png" /></figure><p><em>Documents are parsed into structured elements, chunked, embedded, and written to the vector store along with their source metadata.</em></p><p><strong>Inference</strong> runs per question. The user’s question goes through the same embedding model. That question embedding is used to search and retrieve the vector store for the closest chunks, these are the ones that are semantically nearest to the question. These retrieved chunks are assembled into a prompt alongside the original question and sent to the LLM, which generates the answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2eOVz-QqjS4Hj3B6PlueiQ.png" /></figure><h3>When RAG fits</h3><ul><li>Your LLM needs knowledge that is specific or proprietary to your organisation: policies, procedures, run books, guidelines, anything not in the public domain and therefore not in the training data.</li><li>You’re dealing with events or data that post-date the model’s training cut-off.</li><li>You need answers traceable to a source. Citation is a first-class benefit of RAG.</li></ul><h3>When RAG does not fit</h3><ul><li><strong>Public, historic facts.</strong> If you’re only asking about past US presidents, the model already knows. Adding retrieval adds latency and failure surface for nothing.</li><li><strong>Aggregate questions over the whole corpus of data.</strong> “How many of our contracts expire in Q3?” cannot be answered by retrieving five chunks, because no single chunk contains the answer.</li><li><strong>Structured data.</strong> If it lives in tables with a schema, querying the schema would provide better, faster results rather than trying to embed them in a vector store.</li><li><strong>When Latency is a concern. </strong>Embedding the query and searching the index adds a round trip before the LLM even starts.</li></ul><h3>Alternatives to RAG</h3><p>The most obvious alternative is to skip retrieval and put the whole document in the context window. This works fine for a handful of documents. It stops working at hundreds or thousands, and long contexts cost tokens on every single request.</p><p>The other alternative is fine-tuning. It’s more tedious, slower and more expensive, and it teaches the model behaviour and style far better than it teaches facts. If your objective is to give the model knowledge it doesn’t have, fine-tuning is overkill.</p><h3>Components and choices</h3><p>Lets look at the RAG pipeline and explore the component involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEBAlk7VMl_zDWuMdoRYfg.png" /></figure><h3>Parsing</h3><p>Parsing takes a raw document, a PDF or a Word file and turns it into typed, ordered elements you can chunk. Parsing is an important step and would determine the accuracy of the chunks created in the next step that would feed the entire RAG pipeline.</p><p>Open source options:</p><ul><li><a href="https://github.com/docling-project/docling"><strong>Docling</strong></a> <strong>(MIT)</strong> — good general-purpose choice, strong on tables and document structure.</li><li><a href="https://github.com/Unstructured-IO/unstructured"><strong>Unstructured</strong></a><strong> (Apache 2.0)</strong> — broad format coverage. Check the current licence terms against your use case before committing; the open source library and the hosted API are licensed differently.</li><li><a href="https://github.com/pymupdf/pymupdf"><strong>PyMuPDF</strong></a> (<strong>AGPL-3.0</strong>) — specialises in PDFs and is very fast. Note the licence: AGPL is fine for internal or open source use, but a closed-source product or SaaS that processes user PDFs on your servers triggers the source-disclosure obligation.</li></ul><h3>Chunking</h3><p>Chunking breaks a parsed document into smaller pieces at ingestion time. Three common strategies:</p><p><strong>Fixed-length chunking</strong> splits text at a fixed size. A 512 token chunk with a 50 token overlap is a reasonable starting point. It’s the simplest option and the fastest to run. Its weakness is that it splits on character counts, not meaning, so it will happily cut a policy clause or a table in half and leave you retrieving incomplete chunks. The overlap exists to soften this, the tail of one chunk is repeated at the head of the next.</p><p><strong>Hierarchical chunking</strong> indexes small, precise chunks for retrieval but returns the larger parent section to the LLM. You get precise matching and complete context. Ideal for anything with a nested structure such as legal documents, technical manuals, anything with numbered sections.</p><p><strong>Semantic chunking</strong> places boundaries based on meaning rather than length. It embeds each sentence and splits where the similarity between adjacent sentences drops below a threshold. It costs more than fixed-length chunking because you run the embedding model over the whole document before you chunk it.</p><p>Open source options:</p><ul><li><a href="https://github.com/docling-project/docling"><strong>Docling</strong></a> (MIT) — the natural choice if you’re already using it for parsing.</li><li><a href="https://github.com/feyninc/chonkie"><strong>Chonkie</strong></a> (MIT) — purpose-built chunking library, covers all three strategies above.</li></ul><h3>Embedding</h3><p>Once you have chunks, an embedding model turns each one into a vector. An embedding is a fixed-length array of floats representing its meaning. Chunks with similar meaning land close together in that vector space, so embedding a question and searching for its nearest neighbours returns chunks that are about the same thing.</p><p>Here’s a worked example, simplified to 4 dimensions (real models produce 384 to 1536):</p><pre>chunk 1 — &quot;the cat sat on the mat&quot;                → [-0.5178, -0.4959,  0.2511,  0.6504]<br>chunk 2 — &quot;a feline rested on the rug&quot;            → [-0.0361, -0.9406,  0.0115, -0.3374]<br>chunk 3 — &quot;quarterly revenue grew by twelve percent&quot; → [ 0.3266, -0.4828,  0.3377, -0.7390]</pre><pre>Cosine similarity:<br>  cat/mat  vs  feline/rug  →  +0.2685<br>  cat/mat  vs  revenue     →  -0.3255</pre><p>What matters here is the <em>relative</em> distance, not the absolute number: chunk 1 and chunk 2 describe the same scene in different words and score positive, while chunk 1 and chunk 3 are unrelated and score negative. With only 4 dimensions the scores are compressed; a real embedding model would put the first pair closer to 0.8.</p><p>Open source embedding models worth looking at:</p><ul><li>nomic-embed-text v1.5 / v2 (768 dimensions)</li><li>Qwen3-Embedding (available in several sizes, up to 8B)</li><li>EmbeddingGemma-300M</li><li>all-MiniLM-L6-v2 (384 dimensions, the lightweight default)</li></ul><p>Three key points to highlight. First, your embedding model is not your inference model. They’re trained for different jobs and are not interchangeable. Second, most small embedding models run comfortably on CPU. Third, changing your embedding model means re-embedding your entire corpus of data. Questions and documents must go through the same model or the vectors aren’t comparable.</p><h3>Vector stores</h3><p>A vector store is a database built to store high-dimensional embeddings and return the ones closest to a query vector. Query for “pets” and it returns “dogs” and “cats” before “coffee”.</p><p>Below image depicts how vector store would store data. As you could see cats and dogs sits closer to each other compared to words like coffee, mountain or universe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJIPDYG9vBRJnRrYjCIl8A.png" /></figure><p>Alongside the vector, a vector store keeps the original chunk text. These chunks are the ones that you would send to the LLM. You would also store metadata about the document name, page number and the location. The metadata is what lets you cite sources and, if needed, pull the full document.</p><p>A query to a vector store returns the matching chunks, a relevance score you can rank on, and that metadata.</p><p>Open source options:</p><ul><li><a href="https://github.com/chroma-core/chroma"><strong>ChromaDB</strong></a> (Apache-2.0) — easiest to start with, runs embedded.</li><li><a href="https://github.com/qdrant/qdrant"><strong>Qdrant</strong></a> (Apache-2.0) — strong filtering, scales well.</li><li><a href="https://github.com/pgvector/pgvector"><strong>pgvector</strong></a> (PostgreSQL licence) — an extension to Postgres. If you already run Postgres, this is one less system to operate.</li></ul><h3>Inference LLM</h3><p>The retrieved chunks and the question are assembled into a prompt and passed to an LLM, which now has the context it needs to answer.</p><p>Some open-weight models you can serve yourself are given below</p><ul><li><strong>Gemma 4 </strong>(Apache-2.0) — Now fully Opensource</li><li><strong>GLM-4.5-Air</strong> (MIT)</li><li><strong>Llama 4 Scout</strong> (Meta’s Llama Community Licence) — open weights under Meta’s Llama Community Licence, which carries acceptable-use terms and a monthly-active-user threshold.</li></ul><p>If you’re running locally you also need a serving layer. Ollama and vLLM are two opensource options. I covered model serving in depth in an earlier post [2].</p><h3>The harness</h3><p>The harness is the orchestration layer that ties everything together. It receives the question, embeds it, queries the vector store, assembles the prompt from the question plus the retrieved chunks and calls the inference LLM, and returns the answer. An illustration on how it all works together is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ak4k9jkCObEv1ToEMpdbBA.png" /></figure><p>In practice a harness can be an application written in Python or TypeScript, or an agent built on a framework like LangChain or Strands.</p><h3>Trade-offs</h3><p>Given below are some of the trade offs that you need to consider when using RAGs.</p><ul><li><strong>Latency.</strong> RAG adds an embedding call plus a vector search before the LLM starts, and a longer prompt means more prefill.</li><li><strong>Cost.</strong> Longer prompts on every request. Plus the one-off cost of embedding the corpus, and the cost of doing it again whenever you change embedding models.</li><li><strong>Complexity.</strong> You’ve added a parser, a chunker, an embedding model, a vector store and an orchestration layer. Each is a thing that can break.</li><li><strong>Operational burden.</strong> The vector store is a second stateful system to back up, monitor and scale. Someone has to keep the index in sync with the source documents.</li><li><strong>Accuracy ceiling.</strong> RAG doesn’t make the model smarter. It only changes what’s in the context. Bad retrieval produces confidently wrong answers, and the model gives you no signal that retrieval failed.</li></ul><h3>Failure modes</h3><ul><li><strong>Semantically similar but factually wrong</strong> chunks retrieved. The common scenario is when the retriever finds the <em>2023</em> policy when the question is about <em>2025</em>.</li><li><strong>Chunk boundaries splitting meaning.</strong> A table or a numbered clause cut in half, so the retrieved chunk is incomplete and the answer is wrong in a way that reads fine.</li><li><strong>Stale index.</strong> Source document changed but the the index doesnt change which makes your system retrieve old information from your vector store.</li></ul><h3>What more can be done</h3><p>The flow above is basic RAG. There’s a lot you can add on either side of retrieval, its beyond the scope of this blog and I will cover these properly in part 3 of this blog series. Some pointers on what more can be done are given below.</p><p><strong>Pre-retrieval</strong></p><ul><li><em>Query expansion and decomposition.</em> Expand a short query, or decompose a multi-part query into several. Expansion helps when the query is too terse . Decomposition helps when the user asked three things at once.</li><li><em>HyDE (Hypothetical Document Embedding).</em> Have the LLM write a hypothetical answer first, then embed <em>that</em> for retrieval. Works well when the query is much shorter than the documents you’re searching.</li></ul><p><strong>Post-retrieval</strong></p><ul><li><em>Reranking.</em> Retrieve a wider set of chunks, then rerank them by relevance with a cross-encoder or an LLM before the top few go into the prompt.</li></ul><h3>What’s next</h3><p>This post covered RAG from first principles: the concept, and the open source options at each stage of the pipeline. Part 2 is a working implementation that guides you step by step, running against an open-weight model, with the repo linked. Part 3 covers the advanced retrieval techniques and how you can finetune the entire pipeline to get better results out of your RAG pipeline.</p><h3>References</h3><ol><li>Datacamp, <em>What is a Vector Embedding?</em> — <a href="https://www.datacamp.com/blog/vector-embedding">https://www.datacamp.com/blog/vector-embedding</a></li><li><em>Self-hosting LLMs In-House: A Practical Guide to Hardware, Serving Frameworks and Guardrails</em> — <a href="https://medium.com/fwddeployed/self-hosting-llms-in-house-a-practical-guide-to-hardware-serving-frameworks-and-guardrails-7f15099ec1f1">https://medium.com/fwddeployed/self-hosting-llms-in-house-a-practical-guide-to-hardware-serving-frameworks-and-guardrails-7f15099ec1f1</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=248ec784b0cf" width="1" height="1" alt=""><hr><p><a href="https://fwddeployed.dev/rag-from-first-principles-retrieval-chunking-and-generation-with-open-source-tools-248ec784b0cf">RAG from First Principles: Retrieval, Chunking, and Generation with Open Source Tools</a> was originally published in <a href="https://fwddeployed.dev/">FwdDeployed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Hosting LLMs In-House: A Practical Guide to Hardware, Serving Frameworks, and Guardrails]]></title>
            <link>https://fwddeployed.dev/self-hosting-llms-in-house-a-practical-guide-to-hardware-serving-frameworks-and-guardrails-7f15099ec1f1?source=rss-ab90b3c91bad------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f15099ec1f1</guid>
            <category><![CDATA[guardrail]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-gateway]]></category>
            <dc:creator><![CDATA[Nadeesha Gamage]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 05:51:40 GMT</pubDate>
            <atom:updated>2026-08-06T05:52:29.562Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aycozM7qS6kSIJK_39to_g.png" /></figure><h3>The Problem</h3><p>An organization that wants to put a large language model (LLM) into production can start by calling a cloud-hosted model from an AI lab such as Anthropic or OpenAI. That works well until one of three things happens: the data being sent to the model can’t leave the organization, the per-token cost of a frontier model becomes too expensive as volume grows, or the team wants to fine-tune a model on its own data.</p><p>Self-hosting is turning an open source LLM into a live, queryable service you run yourself. But running your own model is not easy, especially when the system has to be production-ready for enterprise traffic.</p><h3>When This Pattern Fits</h3><p><strong>Data can’t leave the organization.</strong> The model has to run inside your own boundaries rather than sending organizational data to an external provider.</p><p><strong>Token costs at your volume outweigh infrastructure costs.</strong> Self-hosting isn’t free; there is infrastructure and engineering cost to set it up instead of just calling an endpoint. But once it’s running, you don’t pay an incremental cost per query or per token.</p><p><strong>You want to fine-tune the model on your own data.</strong> Fine-tuning requires access to the model weights, which commercial APIs don’t give you. I’ll cover fine-tuning in a later post.</p><h3>When It Does Not Fit</h3><p>This pattern doesn’t fit when none of the three conditions above apply. Low traffic is the clearest disqualifier: at low volume, the setup and infrastructure costs outweigh anything you would have paid per token to a hosted model.</p><h3>The Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wO9DgascuPDAeVQy_wtWyQ.png" /></figure><p>The diagram above shows the typical component architecture of a self-hosted LLM setup. A request comes in and hits the <strong>AI gateway</strong>, which enforces guardrails on the incoming request. The gateway passes the request to the <strong>harness</strong>, which may call a <strong>RAG</strong> (retrieval-augmented generation) pipeline for additional context, call <strong>tools</strong> to perform specific tasks, or both, before calling the <strong>LLM service</strong>. An <strong>observability</strong> tool monitors all of it.</p><p>This is a basic setup and a good starting point. A real deployment looks more complicated, with calls going back and forth across the RAG pipeline, the tools, and the model.</p><h3>Components and Choices</h3><h3>Infrastructure and Models</h3><p>This is where most of the cost sits: the CPU/GPU, network, and other core infrastructure needed to run the model. Requirements vary widely by model. The tiers below focus on models that are realistic for a small-to-midsize company, not ones that need a full datacenter cluster.</p><p><strong>Entry tier — 8–12 GB GPU (~$250–500)</strong></p><ul><li><strong>Gemma 4 E2B / E4B</strong> — 2–4B effective parameters, ~5 GB at Q4. Runs on almost anything, even CPU-only.</li><li><strong>Qwen3 8B</strong> — 8B dense, ~5 GB. Matches the previous generation’s 14B model on quality while using half the memory.</li><li><strong>Mistral 7B</strong> — 7B dense, ~4 GB. The efficient, reliable baseline, and a community favorite since release.</li></ul><p><strong>Mid tier — 32–48 GB, RTX 5090 or dual RTX 3090/4090 (~$2,000–4,000)</strong></p><ul><li><strong>Llama 4 Scout (Q4)</strong> — 109B total / 17B active, ~61 GB. Needs dual GPUs or 64 GB+ of unified memory. The 10-million-token context window is the standout feature.</li><li><strong>Qwen3 72B</strong> — 72B dense, ~36 GB at INT4. Fits on a single RTX 5090.</li><li><strong>Devstral 2 (123B)</strong> — ~64-72 GB at Q4. The ceiling of Mistral’s open coding lineup.</li></ul><p><strong>Top tier — workstation/prosumer, 64–128 GB, dual/quad GPU rig or Mac Studio (~$4,000–7,000)</strong></p><ul><li><strong>GLM-4.5-Air</strong> — 106B total / 12B active, ~67 GB, and aggressive 3-bit quants can squeeze onto a 64 GB Mac.</li><li><strong>Mistral Small 4 (119B)</strong> — ~74 GB. Strong coding and reasoning quality, and realistic on three RTX 4090s or a large-memory Mac Studio.</li></ul><p><strong>A licensing note.</strong> “Open source” and “open weights” are not the same thing, and the difference matters if you’re deploying commercially. Qwen3, Mistral 7B, and GLM-4.5-Air ship under permissive licenses (Apache-2.0 and MIT). Llama 4 and Gemma 4 ship under vendor licenses with usage restrictions — Meta’s and Google’s community license terms. Check the current terms against your deployment scale before committing.</p><h3>Serving Framework</h3><p>The serving framework, also called the inference engine, sits between your application and the model. It loads the model and runs it efficiently: batching requests, managing the KV cache, and handling concurrency. Think of the serving framework as a kitchen and the model as the chef. The kitchen gives the chef the tools and setup to cook as many dishes as possible, as fast as possible.</p><p>Given the hardware tiers above, two options cover most cases:</p><ul><li><strong>Entry-tier models: Ollama.</strong> Ollama provides a UI and a CLI for downloading and running models, plus an API for invoking them programmatically. It’s the simplest way to get a model running locally.</li><li><strong>Mid- and top-tier models: vLLM.</strong> vLLM is built for higher concurrency and is the better choice for serious enterprise workloads.</li></ul><h3>Harness</h3><p>The harness is the orchestration layer between the application and the model. For a simple query, it can be a straight pass-through. But if the query needs information from an internal knowledge base or an external system such as Salesforce, the harness pulls that information from a RAG pipeline or makes a tool call.</p><h3>RAG</h3><p>Retrieval-augmented generation (RAG) pipeline loads the information needed to answer a query into the message context. Consider a user asking about the company’s leave policy. The underline model doesn’t know the leave policy, because it’s specific to your company. RAG loads that document into the message context so the model has what it needs to answer.</p><p>RAG pipelines itself is a much broader topic, and I’ll cover it in a separate post.</p><h3>Tools</h3><p>Tools let the harness pull information from other systems. Take the same leave-policy example: a tool call can use MCP or a direct API call to reach your HR system, pull the user’s available leave, and even let the user apply for leave through a prompt.</p><h3>AI Gateway</h3><p>Serving requests is only part of running an LLM responsibly. An AI gateway covers the rest:</p><ul><li><strong>Guardrails</strong> — control what goes into the model and what comes back out, so responses stay grounded in your organization’s own data and the model is used for its intended purpose. PII redaction matters less here than with an external provider, since the model stays inside your environment, but it’s still worth considering.</li><li><strong>Authentication and authorization</strong> — gate access to the chat interface and the API through an identity provider before users or systems can query the model.</li><li><strong>Request and response logging</strong> — log everything for auditing.</li><li><strong>Semantic caching</strong> — cache and reuse responses to similar questions to improve latency and cut redundant compute.</li><li><strong>Multi-model routing</strong> — route each query to the model best suited to it. Not every task needs your largest model.</li></ul><p>Two open source options cover most of this: <a href="https://wso2.com/api-manager/usecases/ai-gateway/">WSO2 AI Gateway</a> and <a href="https://www.litellm.ai/">LiteLLM</a>.</p><h3>Observability</h3><p>Observability traces and logs requests and responses, and gives you a view of how the model is used across the organization. Paired with GPU, CPU, and RAM utilization on the serving host, it gives end-to-end visibility into how the model is being used and how hard the infrastructure is working. <a href="https://langfuse.com/">Langfuse</a> and <a href="https://arize.com/">Arize</a> are the leading open source options, and both offer a generous free tier to start with.</p><h3>Trade-offs</h3><p>Running your own model isn’t free. There’s infrastructure and engineering cost to set it up that a commercial API call doesn’t have. Once it’s running, though, you don’t pay an incremental cost per query or per token, which changes the economics at volume.</p><h3>Failure Modes</h3><p>Availability matters for a production LLM service. At minimum, run two or more instances of the model so a single host failure doesn’t take the service down.</p><p>If running redundant GPU instances isn’t affordable, the alternative is to queue requests while the model is unavailable and serve them once it’s back online. That requires message orchestration, which is worth its own post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f15099ec1f1" width="1" height="1" alt=""><hr><p><a href="https://fwddeployed.dev/self-hosting-llms-in-house-a-practical-guide-to-hardware-serving-frameworks-and-guardrails-7f15099ec1f1">Self-Hosting LLMs In-House: A Practical Guide to Hardware, Serving Frameworks, and Guardrails</a> was originally published in <a href="https://fwddeployed.dev/">FwdDeployed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>