<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emin Mammadov on Medium]]></title>
        <description><![CDATA[Stories by Emin Mammadov on Medium]]></description>
        <link>https://medium.com/@emin.mammadov?source=rss-d88c3f91e764------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8XmFqciIrT3USYm7</url>
            <title>Stories by Emin Mammadov on Medium</title>
            <link>https://medium.com/@emin.mammadov?source=rss-d88c3f91e764------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:54:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@emin.mammadov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Model Is the Easy Part: What Geotab Learned From Self-Hosting Large Language Models]]></title>
            <link>https://medium.com/geotab/the-model-is-the-easy-part-what-we-learned-from-self-hosting-large-language-models-5c4d76eaf993?source=rss-d88c3f91e764------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c4d76eaf993</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Emin Mammadov]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 11:01:02 GMT</pubDate>
            <atom:updated>2026-10-06T17:10:05.950Z</atom:updated>
            <content:encoded><![CDATA[<p>In April 2026, ChatGPT, Claude, and Gemini went down at roughly the same time [1]. If your product leaned on any of them, it didn’t matter that you’d “diversified” across three vendors — they all blinked out together.</p><p>Two months later, the US Commerce Department ordered Anthropic to restrict foreign-national access to two of its newest models. Because nationality cannot be filtered in real time at the API layer, the only way to comply was to pull the models entirely, for every customer. The notice window was roughly 90 minutes [2]. One of the models had been generally available for just three days.</p><p>That’s when “<em>should we self-host?</em>” stops being a cost question and becomes a resilience question.</p><p>At Geotab, where our platform analyzes data from hundreds of thousands of vehicles around the clock, that question had a clear answer.</p><h3>Independence is the point</h3><p>Let me start with the money, because it’s the easy sell. Hosted APIs charge per token, so your bill scales linearly with every user, prompt, and context window. Self-hosting inverts that: you pay a fixed price for a GPU, and the marginal cost of the next token falls to nearly zero. Prefix caching sharpens the edge. Most production traffic shares long prefixes — system instructions, a codebase, a chat history. Hosted APIs give you a discount on those cached tokens. On your own hardware, a cache hit is free.</p><p>However, it’s not only about the budget. The reasons that matter don’t fit on a spreadsheet:</p><ul><li><strong>Control over versions and behavior</strong>, instead of unannounced updates.</li><li><strong>Privacy</strong> — sensitive prompts never leave our perimeter.</li><li><strong>Resilience</strong> — an open-weight model on your own servers is the one option nobody else can switch off.</li></ul><p>The gap between open and closed models has narrowed to a several-month lag. When the models are that close, the model stops being the battleground. The <em>platform that serves it</em> becomes the whole game.</p><p>So this is where we focused our efforts: <strong>building a platform that allows us to onboard open-source models quickly, while ensuring that no data leaves our control.</strong></p><p>For Geotab, the motivation is specific. Our platform processes telematics data — trips, driver behavior, location, vehicle health — on behalf of fleet operators who treat that as competitive intelligence. Self-hosting ensures that data processed by the model stays inside of our perimeter. It also means GIA, Geotab’s internal AI assistant, runs on infrastructure we own — not a commercial API that can be rate-limited, updated without notice, or taken offline. For a platform serving 24/7 fleet operations, that control is not a nice-to-have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wv3EHW06mzDvTvU1YK9G1w.png" /></figure><h3><strong>Smart routing</strong></h3><p>When it comes to LLMs, standard load balancing techniques such as round-robin are actively wrong. Round-robin assumes two things: that requests are interchangeable, and that your backends are stateless. Both are false here. One request is a 256-token prompt and the next can be even longer prefill costing orders of magnitude more compute. And each vLLM replica isn’t a blank slate — its KV cache already holds the prefixes it recently processed.</p><p><strong>Cache affinity</strong> therefore becomes the cornerstone of proper routing, and it’s the heart of smart routing. By sending a request to the exact replica that already holds its prefix, the GPU skips the expensive prefill phase entirely, saving on computational costs.</p><p>We built it with the Gateway API Inference Extension and the llm-d router as our Endpoint Picker. Every request runs through a weighted scoring pipeline: a prefix-cache scorer (weighted highest, at <strong>2.0</strong>) finds which replica already holds the prompt, while KV-utilization and queue scorers stop everything from being crammed onto one hot replica. Tuned right, we’ve measured cache-hit scores reaching a perfect <strong>1.0</strong>.</p><p><strong>The smart routing layer exposed one of the most challenging problems: silent failures</strong>. The router and vLLM have to agree on the KV-cache block size. vLLM emits cache events; the router hashes incoming prompts into blocks of the same size to figure out who holds what. The challenge is what happens if those two numbers are not configured to match. Nothing errors. The router simply resolves prefix matches badly, cache affinity degrades, and you’re paying for an inference-aware router that is silently routing blind.</p><p>We only trust it because we verified the other direction: watching cache-hit scores climb to 1.0 confirmed the two sides were genuinely aligned. The lesson generalizes: when a system fails silently, “no errors” is not evidence it works. <em>Reviewing logs and running load tests</em> is the only way to detect issues like this.</p><h3>Making it feel like just another model</h3><p>None of this matters if it’s painful to use. The whole platform hides behind our GenAI Gateway — a proxy that speaks the OpenAI API — so a self-hosted large language model shows up right beside commercial ones as an interchangeable model before it was widely available on the major hosted platforms.</p><p>For our coding agent, OpenCode, adoption meant changing a URL and a model name. That’s the bar. If switching to your own infrastructure is harder than switching a config value, people won’t do it.</p><p>We did not keep security simple. Instead of static API keys between the gateway and the inference endpoint, we validate short-lived, identity-based tokens at the network edge: only a small set of authorised service identities can reach the inference path, and everything else is rejected before it ever touches a GPU.</p><h3>You can’t operate what you can’t see</h3><p>We split observability into three layers for three audiences. Business metrics go to leadership: cost per million tokens, adoption, cost versus commercial APIs. Hardware health comes from DCGM, where a hardware fault code is our single most important fault signal. And vLLM’s own serving metrics give us the truth about latency: time-to-first-token, time-per-output-token, queue depth, and KV-cache pressure.</p><p>One design principle this work taught us is worth stating plainly. A heartbeat check answers one question: <em>is the process alive?</em> For an inference system, that is not the same question as:<strong> </strong><em>is it doing useful work? </em>Our principle for LLM serving is that <strong>liveness must test useful work</strong>, not just process health — a probe is only meaningful if it verifies the model actually completes a request end to end.</p><p>The infrastructure is still young — but for a platform built on fleet data that never leaves our perimeter, owning the AI layer isn’t a technical preference. It’s a trust commitment to every operator we serve.</p><p><em>Pilot scope: This experimental pilot operates only in Geotab’s commercial environment. It cannot access Geotab’s FedRAMP environment and is not authorized for FedRAMP, DoD, CUI, or other regulated government workloads, data, metadata, or code development. Use remains subject to Geotab’s software approval, data classification, security, and AI governance requirements.</em></p><ol><li><a href="https://www.pivotpointsecurity.com/ai-business-continuity-planning-model-outages/">https://www.pivotpointsecurity.com/ai-business-continuity-planning-model-outages</a></li><li><a href="https://www.techtarget.com/searchcio/feature/Anthropic-models-foreign-national-ban-exposes-AI-dependency-risks">https://www.techtarget.com/searchcio/feature/Anthropic-models-foreign-national-ban-exposes-AI-dependency-risks</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c4d76eaf993" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geotab/the-model-is-the-easy-part-what-we-learned-from-self-hosting-large-language-models-5c4d76eaf993">The Model Is the Easy Part: What Geotab Learned From Self-Hosting Large Language Models</a> was originally published in <a href="https://medium.com/geotab">Geotab Innovators</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>