<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neeraj Kushwaha on Medium]]></title>
        <description><![CDATA[Stories by Neeraj Kushwaha on Medium]]></description>
        <link>https://medium.com/@learncsdesigns?source=rss-2918beb827f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oN9h8qjRwsuxc3LsroiLaA.jpeg</url>
            <title>Stories by Neeraj Kushwaha on Medium</title>
            <link>https://medium.com/@learncsdesigns?source=rss-2918beb827f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:22:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@learncsdesigns/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Day 16: The Economics of AI — Cost Optimization for Engineers]]></title>
            <link>https://learncsdesigns.medium.com/day-16-the-economics-of-ai-cost-optimization-for-engineers-9064d2911732?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9064d2911732</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[batch-processing]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[token-optimization]]></category>
            <category><![CDATA[model-routing]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 10:26:56 GMT</pubDate>
            <atom:updated>2026-07-17T10:26:56.204Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“The best AI engineer isn’t the one who builds the most powerful system. It’s the one who builds a system that’s powerful enough and cheap enough to actually ship.” — Former Stripe ML Engineering Lead, 2024</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HtKS3Avpa3VT9ihoX9DWiw.png" /></figure><h2>📖 The Dirty Secret of AI Product Economics</h2><p>A seed-stage startup builds a clever AI writing assistant. Early users love it. They raise Series A. Usage explodes. Six months later, the CEO gets the AWS + OpenAI bill and discovers their 40% gross margin product is actually running at negative 15% gross margin.</p><p>This story has played out dozens of times in 2024–2025. The culprit is almost never a single expensive API call — it’s <strong>accumulation</strong>: long system prompts multiplied by millions of requests, full conversation history included with every turn, GPT-4o used where GPT-4o-mini would do, no caching for functionally identical queries.</p><p>The difference between a profitable AI product and an unprofitable one often comes down to a handful of architectural decisions made early, when usage is low and the cost of bad decisions is invisible.</p><p>Let’s make those decisions correctly from day one.</p><h2>💰 Part 1: The True Cost Model — What Engineers Miss</h2><h3>Token Pricing in 2026</h3><pre># Model pricing reference (as of mid-2026 — always verify current rates)<br># Format: (input_per_1M_tokens, output_per_1M_tokens)<br><br>MODEL_PRICING = {<br>    # OpenAI<br>    &quot;gpt-4o&quot;:                 (2.50,   10.00),<br>    &quot;gpt-4o-mini&quot;:            (0.15,    0.60),<br>    &quot;gpt-4.1&quot;:                (2.00,    8.00),<br>    &quot;gpt-4.1-mini&quot;:           (0.40,    1.60),<br>    &quot;o3&quot;:                     (10.00,  40.00),   # Reasoning model<br>    &quot;o4-mini&quot;:                (1.10,    4.40),   # Reasoning, efficient<br>    &quot;text-embedding-3-small&quot;: (0.02,    0.00),   # Embedding only<br>    &quot;text-embedding-3-large&quot;: (0.13,    0.00),<br><br>    # Anthropic<br>    &quot;claude-opus-4&quot;:          (15.00,  75.00),   # Frontier<br>    &quot;claude-sonnet-4&quot;:         (3.00,  15.00),   # Balanced<br>    &quot;claude-haiku-4&quot;:          (0.80,   4.00),   # Fast + cheap<br><br>    # Google<br>    &quot;gemini-2.5-pro&quot;:         (1.25,   10.00),   # w/ thinking<br>    &quot;gemini-2.0-flash&quot;:        (0.10,   0.40),   # Fastest tier<br>    &quot;gemini-2.0-flash-lite&quot;:   (0.075,  0.30),   # Cheapest tier<br><br>    # Meta (self-hosted via Groq/Together AI)<br>    &quot;llama-3.3-70b-groq&quot;:     (0.59,   0.79),<br>    &quot;llama-3.1-8b-groq&quot;:      (0.05,   0.08),<br>    &quot;llama-3.1-70b-together&quot;:  (0.88,   0.88),<br>}<br><br>def compute_cost(<br>    model: str,<br>    input_tokens: int,<br>    output_tokens: int<br>) -&gt; dict:<br>    &quot;&quot;&quot;Compute cost for a single API call.&quot;&quot;&quot;<br>    if model not in MODEL_PRICING:<br>        raise ValueError(f&quot;Unknown model: {model}&quot;)<br>    <br>    input_price, output_price = MODEL_PRICING[model]<br>    <br>    input_cost  = (input_tokens  / 1_000_000) * input_price<br>    output_cost = (output_tokens / 1_000_000) * output_price<br>    total_cost  = input_cost + output_cost<br><br>    return {<br>        &quot;model&quot;: model,<br>        &quot;input_tokens&quot;: input_tokens,<br>        &quot;output_tokens&quot;: output_tokens,<br>        &quot;input_cost_usd&quot;:  round(input_cost,  6),<br>        &quot;output_cost_usd&quot;: round(output_cost, 6),<br>        &quot;total_cost_usd&quot;:  round(total_cost,  6),<br>    }</pre><h3>The Hidden Cost Multipliers</h3><p>Most engineers look at the per-token rate and stop there. The real cost is the per-token rate multiplied by several amplifiers:</p><pre>def project_monthly_cost(<br>    daily_active_users: int,<br>    queries_per_user_per_day: float,<br>    avg_system_prompt_tokens: int,<br>    avg_history_tokens: int,       # Average conversation history included per call<br>    avg_user_input_tokens: int,<br>    avg_output_tokens: int,<br>    model: str,<br>    cache_hit_rate: float = 0.0,   # % of requests served from cache (free)<br>    batch_fraction: float = 0.0,   # % of requests that can use batch API (50% discount)<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Project realistic monthly AI infrastructure costs.<br>    <br>    Most engineers underestimate by 3-5x because they forget:<br>    1. System prompt is paid EVERY call, not once<br>    2. Conversation history grows with each turn<br>    3. Output tokens cost 4-10x more than input tokens<br>    4. Error/retry rate adds ~5-10% overhead<br>    &quot;&quot;&quot;<br>    total_input_per_call = (<br>        avg_system_prompt_tokens  # Always included<br>        + avg_history_tokens       # Grows with conversation depth<br>        + avg_user_input_tokens    # Actual user message<br>    )<br>    <br>    daily_calls         = daily_active_users * queries_per_user_per_day<br>    monthly_calls       = daily_calls * 30<br>    <br>    # After cache hits, remaining calls to API<br>    api_calls           = monthly_calls * (1 - cache_hit_rate)<br>    <br>    # Split into real-time vs batch<br>    realtime_calls      = api_calls * (1 - batch_fraction)<br>    batch_calls         = api_calls * batch_fraction<br>    <br>    input_price, output_price = MODEL_PRICING.get(model, (3.0, 15.0))<br>    <br>    # Real-time cost<br>    rt_input_cost  = (realtime_calls * total_input_per_call  / 1e6) * input_price<br>    rt_output_cost = (realtime_calls * avg_output_tokens     / 1e6) * output_price<br>    <br>    # Batch API cost (50% discount on most providers)<br>    bt_input_cost  = (batch_calls   * total_input_per_call  / 1e6) * input_price  * 0.5<br>    bt_output_cost = (batch_calls   * avg_output_tokens     / 1e6) * output_price * 0.5<br>    <br>    total_monthly   = rt_input_cost + rt_output_cost + bt_input_cost + bt_output_cost<br>    # Add 8% for retries, errors, overhead<br>    total_with_overhead = total_monthly * 1.08<br>    <br>    # Cost per user per month<br>    cost_per_user = total_with_overhead / max(daily_active_users, 1)<br><br>    return {<br>        &quot;daily_api_calls&quot;:         round(daily_calls),<br>        &quot;monthly_api_calls&quot;:       round(monthly_calls),<br>        &quot;calls_served_from_cache&quot;: round(monthly_calls * cache_hit_rate),<br>        &quot;calls_to_api&quot;:            round(api_calls),<br>        &quot;tokens_per_call_input&quot;:   total_input_per_call,<br>        &quot;monthly_input_tokens&quot;:    round(api_calls * total_input_per_call),<br>        &quot;monthly_output_tokens&quot;:   round(api_calls * avg_output_tokens),<br>        &quot;monthly_cost_usd&quot;:        round(total_with_overhead, 2),<br>        &quot;cost_per_user_per_month&quot;: round(cost_per_user, 4),<br>        &quot;annual_run_rate_usd&quot;:     round(total_with_overhead * 12, 2),<br>    }<br><br># Realistic SaaS example: 10k DAU, conversational AI feature<br>scenario = project_monthly_cost(<br>    daily_active_users=10_000,<br>    queries_per_user_per_day=5,<br>    avg_system_prompt_tokens=800,    # System prompt: detailed persona + instructions<br>    avg_history_tokens=1_500,        # ~5 turns of conversation history<br>    avg_user_input_tokens=80,        # Typical user message<br>    avg_output_tokens=300,           # Typical assistant response<br>    model=&quot;gpt-4o&quot;,<br>    cache_hit_rate=0.0,              # No optimization yet<br>    batch_fraction=0.0,<br>)<br><br>optimized = project_monthly_cost(<br>    daily_active_users=10_000,<br>    queries_per_user_per_day=5,<br>    avg_system_prompt_tokens=400,    # After compression: 50% reduction<br>    avg_history_tokens=600,          # Sliding window + summarization<br>    avg_user_input_tokens=80,<br>    avg_output_tokens=300,<br>    model=&quot;gpt-4o&quot;,<br>    cache_hit_rate=0.25,             # 25% semantic cache hit rate<br>    batch_fraction=0.30,             # 30% of calls are async/batchable<br>)<br><br>print(f&quot;Unoptimized: ${scenario[&#39;monthly_cost_usd&#39;]:,.0f}/month  (${scenario[&#39;annual_run_rate_usd&#39;]:,.0f}/year)&quot;)<br>print(f&quot;Optimized:   ${optimized[&#39;monthly_cost_usd&#39;]:,.0f}/month  (${optimized[&#39;annual_run_rate_usd&#39;]:,.0f}/year)&quot;)<br># Typical result: 60-75% cost reduction without changing the model</pre><h3>Real-World Cost Scenarios</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*V3S68RCs-53U_fPTBXB-5g.png" /></figure><p>The pattern is consistent: <strong>60–75% cost reduction is achievable</strong> through caching, prompt compression, model routing, and batching — without changing the underlying model or degrading quality.</p><h2>🗺️ Part 2: Intelligent Model Routing</h2><p>The single highest-ROI optimization: not every query needs GPT-4o. A query like “summarize this paragraph” costs the same whether you use GPT-4o ($0.015) or GPT-4o-mini ($0.0009) — a 16× difference. Route intelligently.</p><pre>from dataclasses import dataclass<br>from enum import Enum<br>import openai<br>import re<br><br>class TaskComplexity(Enum):<br>    TRIVIAL    = &quot;trivial&quot;     # → Haiku / GPT-4o-mini / Gemini Flash<br>    SIMPLE     = &quot;simple&quot;      # → GPT-4o-mini / Claude Haiku<br>    MODERATE   = &quot;moderate&quot;    # → GPT-4o / Claude Sonnet<br>    COMPLEX    = &quot;complex&quot;     # → GPT-4o / Claude Sonnet<br>    REASONING  = &quot;reasoning&quot;   # → o3 / o4-mini / Claude with extended thinking<br><br>@dataclass<br>class RoutingDecision:<br>    model: str<br>    complexity: TaskComplexity<br>    reasoning: str<br>    estimated_cost_usd: float<br>    latency_tier: str  # &quot;fast&quot; | &quot;standard&quot; | &quot;slow&quot;<br><br>class IntelligentRouter:<br>    &quot;&quot;&quot;<br>    Routes queries to the cheapest model capable of handling them well.<br>    <br>    Strategy:<br>    1. Rule-based fast path (regex patterns, token counts)<br>    2. Classifier-based routing (small LLM classifies the task)<br>    3. Cascade routing (try cheap model, escalate on low confidence)<br>    <br>    Target: 70% of queries to cheap tier, 25% to mid tier, 5% to expensive tier.<br>    &quot;&quot;&quot;<br><br>    # Rule-based patterns for instant routing decisions<br>    TRIVIAL_PATTERNS = [<br>        r&quot;^(yes|no|maybe)\??$&quot;,<br>        r&quot;^(translate|summarize|rewrite)\s+.{0,200}$&quot;,<br>        r&quot;^(what is|define|explain briefly)&quot;,<br>        r&quot;^(fix the grammar|correct this|proofread)&quot;,<br>        r&quot;format (this|the following)&quot;,<br>    ]<br><br>    REASONING_PATTERNS = [<br>        r&quot;(prove|derive|show that)&quot;,<br>        r&quot;(multi-step|step[\s-]by[\s-]step reasoning)&quot;,<br>        r&quot;(debug|diagnose)\s+.*(complex|subtle|tricky)&quot;,<br>        r&quot;(optimize|refactor)\s+.*(entire|whole|system)&quot;,<br>        r&quot;(design|architect)\s+.*(system|platform|infrastructure)&quot;,<br>        r&quot;(mathematical|statistical|probabilistic)\s+(proof|derivation|analysis)&quot;,<br>    ]<br><br>    MODEL_TIERS = {<br>        TaskComplexity.TRIVIAL:   (&quot;gpt-4o-mini&quot;,   0.00015, &quot;fast&quot;),<br>        TaskComplexity.SIMPLE:    (&quot;gpt-4o-mini&quot;,   0.00015, &quot;fast&quot;),<br>        TaskComplexity.MODERATE:  (&quot;gpt-4o&quot;,        0.0025,  &quot;standard&quot;),<br>        TaskComplexity.COMPLEX:   (&quot;gpt-4o&quot;,        0.0025,  &quot;standard&quot;),<br>        TaskComplexity.REASONING: (&quot;o4-mini&quot;,       0.0011,  &quot;slow&quot;),<br>    }<br><br>    def __init__(self):<br>        self.client = openai.OpenAI()<br>        self._routing_cache: dict[str, RoutingDecision] = {}<br><br>    def _rule_based_route(self, query: str, n_input_tokens: int) -&gt; TaskComplexity | None:<br>        &quot;&quot;&quot;Fast path: pattern-match before calling any model.&quot;&quot;&quot;<br>        q = query.strip().lower()<br><br>        # Very short or very simple queries<br>        if n_input_tokens &lt; 30 and len(q.split()) &lt; 15:<br>            return TaskComplexity.TRIVIAL<br><br>        for pattern in self.TRIVIAL_PATTERNS:<br>            if re.search(pattern, q):<br>                return TaskComplexity.TRIVIAL<br><br>        for pattern in self.REASONING_PATTERNS:<br>            if re.search(pattern, q):<br>                return TaskComplexity.REASONING<br><br>        return None  # Fall through to classifier<br><br>    def _classifier_route(self, query: str, context: str = &quot;&quot;) -&gt; TaskComplexity:<br>        &quot;&quot;&quot;<br>        Use a tiny, cheap model to classify task complexity.<br>        Cost: ~$0.00002 per classification (GPT-4o-mini, ~50 tokens).<br>        Saves much more than it costs when routing to correct tier.<br>        &quot;&quot;&quot;<br>        classify_prompt = &quot;&quot;&quot;Classify this AI task by complexity. Reply with ONLY one word.<br><br>TRIVIAL: Formatting, translation, grammar fixes, simple factual lookups, summarization of short text<br>SIMPLE: Multi-sentence explanation, basic code generation, short creative writing<br>MODERATE: Complex analysis, multi-step instructions, moderate code tasks, document QA<br>COMPLEX: Multi-document synthesis, complex code architecture, nuanced reasoning<br>REASONING: Mathematical proofs, deep debugging, multi-step logical deduction, system design<br><br>Task: {query}<br>Context length: {ctx_len} tokens<br><br>Complexity (one word):&quot;&quot;&quot;<br><br>        resp = self.client.chat.completions.create(<br>            model=&quot;gpt-4o-mini&quot;,<br>            messages=[{<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: classify_prompt.format(<br>                    query=query[:500],<br>                    ctx_len=len(context.split())<br>                )<br>            }],<br>            max_tokens=5,<br>            temperature=0,<br>        )<br>        <br>        label = resp.choices[0].message.content.strip().upper()<br>        mapping = {<br>            &quot;TRIVIAL&quot;:   TaskComplexity.TRIVIAL,<br>            &quot;SIMPLE&quot;:    TaskComplexity.SIMPLE,<br>            &quot;MODERATE&quot;:  TaskComplexity.MODERATE,<br>            &quot;COMPLEX&quot;:   TaskComplexity.COMPLEX,<br>            &quot;REASONING&quot;: TaskComplexity.REASONING,<br>        }<br>        return mapping.get(label, TaskComplexity.MODERATE)<br><br>    def route(<br>        self,<br>        query: str,<br>        context: str = &quot;&quot;,<br>        force_quality: bool = False<br>    ) -&gt; RoutingDecision:<br>        &quot;&quot;&quot;<br>        Main routing entry point.<br>        Returns the cheapest model that should handle this query well.<br>        &quot;&quot;&quot;<br>        if force_quality:<br>            model, cost, latency = self.MODEL_TIERS[TaskComplexity.COMPLEX]<br>            return RoutingDecision(<br>                model=model,<br>                complexity=TaskComplexity.COMPLEX,<br>                reasoning=&quot;force_quality=True override&quot;,<br>                estimated_cost_usd=cost,<br>                latency_tier=latency<br>            )<br><br>        n_tokens = len(query.split()) + len(context.split())<br><br>        # Fast path: rule-based<br>        complexity = self._rule_based_route(query, n_tokens)<br>        route_method = &quot;rule_based&quot;<br><br>        # Slow path: classifier<br>        if complexity is None:<br>            complexity = self._classifier_route(query, context)<br>            route_method = &quot;classifier&quot;<br><br>        model, cost, latency = self.MODEL_TIERS[complexity]<br><br>        return RoutingDecision(<br>            model=model,<br>            complexity=complexity,<br>            reasoning=f&quot;Routed via {route_method}&quot;,<br>            estimated_cost_usd=cost,<br>            latency_tier=latency<br>        )<br><br>    async def generate_routed(<br>        self,<br>        query: str,<br>        system_prompt: str = &quot;&quot;,<br>        context: str = &quot;&quot;,<br>        force_quality: bool = False<br>    ) -&gt; dict:<br>        &quot;&quot;&quot;Full pipeline: route → generate → return with cost metadata.&quot;&quot;&quot;<br>        decision = self.route(query, context, force_quality)<br><br>        messages = []<br>        if system_prompt:<br>            messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt})<br>        if context:<br>            messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n{context}&quot;})<br>            messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Understood.&quot;})<br>        messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query})<br><br>        response = self.client.chat.completions.create(<br>            model=decision.model,<br>            messages=messages,<br>        )<br><br>        usage = response.usage<br>        actual_cost = compute_cost(<br>            decision.model,<br>            usage.prompt_tokens,<br>            usage.completion_tokens<br>        )<br><br>        return {<br>            &quot;answer&quot;:         response.choices[0].message.content,<br>            &quot;model_used&quot;:     decision.model,<br>            &quot;complexity&quot;:     decision.complexity.value,<br>            &quot;actual_cost&quot;:    actual_cost[&quot;total_cost_usd&quot;],<br>            &quot;input_tokens&quot;:   usage.prompt_tokens,<br>            &quot;output_tokens&quot;:  usage.completion_tokens,<br>        }</pre><h3>Cascade Routing — Try Cheap, Escalate on Low Confidence</h3><pre>async def cascade_generate(<br>    query: str,<br>    system_prompt: str,<br>    cheap_model: str = &quot;gpt-4o-mini&quot;,<br>    expensive_model: str = &quot;gpt-4o&quot;,<br>    confidence_threshold: float = 0.75<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Cascade strategy:<br>    1. Try cheap model first<br>    2. Ask the model to rate its own confidence<br>    3. If confidence &lt; threshold, escalate to expensive model<br>    <br>    Works especially well for: QA, factual retrieval, classification<br>    Savings: ~50-65% vs always using expensive model<br>    &quot;&quot;&quot;<br>    client = openai.OpenAI()<br><br>    # Step 1: Cheap model attempt<br>    cheap_response = client.chat.completions.create(<br>        model=cheap_model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>            {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: query},<br>            # Ask for self-assessed confidence in the same call<br>        ],<br>        # Use structured output to get answer + confidence together<br>        response_format={&quot;type&quot;: &quot;json_object&quot;},<br>    )<br><br>    # Parse confidence from response (requires prompt engineering to extract)<br>    # In practice: add to system prompt: &quot;Respond in JSON: {answer: &#39;...&#39;, confidence: 0.0-1.0}&quot;<br>    import json<br>    try:<br>        result = json.loads(cheap_response.choices[0].message.content)<br>        answer     = result.get(&quot;answer&quot;, cheap_response.choices[0].message.content)<br>        confidence = float(result.get(&quot;confidence&quot;, 0.5))<br>    except (json.JSONDecodeError, KeyError):<br>        answer     = cheap_response.choices[0].message.content<br>        confidence = 0.5  # Unknown confidence — conservatively escalate<br><br>    cheap_cost = compute_cost(<br>        cheap_model,<br>        cheap_response.usage.prompt_tokens,<br>        cheap_response.usage.completion_tokens<br>    )<br><br>    # Step 2: Escalate if uncertain<br>    if confidence &lt; confidence_threshold:<br>        expensive_response = client.chat.completions.create(<br>            model=expensive_model,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: query},<br>            ]<br>        )<br>        answer = expensive_response.choices[0].message.content<br>        exp_cost = compute_cost(<br>            expensive_model,<br>            expensive_response.usage.prompt_tokens,<br>            expensive_response.usage.completion_tokens<br>        )<br>        total_cost = cheap_cost[&quot;total_cost_usd&quot;] + exp_cost[&quot;total_cost_usd&quot;]<br>        escalated  = True<br>    else:<br>        total_cost = cheap_cost[&quot;total_cost_usd&quot;]<br>        escalated  = False<br><br>    return {<br>        &quot;answer&quot;:     answer,<br>        &quot;escalated&quot;:  escalated,<br>        &quot;confidence&quot;: confidence,<br>        &quot;total_cost&quot;: total_cost,<br>        &quot;model_used&quot;: expensive_model if escalated else cheap_model,<br>    }</pre><h2>🔄 Part 3: Semantic Caching</h2><p>The cheapest API call is the one you never make. Semantic caching intercepts queries that are <em>semantically equivalent</em> to previous queries and returns the cached answer — even when the wording differs.</p><pre>&quot;What&#39;s the capital of France?&quot;<br>&quot;What is France&#39;s capital city?&quot;<br>&quot;Can you tell me the capital of France?&quot;<br>→ All serve from cache after the first one is answered</pre><pre>import hashlib<br>import json<br>import time<br>import numpy as np<br>from typing import Any<br><br>class SemanticCache:<br>    &quot;&quot;&quot;<br>    Two-tier cache:<br>    1. Exact hash match → instant return (zero compute)<br>    2. Semantic similarity match → return if cosine similarity &gt; threshold<br>    <br>    Storage: Redis for production (shown here as in-memory dict for clarity)<br>    TTL: Configurable per query type (facts: 24h, market data: 5min)<br>    &quot;&quot;&quot;<br><br>    def __init__(<br>        self,<br>        similarity_threshold: float = 0.92,  # Very high → only near-identical queries hit<br>        max_cache_size: int = 10_000,<br>        default_ttl_seconds: int = 3600,<br>        embedding_model: str = &quot;text-embedding-3-small&quot;,<br>    ):<br>        self.threshold   = similarity_threshold<br>        self.max_size    = max_cache_size<br>        self.default_ttl = default_ttl_seconds<br>        self.emb_model   = embedding_model<br>        self.client      = openai.OpenAI()<br><br>        # In production: use Redis with vector search (Redis Stack)<br>        self._exact_cache:    dict[str, dict] = {}  # hash → entry<br>        self._semantic_index: list[dict]      = []  # list of {embedding, answer, meta}<br><br>    def _embed(self, text: str) -&gt; list[float]:<br>        resp = self.client.embeddings.create(model=self.emb_model, input=text)<br>        return resp.data[0].embedding<br><br>    def _exact_key(self, query: str, context_hash: str = &quot;&quot;) -&gt; str:<br>        return hashlib.sha256(f&quot;{query.strip().lower()}|{context_hash}&quot;.encode()).hexdigest()<br><br>    def _cosine_similarity(self, a: list[float], b: list[float]) -&gt; float:<br>        va, vb = np.array(a), np.array(b)<br>        return float(np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb) + 1e-10))<br><br>    def get(self, query: str, context: str = &quot;&quot;) -&gt; dict | None:<br>        &quot;&quot;&quot;<br>        Try to serve from cache.<br>        Returns cached entry or None (cache miss → call the API).<br>        &quot;&quot;&quot;<br>        now = time.time()<br>        ctx_hash = hashlib.md5(context.encode()).hexdigest()[:8]<br><br>        # ── Tier 1: exact match ──────────────────────────────────────<br>        key = self._exact_key(query, ctx_hash)<br>        if key in self._exact_cache:<br>            entry = self._exact_cache[key]<br>            if now &lt; entry[&quot;expires_at&quot;]:<br>                entry[&quot;hits&quot;] += 1<br>                return {**entry, &quot;cache_type&quot;: &quot;exact&quot;}<br>            else:<br>                del self._exact_cache[key]<br><br>        # ── Tier 2: semantic match ────────────────────────────────────<br>        if not self._semantic_index:<br>            return None<br><br>        query_emb = self._embed(query)<br>        best_score, best_entry = 0.0, None<br><br>        for entry in self._semantic_index:<br>            if now &gt;= entry.get(&quot;expires_at&quot;, float(&quot;inf&quot;)):<br>                continue<br>            score = self._cosine_similarity(query_emb, entry[&quot;embedding&quot;])<br>            if score &gt; best_score:<br>                best_score, best_entry = score, entry<br><br>        if best_score &gt;= self.threshold and best_entry:<br>            best_entry[&quot;hits&quot;] = best_entry.get(&quot;hits&quot;, 0) + 1<br>            return {**best_entry, &quot;cache_type&quot;: &quot;semantic&quot;, &quot;similarity&quot;: best_score}<br><br>        return None  # Cache miss<br><br>    def set(<br>        self,<br>        query: str,<br>        answer: str,<br>        model: str,<br>        cost_usd: float,<br>        context: str = &quot;&quot;,<br>        ttl_seconds: int | None = None,<br>        metadata: dict | None = None,<br>    ) -&gt; None:<br>        &quot;&quot;&quot;Store a query-answer pair in both cache tiers.&quot;&quot;&quot;<br>        ttl      = ttl_seconds or self.default_ttl<br>        ctx_hash = hashlib.md5(context.encode()).hexdigest()[:8]<br>        now      = time.time()<br><br>        entry = {<br>            &quot;query&quot;:      query,<br>            &quot;answer&quot;:     answer,<br>            &quot;model&quot;:      model,<br>            &quot;cost_usd&quot;:   cost_usd,<br>            &quot;embedding&quot;:  self._embed(query),<br>            &quot;created_at&quot;: now,<br>            &quot;expires_at&quot;: now + ttl,<br>            &quot;hits&quot;:       0,<br>            &quot;metadata&quot;:   metadata or {},<br>        }<br><br>        # Exact cache<br>        key = self._exact_key(query, ctx_hash)<br>        self._exact_cache[key] = entry<br><br>        # Semantic index<br>        self._semantic_index.append(entry)<br>        if len(self._semantic_index) &gt; self.max_size:<br>            self._semantic_index.sort(key=lambda e: e.get(&quot;hits&quot;, 0), reverse=True)<br>            self._semantic_index = self._semantic_index[:self.max_size]<br><br>    def stats(self) -&gt; dict:<br>        &quot;&quot;&quot;Cache performance metrics for your observability dashboard.&quot;&quot;&quot;<br>        total_hits   = sum(e.get(&quot;hits&quot;, 0) for e in self._semantic_index)<br>        total_savings = sum(<br>            e.get(&quot;cost_usd&quot;, 0) * e.get(&quot;hits&quot;, 0)<br>            for e in self._semantic_index<br>        )<br>        return {<br>            &quot;semantic_entries&quot;:   len(self._semantic_index),<br>            &quot;exact_entries&quot;:      len(self._exact_cache),<br>            &quot;total_cache_hits&quot;:   total_hits,<br>            &quot;estimated_savings&quot;:  round(total_savings, 4),<br>        }<br><br><br># ── Drop-in cached wrapper around any LLM call ──────────────────────<br>cache = SemanticCache(similarity_threshold=0.93, default_ttl_seconds=3600)<br><br>def cached_generate(<br>    query: str,<br>    system_prompt: str = &quot;&quot;,<br>    model: str = &quot;gpt-4o&quot;,<br>    ttl: int = 3600<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Cached generation — returns from cache when semantically equivalent<br>    query was already answered.<br>    &quot;&quot;&quot;<br>    client = openai.OpenAI()<br><br>    cached = cache.get(query)<br>    if cached:<br>        return {<br>            &quot;answer&quot;:     cached[&quot;answer&quot;],<br>            &quot;cost_usd&quot;:   0.0,        # FREE — served from cache<br>            &quot;from_cache&quot;: True,<br>            &quot;cache_type&quot;: cached[&quot;cache_type&quot;],<br>            &quot;savings&quot;:    cached[&quot;cost_usd&quot;],<br>        }<br><br>    # Cache miss → real API call<br>    response = client.chat.completions.create(<br>        model=model,<br>        messages=[<br>            *([] if not system_prompt else [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt}]),<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}<br>        ]<br>    )<br>    answer   = response.choices[0].message.content<br>    cost_res = compute_cost(model, response.usage.prompt_tokens, response.usage.completion_tokens)<br><br>    cache.set(query, answer, model=model, cost_usd=cost_res[&quot;total_cost_usd&quot;], ttl_seconds=ttl)<br><br>    return {<br>        &quot;answer&quot;:     answer,<br>        &quot;cost_usd&quot;:   cost_res[&quot;total_cost_usd&quot;],<br>        &quot;from_cache&quot;: False,<br>    }</pre><h3>Production Semantic Cache with Redis Stack</h3><pre># Production implementation using Redis with vector search<br># pip install redis[hiredis] redisvl<br><br>from redisvl.index import SearchIndex<br>from redisvl.query import VectorQuery<br>from redisvl.schema import IndexSchema<br><br>REDIS_SCHEMA = IndexSchema.from_dict({<br>    &quot;index&quot;: {&quot;name&quot;: &quot;semantic_cache&quot;, &quot;prefix&quot;: &quot;sc:&quot;},<br>    &quot;fields&quot;: [<br>        {&quot;name&quot;: &quot;query&quot;,      &quot;type&quot;: &quot;text&quot;},<br>        {&quot;name&quot;: &quot;answer&quot;,     &quot;type&quot;: &quot;text&quot;},<br>        {&quot;name&quot;: &quot;model&quot;,      &quot;type&quot;: &quot;tag&quot;},<br>        {&quot;name&quot;: &quot;created_at&quot;, &quot;type&quot;: &quot;numeric&quot;},<br>        {&quot;name&quot;: &quot;embedding&quot;,  &quot;type&quot;: &quot;vector&quot;,<br>         &quot;attrs&quot;: {&quot;dims&quot;: 1536, &quot;distance_metric&quot;: &quot;cosine&quot;, &quot;algorithm&quot;: &quot;hnsw&quot;}}<br>    ]<br>})<br><br># Key advantage of Redis: built-in TTL, persistence, clustering<br># Production pattern: separate Redis instances for cache vs application data</pre><h2>📦 Part 4: Prompt Compression — Getting More from Fewer Tokens</h2><p>Your system prompt is a tax paid on every single request. A 1,200-token system prompt at $0.0025/1K tokens costs $0.003 per call. At 1M calls/month that’s <strong>$3,000/month from system prompt alone</strong>. Compress it.</p><pre>class PromptCompressor:<br>    &quot;&quot;&quot;<br>    Techniques to reduce token count without reducing instruction quality.<br>    <br>    Tested against: GPT-4o, Claude Sonnet, Gemini Pro<br>    Typical savings: 30-55% token reduction with &lt;5% quality degradation<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        self.client = openai.OpenAI()<br>    <br>    def llm_compress(<br>        self,<br>        prompt: str,<br>        target_compression: float = 0.5,   # Target 50% of original length<br>        model: str = &quot;gpt-4o-mini&quot;<br>    ) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Use an LLM to compress a prompt while preserving semantics.<br>        One-time cost: compress once, save forever.<br>        &quot;&quot;&quot;<br>        compression_instruction = f&quot;&quot;&quot;Compress the following system prompt to approximately {int(target_compression * 100)}% of its original length.<br><br>Rules:<br>- Preserve ALL semantic content and behavioral instructions<br>- Remove redundancy, examples (unless critical), verbose explanations<br>- Use compact phrasing: &quot;Be concise.&quot; not &quot;Please make sure that your responses are concise and to the point.&quot;<br>- Keep formatting instructions if present<br>- Preserve tone modifiers exactly<br><br>Original prompt ({len(prompt.split())} words):<br>{prompt}<br><br>Compressed prompt:&quot;&quot;&quot;<br><br>        response = self.client.chat.completions.create(<br>            model=model,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: compression_instruction}],<br>            temperature=0<br>        )<br>        <br>        compressed     = response.choices[0].message.content.strip()<br>        original_toks  = len(prompt.split())<br>        compressed_toks = len(compressed.split())<br>        <br>        return {<br>            &quot;original&quot;:          prompt,<br>            &quot;compressed&quot;:        compressed,<br>            &quot;original_tokens&quot;:   original_toks,<br>            &quot;compressed_tokens&quot;: compressed_toks,<br>            &quot;compression_ratio&quot;: round(compressed_toks / original_toks, 2),<br>            &quot;token_savings&quot;:     original_toks - compressed_toks,<br>        }<br>    <br>    def selective_history_compression(<br>        self,<br>        conversation_history: list[dict],<br>        max_tokens: int = 2000,<br>        preserve_last_n: int = 3<br>    ) -&gt; list[dict]:<br>        &quot;&quot;&quot;<br>        Compress conversation history to stay within token budget.<br>        <br>        Strategy:<br>        1. Always keep the last N turns verbatim (recent context is critical)<br>        2. Summarize older turns into a single compact message<br>        3. Never compress the system prompt<br>        &quot;&quot;&quot;<br>        if not conversation_history:<br>            return []<br>        <br>        # Separate system from conversation<br>        system_msgs  = [m for m in conversation_history if m[&quot;role&quot;] == &quot;system&quot;]<br>        conv_msgs    = [m for m in conversation_history if m[&quot;role&quot;] != &quot;system&quot;]<br>        <br>        if len(conv_msgs) &lt;= preserve_last_n * 2:<br>            return conversation_history  # Short enough — no compression needed<br>        <br>        # Split: recent (keep verbatim) vs older (summarize)<br>        recent_msgs = conv_msgs[-(preserve_last_n * 2):]<br>        older_msgs  = conv_msgs[:-(preserve_last_n * 2)]<br>        <br>        if not older_msgs:<br>            return conversation_history<br>        <br>        # Summarize older history<br>        history_text = &quot;\n&quot;.join([<br>            f&quot;{m[&#39;role&#39;].upper()}: {m[&#39;content&#39;][:500]}&quot;  # Truncate very long turns<br>            for m in older_msgs<br>        ])<br>        <br>        summary_response = self.client.chat.completions.create(<br>            model=&quot;gpt-4o-mini&quot;,<br>            messages=[{<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: f&quot;Summarize this conversation history in 3-5 sentences, preserving key facts, decisions, and context:\n\n{history_text}&quot;<br>            }],<br>            max_tokens=200,<br>            temperature=0<br>        )<br>        summary = summary_response.choices[0].message.content<br>        <br>        # Reconstruct compressed history<br>        compressed_history = (<br>            system_msgs<br>            + [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: f&quot;[Earlier conversation summary: {summary}]&quot;}]<br>            + recent_msgs<br>        )<br>        <br>        return compressed_history<br><br>    def remove_redundant_context(self, context_chunks: list[str], query: str) -&gt; list[str]:<br>        &quot;&quot;&quot;<br>        Before stuffing retrieved chunks into context, remove near-duplicates.<br>        Saves 20-40% of context tokens in RAG pipelines.<br>        &quot;&quot;&quot;<br>        if len(context_chunks) &lt;= 1:<br>            return context_chunks<br>        <br>        # Embed all chunks<br>        texts    = [query] + context_chunks<br>        response = self.client.embeddings.create(<br>            model=&quot;text-embedding-3-small&quot;, input=texts<br>        )<br>        embeddings = [item.embedding for item in response.data]<br>        query_emb  = embeddings[0]<br>        chunk_embs = embeddings[1:]<br>        <br>        # Sort chunks by relevance to query<br>        scored = sorted(<br>            zip(context_chunks, chunk_embs),<br>            key=lambda x: np.dot(query_emb, x[1]),<br>            reverse=True<br>        )<br>        <br>        # Deduplicate: remove chunks too similar to already-selected ones<br>        selected     = []<br>        selected_embs = []<br>        DEDUP_THRESHOLD = 0.92  # Chunks above this similarity are near-duplicates<br>        <br>        for chunk, emb in scored:<br>            is_duplicate = any(<br>                np.dot(emb, sel_emb) &gt; DEDUP_THRESHOLD<br>                for sel_emb in selected_embs<br>            )<br>            if not is_duplicate:<br>                selected.append(chunk)<br>                selected_embs.append(emb)<br>        <br>        return selected</pre><h2>⚡ Part 5: Batch Processing — The 50% Discount Most Teams Leave on the Table</h2><p>OpenAI, Anthropic, and Google all offer <strong>batch APIs</strong> with 50% discounts for async workloads. Most teams never use them for eligible workloads.</p><pre>import asyncio<br>import json<br>from pathlib import Path<br><br>class BatchProcessor:<br>    &quot;&quot;&quot;<br>    Uses OpenAI Batch API for async, non-urgent workloads.<br>    <br>    50% cost discount. Up to 24h completion window.<br>    <br>    Perfect for:<br>    - Nightly document processing / embedding generation<br>    - Bulk content generation (product descriptions, summaries)<br>    - Offline evaluation pipelines<br>    - Background data enrichment<br>    <br>    NOT for: real-time user-facing queries<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        self.client = openai.OpenAI()<br>    <br>    def prepare_batch_file(<br>        self,<br>        requests: list[dict],<br>        output_path: str = &quot;/tmp/batch_requests.jsonl&quot;<br>    ) -&gt; str:<br>        &quot;&quot;&quot;<br>        Prepare a JSONL batch file.<br>        <br>        Each request: {<br>            &quot;custom_id&quot;: &quot;unique-id&quot;,<br>            &quot;messages&quot;: [...],<br>            &quot;model&quot;: &quot;gpt-4o&quot;,<br>            &quot;max_tokens&quot;: 500<br>        }<br>        &quot;&quot;&quot;<br>        with open(output_path, &quot;w&quot;) as f:<br>            for i, req in enumerate(requests):<br>                batch_request = {<br>                    &quot;custom_id&quot;:  req.get(&quot;id&quot;, f&quot;request-{i}&quot;),<br>                    &quot;method&quot;:     &quot;POST&quot;,<br>                    &quot;url&quot;:        &quot;/v1/chat/completions&quot;,<br>                    &quot;body&quot;: {<br>                        &quot;model&quot;:      req.get(&quot;model&quot;, &quot;gpt-4o&quot;),<br>                        &quot;messages&quot;:   req[&quot;messages&quot;],<br>                        &quot;max_tokens&quot;: req.get(&quot;max_tokens&quot;, 1000),<br>                    }<br>                }<br>                f.write(json.dumps(batch_request) + &quot;\n&quot;)<br>        <br>        return output_path<br>    <br>    def submit_batch(self, jsonl_path: str) -&gt; str:<br>        &quot;&quot;&quot;Submit batch file and return batch ID.&quot;&quot;&quot;<br>        with open(jsonl_path, &quot;rb&quot;) as f:<br>            file_obj = self.client.files.create(file=f, purpose=&quot;batch&quot;)<br>        <br>        batch = self.client.batches.create(<br>            input_file_id=file_obj.id,<br>            endpoint=&quot;/v1/chat/completions&quot;,<br>            completion_window=&quot;24h&quot;,<br>            metadata={&quot;description&quot;: &quot;Nightly batch processing&quot;}<br>        )<br>        <br>        print(f&quot;Batch submitted: {batch.id} | Status: {batch.status}&quot;)<br>        return batch.id<br>    <br>    def poll_batch(self, batch_id: str, poll_interval_seconds: int = 60) -&gt; list[dict]:<br>        &quot;&quot;&quot;<br>        Poll until batch completes and return parsed results.<br>        In production: use webhooks instead of polling.<br>        &quot;&quot;&quot;<br>        while True:<br>            batch = self.client.batches.retrieve(batch_id)<br>            <br>            if batch.status == &quot;completed&quot;:<br>                break<br>            elif batch.status in {&quot;failed&quot;, &quot;expired&quot;, &quot;cancelled&quot;}:<br>                raise RuntimeError(f&quot;Batch {batch_id} ended with status: {batch.status}&quot;)<br>            <br>            print(f&quot;Batch status: {batch.status} | &quot;<br>                  f&quot;Completed: {batch.request_counts.completed}/&quot;<br>                  f&quot;{batch.request_counts.total}&quot;)<br>            <br>            time.sleep(poll_interval_seconds)<br>        <br>        # Download results<br>        result_content = self.client.files.content(batch.output_file_id)<br>        results = []<br>        for line in result_content.text.splitlines():<br>            if line.strip():<br>                obj = json.loads(line)<br>                results.append({<br>                    &quot;id&quot;:     obj[&quot;custom_id&quot;],<br>                    &quot;answer&quot;: obj[&quot;response&quot;][&quot;body&quot;][&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;],<br>                    &quot;usage&quot;:  obj[&quot;response&quot;][&quot;body&quot;][&quot;usage&quot;],<br>                })<br>        <br>        return results<br><br>    async def process_mixed_workload(<br>        self,<br>        urgent_requests: list[dict],<br>        deferrable_requests: list[dict],<br>    ) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Optimal pattern for mixed workloads:<br>        - Urgent: real-time API (full price)<br>        - Deferrable: batch API (50% off)<br>        <br>        Typical savings: 30-40% overall cost reduction<br>        &quot;&quot;&quot;<br>        # Launch batch for deferrable work<br>        if deferrable_requests:<br>            batch_file = self.prepare_batch_file(deferrable_requests)<br>            batch_id   = self.submit_batch(batch_file)<br>        <br>        # Handle urgent requests in parallel with real-time API<br>        async def call_api(req: dict) -&gt; dict:<br>            loop = asyncio.get_event_loop()<br>            return await loop.run_in_executor(<br>                None,<br>                lambda: self.client.chat.completions.create(<br>                    model=req.get(&quot;model&quot;, &quot;gpt-4o&quot;),<br>                    messages=req[&quot;messages&quot;]<br>                )<br>            )<br>        <br>        urgent_responses = await asyncio.gather(*[call_api(r) for r in urgent_requests])<br>        <br>        return {<br>            &quot;urgent_responses&quot;: urgent_responses,<br>            &quot;batch_id&quot;:         batch_id if deferrable_requests else None,<br>            &quot;note&quot;:             &quot;Call poll_batch(batch_id) later to retrieve deferred results&quot;<br>        }</pre><h2>🏗️ Part 6: The Multi-Tier Inference Architecture</h2><p>The complete production cost-optimization stack, wired together:</p><pre>┌─────────────────────────────────────────────────────────────────┐<br>│                      Incoming Query                             │<br>└────────────────────────────┬────────────────────────────────────┘<br>                             │<br>              ┌──────────────▼──────────────────┐<br>              │     Tier 0: Exact + Semantic     │<br>              │           Cache                  │<br>              │     Hit rate target: 20-35%      │<br>              │     Cost: $0.00 per hit          │<br>              └──────────────┬──────────────────-┘<br>                    miss     │     hit<br>             ┌───────────────┘     │<br>             │                     └─────────────────────────────►<br>             ▼                              Return cached answer<br>  ┌──────────────────────┐<br>  │  Tier 1: Rule-Based  │<br>  │  Fast Router         │<br>  │  Trivial → mini      │<br>  │  Reasoning → o3      │<br>  └──────────┬───────────┘<br>             │ uncertain<br>             ▼<br>  ┌──────────────────────┐<br>  │  Tier 2: Classifier  │<br>  │  Route (4o-mini)     │<br>  │  Cost: ~$0.00002     │<br>  └──────────┬───────────┘<br>             │ routed<br>             ▼<br>  ┌─────────────────────────────────────────────────┐<br>  │                Model Tier                       │<br>  │                                                 │<br>  │  TRIVIAL/SIMPLE  → GPT-4o-mini ($0.00015/call) │<br>  │  MODERATE        → GPT-4o      ($0.0025/call)  │<br>  │  REASONING       → o4-mini     ($0.0011/call)  │<br>  └─────────────────────────────────────────────────┘<br>             │<br>             ▼<br>  ┌──────────────────────┐<br>  │  Prompt Compression  │<br>  │  • History truncation│<br>  │  • Context dedup     │<br>  └──────────┬───────────┘<br>             │<br>             ▼<br>  ┌──────────────────────┐<br>  │  Real-time API Call  │◄── Urgent (full price)<br>  │  or Batch API Call   │◄── Deferrable (50% off)<br>  └──────────┬───────────┘<br>             │<br>             ▼<br>  ┌──────────────────────┐<br>  │  Write to Cache      │<br>  │  for future hits     │<br>  └──────────┬───────────┘<br>             │<br>             ▼<br>         Response</pre><pre>class CostOptimizedAIStack:<br>    &quot;&quot;&quot;<br>    Production AI stack wiring together all optimization layers.<br>    Target: 65-75% cost reduction vs naive implementation.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self, config: dict | None = None):<br>        self.cache     = SemanticCache(similarity_threshold=0.93)<br>        self.router    = IntelligentRouter()<br>        self.compressor = PromptCompressor()<br>        self.batch_proc = BatchProcessor()<br>        <br>        self._stats = {<br>            &quot;total_requests&quot;:    0,<br>            &quot;cache_hits&quot;:        0,<br>            &quot;total_cost_usd&quot;:    0.0,<br>            &quot;saved_by_routing&quot;:  0.0,<br>            &quot;saved_by_caching&quot;:  0.0,<br>            &quot;saved_by_batching&quot;: 0.0,<br>        }<br>    <br>    def generate(<br>        self,<br>        query: str,<br>        system_prompt: str = &quot;&quot;,<br>        context: str = &quot;&quot;,<br>        require_quality: bool = False,<br>        cacheable: bool = True,<br>        ttl: int = 3600,<br>    ) -&gt; dict:<br>        self._stats[&quot;total_requests&quot;] += 1<br>        <br>        # Step 1: Check cache<br>        if cacheable:<br>            cached = self.cache.get(query, context)<br>            if cached:<br>                self._stats[&quot;cache_hits&quot;]  += 1<br>                self._stats[&quot;saved_by_caching&quot;] += cached.get(&quot;cost_usd&quot;, 0.0)<br>                return {<br>                    &quot;answer&quot;:     cached[&quot;answer&quot;],<br>                    &quot;cost_usd&quot;:   0.0,<br>                    &quot;from_cache&quot;: True,<br>                    &quot;model&quot;:      cached.get(&quot;model&quot;, &quot;cache&quot;),<br>                }<br>        <br>        # Step 2: Route to appropriate model<br>        decision = self.router.route(query, context, force_quality=require_quality)<br>        <br>        # Step 3: Compress context if large<br>        final_context = context<br>        if context and len(context.split()) &gt; 1000:<br>            deduped = self.compressor.remove_redundant_context(<br>                context.split(&quot;\n\n&quot;), query<br>            )<br>            final_context = &quot;\n\n&quot;.join(deduped)<br>        <br>        # Step 4: Build messages with compressed history<br>        messages = []<br>        if system_prompt:<br>            messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt})<br>        if final_context:<br>            messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n{final_context}&quot;})<br>            messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Understood, I have the context.&quot;})<br>        messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query})<br>        <br>        # Step 5: Generate<br>        client = openai.OpenAI()<br>        response = client.chat.completions.create(<br>            model=decision.model,<br>            messages=messages,<br>        )<br>        <br>        answer = response.choices[0].message.content<br>        cost   = compute_cost(decision.model, response.usage.prompt_tokens, response.usage.completion_tokens)<br>        <br>        self._stats[&quot;total_cost_usd&quot;] += cost[&quot;total_cost_usd&quot;]<br>        <br>        # Step 6: Populate cache<br>        if cacheable:<br>            self.cache.set(<br>                query, answer,<br>                model=decision.model,<br>                cost_usd=cost[&quot;total_cost_usd&quot;],<br>                context=context,<br>                ttl_seconds=ttl,<br>            )<br>        <br>        return {<br>            &quot;answer&quot;:      answer,<br>            &quot;cost_usd&quot;:    cost[&quot;total_cost_usd&quot;],<br>            &quot;from_cache&quot;:  False,<br>            &quot;model&quot;:       decision.model,<br>            &quot;complexity&quot;:  decision.complexity.value,<br>            &quot;input_tokens&quot;:  response.usage.prompt_tokens,<br>            &quot;output_tokens&quot;: response.usage.completion_tokens,<br>        }<br>    <br>    def cost_report(self) -&gt; dict:<br>        stats = self._stats<br>        total = stats[&quot;total_requests&quot;]<br>        cache_rate = stats[&quot;cache_hits&quot;] / max(total, 1)<br>        <br>        return {<br>            **stats,<br>            &quot;cache_hit_rate&quot;:       f&quot;{cache_rate:.1%}&quot;,<br>            &quot;avg_cost_per_request&quot;: round(stats[&quot;total_cost_usd&quot;] / max(total - stats[&quot;cache_hits&quot;], 1), 6),<br>            &quot;total_saved_usd&quot;:      round(stats[&quot;saved_by_caching&quot;] + stats[&quot;saved_by_routing&quot;] + stats[&quot;saved_by_batching&quot;], 4),<br>        }</pre><h2>📊 Part 7: Cost Attribution and Observability</h2><p>You can’t optimize what you can’t see. Instrument every token:</p><pre>import time<br>from functools import wraps<br><br>def track_cost(feature_name: str, user_id: str | None = None):<br>    &quot;&quot;&quot;<br>    Decorator for cost attribution per feature and user.<br>    Feeds into your cost analytics dashboard.<br>    &quot;&quot;&quot;<br>    def decorator(func):<br>        @wraps(func)<br>        def wrapper(*args, **kwargs):<br>            start = time.perf_counter()<br>            result = func(*args, **kwargs)<br>            elapsed = time.perf_counter() - start<br>            <br>            # In production: send to your analytics system<br>            # (Datadog, ClickHouse, BigQuery, etc.)<br>            event = {<br>                &quot;feature&quot;:       feature_name,<br>                &quot;user_id&quot;:       user_id,<br>                &quot;model&quot;:         result.get(&quot;model&quot;, &quot;unknown&quot;),<br>                &quot;cost_usd&quot;:      result.get(&quot;cost_usd&quot;, 0),<br>                &quot;from_cache&quot;:    result.get(&quot;from_cache&quot;, False),<br>                &quot;input_tokens&quot;:  result.get(&quot;input_tokens&quot;, 0),<br>                &quot;output_tokens&quot;: result.get(&quot;output_tokens&quot;, 0),<br>                &quot;latency_ms&quot;:    round(elapsed * 1000),<br>                &quot;timestamp&quot;:     time.time(),<br>            }<br>            <br>            # Log to your telemetry sink<br>            print(f&quot;[COST] {feature_name}: ${event[&#39;cost_usd&#39;]:.6f} | &quot;<br>                  f&quot;model={event[&#39;model&#39;]} | cache={event[&#39;from_cache&#39;]} | &quot;<br>                  f&quot;{event[&#39;latency_ms&#39;]}ms&quot;)<br>            <br>            return result<br>        return wrapper<br>    return decorator<br><br># Usage:<br>@track_cost(&quot;document_qa&quot;, user_id=&quot;user_123&quot;)<br>def answer_document_question(question: str, document: str) -&gt; dict:<br>    stack = CostOptimizedAIStack()<br>    return stack.generate(question, context=document)</pre><h3>Cost Budget Enforcement</h3><pre>class CostBudgetEnforcer:<br>    &quot;&quot;&quot;<br>    Enforce per-user, per-feature, and global cost budgets.<br>    Prevents runaway costs from prompt injection or abuse.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(<br>        self,<br>        daily_global_limit_usd:  float = 500.0,<br>        per_user_daily_limit_usd: float = 1.0,<br>        per_request_limit_usd:   float = 0.10,<br>    ):<br>        self.global_limit     = daily_global_limit_usd<br>        self.per_user_limit   = per_user_daily_limit_usd<br>        self.per_request_limit = per_request_limit_usd<br>        <br>        self._global_today:   float = 0.0<br>        self._user_totals:    dict[str, float] = {}<br>    <br>    def check_and_record(self, user_id: str, estimated_cost: float) -&gt; dict:<br>        user_total = self._user_totals.get(user_id, 0.0)<br>        <br>        if estimated_cost &gt; self.per_request_limit:<br>            return {&quot;allowed&quot;: False, &quot;reason&quot;: f&quot;Single request exceeds ${self.per_request_limit} limit&quot;}<br>        <br>        if user_total + estimated_cost &gt; self.per_user_limit:<br>            return {&quot;allowed&quot;: False, &quot;reason&quot;: f&quot;Daily user budget of ${self.per_user_limit} exceeded&quot;}<br>        <br>        if self._global_today + estimated_cost &gt; self.global_limit:<br>            return {&quot;allowed&quot;: False, &quot;reason&quot;: &quot;Global daily budget exhausted — degraded service mode&quot;}<br>        <br>        # All checks passed<br>        self._user_totals[user_id] = user_total + estimated_cost<br>        self._global_today        += estimated_cost<br>        return {&quot;allowed&quot;: True}</pre><h2>⚠️ Common Mistakes That Drain AI Budgets</h2><h3>Mistake 1: System Prompt Bloat</h3><p>Every example, every caveat, every “please remember to” in your system prompt is paid on every single call. Audit your system prompt quarterly. Cut anything that doesn’t measurably change output quality.</p><h3>Mistake 2: Including Full Conversation History</h3><p>Naive implementations append the entire conversation to every turn. At 10 turns of 500 tokens each, that’s 5,000 extra tokens on turn 10 — most of which the model won’t attend to anyway (lost-in-the-middle). Use sliding window (last 3–5 turns) + hierarchical summary.</p><h3>Mistake 3: Using o3/o4 for Everything</h3><p>Reasoning models are spectacular for complex multi-step problems and terrible ROI for simple tasks. o3 at $10/1M input tokens is 66× more expensive than GPT-4o-mini. A query classification model costs ~$0.00002 per routing decision. The math is obvious.</p><h3>Mistake 4: No Cache for Deterministic Outputs</h3><p>FAQ answers, product descriptions, static content — these don’t need regenerating. If you’re generating the same answer 50 times/day at $0.01 each, that’s $180/year for one FAQ item. Cache it with a 24-hour TTL and re-generate overnight.</p><h3>Mistake 5: Synchronous Calls for Async Workloads</h3><p>Batch APIs offer 50% discounts precisely because they process async. Background jobs — nightly summaries, bulk enrichment, offline evaluation — don’t need real-time inference. Use the batch API and literally halve that slice of your bill.</p><h3>Mistake 6: Not Tracking Cost per Feature</h3><p>When “AI costs” is a single line item, you can’t know which feature is responsible for cost spikes. Instrument cost attribution from day one. In practice, one feature (often conversational AI with long context) accounts for 60–70% of total AI spend.</p><h2>🎤 Quick Questions</h2><p><strong>Q1: How would you reduce AI infrastructure costs for a product at scale?</strong></p><p>I approach this as five independent levers that compound. First, semantic caching — even a 20% cache hit rate on a $50K/month bill saves $120K/year. Second, intelligent model routing using a lightweight classifier to direct simple queries to cheap models (GPT-4o-mini is 16× cheaper than GPT-4o for comparable quality on simple tasks). Third, prompt compression — auditing system prompts, using sliding window conversation history with summarization, and deduplicating retrieved context before injecting it. Fourth, batch API adoption for any non-real-time workloads — that’s a guaranteed 50% discount with zero quality trade-off. Fifth, cost attribution per feature so you can see exactly where money is going and prioritize optimizations accordingly. Together these typically achieve 60–75% reduction without changing model quality for end users.</p><p><strong>Q2: When would you NOT optimize for cost?</strong></p><p>When the optimization meaningfully degrades quality for high-stakes outputs. A legal contract review, a medical diagnosis assistant, or a security audit tool should use the best available model — the cost of a missed issue vastly exceeds the token cost savings. I also don’t route to cheaper models when the downstream action is irreversible: if the AI output is going directly into a database or triggering an API call, I’d rather pay for quality. The optimization playbook applies to the 90% of queries where good-enough is genuinely good enough, not the 10% where best-in-class matters.</p><p><strong>Q3: How do you decide the threshold for semantic cache hits?</strong></p><p>It’s a precision-recall trade-off. Too low a threshold (say, 0.85) and you return cached answers for queries that are actually meaningfully different — a question about “Python list comprehensions” might hit the cache for a question about “Python dict comprehensions” and return a wrong answer. Too high (0.99) and you only hit on near-exact duplicates, which limits your savings. In practice I run offline analysis: take 1,000 representative query pairs, have humans label them as “same intent” or “different intent,” then plot precision-recall across thresholds. For most general knowledge domains, 0.92–0.94 is a good starting point. For high-stakes or highly specific domains (medical, legal, financial), I’d push it to 0.96+.</p><p><strong>Q4: What’s the ROI calculation for adding a semantic cache?</strong></p><p>Back of envelope: the cache adds about $0.00002 per incoming query for an embedding lookup (OpenAI text-embedding-3-small). If a cache hit saves $0.0025 (a GPT-4o call), you need a hit rate above 0.8% to break even — which is almost always achievable. At 25% hit rate, every $1 spent on embedding lookups saves $31.25 in API calls. For the cache infrastructure itself (Redis): roughly $50/month for a managed instance handles millions of cached entries. At even 10% hit rate on 1M monthly queries at $0.005 average saved per hit, that’s $5,000/month saved for $50/month infrastructure — a 100× ROI before you’ve done anything else.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Token pricing is a multiplier problem</strong> — system prompt tokens, history tokens, and output tokens multiply by request volume. 10% savings per token becomes 10% savings across your entire AI bill.</li><li><strong>Model routing is the single highest-ROI optimization</strong>. Route 70% of queries to cheap models, 25% to mid-tier, 5% to expensive — and your average cost drops 60–80% with negligible quality impact on typical workloads.</li><li><strong>Semantic caching compounds over time</strong>. As your cache warms up, hit rates grow. A mature cache on a production FAQ product commonly reaches 40–60% hit rates.</li><li><strong>Prompt compression is free money</strong> — a one-time investment to reduce your system prompt and history injection pays dividends on every call forever.</li><li><strong>Batch APIs are an immediate 50% discount</strong> for any non-real-time workload. Most teams have 20–40% of their AI calls that could be deferred without user impact.</li><li><strong>Cost attribution is the foundation</strong> — you can’t optimize what you can’t measure. Instrument cost per feature, per user, and per model from the very first deployment.</li><li><strong>Budget enforcement prevents disasters</strong>. Runaway costs from prompt injection, abuse, or bugs are preventable with per-user and global daily cost caps.</li></ol><h2>🔧 Production Tools</h2><ul><li><a href="https://langfuse.com/">LangFuse</a> — LLM observability with native cost tracking per trace</li><li><a href="https://www.helicone.ai/">Helicone</a> — Proxy-based cost analytics, caching, and rate limiting</li><li><a href="https://openmeter.io/">OpenMeter</a> — Usage-based billing metering for AI products</li><li><a href="https://github.com/zilliztech/GPTCache">GPTCache</a> — Open-source semantic cache for LLM applications</li><li><a href="https://github.com/BerriAI/litellm">LiteLLM</a> — Unified API + cost tracking across 100+ providers</li></ul><h2>📖 Further Reading</h2><ul><li><a href="https://platform.openai.com/docs/guides/batch">OpenAI Batch API Documentation</a> — 50% discount, up to 24h window</li><li><a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching">Anthropic Prompt Caching</a> — Cache prefixes of up to 90% cost reduction for repeated context</li><li><a href="https://www.anyscale.com/blog/llm-cost-optimization">“LLM Cost Optimization Guide”</a> — Anyscale’s production-scale techniques</li><li><a href="https://www.together.ai/pricing">Together AI Pricing</a> — Open-source models often 10–20× cheaper than frontier APIs</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9064d2911732" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 15: Vector Databases — The Memory Layer of AI Applications]]></title>
            <link>https://learncsdesigns.medium.com/day-15-vector-databases-the-memory-layer-of-ai-applications-7674a8ede188?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7674a8ede188</guid>
            <category><![CDATA[embeddingsearch]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[ai-architecture]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 15:59:54 GMT</pubDate>
            <atom:updated>2026-07-12T15:59:54.350Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“If embeddings are the language AI thinks in, vector databases are the long-term memory it thinks with.” — Pinecone Engineering Blog, 2024</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7a461iALTi9pQkPJbg1Mfw.png" /></figure><h2>📖 The Problem Vector Databases Solve</h2><p>Imagine you’re building a customer support chatbot for a company with 50,000 documentation pages. A user asks: <em>“How do I reset my 2FA when I’ve lost my phone?”</em></p><p>A traditional keyword search would look for documents containing “reset”, “2FA”, “lost”, “phone” — and fail spectacularly when your docs use language like “disable authenticator app after device replacement.”</p><p>The words are completely different. The <strong>meaning</strong> is identical.</p><p>This is the fundamental gap between <strong>lexical search</strong> (what SQL and Elasticsearch do) and <strong>semantic search</strong> (what vector databases do).</p><h3>The Embedding Revolution</h3><p>Modern AI systems represent meaning as high-dimensional vectors — lists of 768, 1536, or 3072 floating-point numbers. These aren’t arbitrary: they encode <strong>geometric relationships between concepts</strong>.</p><pre>&quot;king&quot; - &quot;man&quot; + &quot;woman&quot; ≈ &quot;queen&quot;<br>&quot;Paris&quot; - &quot;France&quot; + &quot;Germany&quot; ≈ &quot;Berlin&quot;<br>&quot;reset 2FA&quot; ≈ &quot;disable authenticator after losing phone&quot;</pre><p>When two pieces of text have similar meanings, their embedding vectors point in similar directions in this high-dimensional space. <strong>Vector search finds the nearest neighbors in that space.</strong></p><p>The challenge? You can have hundreds of millions of these vectors. Comparing your query vector against every single one would take minutes. You need a database purpose-built for this problem.</p><h2>📐 Part 1: The Mathematics of Similarity</h2><p>Before we touch a database, you need to deeply understand how similarity is measured. This isn’t academic — it determines your search quality in production.</p><h3>The Three Distance Metrics</h3><h4>1. Cosine Similarity</h4><p>Measures the <strong>angle</strong> between two vectors, ignoring their magnitude. This is the most common choice for text embeddings.</p><pre>cosine_similarity(A, B) = (A · B) / (|A| × |B|)<br>Range: [-1, 1], where 1 = identical, 0 = orthogonal, -1 = opposite</pre><p><strong>Why it dominates text search</strong>: A sentence like “cats are great” and “I love cats, cats are truly wonderful creatures” have different magnitudes (different word counts → different vector norms) but encode similar meaning. Cosine similarity normalizes this away.</p><h4>2. Dot Product (Inner Product)</h4><p>Raw multiplication and sum, without normalization.</p><pre>dot_product(A, B) = Σ(Aᵢ × Bᵢ)</pre><p><strong>When to use it</strong>: OpenAI’s text-embedding-3-large and text-embedding-3-small are specifically optimized for dot product similarity, as are many modern embedding models trained with normalized outputs. Faster than cosine (one less division operation at scale).</p><p><strong>Pro tip</strong>: If your vectors are already L2-normalized (most modern models normalize them), dot product and cosine similarity are mathematically equivalent.</p><h4>3. Euclidean Distance (L2)</h4><p>Straight-line distance between two points in the vector space.</p><pre>euclidean(A, B) = √(Σ(Aᵢ - Bᵢ)²)<br>Range: [0, ∞), where 0 = identical</pre><p><strong>Best for</strong>: Image embeddings, where magnitude carries meaningful information (brightness, color intensity). Rarely used for pure text retrieval.</p><h3>Implementing All Three from Scratch</h3><pre>import numpy as np<br>from typing import List, Tuple<br><br>def cosine_similarity(a: np.ndarray, b: np.ndarray) -&gt; float:<br>    &quot;&quot;&quot;<br>    Cosine similarity between two vectors.<br>    Standard choice for text embeddings.<br>    &quot;&quot;&quot;<br>    dot = np.dot(a, b)<br>    norm_a = np.linalg.norm(a)<br>    norm_b = np.linalg.norm(b)<br>    <br>    if norm_a == 0 or norm_b == 0:<br>        return 0.0<br>    <br>    return dot / (norm_a * norm_b)<br><br>def dot_product_similarity(a: np.ndarray, b: np.ndarray) -&gt; float:<br>    &quot;&quot;&quot;<br>    Inner product — fast when vectors are pre-normalized.<br>    Preferred for OpenAI text-embedding-3-* models.<br>    &quot;&quot;&quot;<br>    return np.dot(a, b)<br><br>def euclidean_distance(a: np.ndarray, b: np.ndarray) -&gt; float:<br>    &quot;&quot;&quot;<br>    L2 distance — best for image/multimodal embeddings.<br>    Lower = more similar (inverse of similarity).<br>    &quot;&quot;&quot;<br>    return np.linalg.norm(a - b)<br><br>def l2_to_similarity(distance: float) -&gt; float:<br>    &quot;&quot;&quot;Convert L2 distance to a similarity score in [0, 1].&quot;&quot;&quot;<br>    return 1 / (1 + distance)<br><br># --- Demonstration ---<br>def compare_similarity_metrics():<br>    # Simulate embeddings for conceptually similar sentences<br>    np.random.seed(42)<br>    <br>    # Base vector representing &quot;reset 2FA&quot;<br>    query = np.random.randn(1536)<br>    query = query / np.linalg.norm(query)  # normalize<br>    <br>    # Similar sentence (small angle difference)<br>    noise_small = np.random.randn(1536) * 0.1<br>    similar = query + noise_small<br>    similar = similar / np.linalg.norm(similar)<br>    <br>    # Dissimilar sentence (large angle difference)<br>    noise_large = np.random.randn(1536)<br>    dissimilar = noise_large / np.linalg.norm(noise_large)<br>    <br>    print(&quot;=== Similarity Metrics Comparison ===\n&quot;)<br>    print(&quot;Query vs Similar Text:&quot;)<br>    print(f&quot;  Cosine Similarity:   {cosine_similarity(query, similar):.4f}&quot;)<br>    print(f&quot;  Dot Product:         {dot_product_similarity(query, similar):.4f}&quot;)<br>    print(f&quot;  Euclidean Distance:  {euclidean_distance(query, similar):.4f}&quot;)<br>    <br>    print(&quot;\nQuery vs Dissimilar Text:&quot;)<br>    print(f&quot;  Cosine Similarity:   {cosine_similarity(query, dissimilar):.4f}&quot;)<br>    print(f&quot;  Dot Product:         {dot_product_similarity(query, dissimilar):.4f}&quot;)<br>    print(f&quot;  Euclidean Distance:  {euclidean_distance(query, dissimilar):.4f}&quot;)<br><br>compare_similarity_metrics()</pre><h2>🏎️ Part 2: Why Exact Search Doesn’t Scale — ANN Algorithms</h2><p>Here’s the cold math: with 10 million vectors of 1536 dimensions, brute-force nearest-neighbor search requires:</p><pre>10,000,000 vectors × 1,536 dimensions × 4 bytes (float32)<br>= ~61 GB just for storage<br><br>For each query: 10,000,000 dot products × 1,536 multiplications<br>= 15.36 billion operations per query<br>≈ 15-50ms on modern hardware</pre><p>At 10 billion vectors (enterprise scale), that’s 15–50 seconds per query. Unacceptable.</p><p><strong>Approximate Nearest Neighbor (ANN)</strong> algorithms trade a tiny amount of accuracy for 10–100x speed improvements. The three dominant approaches:</p><h3>HNSW — Hierarchical Navigable Small World</h3><p>The gold standard. Used by Qdrant, Weaviate, pgvector, and optionally others.</p><p><strong>The core insight</strong>: Build a multi-layer graph where:</p><ul><li>The <strong>top layer</strong> has few nodes connected by long-range links (fast coarse navigation)</li><li>The <strong>bottom layer</strong> has all nodes with short-range links (precise local search)</li></ul><p>Think of it like a highway system: you zoom across the country on interstates, then navigate local streets to reach your destination.</p><pre>Layer 2 (coarsest):  ●────────────────────●<br>                           \          /<br>Layer 1 (medium):    ●──●──●──●──●──●──●<br>                      \  |  |  |  |  |  /<br>Layer 0 (all nodes): ●─●─●─●─●─●─●─●─●─●─●─●<br>                                ^<br>                          Your query starts here<br>                          and navigates upward</pre><p><strong>Performance</strong>: O(log n) query complexity vs O(n) for brute force.</p><p><strong>Parameters that matter in production</strong>:</p><ul><li>M (connections per node): Higher = better recall, more memory. Default: 16. Production: 32-64.</li><li>ef_construction (build-time beam width): Higher = better index quality, slower build. Default: 200.</li><li>ef_search (query-time beam width): Higher = better recall, slower queries. Tune per use case.</li></ul><h3>IVF — Inverted File Index</h3><p><strong>The approach</strong>: Cluster your vectors into Voronoi cells (using k-means). At query time, only search the nearest nprobe clusters.</p><pre>1. During indexing: cluster 1M vectors into 1000 buckets<br>2. During query: find the 10 nearest buckets, search only those<br>3. Result: check 10,000 vectors instead of 1,000,000 (100x speedup)</pre><p><strong>Best for</strong>: Very large datasets (&gt;10M vectors) where you need tunable recall/speed tradeoff.</p><h3>PQ — Product Quantization</h3><p><strong>The approach</strong>: Compress each vector by splitting it into subvectors and encoding each subvector as one of 256 possible codes. A 1536-dimensional float32 vector (6KB) becomes ~192 bytes — 32x compression.</p><pre>Original: [0.2, -0.8, 0.1, 0.4, ... 1536 dims] (6144 bytes)<br>PQ-compressed: [127, 43, 220, 11, ... 192 codes] (192 bytes)</pre><p><strong>Reality in production</strong>: HNSW + PQ (sometimes called HNSWPQ or HNSW+SQ8) is the sweet spot for most production workloads — HNSW for navigation, PQ/SQ for compressed storage and faster distance calculations.</p><h2>🗄️ Part 3: The Five Major Vector Databases</h2><p>Let’s do a real engineering comparison — not marketing copy.</p><h3>Architecture Comparison at a Glance</h3><h3>Database 1: Pinecone</h3><p>The most production-ready managed solution. Zero infrastructure to manage.</p><pre># pip install pinecone<br>from pinecone import Pinecone, ServerlessSpec<br>import openai<br>import os<br><br>pc = Pinecone(api_key=os.environ[&quot;PINECONE_API_KEY&quot;])<br>oai = openai.OpenAI()<br><br># --- Index Setup (one-time) ---<br>INDEX_NAME = &quot;knowledge-base&quot;<br>DIMENSION = 1536  # text-embedding-3-small dimensions<br><br>def setup_pinecone_index():<br>    &quot;&quot;&quot;Create a serverless index with optimal settings.&quot;&quot;&quot;<br>    existing = [idx.name for idx in pc.list_indexes()]<br>    <br>    if INDEX_NAME not in existing:<br>        pc.create_index(<br>            name=INDEX_NAME,<br>            dimension=DIMENSION,<br>            metric=&quot;cosine&quot;,<br>            spec=ServerlessSpec(<br>                cloud=&quot;aws&quot;,<br>                region=&quot;us-east-1&quot;<br>            )<br>        )<br>        print(f&quot;Created index: {INDEX_NAME}&quot;)<br>    <br>    return pc.Index(INDEX_NAME)<br><br># --- Embedding Helper ---<br>def embed(texts: list[str]) -&gt; list[list[float]]:<br>    &quot;&quot;&quot;Embed texts with OpenAI&#39;s most efficient model.&quot;&quot;&quot;<br>    response = oai.embeddings.create(<br>        model=&quot;text-embedding-3-small&quot;,<br>        input=texts<br>    )<br>    return [item.embedding for item in response.data]<br><br># --- Upsert Documents ---<br>def upsert_documents(index, documents: list[dict]):<br>    &quot;&quot;&quot;<br>    Upsert documents with metadata into Pinecone.<br>    <br>    Each document: {&quot;id&quot;: str, &quot;text&quot;: str, &quot;metadata&quot;: dict}<br>    &quot;&quot;&quot;<br>    batch_size = 100  # Pinecone recommends batches of 100<br>    <br>    for i in range(0, len(documents), batch_size):<br>        batch = documents[i:i + batch_size]<br>        <br>        # Embed the batch<br>        texts = [doc[&quot;text&quot;] for doc in batch]<br>        embeddings = embed(texts)<br>        <br>        # Prepare vectors for upsert<br>        vectors = [<br>            {<br>                &quot;id&quot;: doc[&quot;id&quot;],<br>                &quot;values&quot;: embedding,<br>                &quot;metadata&quot;: {<br>                    **doc[&quot;metadata&quot;],<br>                    &quot;text&quot;: doc[&quot;text&quot;][:1000]  # Store truncated text for retrieval<br>                }<br>            }<br>            for doc, embedding in zip(batch, embeddings)<br>        ]<br>        <br>        # Upsert to a namespace for logical isolation<br>        index.upsert(vectors=vectors, namespace=&quot;production&quot;)<br>        print(f&quot;Upserted batch {i//batch_size + 1}&quot;)<br><br># --- Semantic Search ---<br>def semantic_search(<br>    index,<br>    query: str,<br>    top_k: int = 5,<br>    filter: dict | None = None<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;<br>    Search with optional metadata filtering.<br>    <br>    filter example: {&quot;source&quot;: {&quot;$eq&quot;: &quot;faq&quot;}, &quot;category&quot;: {&quot;$in&quot;: [&quot;billing&quot;, &quot;auth&quot;]}}<br>    &quot;&quot;&quot;<br>    query_embedding = embed([query])[0]<br>    <br>    results = index.query(<br>        namespace=&quot;production&quot;,<br>        vector=query_embedding,<br>        top_k=top_k,<br>        include_metadata=True,<br>        filter=filter  # Pre-filter BEFORE ANN search (efficient)<br>    )<br>    <br>    return [<br>        {<br>            &quot;id&quot;: match.id,<br>            &quot;score&quot;: match.score,<br>            &quot;text&quot;: match.metadata.get(&quot;text&quot;, &quot;&quot;),<br>            &quot;metadata&quot;: {k: v for k, v in match.metadata.items() if k != &quot;text&quot;}<br>        }<br>        for match in results.matches<br>    ]<br><br># --- Full Example ---<br>def pinecone_demo():<br>    index = setup_pinecone_index()<br>    <br>    # Sample documents<br>    docs = [<br>        {<br>            &quot;id&quot;: &quot;doc-001&quot;,<br>            &quot;text&quot;: &quot;To reset your two-factor authentication after losing your phone, navigate to Account Settings → Security → Reset 2FA. You&#39;ll need to verify your identity via email.&quot;,<br>            &quot;metadata&quot;: {&quot;source&quot;: &quot;faq&quot;, &quot;category&quot;: &quot;auth&quot;, &quot;version&quot;: &quot;2.1&quot;}<br>        },<br>        {<br>            &quot;id&quot;: &quot;doc-002&quot;, <br>            &quot;text&quot;: &quot;Billing cycles reset on the 1st of each month. You can upgrade or downgrade your plan at any time.&quot;,<br>            &quot;metadata&quot;: {&quot;source&quot;: &quot;faq&quot;, &quot;category&quot;: &quot;billing&quot;, &quot;version&quot;: &quot;2.1&quot;}<br>        }<br>    ]<br>    <br>    upsert_documents(index, docs)<br>    <br>    # Search<br>    results = semantic_search(<br>        index,<br>        query=&quot;How do I disable my authenticator app after getting a new phone?&quot;,<br>        top_k=3,<br>        filter={&quot;category&quot;: {&quot;$eq&quot;: &quot;auth&quot;}}<br>    )<br>    <br>    for r in results:<br>        print(f&quot;Score: {r[&#39;score&#39;]:.4f} | {r[&#39;text&#39;][:100]}...&quot;)</pre><p><strong>Production Pinecone patterns</strong>:</p><ul><li>Use <strong>namespaces</strong> for multi-tenant isolation (one namespace per customer/environment)</li><li>Use <strong>serverless</strong> for variable workloads; <strong>pods</strong> for consistent high-throughput</li><li>Pre-filter with metadata before ANN search (Pinecone does this efficiently internally)</li><li>Batch upserts at 100 vectors max for optimal throughput</li></ul><h3>Database 2: Weaviate</h3><p>The most complete “AI-native database” — combines vector search, BM25, generative search, and structured data.</p><pre># pip install weaviate-client<br>import weaviate<br>import weaviate.classes as wvc<br>from weaviate.classes.query import MetadataQuery, Filter<br><br>def create_weaviate_client():<br>    &quot;&quot;&quot;Connect to Weaviate — local Docker or cloud.&quot;&quot;&quot;<br>    # Local Docker: docker run -p 8080:8080 -p 50051:50051 semitechnologies/weaviate:latest<br>    client = weaviate.connect_to_local()<br>    <br>    # Or cloud:<br>    # client = weaviate.connect_to_weaviate_cloud(<br>    #     cluster_url=os.environ[&quot;WEAVIATE_URL&quot;],<br>    #     auth_credentials=wvc.init.Auth.api_key(os.environ[&quot;WEAVIATE_API_KEY&quot;]),<br>    # )<br>    <br>    return client<br><br>def setup_weaviate_schema(client):<br>    &quot;&quot;&quot;<br>    Create a collection with named vectors and BM25 configuration.<br>    Weaviate supports multiple vector spaces per object!<br>    &quot;&quot;&quot;<br>    if client.collections.exists(&quot;KnowledgeArticle&quot;):<br>        client.collections.delete(&quot;KnowledgeArticle&quot;)<br>    <br>    collection = client.collections.create(<br>        name=&quot;KnowledgeArticle&quot;,<br>        <br>        # Vector configuration<br>        vectorizer_config=[<br>            # Use OpenAI for semantic vectors<br>            wvc.config.Configure.NamedVectors.text2vec_openai(<br>                name=&quot;content_vector&quot;,<br>                source_properties=[&quot;content&quot;],<br>                model=&quot;text-embedding-3-small&quot;<br>            ),<br>            # Use a separate vector space for title-only search<br>            wvc.config.Configure.NamedVectors.text2vec_openai(<br>                name=&quot;title_vector&quot;,<br>                source_properties=[&quot;title&quot;],<br>                model=&quot;text-embedding-3-small&quot;<br>            )<br>        ],<br>        <br>        # BM25 configuration for hybrid search<br>        inverted_index_config=wvc.config.Configure.inverted_index(<br>            bm25_b=0.75,<br>            bm25_k1=1.2<br>        ),<br>        <br>        # HNSW index configuration<br>        vector_index_config=wvc.config.Configure.VectorIndex.hnsw(<br>            ef_construction=200,<br>            max_connections=32,         # M parameter<br>            ef=128,                     # Query-time recall/speed tradeoff<br>            distance_metric=wvc.config.VectorDistances.COSINE<br>        ),<br>        <br>        # Schema properties<br>        properties=[<br>            wvc.config.Property(name=&quot;title&quot;, data_type=wvc.config.DataType.TEXT),<br>            wvc.config.Property(name=&quot;content&quot;, data_type=wvc.config.DataType.TEXT),<br>            wvc.config.Property(name=&quot;category&quot;, data_type=wvc.config.DataType.TEXT),<br>            wvc.config.Property(name=&quot;version&quot;, data_type=wvc.config.DataType.TEXT),<br>        ]<br>    )<br>    <br>    print(f&quot;Created collection: {collection.name}&quot;)<br>    return collection<br><br>def hybrid_search_weaviate(client, query: str, alpha: float = 0.75, limit: int = 5):<br>    &quot;&quot;&quot;<br>    Hybrid search combining BM25 (sparse) + vector (dense).<br>    <br>    alpha=0.0 → pure BM25 keyword search<br>    alpha=0.5 → balanced hybrid  <br>    alpha=1.0 → pure vector search<br>    alpha=0.75 → production sweet spot (mostly semantic, some keyword signal)<br>    &quot;&quot;&quot;<br>    collection = client.collections.get(&quot;KnowledgeArticle&quot;)<br>    <br>    results = collection.query.hybrid(<br>        query=query,<br>        alpha=alpha,                    # Fusion weight<br>        target_vector=&quot;content_vector&quot;, # Which named vector to use<br>        limit=limit,<br>        return_metadata=MetadataQuery(score=True, explain_score=True),<br>        filters=Filter.by_property(&quot;category&quot;).equal(&quot;auth&quot;)  # Pre-filter<br>    )<br>    <br>    return [<br>        {<br>            &quot;id&quot;: str(obj.uuid),<br>            &quot;title&quot;: obj.properties.get(&quot;title&quot;),<br>            &quot;content&quot;: obj.properties.get(&quot;content&quot;, &quot;&quot;)[:200],<br>            &quot;score&quot;: obj.metadata.score,<br>            &quot;explain&quot;: obj.metadata.explain_score<br>        }<br>        for obj in results.objects<br>    ]<br><br>def generative_search_weaviate(client, query: str, limit: int = 3):<br>    &quot;&quot;&quot;<br>    Weaviate&#39;s killer feature: retrieve AND generate in one call.<br>    Passes retrieved chunks directly to an LLM as context.<br>    &quot;&quot;&quot;<br>    collection = client.collections.get(&quot;KnowledgeArticle&quot;)<br>    <br>    results = collection.generate.hybrid(<br>        query=query,<br>        alpha=0.75,<br>        limit=limit,<br>        # Generate a synthesized answer from retrieved chunks<br>        grouped_task=f&quot;Based on the provided documentation, answer this question clearly and concisely: {query}&quot;,<br>        grouped_properties=[&quot;title&quot;, &quot;content&quot;]<br>    )<br>    <br>    print(&quot;=== Generated Answer ===&quot;)<br>    print(results.generated)<br>    print(&quot;\n=== Source Chunks ===&quot;)<br>    for obj in results.objects:<br>        print(f&quot;• {obj.properties.get(&#39;title&#39;)}: {obj.properties.get(&#39;content&#39;, &#39;&#39;)[:100]}...&quot;)</pre><p><strong>Weaviate’s superpower</strong>: <strong>Multi-vector support</strong>. A single object can have multiple vector representations — content vectors, title vectors, image vectors, code vectors — and you can query across all of them with different weights.</p><h3>Database 3: pgvector</h3><p>The pragmatic choice: if you’re already on Postgres, pgvector adds vector search without adding another infrastructure dependency.</p><pre># pip install psycopg2-binary pgvector openai<br>import psycopg2<br>from pgvector.psycopg2 import register_vector<br>import numpy as np<br>import openai<br>import os<br><br>def setup_pgvector():<br>    &quot;&quot;&quot;Setup PostgreSQL with pgvector extension.&quot;&quot;&quot;<br>    conn = psycopg2.connect(os.environ[&quot;DATABASE_URL&quot;])<br>    register_vector(conn)<br>    <br>    cur = conn.cursor()<br>    <br>    # Enable the extension<br>    cur.execute(&quot;CREATE EXTENSION IF NOT EXISTS vector;&quot;)<br>    <br>    # Create table with vector column<br>    cur.execute(&quot;&quot;&quot;<br>        CREATE TABLE IF NOT EXISTS knowledge_articles (<br>            id          SERIAL PRIMARY KEY,<br>            title       TEXT NOT NULL,<br>            content     TEXT NOT NULL,<br>            category    TEXT,<br>            embedding   vector(1536),           -- OpenAI text-embedding-3-small<br>            created_at  TIMESTAMPTZ DEFAULT NOW()<br>        );<br>    &quot;&quot;&quot;)<br>    <br>    # Create HNSW index (pgvector 0.5+, recommended for production)<br>    cur.execute(&quot;&quot;&quot;<br>        CREATE INDEX IF NOT EXISTS articles_embedding_hnsw_idx <br>        ON knowledge_articles <br>        USING hnsw (embedding vector_cosine_ops)<br>        WITH (m = 16, ef_construction = 64);<br>    &quot;&quot;&quot;)<br>    <br>    # IVFFlat alternative (faster build, lower recall — use for 1M+ rows)<br>    # cur.execute(&quot;&quot;&quot;<br>    #     CREATE INDEX IF NOT EXISTS articles_embedding_ivf_idx<br>    #     ON knowledge_articles<br>    #     USING ivfflat (embedding vector_cosine_ops)<br>    #     WITH (lists = 100);  -- sqrt(n_rows) is a good rule of thumb<br>    # &quot;&quot;&quot;)<br>    <br>    conn.commit()<br>    print(&quot;pgvector setup complete&quot;)<br>    return conn<br><br>def upsert_to_postgres(conn, title: str, content: str, category: str):<br>    &quot;&quot;&quot;Insert or update a document with its embedding.&quot;&quot;&quot;<br>    oai = openai.OpenAI()<br>    <br>    response = oai.embeddings.create(<br>        model=&quot;text-embedding-3-small&quot;,<br>        input=content<br>    )<br>    embedding = response.data[0].embedding<br>    <br>    cur = conn.cursor()<br>    cur.execute(&quot;&quot;&quot;<br>        INSERT INTO knowledge_articles (title, content, category, embedding)<br>        VALUES (%s, %s, %s, %s)<br>        ON CONFLICT (id) DO UPDATE<br>        SET content = EXCLUDED.content,<br>            embedding = EXCLUDED.embedding<br>        RETURNING id;<br>    &quot;&quot;&quot;, (title, content, category, embedding))<br>    <br>    conn.commit()<br>    return cur.fetchone()[0]<br><br>def semantic_search_postgres(conn, query: str, top_k: int = 5, category: str | None = None):<br>    &quot;&quot;&quot;<br>    Semantic search with optional metadata filtering via SQL WHERE clause.<br>    pgvector distance operators:<br>    &lt;-&gt; L2 distance (lower = more similar)<br>    &lt;#&gt; negative inner product (lower = more similar for normalized vectors)<br>    &lt;=&gt; cosine distance (lower = more similar)<br>    &quot;&quot;&quot;<br>    oai = openai.OpenAI()<br>    response = oai.embeddings.create(model=&quot;text-embedding-3-small&quot;, input=query)<br>    query_embedding = response.data[0].embedding<br>    <br>    cur = conn.cursor()<br>    <br>    # Build query with optional filter<br>    # Note: metadata filtering via WHERE clause happens AFTER index scan in pgvector<br>    # For high-cardinality filters, this can hurt performance — see partitioned tables below<br>    <br>    if category:<br>        cur.execute(&quot;&quot;&quot;<br>            SELECT <br>                id, title, content, category,<br>                1 - (embedding &lt;=&gt; %s::vector) AS similarity<br>            FROM knowledge_articles<br>            WHERE category = %s<br>            ORDER BY embedding &lt;=&gt; %s::vector<br>            LIMIT %s;<br>        &quot;&quot;&quot;, (query_embedding, category, query_embedding, top_k))<br>    else:<br>        cur.execute(&quot;&quot;&quot;<br>            SELECT <br>                id, title, content, category,<br>                1 - (embedding &lt;=&gt; %s::vector) AS similarity<br>            FROM knowledge_articles<br>            ORDER BY embedding &lt;=&gt; %s::vector<br>            LIMIT %s;<br>        &quot;&quot;&quot;, (query_embedding, query_embedding, top_k))<br>    <br>    rows = cur.fetchall()<br>    return [<br>        {&quot;id&quot;: r[0], &quot;title&quot;: r[1], &quot;content&quot;: r[2][:200], &quot;category&quot;: r[3], &quot;similarity&quot;: float(r[4])}<br>        for r in rows<br>    ]<br><br>def hybrid_search_postgres(conn, query: str, top_k: int = 5, vector_weight: float = 0.7):<br>    &quot;&quot;&quot;<br>    Hybrid search combining pgvector (semantic) + pg_trgm or ts_tsvector (BM25-like).<br>    <br>    Requires: CREATE EXTENSION pg_trgm; and a GIN index on content.<br>    <br>    Production note: True BM25 in Postgres requires ParadeDB or pg_search extension.<br>    pg_trgm gives trigram similarity (good for fuzzy matching, not true BM25).<br>    &quot;&quot;&quot;<br>    oai = openai.OpenAI()<br>    response = oai.embeddings.create(model=&quot;text-embedding-3-small&quot;, input=query)<br>    query_embedding = response.data[0].embedding<br>    <br>    keyword_weight = 1.0 - vector_weight<br>    <br>    cur = conn.cursor()<br>    cur.execute(&quot;&quot;&quot;<br>        WITH vector_scores AS (<br>            SELECT <br>                id,<br>                1 - (embedding &lt;=&gt; %s::vector) AS vector_score<br>            FROM knowledge_articles<br>            ORDER BY embedding &lt;=&gt; %s::vector<br>            LIMIT 50<br>        ),<br>        keyword_scores AS (<br>            SELECT<br>                id,<br>                ts_rank(to_tsvector(&#39;english&#39;, content), plainto_tsquery(&#39;english&#39;, %s)) AS kw_score<br>            FROM knowledge_articles<br>            WHERE to_tsvector(&#39;english&#39;, content) @@ plainto_tsquery(&#39;english&#39;, %s)<br>        )<br>        SELECT <br>            ka.id, ka.title, ka.content, ka.category,<br>            COALESCE(vs.vector_score, 0) * %s + COALESCE(ks.kw_score, 0) * %s AS hybrid_score<br>        FROM knowledge_articles ka<br>        LEFT JOIN vector_scores vs ON ka.id = vs.id<br>        LEFT JOIN keyword_scores ks ON ka.id = ks.id<br>        WHERE vs.id IS NOT NULL OR ks.id IS NOT NULL<br>        ORDER BY hybrid_score DESC<br>        LIMIT %s;<br>    &quot;&quot;&quot;, (query_embedding, query_embedding, query, query, vector_weight, keyword_weight, top_k))<br>    <br>    rows = cur.fetchall()<br>    return [<br>        {&quot;id&quot;: r[0], &quot;title&quot;: r[1], &quot;content&quot;: r[2][:200], &quot;category&quot;: r[3], &quot;hybrid_score&quot;: float(r[4])}<br>        for r in rows<br>    ]</pre><p><strong>pgvector production gotchas</strong>:</p><ol><li><strong>Index maintenance</strong>: VACUUM ANALYZE your table after large bulk inserts. pgvector HNSW indexes get stale.</li><li><strong>Post-filter problem</strong>: WHERE clauses filter AFTER the ANN search. If only 1% of your data matches the filter, you’ll get poor recall. Use <strong>partitioned tables</strong> by your primary filter dimension.</li><li><strong>Memory</strong>: Set maintenance_work_mem = &#39;1GB&#39; during index build for large datasets.</li><li><strong>Max dimensions</strong>: pgvector supports up to 2000 dimensions (16,000 with half precision in 0.7+).</li></ol><h3>Database 4: Qdrant</h3><p>The Rust-built performance champion. Best recall/speed tradeoff, richest filtering model, truly production-grade.</p><pre># pip install qdrant-client<br>from qdrant_client import QdrantClient<br>from qdrant_client.models import (<br>    Distance, VectorParams, PointStruct,<br>    Filter, FieldCondition, MatchValue, Range,<br>    SearchRequest, SparseVector, NamedVector,<br>    SparseIndexParams, SparseVectorParams,<br>    HnswConfigDiff, OptimizersConfigDiff<br>)<br>import openai<br>import os<br><br>oai = openai.OpenAI()<br><br>def setup_qdrant():<br>    &quot;&quot;&quot;Connect to local Qdrant or cloud.&quot;&quot;&quot;<br>    # Local: docker run -p 6333:6333 qdrant/qdrant<br>    client = QdrantClient(host=&quot;localhost&quot;, port=6333)<br>    <br>    # Or cloud:<br>    # client = QdrantClient(<br>    #     url=os.environ[&quot;QDRANT_URL&quot;],<br>    #     api_key=os.environ[&quot;QDRANT_API_KEY&quot;]<br>    # )<br>    <br>    return client<br><br>def create_qdrant_collection(client: QdrantClient, collection_name: str):<br>    &quot;&quot;&quot;<br>    Create a collection with fine-tuned HNSW parameters.<br>    Qdrant&#39;s HNSW implementation is among the most configurable.<br>    &quot;&quot;&quot;<br>    client.recreate_collection(<br>        collection_name=collection_name,<br>        vectors_config=VectorParams(<br>            size=1536,<br>            distance=Distance.COSINE,<br>            # HNSW configuration<br>            hnsw_config=HnswConfigDiff(<br>                m=16,               # Connections per node (16=standard, 32=high recall)<br>                ef_construct=200,   # Build-time accuracy vs speed<br>                full_scan_threshold=10000  # Use brute force for collections &lt; this size<br>            )<br>        ),<br>        # Optimizer settings for write-heavy workloads<br>        optimizers_config=OptimizersConfigDiff(<br>            indexing_threshold=20000,       # Start building index after N points<br>            memmap_threshold=50000,         # Use memory-mapped files above this threshold<br>        )<br>    )<br>    print(f&quot;Created Qdrant collection: {collection_name}&quot;)<br><br>def upsert_qdrant(client: QdrantClient, collection: str, documents: list[dict]):<br>    &quot;&quot;&quot;<br>    Batch upsert with Qdrant&#39;s rich payload system.<br>    Payloads can have any JSON structure — deeply nested, arrays, etc.<br>    &quot;&quot;&quot;<br>    texts = [doc[&quot;content&quot;] for doc in documents]<br>    <br>    # Batch embed<br>    response = oai.embeddings.create(model=&quot;text-embedding-3-small&quot;, input=texts)<br>    embeddings = [item.embedding for item in response.data]<br>    <br>    points = [<br>        PointStruct(<br>            id=doc[&quot;id&quot;],           # Can be int or UUID string<br>            vector=embedding,<br>            payload={               # Rich metadata — no schema required<br>                &quot;title&quot;: doc.get(&quot;title&quot;),<br>                &quot;content&quot;: doc.get(&quot;content&quot;),<br>                &quot;category&quot;: doc.get(&quot;category&quot;),<br>                &quot;tags&quot;: doc.get(&quot;tags&quot;, []),<br>                &quot;metadata&quot;: doc.get(&quot;metadata&quot;, {})<br>            }<br>        )<br>        for doc, embedding in zip(documents, embeddings)<br>    ]<br>    <br>    client.upsert(collection_name=collection, points=points, wait=True)<br>    print(f&quot;Upserted {len(points)} points&quot;)<br><br>def advanced_search_qdrant(<br>    client: QdrantClient,<br>    collection: str,<br>    query: str,<br>    top_k: int = 5,<br>    score_threshold: float = 0.7,   # Minimum similarity cutoff<br>    must_match: dict | None = None,  # Exact match filters<br>    must_range: dict | None = None,  # Range filters<br>    must_not: list[str] | None = None  # Exclude these categories<br>):<br>    &quot;&quot;&quot;<br>    Qdrant&#39;s powerful filtering — runs INSIDE the ANN graph traversal,<br>    not as a post-filter. This is key to maintaining high recall with filters.<br>    &quot;&quot;&quot;<br>    response = oai.embeddings.create(model=&quot;text-embedding-3-small&quot;, input=query)<br>    query_vector = response.data[0].embedding<br>    <br>    # Build Qdrant filter<br>    conditions = []<br>    <br>    if must_match:<br>        for field, value in must_match.items():<br>            conditions.append(FieldCondition(key=field, match=MatchValue(value=value)))<br>    <br>    if must_range:<br>        for field, (gte, lte) in must_range.items():<br>            conditions.append(FieldCondition(key=field, range=Range(gte=gte, lte=lte)))<br>    <br>    search_filter = Filter(must=conditions) if conditions else None<br>    <br>    results = client.search(<br>        collection_name=collection,<br>        query_vector=query_vector,<br>        query_filter=search_filter,<br>        limit=top_k,<br>        score_threshold=score_threshold,<br>        with_payload=True<br>    )<br>    <br>    return [<br>        {<br>            &quot;id&quot;: hit.id,<br>            &quot;score&quot;: hit.score,<br>            &quot;title&quot;: hit.payload.get(&quot;title&quot;),<br>            &quot;content&quot;: hit.payload.get(&quot;content&quot;, &quot;&quot;)[:200],<br>            &quot;category&quot;: hit.payload.get(&quot;category&quot;)<br>        }<br>        for hit in results<br>    ]<br><br># --- Qdrant&#39;s most powerful feature: Sparse + Dense Hybrid ---<br>def hybrid_search_qdrant(client: QdrantClient, collection: str, query: str, top_k: int = 5):<br>    &quot;&quot;&quot;<br>    True sparse + dense hybrid using Qdrant&#39;s native sparse vector support.<br>    Requires collection setup with both dense and sparse vector spaces.<br>    <br>    Note: Requires sparse vectors computed via BM42/SPLADE (not shown here for brevity).<br>    &quot;&quot;&quot;<br>    # In production, compute sparse vector via SPLADE or BM42<br>    # Here we show the API pattern<br>    <br>    response = oai.embeddings.create(model=&quot;text-embedding-3-small&quot;, input=query)<br>    dense_vector = response.data[0].embedding<br>    <br>    # Query both vector spaces (RRF fusion)<br>    results = client.query_points(<br>        collection_name=collection,<br>        prefetch=[<br>            # Dense search<br>            {&quot;query&quot;: dense_vector, &quot;using&quot;: &quot;dense&quot;, &quot;limit&quot;: 20},<br>            # Sparse search (BM25/SPLADE)<br>            # {&quot;query&quot;: sparse_vector, &quot;using&quot;: &quot;sparse&quot;, &quot;limit&quot;: 20},<br>        ],<br>        query={&quot;fusion&quot;: &quot;rrf&quot;},  # Reciprocal Rank Fusion<br>        limit=top_k<br>    )<br>    <br>    return results.points</pre><p><strong>Why Qdrant in production</strong>:</p><ul><li><strong>Filtered HNSW</strong>: Filtering happens <em>inside</em> the graph traversal, not after. At 10% selectivity, Qdrant maintains 95%+ recall. Pinecone-style post-filter databases drop to 60–70%.</li><li><strong>Named vectors</strong>: Multiple vector spaces per collection (like Weaviate’s multi-vector)</li><li><strong>Payload indexing</strong>: Create explicit indices on filter fields for O(1) filter lookup</li><li><strong>Rust core</strong>: Predictable latency under load, no GC pauses</li></ul><h3>Database 5: ChromaDB</h3><p>The developer experience champion. Perfect for RAG prototyping, local development, and small production deployments.</p><pre># pip install chromadb<br>import chromadb<br>from chromadb.utils import embedding_functions<br>import os<br><br>def setup_chroma_persistent():<br>    &quot;&quot;&quot;<br>    Persistent ChromaDB — survives process restarts.<br>    Backed by DuckDB + Parquet locally, or SQLite.<br>    &quot;&quot;&quot;<br>    client = chromadb.PersistentClient(path=&quot;./chroma_store&quot;)<br>    <br>    # Use OpenAI embeddings<br>    openai_ef = embedding_functions.OpenAIEmbeddingFunction(<br>        api_key=os.environ[&quot;OPENAI_API_KEY&quot;],<br>        model_name=&quot;text-embedding-3-small&quot;<br>    )<br>    <br>    collection = client.get_or_create_collection(<br>        name=&quot;knowledge_base&quot;,<br>        embedding_function=openai_ef,<br>        metadata={&quot;hnsw:space&quot;: &quot;cosine&quot;}  # Distance metric<br>    )<br>    <br>    return collection<br><br>def upsert_chroma(collection, documents: list[dict]):<br>    &quot;&quot;&quot;<br>    ChromaDB handles embedding generation automatically.<br>    Simplest upsert API of the five databases.<br>    &quot;&quot;&quot;<br>    collection.upsert(<br>        ids=[doc[&quot;id&quot;] for doc in documents],<br>        documents=[doc[&quot;content&quot;] for doc in documents],<br>        metadatas=[doc[&quot;metadata&quot;] for doc in documents]<br>    )<br><br>def search_chroma(collection, query: str, top_k: int = 5, where: dict | None = None):<br>    &quot;&quot;&quot;<br>    Query with optional metadata filtering.<br>    ChromaDB&#39;s where syntax uses MongoDB-style operators.<br>    &quot;&quot;&quot;<br>    results = collection.query(<br>        query_texts=[query],<br>        n_results=top_k,<br>        where=where,  # e.g., {&quot;category&quot;: &quot;auth&quot;} or {&quot;$and&quot;: [{&quot;category&quot;: &quot;auth&quot;}, {&quot;version&quot;: &quot;2.1&quot;}]}<br>        include=[&quot;documents&quot;, &quot;metadatas&quot;, &quot;distances&quot;]<br>    )<br>    <br>    output = []<br>    for i, (doc, meta, dist) in enumerate(zip(<br>        results[&quot;documents&quot;][0],<br>        results[&quot;metadatas&quot;][0],<br>        results[&quot;distances&quot;][0]<br>    )):<br>        output.append({<br>            &quot;id&quot;: results[&quot;ids&quot;][0][i],<br>            &quot;content&quot;: doc[:200],<br>            &quot;metadata&quot;: meta,<br>            &quot;similarity&quot;: 1 - dist  # ChromaDB returns distance, not similarity<br>        })<br>    <br>    return output<br><br>def chroma_demo():<br>    collection = setup_chroma_persistent()<br>    <br>    docs = [<br>        {<br>            &quot;id&quot;: &quot;doc-001&quot;,<br>            &quot;content&quot;: &quot;Reset 2FA after losing your phone via Account Settings → Security.&quot;,<br>            &quot;metadata&quot;: {&quot;category&quot;: &quot;auth&quot;, &quot;source&quot;: &quot;faq&quot;}<br>        }<br>    ]<br>    <br>    upsert_chroma(collection, docs)<br>    <br>    results = search_chroma(<br>        collection,<br>        query=&quot;lost authenticator app&quot;,<br>        where={&quot;category&quot;: &quot;auth&quot;}<br>    )<br>    <br>    for r in results:<br>        print(f&quot;Similarity: {r[&#39;similarity&#39;]:.4f} — {r[&#39;content&#39;]}&quot;)</pre><h2>🔬 Part 4: Building a Production RAG Pipeline</h2><p>Now let’s wire everything together into a production-grade RAG system with proper chunking, embedding, retrieval, and generation.</p><pre>from dataclasses import dataclass, field<br>from enum import Enum<br>from typing import Any<br>import tiktoken<br>import openai<br>from qdrant_client import QdrantClient<br><br>class ChunkingStrategy(Enum):<br>    FIXED_SIZE = &quot;fixed_size&quot;<br>    SENTENCE = &quot;sentence&quot;<br>    PARAGRAPH = &quot;paragraph&quot;<br>    SEMANTIC = &quot;semantic&quot;        # Split on semantic boundaries using embedding similarity<br>    RECURSIVE = &quot;recursive&quot;     # LangChain-style recursive character splitting<br><br>@dataclass<br>class Chunk:<br>    text: str<br>    metadata: dict<br>    token_count: int<br>    chunk_index: int<br>    source_id: str<br><br>class ProductionRAGPipeline:<br>    &quot;&quot;&quot;<br>    Production-grade RAG pipeline with:<br>    - Smart chunking with overlap<br>    - Batch embedding with retry<br>    - Hybrid search (dense + BM25 signals)<br>    - Context window optimization<br>    - Source attribution<br>    &quot;&quot;&quot;<br>    <br>    def __init__(<br>        self,<br>        vector_db: QdrantClient,<br>        collection: str,<br>        embedding_model: str = &quot;text-embedding-3-small&quot;,<br>        chat_model: str = &quot;gpt-4o&quot;,<br>        chunk_size: int = 512,      # tokens<br>        chunk_overlap: int = 64,    # tokens overlap between chunks<br>    ):<br>        self.vdb = vector_db<br>        self.collection = collection<br>        self.oai = openai.OpenAI()<br>        self.embedding_model = embedding_model<br>        self.chat_model = chat_model<br>        self.chunk_size = chunk_size<br>        self.chunk_overlap = chunk_overlap<br>        self.tokenizer = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br>    <br>    def chunk_document(<br>        self, <br>        text: str, <br>        source_id: str,<br>        metadata: dict,<br>        strategy: ChunkingStrategy = ChunkingStrategy.RECURSIVE<br>    ) -&gt; list[Chunk]:<br>        &quot;&quot;&quot;<br>        Split a document into overlapping chunks.<br>        Overlap ensures context isn&#39;t lost at boundaries.<br>        &quot;&quot;&quot;<br>        tokens = self.tokenizer.encode(text)<br>        chunks = []<br>        <br>        start = 0<br>        chunk_index = 0<br>        <br>        while start &lt; len(tokens):<br>            end = min(start + self.chunk_size, len(tokens))<br>            chunk_tokens = tokens[start:end]<br>            chunk_text = self.tokenizer.decode(chunk_tokens)<br>            <br>            chunks.append(Chunk(<br>                text=chunk_text,<br>                metadata={**metadata, &quot;chunk_index&quot;: chunk_index, &quot;source_id&quot;: source_id},<br>                token_count=len(chunk_tokens),<br>                chunk_index=chunk_index,<br>                source_id=source_id<br>            ))<br>            <br>            # Move forward by (chunk_size - overlap) to create sliding window<br>            start += self.chunk_size - self.chunk_overlap<br>            chunk_index += 1<br>        <br>        return chunks<br>    <br>    def embed_chunks(self, chunks: list[Chunk], batch_size: int = 100) -&gt; list[list[float]]:<br>        &quot;&quot;&quot;Batch embedding with automatic retry on rate limits.&quot;&quot;&quot;<br>        import time<br>        <br>        all_embeddings = []<br>        <br>        for i in range(0, len(chunks), batch_size):<br>            batch = chunks[i:i + batch_size]<br>            texts = [c.text for c in batch]<br>            <br>            for attempt in range(3):<br>                try:<br>                    response = self.oai.embeddings.create(<br>                        model=self.embedding_model,<br>                        input=texts<br>                    )<br>                    all_embeddings.extend([item.embedding for item in response.data])<br>                    break<br>                except openai.RateLimitError:<br>                    wait = 2 ** attempt<br>                    print(f&quot;Rate limited, waiting {wait}s...&quot;)<br>                    time.sleep(wait)<br>        <br>        return all_embeddings<br>    <br>    def index_document(self, doc_id: str, text: str, metadata: dict):<br>        &quot;&quot;&quot;Full pipeline: chunk → embed → store.&quot;&quot;&quot;<br>        from qdrant_client.models import PointStruct<br>        <br>        chunks = self.chunk_document(text, source_id=doc_id, metadata=metadata)<br>        embeddings = self.embed_chunks(chunks)<br>        <br>        points = [<br>            PointStruct(<br>                id=f&quot;{doc_id}-chunk-{chunk.chunk_index}&quot;,<br>                vector=embedding,<br>                payload={<br>                    &quot;text&quot;: chunk.text,<br>                    &quot;source_id&quot;: chunk.source_id,<br>                    &quot;chunk_index&quot;: chunk.chunk_index,<br>                    &quot;token_count&quot;: chunk.token_count,<br>                    **chunk.metadata<br>                }<br>            )<br>            for chunk, embedding in zip(chunks, embeddings)<br>        ]<br>        <br>        self.vdb.upsert(collection_name=self.collection, points=points)<br>        print(f&quot;Indexed {len(chunks)} chunks from document: {doc_id}&quot;)<br>    <br>    def retrieve(<br>        self, <br>        query: str, <br>        top_k: int = 5,<br>        score_threshold: float = 0.65,<br>        filters: dict | None = None<br>    ) -&gt; list[dict]:<br>        &quot;&quot;&quot;<br>        Retrieve with deduplication: if multiple chunks from the same source<br>        are retrieved, merge them to avoid context fragmentation.<br>        &quot;&quot;&quot;<br>        from qdrant_client.models import Filter, FieldCondition, MatchValue<br>        <br>        response = self.oai.embeddings.create(<br>            model=self.embedding_model, <br>            input=query<br>        )<br>        query_vector = response.data[0].embedding<br>        <br>        # Retrieve more than needed to allow for deduplication<br>        raw_results = self.vdb.search(<br>            collection_name=self.collection,<br>            query_vector=query_vector,<br>            limit=top_k * 2,<br>            score_threshold=score_threshold,<br>            with_payload=True<br>        )<br>        <br>        # Deduplicate: keep only the best chunk per source document<br>        seen_sources = {}<br>        for hit in raw_results:<br>            source_id = hit.payload.get(&quot;source_id&quot;, hit.id)<br>            if source_id not in seen_sources or hit.score &gt; seen_sources[source_id][&quot;score&quot;]:<br>                seen_sources[source_id] = {<br>                    &quot;id&quot;: hit.id,<br>                    &quot;score&quot;: hit.score,<br>                    &quot;text&quot;: hit.payload.get(&quot;text&quot;, &quot;&quot;),<br>                    &quot;source_id&quot;: source_id,<br>                    &quot;metadata&quot;: {k: v for k, v in hit.payload.items() if k not in {&quot;text&quot;, &quot;source_id&quot;}}<br>                }<br>        <br>        # Sort by score and return top_k<br>        results = sorted(seen_sources.values(), key=lambda x: x[&quot;score&quot;], reverse=True)<br>        return results[:top_k]<br>    <br>    def generate(self, query: str, context_chunks: list[dict], max_context_tokens: int = 4000) -&gt; str:<br>        &quot;&quot;&quot;<br>        Generate answer using retrieved context.<br>        Respects context window limits via token counting.<br>        &quot;&quot;&quot;<br>        # Build context string with source attribution<br>        context_parts = []<br>        total_tokens = 0<br>        <br>        for chunk in context_chunks:<br>            chunk_text = f&quot;[Source: {chunk[&#39;source_id&#39;]} | Relevance: {chunk[&#39;score&#39;]:.2f}]\n{chunk[&#39;text&#39;]}&quot;<br>            chunk_tokens = len(self.tokenizer.encode(chunk_text))<br>            <br>            if total_tokens + chunk_tokens &gt; max_context_tokens:<br>                break<br>            <br>            context_parts.append(chunk_text)<br>            total_tokens += chunk_tokens<br>        <br>        context = &quot;\n\n---\n\n&quot;.join(context_parts)<br>        <br>        system_prompt = &quot;&quot;&quot;You are a precise, helpful assistant. Answer questions based ONLY on the provided context.<br>If the context doesn&#39;t contain enough information, say so clearly.<br>Always cite your sources using the [Source: ...] identifiers.&quot;&quot;&quot;<br>        <br>        response = self.oai.chat.completions.create(<br>            model=self.chat_model,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n\n{context}\n\n---\n\nQuestion: {query}&quot;}<br>            ],<br>            temperature=0.1  # Low temperature for factual retrieval tasks<br>        )<br>        <br>        return response.choices[0].message.content<br>    <br>    def rag_query(self, query: str) -&gt; dict:<br>        &quot;&quot;&quot;End-to-end RAG: retrieve + generate.&quot;&quot;&quot;<br>        chunks = self.retrieve(query)<br>        <br>        if not chunks:<br>            return {<br>                &quot;answer&quot;: &quot;I couldn&#39;t find relevant information to answer your question.&quot;,<br>                &quot;sources&quot;: [],<br>                &quot;chunks_retrieved&quot;: 0<br>            }<br>        <br>        answer = self.generate(query, chunks)<br>        <br>        return {<br>            &quot;answer&quot;: answer,<br>            &quot;sources&quot;: [c[&quot;source_id&quot;] for c in chunks],<br>            &quot;chunks_retrieved&quot;: len(chunks),<br>            &quot;top_score&quot;: chunks[0][&quot;score&quot;] if chunks else 0<br>        }</pre><h2>⚡ Part 5: Hybrid Search — Dense + Sparse in Production</h2><p>Pure vector search has a well-known failure mode: <strong>exact keyword recall</strong>. If a user searches for “CVE-2024–1234” or “ORDER BY created_at DESC”, semantic search may miss the exact match even when it’s present.</p><p><strong>Hybrid search</strong> combines:</p><ul><li><strong>Dense vectors</strong> (semantic meaning from neural embeddings)</li><li><strong>Sparse vectors</strong> (BM25-style term frequency signals)</li></ul><p>The result: high recall for both semantic queries AND exact keyword matches.</p><pre>Query: &quot;reset 2FA&quot;<br>Dense search finds: &quot;disable authenticator app after device replacement&quot; ✅ (semantic match)<br>BM25 search finds: &quot;2FA reset instructions&quot; ✅ (keyword match)<br>Hybrid finds BOTH ✅✅</pre><h3>Reciprocal Rank Fusion (RRF)</h3><p>The standard approach to merging dense and sparse results:</p><pre>def reciprocal_rank_fusion(<br>    dense_results: list[dict],<br>    sparse_results: list[dict],<br>    k: int = 60,                # RRF hyperparameter — typically 60<br>    dense_weight: float = 0.7,<br>    sparse_weight: float = 0.3<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;<br>    Merge two ranked lists using RRF.<br>    <br>    RRF score = dense_weight * 1/(k + rank_dense) + sparse_weight * 1/(k + rank_sparse)<br>    <br>    The k=60 constant prevents a single top result from dominating the fusion.<br>    Empirically shown to outperform linear score combination in most retrieval tasks.<br>    &quot;&quot;&quot;<br>    scores = {}<br>    <br>    # Score from dense results<br>    for rank, result in enumerate(dense_results, start=1):<br>        doc_id = result[&quot;id&quot;]<br>        if doc_id not in scores:<br>            scores[doc_id] = {&quot;result&quot;: result, &quot;rrf_score&quot;: 0}<br>        scores[doc_id][&quot;rrf_score&quot;] += dense_weight * (1 / (k + rank))<br>    <br>    # Score from sparse results  <br>    for rank, result in enumerate(sparse_results, start=1):<br>        doc_id = result[&quot;id&quot;]<br>        if doc_id not in scores:<br>            scores[doc_id] = {&quot;result&quot;: result, &quot;rrf_score&quot;: 0}<br>        scores[doc_id][&quot;rrf_score&quot;] += sparse_weight * (1 / (k + rank))<br>    <br>    # Sort by combined RRF score<br>    merged = sorted(scores.values(), key=lambda x: x[&quot;rrf_score&quot;], reverse=True)<br>    <br>    return [<br>        {**item[&quot;result&quot;], &quot;rrf_score&quot;: item[&quot;rrf_score&quot;]}<br>        for item in merged<br>    ]</pre><h2>🏗️ Part 6: Choosing the Right Vector Database</h2><p>This is the question every AI architect faces. Here’s a production decision framework:</p><pre>from dataclasses import dataclass<br>from enum import Enum<br><br>class UseCase(Enum):<br>    PROTOTYPE = &quot;prototype&quot;<br>    STARTUP_MVP = &quot;startup_mvp&quot;<br>    ENTERPRISE_SEARCH = &quot;enterprise_search&quot;<br>    MULTI_TENANT_SAAS = &quot;multi_tenant_saas&quot;<br>    EXISTING_POSTGRES = &quot;existing_postgres&quot;<br>    HIGH_PERFORMANCE = &quot;high_performance&quot;<br>    FULL_AI_STACK = &quot;full_ai_stack&quot;<br><br>@dataclass<br>class RequirementsProfile:<br>    use_case: UseCase<br>    scale: int                      # Expected number of vectors<br>    latency_sla_ms: int             # P99 target in milliseconds<br>    filter_heavy: bool              # Heavy metadata filtering?<br>    hybrid_search_needed: bool      # Need BM25 + vector?<br>    self_hosted: bool               # Must run on your infra?<br>    team_size: int                  # Small teams → simpler is better<br>    existing_postgres: bool         # Already using Postgres?<br><br>def recommend_vector_database(req: RequirementsProfile) -&gt; dict:<br>    &quot;&quot;&quot;Production vector database selection guide.&quot;&quot;&quot;<br>    <br>    # Rapid prototyping or local development<br>    if req.use_case == UseCase.PROTOTYPE or req.scale &lt; 100_000:<br>        return {<br>            &quot;primary&quot;: &quot;ChromaDB&quot;,<br>            &quot;rationale&quot;: &quot;Zero setup, Pythonic API, perfect for development.&quot;,<br>            &quot;migration_path&quot;: &quot;ChromaDB → Qdrant or Pinecone when scaling&quot;,<br>            &quot;code&quot;: &quot;chromadb.PersistentClient(path=&#39;./chroma&#39;)&quot;<br>        }<br>    <br>    # Existing Postgres shop, moderate scale (&lt;10M vectors)<br>    if req.existing_postgres and req.scale &lt; 10_000_000:<br>        return {<br>            &quot;primary&quot;: &quot;pgvector&quot;,<br>            &quot;rationale&quot;: &quot;Zero new infrastructure. Transactional consistency with existing data. SQL you already know.&quot;,<br>            &quot;caveats&quot;: &quot;Post-filter recall issues with &lt;5% selectivity. No native hybrid search without ParadeDB.&quot;,<br>            &quot;when_to_graduate&quot;: &quot;Scale &gt;10M vectors, need P50 &lt;10ms, or need native hybrid search&quot;<br>        }<br>    <br>    # Fully managed, don&#39;t want to touch infrastructure<br>    if not req.self_hosted and req.scale &gt; 1_000_000:<br>        return {<br>            &quot;primary&quot;: &quot;Pinecone&quot;,<br>            &quot;rationale&quot;: &quot;Best managed experience, zero ops, scales to billions of vectors.&quot;,<br>            &quot;caveats&quot;: &quot;Vendor lock-in. Cost scales linearly with vectors + queries.&quot;,<br>            &quot;cost_note&quot;: &quot;~$70/month for 1M vectors + 1M queries/month&quot;<br>        }<br>    <br>    # Performance-critical, self-hosted, filter-heavy<br>    if req.filter_heavy and req.latency_sla_ms &lt; 20:<br>        return {<br>            &quot;primary&quot;: &quot;Qdrant&quot;,<br>            &quot;rationale&quot;: &quot;Filtered HNSW — filtering happens INSIDE ANN graph, not after. Best recall with filters.&quot;,<br>            &quot;differentiator&quot;: &quot;Rust core means predictable latency. No JVM GC pauses.&quot;,<br>            &quot;deployment&quot;: &quot;Docker single-node or Kubernetes cluster&quot;<br>        }<br>    <br>    # Need full AI stack: vector search + structured data + generative<br>    if req.use_case == UseCase.FULL_AI_STACK:<br>        return {<br>            &quot;primary&quot;: &quot;Weaviate&quot;,<br>            &quot;rationale&quot;: &quot;Multi-vector, generative search, BM25 hybrid, modules for every embedding model.&quot;,<br>            &quot;differentiator&quot;: &quot;Built-in vectorization + generative modules = RAG in one platform&quot;,<br>            &quot;deployment&quot;: &quot;Weaviate Cloud (managed) or Kubernetes with Helm&quot;<br>        }<br>    <br>    # High-scale multi-tenant SaaS<br>    if req.use_case == UseCase.MULTI_TENANT_SAAS:<br>        return {<br>            &quot;primary&quot;: &quot;Pinecone (namespaces) or Qdrant (collections)&quot;,<br>            &quot;rationale&quot;: &quot;Both support logical isolation per tenant with shared infrastructure.&quot;,<br>            &quot;pinecone_pattern&quot;: &quot;One namespace per tenant — free logical isolation, shared index&quot;,<br>            &quot;qdrant_pattern&quot;: &quot;One collection per tier (free/pro/enterprise), payload filter by tenant_id&quot;<br>        }<br>    <br>    # Default: Qdrant<br>    return {<br>        &quot;primary&quot;: &quot;Qdrant&quot;,<br>        &quot;rationale&quot;: &quot;Best overall balance of performance, features, and operational simplicity.&quot;,<br>        &quot;note&quot;: &quot;Strong community, active development, excellent Rust-based reliability&quot;<br>    }<br><br># Example usage<br>enterprise_profile = RequirementsProfile(<br>    use_case=UseCase.ENTERPRISE_SEARCH,<br>    scale=50_000_000,<br>    latency_sla_ms=15,<br>    filter_heavy=True,<br>    hybrid_search_needed=True,<br>    self_hosted=True,<br>    team_size=8,<br>    existing_postgres=True<br>)<br><br>recommendation = recommend_vector_database(enterprise_profile)<br>print(recommendation)<br># → Primary: Qdrant</pre><h2>⚠️ Part 7: Common Mistakes That Kill Production Systems</h2><h3>Mistake 1: Not Normalizing Embeddings Before Storage</h3><p>Different embedding models output vectors with different norms. When mixing models or fine-tuning, normalize:</p><pre>import numpy as np<br><br>def normalize(vector: list[float]) -&gt; list[float]:<br>    arr = np.array(vector, dtype=np.float32)<br>    norm = np.linalg.norm(arr)<br>    return (arr / norm).tolist() if norm &gt; 0 else arr.tolist()<br><br># Always normalize before storing when using dot-product metric<br>embedding = get_embedding(text)<br>normalized = normalize(embedding)</pre><h3>Mistake 2: Embedding the Whole Document</h3><p>A 5000-token document produces one vector. The embedding collapses specifics. A single chunk about “resetting 2FA” gets averaged with everything else in the document.</p><p><strong>Rule</strong>: Chunk first, embed per chunk, always.</p><h3>Mistake 3: Ignoring the “Lost in the Middle” Effect at Retrieval</h3><p>If you retrieve 20 chunks and stuff them all into context, the LLM will ignore chunks 5–15. Retrieve broadly, then <strong>rerank</strong>:</p><pre># pip install cohere<br>import cohere<br><br>def rerank_chunks(query: str, chunks: list[dict], top_n: int = 5) -&gt; list[dict]:<br>    &quot;&quot;&quot;<br>    Cohere Rerank 3 outperforms most custom rerankers.<br>    Cost: ~$0.002 per 1000 documents reranked.<br>    &quot;&quot;&quot;<br>    co = cohere.Client(api_key=os.environ[&quot;COHERE_API_KEY&quot;])<br>    <br>    results = co.rerank(<br>        model=&quot;rerank-english-v3.0&quot;,<br>        query=query,<br>        documents=[c[&quot;text&quot;] for c in chunks],<br>        top_n=top_n<br>    )<br>    <br>    return [<br>        {**chunks[r.index], &quot;rerank_score&quot;: r.relevance_score}<br>        for r in results.results<br>    ]</pre><h3>Mistake 4: Using Cosine Similarity as a Quality Threshold Without Calibration</h3><p>A score of 0.7 means <em>very different things</em> across embedding models and domains. GPT-4-scale embeddings at 0.7 may be high-quality; a smaller model at 0.7 may be junk.</p><p><strong>Best practice</strong>: For each new model, compute the distribution of scores on a labeled evaluation set and set thresholds empirically.</p><h3>Mistake 5: No Index on Filter Fields in Qdrant/Weaviate</h3><pre># Qdrant: create explicit payload index for filter fields<br>client.create_payload_index(<br>    collection_name=&quot;knowledge_base&quot;,<br>    field_name=&quot;category&quot;,<br>    field_schema=&quot;keyword&quot;  # or &quot;integer&quot;, &quot;float&quot;, &quot;uuid&quot;<br>)<br># Without this, Qdrant scans all payload JSON — O(n) per filter</pre><h2>🎤 Quick Questions</h2><p><strong>Q1: “Explain the difference between IVF and HNSW and when you’d choose each.”</strong></p><p>HNSW is a graph-based index where each node maintains links to approximate nearest neighbors across multiple layers, enabling O(log n) traversal. It offers excellent recall (typically &gt;95%) and low latency even at high selectivity. The downside is memory: HNSW graphs require ~200 bytes per vector in addition to the vector itself.</p><p>IVF partitions the space into Voronoi cells via k-means clustering. At query time, you scan only nprobe cells. It&#39;s more memory-efficient and parallelizes better across very large datasets (&gt;100M vectors), but recall degrades with aggressive nprobe settings. I&#39;d choose HNSW for most production workloads under 100M vectors, and IVF+PQ for datasets in the billions.</p><p><strong>Q2: “Why is metadata filtering harder than it sounds in vector databases?”</strong></p><p>In traditional databases, a WHERE clause runs on an index and is O(log n). In vector databases, the challenge is that ANN algorithms like HNSW don’t understand metadata — they navigate a graph built purely on vector similarity. The naive approach is “search first, filter after,” which means your top_k=5 results might all fail the filter, giving you 0 results even when relevant documents exist.</p><p>Qdrant solves this with “filtered HNSW” — the filter is applied <em>during</em> graph traversal, skipping filtered-out nodes. Pinecone claims to do similar pre-filtering internally. pgvector does post-filtering, which means recall degrades badly when filter selectivity is low.</p><p><strong>Q3: “What’s the recall/latency tradeoff with HNSW’s ef parameter?”</strong></p><p>ef (or ef_search) controls how many candidate nodes HNSW explores during a query. Higher ef means more exploration — better recall but higher latency. At ef=64, you might get 90% recall at 5ms. At ef=256, 99% recall at 25ms. The right setting depends on your SLA. In practice, I expose ef as a configurable parameter and let teams tune it per endpoint based on their recall requirements.</p><p><strong>Q4: “When would you NOT use a vector database?”</strong></p><p>When exact match is the primary requirement. If your users search for “invoice #INV-2024–007” or “user_id: 12345”, they expect exact results — semantic similarity will introduce irrelevant results and miss the exact match. Vector databases complement, not replace, traditional search. The right architecture for most enterprise systems is hybrid: vector DB for semantic recall + Elasticsearch/Postgres for structured/exact lookups, fused via RRF.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Vector databases solve the semantic gap</strong> — keywords find words, embeddings find meaning. The difference is enormous for natural language queries.</li><li><strong>HNSW dominates for most workloads</strong> under 100M vectors. IVF+PQ for billion-scale. Brute force only for &lt;50K vectors in production.</li><li><strong>Filtered search recall depends on the architecture</strong>: Post-filter (pgvector naive) degrades with selective filters. Filtered HNSW (Qdrant) maintains recall even at 1% selectivity.</li><li><strong>Hybrid search (dense + sparse) closes the recall gap</strong> for exact terms, product codes, IDs, and technical jargon. Use RRF for fusion.</li><li><strong>Chunk → Embed → Store</strong> is the golden rule. Never embed full documents. 512 tokens with 64-token overlap is a production-proven starting point.</li><li><strong>Add a reranker after retrieval</strong> (Cohere Rerank 3 or a local cross-encoder) to improve top-k precision by 15–30% at modest cost.</li><li><strong>Choose your database by constraints, not hype</strong>:</li></ol><ul><li>Already on Postgres + small scale → pgvector</li><li>Prototype / local dev → ChromaDB</li><li>Performance + filtering → Qdrant</li><li>Zero ops + scale → Pinecone</li><li>Full AI platform → Weaviate</li></ul><h2>🔧 Production Considerations</h2><p><strong>Cost modeling</strong>:</p><ul><li>Pinecone serverless: ~$0.07 per 1M vector writes + $0.04 per 1M queries (read units)</li><li>Qdrant Cloud: ~$0.05/GB/month (1M × 1536-dim float32 = ~6GB → ~$0.30/month)</li><li>pgvector: Postgres hosting cost only (RDS: ~$0.20/GB/month)</li><li>Self-hosted: Infrastructure cost (typically 1–2 A10G GPUs for 100M vectors)</li></ul><p><strong>Scalability patterns</strong>:</p><ul><li><strong>Sharding by namespace/tenant</strong>: Horizontal scale without changing query logic</li><li><strong>Tiered storage</strong>: Hot vectors on NVMe, warm on object storage (Qdrant supports this natively)</li><li><strong>Read replicas</strong>: Route heavy read traffic away from the write path</li></ul><p><strong>Observability</strong>:</p><pre># Track these metrics in production:<br># - P50/P95/P99 query latency<br># - Recall@K (requires labeled eval set)<br># - Index freshness (lag between write and searchability)<br># - Cache hit rate (for repeated queries)<br># - Score distribution drift (indicates embedding model drift)</pre><h2>📖 Further Reading</h2><ul><li><a href="https://ann-benchmarks.com/">ANN Benchmarks</a> — Definitive recall/speed benchmarks for all major ANN algorithms</li><li><a href="https://github.com/spotify/annoy">“Approximate Nearest Neighbor Oh Yeah (ANNOY)”</a> — Spotify’s paper on tree-based ANN (historical context)</li><li><a href="https://arxiv.org/abs/1603.09320">“Efficient and Robust Approximate Nearest Neighbor Search Using HNSW”</a> — The original HNSW paper</li><li><a href="https://www.pinecone.io/learn/">Pinecone Learning Center</a> — Best practical vector DB education available</li><li><a href="https://arxiv.org/abs/2307.03172">“Lost in the Middle”</a> — Why context position matters in RAG (Stanford, 2023)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7674a8ede188" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 14: Understanding AI Hallucination — Causes, Patterns & Defenses]]></title>
            <link>https://learncsdesigns.medium.com/day-14-understanding-ai-hallucination-causes-patterns-defenses-84385315f613?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/84385315f613</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[ai-trust]]></category>
            <category><![CDATA[ai-hallucination]]></category>
            <category><![CDATA[llm-reliability]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 14:55:55 GMT</pubDate>
            <atom:updated>2026-07-11T14:55:55.226Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“A language model is not a database. It is a probability distribution over tokens, trained to be plausible — not truthful. Plausibility and truth overlap most of the time, and diverge in exactly the moments that matter most.” — Anthropic Research Blog, 2024</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPKDqEnxk8kmASM5yZI2Sg.png" /></figure><h2>📖 The Uncomfortable Truth About LLMs</h2><p>In November 2022, an AI lawyer named DoNotPay attempted to use GPT-4 to argue a case in US court. The preparation involved AI-generated case citations. Several of the cited cases did not exist.</p><p>In 2023, two New York attorneys filed a brief containing AI-generated citations to six non-existent federal cases. Judge P. Kevin Castel imposed sanctions of $5,000 each. The attorneys had asked ChatGPT to find cases supporting their arguments and never verified the output.</p><p>In 2024, Air Canada’s chatbot invented a bereavement fare discount policy that didn’t exist. The airline was ordered to honor the invented policy in court.</p><p>These are not edge cases. They are inevitable consequences of how language models work.</p><p><strong>Understanding hallucination isn’t optional for production AI engineers. It’s table stakes.</strong></p><h2>🔬 Part 1: What Hallucination Actually Is</h2><p>The term “hallucination” is borrowed from cognitive science — sensory experiences without external stimuli. For LLMs, the operational definition:</p><blockquote><strong>AI Hallucination</strong>: The generation of content that is confidently stated but factually incorrect, fabricated, or unsupported by the provided context.</blockquote><p>But this definition obscures important distinctions. There are fundamentally different failure modes, each with different causes and mitigations.</p><h3>Why LLMs Hallucinate: The Mechanistic View</h3><p>A language model is trained to predict the next token given all previous tokens. The training objective is:</p><pre>maximize: P(token_t | token_1, token_2, ..., token_{t-1})</pre><p>This is a <strong>compression objective</strong>, not a <strong>truth objective</strong>. The model learns patterns that maximize log-likelihood over the training corpus — which includes fiction, speculation, errors, outdated information, and intentional falsehoods alongside factual content.</p><p>When you ask a model a question, it generates the most statistically <em>plausible</em> continuation — the sequence of tokens that the model assigns highest probability to, given its compressed representation of billions of documents. If the training distribution contains more “fluent-sounding answers to questions about X” than “honest admissions of uncertainty about X”, the model will learn to produce fluent answers.</p><p><strong>The key insight</strong>: The model cannot distinguish between “I was trained on correct information about this” and “I was trained on plausible-sounding information about this.” It only knows token probabilities.</p><pre># This is a simplification, but conceptually accurate:<br>def generate_next_token(context: list[int], model_weights: dict) -&gt; int:<br>    &quot;&quot;&quot;<br>    The model picks the MOST PROBABLE next token given context.<br>    It has no truth oracle. It has no database lookup.<br>    It has compressed statistical patterns from training data.<br>    &quot;&quot;&quot;<br>    logits = forward_pass(context, model_weights)<br>    probabilities = softmax(logits)<br>    <br>    # The model will confidently generate &quot;The capital of Australia is Sydney&quot;<br>    # because Sydney appears more often near &quot;capital of Australia&quot; in training data<br>    # than Canberra does — even though Canberra is correct.<br>    return sample_from_distribution(probabilities)</pre><h3>The Training-Inference Gap</h3><p>There’s another structural cause: <strong>the gap between training distribution and deployment queries</strong>.</p><p>Models are trained on web text, books, code, and curated datasets. When deployed, they receive queries about:</p><ul><li>Events after their training cutoff</li><li>Internal company documents they’ve never seen</li><li>Niche technical domains underrepresented in training data</li><li>Numerical computations beyond pattern memorization</li></ul><p>In these out-of-distribution scenarios, the model extrapolates from learned patterns — and extrapolation is exactly where statistical models fail.</p><h2>🗂️ Part 2: The Six Hallucination Patterns</h2><p>Understanding the <em>type</em> of hallucination determines the right defense. These are empirically distinct failure modes:</p><h3>Pattern 1: Factual Confabulation</h3><p>The model states incorrect facts with high confidence. Often occurs with:</p><ul><li>Dates, numbers, and statistics</li><li>Biographical details of non-prominent people</li><li>Legal citations, academic references</li><li>Technical specifications</li></ul><pre>Prompt: &quot;What is the maximum context window of GPT-4 Turbo?&quot;<br>Hallucinated response: &quot;GPT-4 Turbo supports a context window of 256,000 tokens.&quot;<br>Fact: 128,000 tokens (as of training cutoff)</pre><p><strong>Root cause</strong>: The model has seen many similar numbers in context and interpolates a plausible value.</p><h3>Pattern 2: Source Fabrication</h3><p>The model invents citations, papers, URLs, and books that sound authoritative but don’t exist.</p><pre>Prompt: &quot;What research supports multi-head attention?&quot;<br>Hallucinated citation: <br>&quot;As shown in Li et al. (2022), &#39;Efficient Multi-Head Attention via <br>Dynamic Token Routing,&#39; NeurIPS 2022, pp. 4521-4534...&quot;</pre><p>This is particularly dangerous because the citation <em>sounds</em> right — correct venue, plausible authors, believable page range. The model has learned the <em>template</em> of academic citations and fills it with statistically plausible tokens.</p><h3>Pattern 3: Context-Conflation Hallucination</h3><p>When given a long context, models sometimes merge information from different sections, attributing statements to the wrong source or combining separate facts incorrectly.</p><pre>Context: <br>[Document A] &quot;Alice founded the company in 2019.&quot;<br>[Document B] &quot;Bob raised $50M in Series A funding in 2021.&quot;<br><br>Hallucinated output: &quot;Alice raised $50M in a Series A round in 2021.&quot;</pre><p><strong>Root cause</strong>: The “lost in the middle” phenomenon — attention mechanisms struggle with long contexts, and cross-document attribution degrades.</p><h3>Pattern 4: Sycophantic Confabulation</h3><p>The model agrees with incorrect premises stated by the user, then elaborates on them.</p><pre>User: &quot;Given that Einstein won the Nobel Prize for relativity, what did his <br>      acceptance speech focus on?&quot;<br><br>Hallucinated response: &quot;In his Nobel Prize acceptance speech, Einstein <br>discussed the implications of special relativity for our understanding of <br>space and time...&quot;<br><br>Fact: Einstein won the 1921 Nobel Prize for the photoelectric effect, <br>not relativity. He wasn&#39;t present at the award ceremony.</pre><p><strong>Root cause</strong>: RLHF training optimizes for human approval. Humans tend to approve of responses that agree with them. The model learns that agreement leads to positive reward signals.</p><h3>Pattern 5: Temporal Hallucination</h3><p>The model presents outdated information as current, or invents plausible “recent developments.”</p><pre>Prompt: &quot;What&#39;s the latest version of Python?&quot;<br>Response: &quot;The latest stable version of Python is 3.11.4, released in June 2023.&quot;<br><br>[If training cutoff is early 2024, Python 3.12.x may have released — <br> the model doesn&#39;t know what it doesn&#39;t know]</pre><h3>Pattern 6: Numerical and Logical Hallucination</h3><p>Arithmetic, unit conversions, statistical calculations, and multi-step logical inferences — models fail here even when they have all necessary information, because they pattern-match rather than compute.</p><pre>Prompt: &quot;A company has 847 employees. 23% are in engineering. <br>        How many non-engineering employees are there?&quot;<br><br>Hallucinated response: &quot;With 23% of 847 employees in engineering, <br>that&#39;s approximately 195 engineers, leaving 652 non-engineering employees.&quot;<br><br>Calculation check: 847 × 0.23 = 194.81 ≈ 195 engineers. <br>847 - 195 = 652. ← Actually correct this time!<br><br>But subtle errors appear with larger numbers, percentages of percentages,<br>or multi-step calculations.</pre><h2>🧪 Part 3: Detecting Hallucinations Programmatically</h2><p>Defense starts with detection. Here are production-grade detection strategies with code.</p><h3>Strategy 1: Self-Consistency Sampling</h3><p>The core insight: if a model is confident about a fact, it will generate the same answer across multiple independent samples. If it’s confabulating, answers will diverge.</p><pre>import openai<br>from collections import Counter<br>from typing import Any<br>import json<br>import re<br><br>client = openai.OpenAI()<br><br>def self_consistency_check(<br>    question: str,<br>    n_samples: int = 5,<br>    temperature: float = 0.7,<br>    model: str = &quot;gpt-4o&quot;<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Sample the model N times at non-zero temperature.<br>    High agreement across samples → high confidence.<br>    High divergence → likely hallucination or genuinely uncertain.<br>    <br>    Best for: factual questions with discrete answers<br>    Not ideal for: long-form generation (too expensive)<br>    &quot;&quot;&quot;<br>    responses = []<br>    <br>    for _ in range(n_samples):<br>        response = client.chat.completions.create(<br>            model=model,<br>            messages=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;, <br>                    &quot;content&quot;: &quot;Answer the question concisely. Provide only the factual answer, no explanation.&quot;<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}<br>            ],<br>            temperature=temperature,<br>            max_tokens=100<br>        )<br>        responses.append(response.choices[0].message.content.strip().lower())<br>    <br>    # Count agreement<br>    counter = Counter(responses)<br>    most_common, count = counter.most_common(1)[0]<br>    agreement_rate = count / n_samples<br>    <br>    # Confidence interpretation<br>    if agreement_rate &gt;= 0.8:<br>        confidence = &quot;HIGH&quot;<br>        hallucination_risk = &quot;LOW&quot;<br>    elif agreement_rate &gt;= 0.6:<br>        confidence = &quot;MEDIUM&quot;<br>        hallucination_risk = &quot;MEDIUM&quot;<br>    else:<br>        confidence = &quot;LOW&quot;<br>        hallucination_risk = &quot;HIGH&quot;<br>    <br>    return {<br>        &quot;question&quot;: question,<br>        &quot;consensus_answer&quot;: most_common,<br>        &quot;agreement_rate&quot;: agreement_rate,<br>        &quot;confidence&quot;: confidence,<br>        &quot;hallucination_risk&quot;: hallucination_risk,<br>        &quot;all_responses&quot;: responses,<br>        &quot;response_distribution&quot;: dict(counter)<br>    }<br><br># Example<br>result = self_consistency_check(<br>    &quot;What year was the Transformer architecture introduced in &#39;Attention Is All You Need&#39;?&quot;<br>)<br>print(f&quot;Answer: {result[&#39;consensus_answer&#39;]}&quot;)<br>print(f&quot;Confidence: {result[&#39;confidence&#39;]} ({result[&#39;agreement_rate&#39;]*100:.0f}% agreement)&quot;)<br># Expected: 2017, HIGH confidence (&gt;80% agreement)</pre><h3>Strategy 2: NLI-Based Grounding Check</h3><p>Natural Language Inference (NLI) classifies whether a premise <em>entails</em>, <em>contradicts</em>, or is <em>neutral to</em> a hypothesis. Use it to check if model outputs are supported by provided context.</p><pre>from transformers import pipeline<br>import torch<br><br># Load NLI model (small, fast, production-viable)<br># microsoft/deberta-large-mnli has excellent accuracy (90.9% on MNLI)<br>nli_pipeline = pipeline(<br>    &quot;text-classification&quot;,<br>    model=&quot;cross-encoder/nli-deberta-v3-small&quot;,<br>    device=0 if torch.cuda.is_available() else -1<br>)<br><br>def check_grounding(<br>    context: str,<br>    claim: str,<br>    threshold: float = 0.8<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Check if a claim is grounded in the provided context.<br>    <br>    Labels: ENTAILMENT (supported), NEUTRAL (not mentioned), CONTRADICTION (false)<br>    <br>    Production use: Call this on every LLM output claim before returning to user.<br>    &quot;&quot;&quot;<br>    # NLI takes (premise, hypothesis)<br>    # Premise = your retrieved context<br>    # Hypothesis = the claim the LLM made<br>    result = nli_pipeline(<br>        f&quot;{context} [SEP] {claim}&quot;,<br>        truncation=True,<br>        max_length=512<br>    )<br>    <br>    label = result[0][&quot;label&quot;].upper()<br>    score = result[0][&quot;score&quot;]<br>    <br>    grounded = label == &quot;ENTAILMENT&quot; and score &gt;= threshold<br>    <br>    return {<br>        &quot;claim&quot;: claim,<br>        &quot;label&quot;: label,<br>        &quot;confidence&quot;: score,<br>        &quot;grounded&quot;: grounded,<br>        &quot;risk&quot;: &quot;LOW&quot; if grounded else (&quot;HIGH&quot; if label == &quot;CONTRADICTION&quot; else &quot;MEDIUM&quot;)<br>    }<br><br>def check_response_grounding(<br>    retrieved_context: str,<br>    llm_response: str,<br>    sentence_split: bool = True<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Check an entire LLM response claim-by-claim.<br>    Splits response into sentences and checks each independently.<br>    &quot;&quot;&quot;<br>    if sentence_split:<br>        # Simple sentence splitting — use spaCy for production<br>        sentences = [s.strip() for s in re.split(r&#39;(?&lt;=[.!?])\s+&#39;, llm_response) if len(s.strip()) &gt; 20]<br>    else:<br>        sentences = [llm_response]<br>    <br>    claim_results = [check_grounding(retrieved_context, s) for s in sentences]<br>    <br>    contradicted = [r for r in claim_results if r[&quot;label&quot;] == &quot;CONTRADICTION&quot;]<br>    ungrounded = [r for r in claim_results if r[&quot;label&quot;] == &quot;NEUTRAL&quot;]<br>    supported = [r for r in claim_results if r[&quot;grounded&quot;]]<br>    <br>    overall_risk = &quot;HIGH&quot; if contradicted else (&quot;MEDIUM&quot; if len(ungrounded) &gt; len(supported) else &quot;LOW&quot;)<br>    <br>    return {<br>        &quot;overall_risk&quot;: overall_risk,<br>        &quot;total_claims&quot;: len(sentences),<br>        &quot;supported&quot;: len(supported),<br>        &quot;ungrounded&quot;: len(ungrounded),<br>        &quot;contradicted&quot;: len(contradicted),<br>        &quot;claim_details&quot;: claim_results<br>    }<br><br># Example<br>context = &quot;&quot;&quot;<br>Anthropic&#39;s Claude 3.5 Sonnet was released in June 2024. <br>It achieved state-of-the-art performance on coding benchmarks <br>and was available via API at $3 per million input tokens.<br>&quot;&quot;&quot;<br><br>llm_response = &quot;&quot;&quot;<br>Claude 3.5 Sonnet, released in mid-2024, is Anthropic&#39;s flagship model <br>with exceptional coding capabilities. It costs $3 per million tokens for input.<br>It also supports real-time video understanding natively.<br>&quot;&quot;&quot;<br><br>result = check_response_grounding(context, llm_response)<br>print(f&quot;Overall risk: {result[&#39;overall_risk&#39;]}&quot;)<br>print(f&quot;Supported: {result[&#39;supported&#39;]}/{result[&#39;total_claims&#39;]} claims&quot;)<br># The &quot;real-time video understanding&quot; claim would be flagged as NEUTRAL or CONTRADICTION</pre><h3>Strategy 3: LLM-as-Judge Hallucination Detection</h3><p>Use a separate, higher-quality model to evaluate the output of your primary model. More expensive but highly accurate.</p><pre>def llm_judge_hallucination(<br>    question: str,<br>    context: str,<br>    answer: str,<br>    judge_model: str = &quot;gpt-4o&quot;<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Use a judge LLM to evaluate factual grounding.<br>    <br>    Cost: ~$0.002 per evaluation at gpt-4o pricing.<br>    Latency: ~1-2 seconds additional per query.<br>    <br>    Worth it for: high-stakes outputs (legal, medical, financial)<br>    Not worth it for: high-volume, low-stakes applications<br>    &quot;&quot;&quot;<br>    judge_prompt = &quot;&quot;&quot;You are a rigorous fact-checker evaluating an AI assistant&#39;s response.<br><br>QUESTION: {question}<br><br>RETRIEVED CONTEXT:<br>{context}<br><br>AI ASSISTANT&#39;S ANSWER:<br>{answer}<br><br>Evaluate the answer strictly against ONLY the provided context. Do NOT use outside knowledge.<br><br>For each factual claim in the answer, determine:<br>1. SUPPORTED: The claim is directly stated or clearly implied by the context<br>2. UNSUPPORTED: The claim is not mentioned in the context (may still be true)  <br>3. CONTRADICTED: The claim directly conflicts with the context<br><br>Respond in JSON format:<br>{{<br>  &quot;verdict&quot;: &quot;PASS&quot; | &quot;WARN&quot; | &quot;FAIL&quot;,<br>  &quot;overall_assessment&quot;: &quot;brief explanation&quot;,<br>  &quot;claims&quot;: [<br>    {{&quot;text&quot;: &quot;the claim&quot;, &quot;status&quot;: &quot;SUPPORTED|UNSUPPORTED|CONTRADICTED&quot;, &quot;reasoning&quot;: &quot;why&quot;}}<br>  ],<br>  &quot;hallucination_risk&quot;: &quot;LOW&quot; | &quot;MEDIUM&quot; | &quot;HIGH&quot;,<br>  &quot;recommended_action&quot;: &quot;USE_AS_IS&quot; | &quot;ADD_DISCLAIMER&quot; | &quot;REGENERATE&quot; | &quot;ESCALATE_TO_HUMAN&quot;<br>}}&quot;&quot;&quot;<br><br>    response = client.chat.completions.create(<br>        model=judge_model,<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: &quot;You are a precise fact-checking judge. Respond only with valid JSON.&quot;<br>            },<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: judge_prompt.format(<br>                    question=question,<br>                    context=context,<br>                    answer=answer<br>                )<br>            }<br>        ],<br>        temperature=0,<br>        response_format={&quot;type&quot;: &quot;json_object&quot;}<br>    )<br>    <br>    return json.loads(response.choices[0].message.content)</pre><h3>Strategy 4: Citation Extraction and Verification</h3><p>For research and document-grounded tasks, force citations and verify them:</p><pre>def generate_with_citations(<br>    query: str,<br>    context_chunks: list[dict],<br>    model: str = &quot;gpt-4o&quot;<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Force the model to cite specific chunks for every claim.<br>    Then verify that the citation actually supports the claim.<br>    <br>    Each chunk must have &quot;id&quot; and &quot;text&quot; fields.<br>    &quot;&quot;&quot;<br>    # Build numbered context<br>    numbered_context = &quot;\n\n&quot;.join([<br>        f&quot;[CHUNK {i+1}]: {chunk[&#39;text&#39;]}&quot;<br>        for i, chunk in enumerate(context_chunks)<br>    ])<br>    <br>    system_prompt = &quot;&quot;&quot;You are a precise research assistant. <br>You MUST cite the specific chunk number [CHUNK N] for EVERY factual claim you make.<br>If you cannot find support for a claim in the provided chunks, <br>write [UNCERTAIN] instead of a citation and acknowledge the uncertainty.<br>Format: &quot;The model was released in 2024 [CHUNK 3].&quot; &quot;&quot;&quot;<br>    <br>    response = client.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n\n{numbered_context}\n\nQuestion: {query}&quot;}<br>        ],<br>        temperature=0.1<br>    )<br>    <br>    answer = response.choices[0].message.content<br>    <br>    # Extract and verify citations<br>    citation_pattern = r&#39;\[CHUNK (\d+)\]&#39;<br>    cited_chunks = set(int(m) for m in re.findall(citation_pattern, answer))<br>    <br>    # Check for uncertain claims<br>    uncertain_count = answer.count(&quot;[UNCERTAIN]&quot;)<br>    <br>    # Verify each citation is plausible (simple: check chunk exists)<br>    invalid_citations = [c for c in cited_chunks if c &gt; len(context_chunks)]<br>    <br>    return {<br>        &quot;answer&quot;: answer,<br>        &quot;cited_chunks&quot;: sorted(cited_chunks),<br>        &quot;uncertain_claims&quot;: uncertain_count,<br>        &quot;invalid_citations&quot;: invalid_citations,<br>        &quot;grounding_score&quot;: len(cited_chunks) / max(1, len(cited_chunks) + uncertain_count),<br>        &quot;verified&quot;: len(invalid_citations) == 0<br>    }</pre><h2>🛡️ Part 4: The Production Hallucination Mitigation Stack</h2><p>Defense in depth — no single technique eliminates hallucination. Layer these strategies:</p><pre>┌─────────────────────────────────────────────────────────┐<br>│                     User Query                          │<br>└────────────────────────┬────────────────────────────────┘<br>                         │<br>          ┌──────────────▼──────────────┐<br>          │   Layer 1: Query Analysis   │<br>          │  • Intent classification    │<br>          │  • Hallucination-prone type?│<br>          │  • Route to retrieval?      │<br>          └──────────────┬──────────────┘<br>                         │<br>          ┌──────────────▼──────────────┐<br>          │  Layer 2: Context Grounding │<br>          │  • RAG retrieval            │<br>          │  • Chunk verification       │<br>          │  • Source freshness check   │<br>          └──────────────┬──────────────┘<br>                         │<br>          ┌──────────────▼──────────────┐<br>          │    Layer 3: Generation      │<br>          │  • Constrained system prompt│<br>          │  • Forced citations         │<br>          │  • Temperature tuning       │<br>          └──────────────┬──────────────┘<br>                         │<br>          ┌──────────────▼──────────────┐<br>          │  Layer 4: Post-Generation   │<br>          │  • NLI grounding check      │<br>          │  • Self-consistency sample  │<br>          │  • LLM judge (for high-     │<br>          │    stakes outputs)          │<br>          └──────────────┬──────────────┘<br>                         │<br>          ┌──────────────▼──────────────┐<br>          │  Layer 5: Response Policy   │<br>          │  • Risk-based gating        │<br>          │  • Disclaimer injection     │<br>          │  • Escalation triggers      │<br>          └──────────────┬──────────────┘<br>                         │<br>                    User Response</pre><h3>Full Production Implementation</h3><pre>import asyncio<br>from dataclasses import dataclass, field<br>from enum import Enum<br>from typing import Callable<br>import logging<br><br>logger = logging.getLogger(__name__)<br><br>class HallucinationRisk(Enum):<br>    LOW = &quot;low&quot;<br>    MEDIUM = &quot;medium&quot;<br>    HIGH = &quot;high&quot;<br>    CRITICAL = &quot;critical&quot;<br><br>class ResponseAction(Enum):<br>    USE_AS_IS = &quot;use_as_is&quot;<br>    ADD_DISCLAIMER = &quot;add_disclaimer&quot;<br>    REGENERATE = &quot;regenerate&quot;<br>    REFUSE = &quot;refuse&quot;<br>    ESCALATE = &quot;escalate&quot;<br><br>@dataclass<br>class SafeGenerationConfig:<br>    # Model settings<br>    model: str = &quot;gpt-4o&quot;<br>    temperature: float = 0.1          # Low temperature for factual tasks<br>    <br>    # RAG settings<br>    enable_rag: bool = True<br>    min_retrieval_score: float = 0.7  # Don&#39;t use chunks below this similarity<br>    <br>    # Detection settings<br>    enable_nli_check: bool = True<br>    enable_self_consistency: bool = False  # Expensive — enable for critical paths<br>    n_consistency_samples: int = 3<br>    enable_llm_judge: bool = False         # Most expensive — high-stakes only<br>    <br>    # Policy settings<br>    max_uncertainty_fraction: float = 0.3  # Max fraction of claims that can be UNCERTAIN<br>    auto_disclaimer_threshold: HallucinationRisk = HallucinationRisk.MEDIUM<br>    auto_refuse_threshold: HallucinationRisk = HallucinationRisk.CRITICAL<br>    <br>    # Domain-specific<br>    domain: str = &quot;general&quot;  # &quot;medical&quot;, &quot;legal&quot;, &quot;financial&quot; get stricter defaults<br><br>@dataclass<br>class SafeGenerationResult:<br>    answer: str<br>    risk_level: HallucinationRisk<br>    action_taken: ResponseAction<br>    confidence_score: float<br>    sources_used: list[str]<br>    detection_results: dict<br>    final_response: str            # Answer + any injected disclaimers<br>    metadata: dict = field(default_factory=dict)<br><br>class HallucinationMitigatedLLM:<br>    &quot;&quot;&quot;<br>    Production-grade LLM wrapper with hallucination mitigation.<br>    <br>    Architecture: Query Analysis → RAG → Constrained Generation → <br>                  NLI Check → Policy Gate → Safe Response<br>    &quot;&quot;&quot;<br>    <br>    DOMAIN_DISCLAIMERS = {<br>        &quot;medical&quot;: &quot;\n\n⚠️ *This information is for educational purposes only and does not constitute medical advice. Always consult a qualified healthcare provider.*&quot;,<br>        &quot;legal&quot;: &quot;\n\n⚠️ *This is not legal advice. Consult a qualified attorney for legal matters specific to your situation.*&quot;,<br>        &quot;financial&quot;: &quot;\n\n⚠️ *This is not financial advice. Consult a qualified financial advisor before making investment decisions.*&quot;,<br>        &quot;general&quot;: &quot;\n\n*Note: AI responses may contain inaccuracies. Please verify important information independently.*&quot;<br>    }<br>    <br>    HALLUCINATION_PRONE_INTENTS = {<br>        &quot;citation_request&quot;: [&quot;cite&quot;, &quot;reference&quot;, &quot;paper&quot;, &quot;study&quot;, &quot;research shows&quot;, &quot;according to&quot;],<br>        &quot;numerical_claim&quot;: [&quot;how many&quot;, &quot;percentage&quot;, &quot;statistics&quot;, &quot;number of&quot;, &quot;how much&quot;],<br>        &quot;biographical&quot;: [&quot;who is&quot;, &quot;biography&quot;, &quot;born&quot;, &quot;founded&quot;, &quot;created by&quot;],<br>        &quot;current_events&quot;: [&quot;latest&quot;, &quot;current&quot;, &quot;recent&quot;, &quot;today&quot;, &quot;now&quot;, &quot;2025&quot;, &quot;2026&quot;],<br>        &quot;legal_medical&quot;: [&quot;dosage&quot;, &quot;diagnosis&quot;, &quot;liable&quot;, &quot;regulation&quot;, &quot;compliance&quot;]<br>    }<br>    <br>    def __init__(<br>        self,<br>        retriever: Callable,    # Function: query → list[dict] (chunks with id, text, score)<br>        config: SafeGenerationConfig = SafeGenerationConfig()<br>    ):<br>        self.retriever = retriever<br>        self.config = config<br>        self.client = openai.OpenAI()<br>    <br>    def _classify_hallucination_risk(self, query: str) -&gt; tuple[HallucinationRisk, list[str]]:<br>        &quot;&quot;&quot;<br>        Pre-flight: classify query for hallucination-prone patterns.<br>        Fast, local, no LLM call needed.<br>        &quot;&quot;&quot;<br>        query_lower = query.lower()<br>        triggered_patterns = []<br>        <br>        for pattern_name, keywords in self.HALLUCINATION_PRONE_INTENTS.items():<br>            if any(kw in query_lower for kw in keywords):<br>                triggered_patterns.append(pattern_name)<br>        <br>        # Risk escalation logic<br>        if &quot;legal_medical&quot; in triggered_patterns:<br>            return HallucinationRisk.CRITICAL, triggered_patterns<br>        elif len(triggered_patterns) &gt;= 2:<br>            return HallucinationRisk.HIGH, triggered_patterns<br>        elif triggered_patterns:<br>            return HallucinationRisk.MEDIUM, triggered_patterns<br>        else:<br>            return HallucinationRisk.LOW, triggered_patterns<br>    <br>    def _build_grounded_system_prompt(<br>        self, <br>        risk_level: HallucinationRisk,<br>        has_context: bool<br>    ) -&gt; str:<br>        &quot;&quot;&quot;Build a system prompt calibrated to the risk level.&quot;&quot;&quot;<br>        <br>        base = &quot;You are a precise, helpful assistant.&quot;<br>        <br>        if has_context:<br>            grounding = &quot;&quot;&quot;<br>Answer questions based ONLY on the provided context.<br>If the context doesn&#39;t contain sufficient information to answer confidently, <br>explicitly say: &quot;Based on the provided information, I cannot fully answer this — [explain what&#39;s missing].&quot;<br>Do NOT extrapolate beyond what the context states.&quot;&quot;&quot;<br>        else:<br>            grounding = &quot;&quot;&quot;<br>Be precise and factual. For any claim you&#39;re uncertain about, <br>use phrases like &quot;I believe&quot;, &quot;I think&quot;, or &quot;I&#39;m not certain, but...&quot;<br>Do not present uncertain information as established fact.&quot;&quot;&quot;<br>        <br>        if risk_level in [HallucinationRisk.HIGH, HallucinationRisk.CRITICAL]:<br>            risk_instructions = &quot;&quot;&quot;<br>CRITICAL ACCURACY REQUIREMENT:<br>- Do not cite any papers, studies, or sources unless explicitly provided in context<br>- Do not provide specific dates, numbers, or statistics unless from the provided context<br>- If asked for information you don&#39;t have in context, clearly state what you don&#39;t know<br>- Prefer &quot;I don&#39;t have reliable information about X&quot; over a potentially incorrect answer&quot;&quot;&quot;<br>        else:<br>            risk_instructions = &quot;&quot;<br>        <br>        return f&quot;{base}\n{grounding}\n{risk_instructions}&quot;.strip()<br>    <br>    async def generate(self, query: str) -&gt; SafeGenerationResult:<br>        &quot;&quot;&quot;<br>        Full hallucination-mitigated generation pipeline.<br>        &quot;&quot;&quot;<br>        detection_results = {}<br>        <br>        # === Layer 1: Pre-flight Risk Classification ===<br>        query_risk, triggered_patterns = self._classify_hallucination_risk(query)<br>        detection_results[&quot;query_risk&quot;] = {&quot;level&quot;: query_risk.value, &quot;patterns&quot;: triggered_patterns}<br>        <br>        # === Layer 2: RAG Retrieval ===<br>        context_chunks = []<br>        sources = []<br>        <br>        if self.config.enable_rag:<br>            try:<br>                raw_chunks = self.retriever(query)<br>                # Filter by minimum score<br>                context_chunks = [<br>                    c for c in raw_chunks <br>                    if c.get(&quot;score&quot;, 1.0) &gt;= self.config.min_retrieval_score<br>                ]<br>                sources = [c.get(&quot;source_id&quot;, c.get(&quot;id&quot;, &quot;unknown&quot;)) for c in context_chunks]<br>            except Exception as e:<br>                logger.warning(f&quot;Retrieval failed: {e}. Proceeding without context.&quot;)<br>        <br>        # Build context string<br>        context_str = &quot;&quot;<br>        if context_chunks:<br>            context_str = &quot;\n\n&quot;.join([<br>                f&quot;[SOURCE {i+1}: {c.get(&#39;source_id&#39;, &#39;unknown&#39;)}]\n{c[&#39;text&#39;]}&quot;<br>                for i, c in enumerate(context_chunks)<br>            ])<br>        <br>        # === Layer 3: Constrained Generation ===<br>        system_prompt = self._build_grounded_system_prompt(<br>            risk_level=query_risk,<br>            has_context=bool(context_chunks)<br>        )<br>        <br>        user_message = f&quot;{f&#39;Context:{chr(10)}{context_str}{chr(10)*2}&#39; if context_str else &#39;&#39;}Question: {query}&quot;<br>        <br>        response = self.client.chat.completions.create(<br>            model=self.config.model,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_message}<br>            ],<br>            temperature=self.config.temperature<br>        )<br>        <br>        raw_answer = response.choices[0].message.content<br>        <br>        # === Layer 4: Post-Generation Detection ===<br>        output_risk = query_risk  # Start with query-level risk<br>        <br>        # NLI grounding check (if context available)<br>        if self.config.enable_nli_check and context_chunks:<br>            grounding_result = check_response_grounding(context_str, raw_answer)<br>            detection_results[&quot;nli_grounding&quot;] = grounding_result<br>            <br>            # Escalate risk if contradictions found<br>            if grounding_result.get(&quot;contradicted&quot;, 0) &gt; 0:<br>                output_risk = HallucinationRisk.CRITICAL<br>            elif grounding_result.get(&quot;overall_risk&quot;) == &quot;HIGH&quot;:<br>                output_risk = max(output_risk, HallucinationRisk.HIGH, key=lambda r: list(HallucinationRisk).index(r))<br>        <br>        # Self-consistency check (for high-risk queries)<br>        if self.config.enable_self_consistency and query_risk.value in [&quot;high&quot;, &quot;critical&quot;]:<br>            consistency_result = self_consistency_check(<br>                query, <br>                n_samples=self.config.n_consistency_samples<br>            )<br>            detection_results[&quot;self_consistency&quot;] = consistency_result<br>            <br>            if consistency_result[&quot;hallucination_risk&quot;] == &quot;HIGH&quot;:<br>                output_risk = HallucinationRisk.CRITICAL<br>        <br>        # === Layer 5: Response Policy Gate ===<br>        if output_risk == HallucinationRisk.CRITICAL and not context_chunks:<br>            action = ResponseAction.REFUSE<br>            final_response = (<br>                &quot;I don&#39;t have reliable information in my current context to answer this accurately. &quot;<br>                &quot;This type of question carries a high risk of errors. &quot;<br>                &quot;Please consult authoritative sources or provide relevant documents for me to reference.&quot;<br>            )<br>        elif output_risk.value in [&quot;high&quot;, &quot;critical&quot;]:<br>            action = ResponseAction.ADD_DISCLAIMER<br>            disclaimer = self.DOMAIN_DISCLAIMERS.get(<br>                self.config.domain, <br>                self.DOMAIN_DISCLAIMERS[&quot;general&quot;]<br>            )<br>            final_response = raw_answer + disclaimer<br>        else:<br>            action = ResponseAction.USE_AS_IS<br>            final_response = raw_answer<br>        <br>        # Calculate overall confidence score [0, 1]<br>        confidence = {<br>            HallucinationRisk.LOW: 0.9,<br>            HallucinationRisk.MEDIUM: 0.7,<br>            HallucinationRisk.HIGH: 0.4,<br>            HallucinationRisk.CRITICAL: 0.1<br>        }[output_risk]<br>        <br>        return SafeGenerationResult(<br>            answer=raw_answer,<br>            risk_level=output_risk,<br>            action_taken=action,<br>            confidence_score=confidence,<br>            sources_used=sources,<br>            detection_results=detection_results,<br>            final_response=final_response,<br>            metadata={&quot;model&quot;: self.config.model, &quot;context_chunks&quot;: len(context_chunks)}<br>        )</pre><h2>🧠 Part 5: Teaching Models to Say “I Don’t Know”</h2><p>The most underused defense: calibrating models to express genuine uncertainty rather than confabulate. This requires both prompting and fine-tuning strategies.</p><h3>Prompting for Calibrated Uncertainty</h3><pre>UNCERTAINTY_CALIBRATION_PROMPT = &quot;&quot;&quot;You are a knowledgeable assistant with calibrated uncertainty.<br><br>When answering, use these precise phrases based on your confidence:<br><br>HIGH CONFIDENCE (you know this well):<br>→ State the fact directly: &quot;The Transformer was introduced in 2017.&quot;<br><br>MEDIUM CONFIDENCE (fairly sure but not certain):<br>→ &quot;I believe...&quot;, &quot;I think...&quot;, &quot;To my knowledge...&quot;<br>→ &quot;I&#39;m fairly confident that..., but I&#39;d recommend verifying.&quot;<br><br>LOW CONFIDENCE (uncertain, might be wrong):<br>→ &quot;I&#39;m not certain about this — you should verify, but I believe...&quot;<br>→ &quot;I have limited information on this specific point.&quot;<br><br>NO RELIABLE INFORMATION:<br>→ &quot;I don&#39;t have reliable information about [specific aspect].&quot;<br>→ &quot;I cannot answer this with confidence based on what I know.&quot;<br>→ DO NOT guess. DO NOT confabulate.<br><br>NEVER present uncertain information as certain fact.<br>NEVER invent citations, papers, names, dates, or statistics.<br>A confident &quot;I don&#39;t know&quot; is always better than a confident wrong answer.&quot;&quot;&quot;<br><br>def calibrated_generation(query: str, context: str = &quot;&quot;) -&gt; str:<br>    &quot;&quot;&quot;Generate with explicit uncertainty calibration.&quot;&quot;&quot;<br>    messages = [<br>        {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: UNCERTAINTY_CALIBRATION_PROMPT},<br>    ]<br>    <br>    if context:<br>        messages.append({<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: f&quot;Context:\n{context}\n\nQuestion: {query}&quot;<br>        })<br>    else:<br>        messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query})<br>    <br>    response = client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=messages,<br>        temperature=0.1<br>    )<br>    <br>    return response.choices[0].message.content</pre><h3>Semantic Entropy for Uncertainty Quantification</h3><p>A research-grade technique from Kuhn et al. (2023) — measures whether the model generates semantically similar answers across samples, rather than just identical strings:</p><pre>from sentence_transformers import SentenceTransformer<br>from sklearn.cluster import DBSCAN<br>import numpy as np<br><br>embedding_model = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)<br><br>def semantic_entropy(<br>    question: str,<br>    n_samples: int = 10,<br>    model: str = &quot;gpt-4o-mini&quot;,  # Use smaller model for cheap sampling<br>    eps: float = 0.3             # DBSCAN clustering radius — tune per domain<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Measure semantic entropy of model responses.<br>    <br>    Low entropy: all responses say the same thing semantically → confident<br>    High entropy: responses diverge semantically → uncertain / hallucination risk<br>    <br>    From: Kuhn et al., &quot;Semantic Uncertainty: Linguistic Invariances for <br>    Uncertainty Estimation in Natural Language Generation&quot; (ICLR 2023)<br>    &quot;&quot;&quot;<br>    responses = []<br>    for _ in range(n_samples):<br>        r = client.chat.completions.create(<br>            model=model,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}],<br>            temperature=1.0,  # High temperature to sample the distribution<br>            max_tokens=150<br>        )<br>        responses.append(r.choices[0].message.content.strip())<br>    <br>    # Embed all responses<br>    embeddings = embedding_model.encode(responses)<br>    <br>    # Cluster semantically similar responses<br>    clustering = DBSCAN(eps=eps, min_samples=1, metric=&quot;cosine&quot;).fit(embeddings)<br>    <br>    n_clusters = len(set(clustering.labels_))<br>    <br>    # Semantic entropy: log(n_semantic_clusters)<br>    # Low n_clusters → low entropy → confident<br>    # High n_clusters → high entropy → uncertain<br>    semantic_ent = np.log(n_clusters) if n_clusters &gt; 1 else 0.0<br>    <br>    # Normalized entropy [0, 1]<br>    max_entropy = np.log(n_samples)<br>    normalized_entropy = semantic_ent / max_entropy if max_entropy &gt; 0 else 0.0<br>    <br>    confidence = &quot;HIGH&quot; if normalized_entropy &lt; 0.3 else (<br>        &quot;MEDIUM&quot; if normalized_entropy &lt; 0.6 else &quot;LOW&quot;<br>    )<br>    <br>    return {<br>        &quot;question&quot;: question,<br>        &quot;n_responses&quot;: n_samples,<br>        &quot;n_semantic_clusters&quot;: n_clusters,<br>        &quot;semantic_entropy&quot;: round(semantic_ent, 3),<br>        &quot;normalized_entropy&quot;: round(normalized_entropy, 3),<br>        &quot;confidence&quot;: confidence,<br>        &quot;sample_responses&quot;: responses[:3]  # Return first 3 for inspection<br>    }</pre><h2>🏢 Part 6: Enterprise Patterns for Regulated Industries</h2><p>Different industries have different hallucination tolerance. Here’s how leading enterprises approach this:</p><h3>Healthcare: Maximum Citation Requirement</h3><pre>HEALTHCARE_SYSTEM_PROMPT = &quot;&quot;&quot;You are a clinical information assistant.<br><br>ABSOLUTE RULES:<br>1. NEVER provide diagnostic conclusions or treatment recommendations<br>2. EVERY clinical claim must cite a specific provided source [SOURCE N]<br>3. Use &quot;based on the provided clinical literature&quot; not &quot;studies show&quot;<br>4. For any dosage, contraindication, or procedure: state &quot;Verify with current clinical guidelines&quot;<br>5. If information is not in the provided context, state: &quot;This information is not in my current reference materials&quot;<br>6. Always recommend consulting a licensed healthcare provider<br><br>FORBIDDEN OUTPUTS:<br>- Specific drug dosages without direct citation<br>- Diagnostic conclusions (&quot;The patient likely has X&quot;)<br>- Treatment plans<br>- Any claim about recent studies not in provided context&quot;&quot;&quot;</pre><h3>Financial Services: Temporal Grounding</h3><pre>FINANCIAL_SYSTEM_PROMPT = &quot;&quot;&quot;You are a financial information assistant.<br><br>REGULATORY COMPLIANCE REQUIREMENTS:<br>1. All market data must include the timestamp of the source<br>2. Do not extrapolate from historical data to make predictions<br>3. Include &quot;Past performance does not guarantee future results&quot; for any performance data<br>4. Never state that a specific investment &quot;will&quot; do anything<br>5. For regulatory information: specify the jurisdiction and &quot;verify with current regulations&quot;<br>6. If asked for specific investment advice: redirect to licensed financial advisors<br><br>DATE AWARENESS: If a source is dated more than 90 days ago, flag it as: <br>[NOTE: This information is from {date} and may be outdated]&quot;&quot;&quot;</pre><h3>Legal: Jurisdiction and Currency Guards</h3><pre>def legal_safe_generate(query: str, jurisdiction: str, context: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Legal hallucination mitigation: <br>    - Force jurisdiction specificity<br>    - Flag potentially outdated statutes<br>    - Require citation to provided documents<br>    &quot;&quot;&quot;<br>    system = f&quot;&quot;&quot;You are a legal research assistant for {jurisdiction} jurisdiction.<br><br>STRICT REQUIREMENTS:<br>1. ONLY reference laws and cases explicitly in the provided context<br>2. Do not cite cases or statutes not in the provided materials<br>3. Specify: &quot;This reflects law as of [date in source], verify current status&quot;<br>4. Every legal claim needs a citation: [Doc N, Section X]<br>5. Never state what a court &quot;would&quot; decide — only what courts &quot;have&quot; decided per citations<br>6. End all responses with: &quot;This is legal research, not legal advice. Consult a licensed attorney.&quot;<br><br>JURISDICTION: {jurisdiction}&quot;&quot;&quot;<br>    <br>    response = client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n{context}\n\nQuestion: {query}&quot;}<br>        ],<br>        temperature=0     # Zero temperature for legal — deterministic outputs<br>    )<br>    <br>    return {<br>        &quot;response&quot;: response.choices[0].message.content,<br>        &quot;jurisdiction&quot;: jurisdiction,<br>        &quot;disclaimer_required&quot;: True<br>    }</pre><h2>📊 Part 7: Measuring Hallucination in Production</h2><p>You can’t improve what you can’t measure. Build hallucination evaluation into your deployment pipeline:</p><pre>from dataclasses import dataclass<br>from datetime import datetime<br>import json<br><br>@dataclass<br>class HallucinationMetrics:<br>    &quot;&quot;&quot;Track hallucination across your production system over time.&quot;&quot;&quot;<br>    timestamp: str<br>    query: str<br>    model: str<br>    risk_level: str<br>    nli_contradiction_rate: float    # % of claims contradicting context<br>    nli_unsupported_rate: float      # % of claims not grounded in context<br>    consistency_score: float         # Agreement across samples (0-1)<br>    context_coverage: float          # How much of answer came from context<br>    user_id: str                     # For per-user analysis<br>    session_id: str<br><br>class HallucinationMonitor:<br>    &quot;&quot;&quot;<br>    Production hallucination monitoring.<br>    Integrates with your observability stack (Datadog, Langfuse, etc.)<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self, storage_backend=None):<br>        self.storage = storage_backend  # e.g., PostgreSQL, ClickHouse<br>        self._buffer = []<br>    <br>    def log_generation(self, result: SafeGenerationResult, query: str, metadata: dict):<br>        &quot;&quot;&quot;Log a generation event with hallucination signals.&quot;&quot;&quot;<br>        nli_result = result.detection_results.get(&quot;nli_grounding&quot;, {})<br>        consistency = result.detection_results.get(&quot;self_consistency&quot;, {})<br>        <br>        total_claims = nli_result.get(&quot;total_claims&quot;, 1)<br>        <br>        metrics = HallucinationMetrics(<br>            timestamp=datetime.utcnow().isoformat(),<br>            query=query[:200],<br>            model=result.metadata.get(&quot;model&quot;, &quot;unknown&quot;),<br>            risk_level=result.risk_level.value,<br>            nli_contradiction_rate=nli_result.get(&quot;contradicted&quot;, 0) / max(total_claims, 1),<br>            nli_unsupported_rate=nli_result.get(&quot;ungrounded&quot;, 0) / max(total_claims, 1),<br>            consistency_score=consistency.get(&quot;agreement_rate&quot;, 1.0),<br>            context_coverage=result.metadata.get(&quot;context_chunks&quot;, 0) / max(1, result.metadata.get(&quot;context_chunks&quot;, 1)),<br>            user_id=metadata.get(&quot;user_id&quot;, &quot;anonymous&quot;),<br>            session_id=metadata.get(&quot;session_id&quot;, &quot;unknown&quot;)<br>        )<br>        <br>        self._buffer.append(metrics)<br>        <br>        # Flush to storage<br>        if len(self._buffer) &gt;= 100:<br>            self._flush()<br>        <br>        return metrics<br>    <br>    def get_dashboard_metrics(self, window_hours: int = 24) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Metrics for your hallucination dashboard.<br>        <br>        Alert if:<br>        - contradiction_rate &gt; 2% (system-wide)<br>        - high_risk_rate &gt; 10% (queries classified HIGH or CRITICAL)<br>        - consistency_score &lt; 0.7 (average)<br>        &quot;&quot;&quot;<br>        # In production: query your storage backend<br>        # Here: compute from buffer<br>        if not self._buffer:<br>            return {}<br>        <br>        rates = {<br>            &quot;avg_contradiction_rate&quot;: sum(m.nli_contradiction_rate for m in self._buffer) / len(self._buffer),<br>            &quot;avg_unsupported_rate&quot;: sum(m.nli_unsupported_rate for m in self._buffer) / len(self._buffer),<br>            &quot;avg_consistency_score&quot;: sum(m.consistency_score for m in self._buffer) / len(self._buffer),<br>            &quot;high_risk_count&quot;: sum(1 for m in self._buffer if m.risk_level in [&quot;high&quot;, &quot;critical&quot;]),<br>            &quot;total_generations&quot;: len(self._buffer)<br>        }<br>        <br>        rates[&quot;high_risk_rate&quot;] = rates[&quot;high_risk_count&quot;] / rates[&quot;total_generations&quot;]<br>        <br>        # Alert thresholds<br>        rates[&quot;alerts&quot;] = []<br>        if rates[&quot;avg_contradiction_rate&quot;] &gt; 0.02:<br>            rates[&quot;alerts&quot;].append(&quot;ALERT: Contradiction rate above 2% threshold&quot;)<br>        if rates[&quot;high_risk_rate&quot;] &gt; 0.10:<br>            rates[&quot;alerts&quot;].append(&quot;ALERT: High-risk query rate above 10%&quot;)<br>        if rates[&quot;avg_consistency_score&quot;] &lt; 0.70:<br>            rates[&quot;alerts&quot;].append(&quot;ALERT: Average consistency score below 0.70&quot;)<br>        <br>        return rates<br>    <br>    def _flush(self):<br>        if self.storage:<br>            for m in self._buffer:<br>                self.storage.insert(m)<br>        self._buffer = []</pre><h2>⚠️ Common Mistakes in Hallucination Mitigation</h2><h3>Mistake 1: RAG as a Silver Bullet</h3><p>RAG dramatically <em>reduces</em> hallucination but doesn’t eliminate it. The model can still:</p><ul><li>Ignore retrieved context and generate from parametric knowledge</li><li>Merge retrieved chunks incorrectly</li><li>Add unsupported elaborations around correct retrieved facts</li></ul><p><strong>Fix</strong>: Always pair RAG with post-generation NLI grounding checks.</p><h3>Mistake 2: Using Temperature=0 for All Tasks</h3><p>Temperature=0 makes outputs deterministic, not accurate. The model will deterministically produce the same hallucination every time. Worse, it loses the ability to generate reasonable paraphrases when the “modal” token is wrong.</p><p><strong>Fix</strong>: Temperature=0 is appropriate for structured extraction and classification. For factual generation, use temperature=0.1–0.2 with NLI grounding.</p><h3>Mistake 3: Treating All Hallucinations as Equal</h3><p>A model hallucinating that Python was created in 1993 (actually 1991) is very different from a model hallucinating a drug dosage or a legal citation. Risk stratification matters.</p><p><strong>Fix</strong>: Classify queries by domain and consequence severity. Apply heavier mitigation where errors have real-world impact.</p><h3>Mistake 4: Not Testing Your System Prompts</h3><p>System prompts claiming “only answer from context” often fail silently. Models are trained to be helpful and will sneak in parametric knowledge.</p><pre># Test your system prompts with adversarial probes<br>HALLUCINATION_PROBE_QUESTIONS = [<br>    &quot;What is the capital of France?&quot;,           # Should use context or say &quot;not in context&quot;<br>    &quot;Who founded OpenAI?&quot;,                      # Probe for biographical confabulation<br>    &quot;What did the 2024 study by Chen et al. find?&quot;,  # Probe for citation fabrication<br>    &quot;What&#39;s the current stock price of Apple?&quot;,       # Probe for temporal hallucination<br>]</pre><h3>Mistake 5: Evaluating Only on Clean Benchmarks</h3><p>TruthfulQA, HaluEval, and FEVER measure hallucination in controlled settings. Your domain may have different failure patterns.</p><p><strong>Fix</strong>: Build a domain-specific hallucination evaluation set with labeled correct/incorrect/uncertain examples.</p><h2>🎤 Quick Questions</h2><p><strong>Q1: “How would you reduce hallucination in a customer support chatbot?”</strong></p><p>I’d implement a layered approach. First, RAG to ground every response in the company’s actual documentation — retrieve the three most relevant chunks, verify their scores exceed 0.7 similarity. Second, a constrained system prompt that explicitly instructs the model to say “I don’t have information about that” when context doesn’t cover the query, rather than extrapolating. Third, NLI-based post-generation grounding checks using a model like DeBERTa to verify each claim in the response is entailed by the retrieved context. Fourth, for any query that touches account status, billing, or service commitments, I’d add a self-consistency sampling step — if the model disagrees with itself across three samples, escalate to a human agent rather than sending a potentially wrong automated response. The whole pipeline would be logged and monitored with a hallucination rate dashboard alerting if contradictions exceed 2%.</p><p><strong>Q2: “Why can’t we just use temperature=0 to stop hallucinations?”</strong></p><p>Temperature=0 gives you the single most probable token at each step, making generation deterministic. But the model’s “most probable” answer is still trained on the same statistical distribution that produces hallucinations — it just always picks the same wrong answer now. Worse, it makes the system brittle: there’s no ability to detect uncertainty through sampling variance, and the model loses beneficial paraphrase flexibility. I’ve seen temperature=0 systems confidently produce the same fabricated citation every time a specific question is asked, which is more dangerous than a system that varies and can be caught through consistency checking.</p><p><strong>Q3: “What’s the difference between a grounded hallucination and an ungrounded one?”</strong></p><p>A grounded hallucination occurs when the model has relevant context but misinterprets, merges, or extrapolates from it incorrectly — like attributing a statement from Document B to the person mentioned only in Document A. An ungrounded hallucination is pure confabulation from parametric memory — the model generates information that has no connection to any provided context. They require different mitigations: grounded hallucinations are caught by NLI cross-checking claims against the specific context chunks that should support them. Ungrounded hallucinations require semantic entropy sampling or LLM judge evaluation to catch, since there’s no context to compare against.</p><p><strong>Q4: “How do you evaluate hallucination in production?”</strong></p><p>The gold standard is a labeled eval set — a few hundred queries with known correct answers, run weekly against your production system. For continuous monitoring without ground truth, I track three proxies: NLI contradiction rate (claims that conflict with retrieved context, target &lt;2%), self-consistency agreement rate (target &gt;80%), and user correction signals (explicit thumbs-down or follow-up corrections). I also run automated canary queries — questions with known correct answers — through the system daily. If the canary answer changes or NLI grounding drops, it triggers an alert before users notice.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Hallucination is structural, not incidental</strong>. LLMs optimize for plausibility, not truth. No prompt or technique eliminates it — you mitigate and manage.</li><li><strong>There are six distinct hallucination patterns</strong>, each with different root causes and mitigations. Treating them identically leads to wasted effort.</li><li><strong>RAG reduces but doesn’t eliminate hallucination</strong>. Always pair retrieval with post-generation NLI grounding checks.</li><li><strong>Self-consistency sampling is your cheapest confidence signal</strong>. Sample 3–5 times; high disagreement = high hallucination risk. Use for medium-stakes queries.</li><li><strong>NLI-based grounding checks</strong> using DeBERTa or similar are fast (&lt;100ms), cheap (&lt;$0.0001 per check), and catch the most dangerous hallucination type: claims that contradict your own retrieved context.</li><li><strong>Risk stratification matters</strong>. A hallucinated restaurant recommendation is a bad UX. A hallucinated drug dosage or legal citation is a liability. Different consequences demand different mitigation depths.</li><li><strong>Teach models to say “I don’t know”</strong>. A calibrated uncertainty signal, reinforced through system prompting and fine-tuning, is more valuable than a confidently wrong answer.</li><li><strong>Measure in production</strong>. Build hallucination rate dashboards. Set alert thresholds. Hallucination rates increase with model updates, context distribution shifts, and new query patterns.</li></ol><h2>🔧 Production Considerations</h2><p><strong>Latency budget for mitigation</strong>:</p><ul><li>Query risk classification: ~1ms (local)</li><li>RAG retrieval: 50–200ms</li><li>Primary generation: 1–5s</li><li>NLI grounding check: 50–150ms (on GPU)</li><li>Self-consistency (3 samples): +3–10s (parallel: +2–4s)</li><li>LLM judge: +2–4s</li></ul><p>For P99 &lt; 3s SLAs, budget for RAG + NLI only. Self-consistency and LLM judge are async post-hoc audits or high-stakes-only gates.</p><p><strong>Cost implications</strong>:</p><ul><li>NLI check: ~$0.00001 per check (self-hosted DeBERTa)</li><li>Self-consistency (3× gpt-4o-mini samples): ~$0.001</li><li>LLM judge (gpt-4o): ~$0.002–0.005</li><li>Full stack: ~$0.005–0.01 per query — usually worth it</li></ul><h2>📖 Further Reading</h2><ul><li><a href="https://arxiv.org/abs/2109.07958">“TruthfulQA: Measuring How Models Mimic Human Falsehoods”</a> — Lin et al., the original systematic study</li><li><a href="https://arxiv.org/abs/2305.11747">“HaluEval: A Large-Scale Hallucination Evaluation Benchmark”</a> — 35K examples across tasks</li><li><a href="https://arxiv.org/abs/2302.09664">“Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation”</a> — Kuhn et al. (ICLR 2023), foundation of semantic entropy approach</li><li><a href="https://arxiv.org/abs/2309.01219">“A Survey of Hallucination in Large Language Models”</a> — Comprehensive taxonomy</li><li><a href="https://huggingface.co/spaces/vectara/leaderboard">Vectara’s Hallucination Leaderboard</a> — Live benchmarks, updated monthly</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84385315f613" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 13: Small Language Models (SLMs) — Phi, Gemma, and On-Device AI]]></title>
            <link>https://learncsdesigns.medium.com/day-13-small-language-models-slms-phi-gemma-and-on-device-ai-1794d9107b13?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1794d9107b13</guid>
            <category><![CDATA[small-language-model]]></category>
            <category><![CDATA[edge-ai]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ai-on-device]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 14:35:42 GMT</pubDate>
            <atom:updated>2026-07-11T14:35:42.927Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>&quot;A 4B model trained on textbooks beats a 70B model trained on the internet — not because it&#39;s smarter, but because what you learn matters more than how much you&#39;ve seen.&quot;</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JpS0YXfz5ca6Z59lIlYgMg.png" /></figure><h2>Why This Matters</h2><p>The dominant narrative in AI is &quot;bigger is better.&quot; GPT-4 is better than GPT-3. Claude Opus is better than Claude Haiku. More parameters, more training data, more FLOPS — more capability.</p><p>That narrative is incomplete. And for a growing class of applications, it&#39;s just wrong.</p><p>In 2026, the most exciting AI deployments aren&#39;t happening in data centers — they&#39;re happening at the edge:</p><ul><li><strong>On your iPhone</strong>: Apple Intelligence processes most requests on-device using a ~3B parameter model, with no data leaving your phone</li><li><strong>In your browser</strong>: WebLLM runs Phi-3.5-mini at 15 tokens/sec entirely in WebAssembly, zero server cost</li><li><strong>On factory floors</strong>: Industrial IoT devices run 1B-parameter specialized models for defect detection with &lt;10ms latency</li><li><strong>In enterprise</strong>: Companies with strict data residency requirements run Gemma 3 on their own hardware instead of sending data to OpenAI</li></ul><p>The SLM revolution is driven by three converging forces: <strong>data quality research</strong> proving that curation beats scale, <strong>hardware advances</strong> making edge inference practical, and <strong>privacy regulation</strong> making data sovereignty mandatory.</p><h2>Part 1: The Data Quality Paradigm — Why Phi Changed Everything</h2><h3>The Textbooks Are All You Need Hypothesis</h3><p>In June 2023, Microsoft Research published <em>Phi-1: Textbooks Are All You Need</em> — a paper that shook the scaling-laws orthodoxy. Their finding: a 1.3B parameter model trained on carefully curated, high-quality &quot;textbook-like&quot; data outperformed 10× larger models trained on raw internet text on coding benchmarks.</p><p>The insight was radical: <strong>the quality and pedagogical structure of training data matters more than the quantity.</strong></p><pre>Standard LLM training data:<br>  Source: Internet crawl (Common Crawl, C4, RedPajama)<br>  Quality: Highly variable — spam, duplicates, errors, noise<br>  Structure: Random, no curriculum<br>  Signal density: Low — most tokens carry little useful information<br><br>Phi training data:<br>  Source: Synthetic textbooks + filtered high-quality content<br>  Quality: Uniformly high — each document teaches something clearly<br>  Structure: Pedagogically organized — concepts built on prior knowledge<br>  Signal density: High — almost every token is informationally dense</pre><h3>Why High-Quality Data Trains Smaller Models Better</h3><p>Think about how humans learn. A student who reads 1 carefully written textbook learns more than a student who reads 100 random blog posts. The textbook presents concepts clearly, builds on prerequisites, uses good examples, and avoids contradicting itself.</p><p>Language models learn by gradient descent on token prediction. If training data is:</p><ul><li><strong>Consistent</strong>: The model doesn&#39;t waste capacity resolving contradictions</li><li><strong>Pedagogical</strong>: The model learns generalizable reasoning patterns, not surface correlations</li><li><strong>Filtered</strong>: No token budget wasted on noise, repetition, or misinformation</li></ul><p>...then a smaller model can achieve the reasoning capability of a much larger model trained on unfiltered data.</p><pre># The signal density concept illustrated:<br><br># Low signal-density text (from internet):<br>low_quality = &quot;&quot;&quot;<br>lol this is so amazing i cant believe it!!! check out this AI thing its crazy<br>click here to learn more &gt;&gt;&gt; link &lt;&lt;&lt; DONT MISS OUT best AI tools 2024<br>omg the chatGPT is literally changing everything you wont believe it<br>&quot;&quot;&quot;<br><br># High signal-density text (textbook style):<br>high_quality = &quot;&quot;&quot;<br>A transformer&#39;s attention mechanism computes, for each query vector q,<br>a weighted sum of value vectors v_i, where weights are determined by<br>the dot-product similarity between q and each key k_i, normalized<br>via softmax. This enables the model to selectively aggregate information<br>from arbitrary positions in the sequence, regardless of distance.<br>&quot;&quot;&quot;<br><br># Per token, the second text teaches vastly more structured knowledge.<br># Training on the first requires many more parameters to &quot;average out&quot;<br># the noise and extract signal.</pre><h3>The Phi Model Progression</h3><pre>Model        Release   Params   Training Data Key Insight            Score vs<br>                                                                      10× Model<br>──────────────────────────────────────────────────────────────────────────────<br>Phi-1        Jun 2023  1.3B     Synthetic textbooks (code)           Beats 10×<br>                                + StackOverflow filtered              on HumanEval<br><br>Phi-1.5      Sep 2023  1.3B     Extended to natural language         Beats many 7B<br>                                &quot;common sense&quot; textbooks              models<br><br>Phi-2        Dec 2023  2.7B     More diverse textbook data          Beats Llama-2-70B<br>                                + web data (curated)                 on several benchmarks<br><br>Phi-3-mini   Apr 2024  3.8B     Scale-up of Phi approach +          Matches Mixtral 8×7B<br>                                RLHF alignment                       on reasoning tasks<br><br>Phi-3.5-mini Aug 2024  3.8B     128k context, MoE variant,          Competitive with<br>Phi-3.5-MoE           16×3.8B  multilingual improvements            GPT-3.5<br><br>Phi-4        Dec 2024  14B      Synthetic data generated by GPT-4o  Matches/beats GPT-4o<br>                                + strategic instruction tuning        on STEM/math tasks</pre><h2>Part 2: Microsoft Phi-4 — The Current State of the Art</h2><h3>Architecture and Capabilities</h3><p>Phi-4 is the clearest validation of the data quality thesis at a larger scale:</p><pre>Phi-4 Specifications:<br>─────────────────────────────────────────────────────────────────<br>Parameters:          14 billion<br>Context window:      16,384 tokens (base), 64k with RoPE scaling<br>Architecture:        Transformer decoder, dense (not MoE)<br>Training tokens:     ~10 trillion (heavily filtered and synthetic)<br>Training hardware:   512× A100 80GB<br>License:             Microsoft Research License (not fully open)<br><br>Key benchmarks (December 2024):<br>  MATH benchmark:     80.4%  (vs GPT-4o: 74.6%)  ← beats GPT-4o on math!<br>  GPQA (PhD science): 56.1%  (vs GPT-4o: 53.6%)<br>  HumanEval (coding): 82.6%  (vs GPT-3.5: 67.0%)<br>  MMLU:               84.8%  (vs Llama-3.1-70B: 86.0%)<br><br>Memory requirements:<br>  BF16 (full):        ~28 GB VRAM<br>  Q4 quantized:       ~8 GB VRAM  ← fits on consumer GPU<br>  Q8 quantized:       ~14 GB VRAM</pre><h3>What Phi-4&#39;s Training Data Looked Like</h3><p>Microsoft described their Phi-4 synthetic data pipeline:</p><pre>Phi-4 Data Sources (approximate):<br>─────────────────────────────────────────────────────────────────<br>40%  Synthetic data generated by GPT-4o<br>     • Multi-step reasoning problems with worked solutions<br>     • Diverse question-answer pairs across STEM domains<br>     • Deliberately crafted to expose &quot;near-miss&quot; reasoning errors<br>     <br>15%  Filtered web data (web pages graded for quality)<br>     • Educational, scientific, and technical content only<br>     • Automated quality scoring using a classifier trained on human ratings<br><br>15%  Filtered code data<br>     • High-quality GitHub repositories (stars, tests, documentation)<br>     • Algorithmic problems with correct, commented solutions<br><br>15%  Books and academic papers<br>     • STEM textbooks, arxiv papers, educational materials<br><br>15%  Other: instruction-tuning conversations, reasoning traces</pre><h3>Using Phi-4 in Practice</h3><pre># ── Phi-4 via Ollama ──────────────────────────────────────────────────────────<br># ollama pull phi4<br><br>import httpx<br>import asyncio<br><br>async def phi4_complete(<br>    prompt: str,<br>    system: str = &quot;You are a helpful AI assistant.&quot;,<br>    temperature: float = 0.0,    # Phi-4 excels at deterministic reasoning<br>) -&gt; str:<br>    &quot;&quot;&quot;Run Phi-4 locally via Ollama.&quot;&quot;&quot;<br>    <br>    async with httpx.AsyncClient(timeout=120.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/v1/chat/completions&quot;,<br>            json={<br>                &quot;model&quot;: &quot;phi4&quot;,<br>                &quot;messages&quot;: [<br>                    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system},<br>                    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>                ],<br>                &quot;temperature&quot;: temperature,<br>                &quot;max_tokens&quot;: 2048,<br>            },<br>        )<br>        return response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br><br><br># ── Phi-4 via Hugging Face Transformers ──────────────────────────────────────<br>from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline<br>import torch<br><br>def load_phi4(quantize: bool = True):<br>    &quot;&quot;&quot;Load Phi-4 with optional 4-bit quantization for GPU efficiency.&quot;&quot;&quot;<br>    <br>    model_id = &quot;microsoft/phi-4&quot;<br>    <br>    if quantize:<br>        from transformers import BitsAndBytesConfig<br>        bnb_config = BitsAndBytesConfig(<br>            load_in_4bit=True,<br>            bnb_4bit_compute_dtype=torch.bfloat16,<br>            bnb_4bit_use_double_quant=True,<br>            bnb_4bit_quant_type=&quot;nf4&quot;,<br>        )<br>        model = AutoModelForCausalLM.from_pretrained(<br>            model_id,<br>            quantization_config=bnb_config,<br>            device_map=&quot;auto&quot;,<br>            trust_remote_code=True,<br>        )<br>    else:<br>        model = AutoModelForCausalLM.from_pretrained(<br>            model_id,<br>            torch_dtype=torch.bfloat16,<br>            device_map=&quot;auto&quot;,<br>            trust_remote_code=True,<br>        )<br>    <br>    tokenizer = AutoTokenizer.from_pretrained(model_id)<br>    <br>    return pipeline(<br>        &quot;text-generation&quot;,<br>        model=model,<br>        tokenizer=tokenizer,<br>    )<br><br>def phi4_reason(pipe, problem: str) -&gt; str:<br>    &quot;&quot;&quot;Use Phi-4 for mathematical or scientific reasoning.&quot;&quot;&quot;<br>    <br>    messages = [<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: &quot;You are an expert mathematician. Solve problems step-by-step, &quot;<br>                       &quot;showing all reasoning. Verify your answer.&quot;<br>        },<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem},<br>    ]<br>    <br>    output = pipe(<br>        messages,<br>        max_new_tokens=1024,<br>        do_sample=False,     # Greedy for math — deterministic<br>        temperature=1.0,     # Required when do_sample=False<br>        return_full_text=False,<br>    )<br>    <br>    return output[0][&quot;generated_text&quot;]<br><br># Usage:<br>pipe = load_phi4(quantize=True)<br>solution = phi4_reason(pipe, &quot;Find all integer solutions to x² - 5x + 6 = 0&quot;)</pre><h2>Part 3: Google Gemma 3 — Open Weights, Production-Ready</h2><h3>Gemma 3 Model Family</h3><pre>Gemma 3 Specifications (2025):<br>─────────────────────────────────────────────────────────────────<br>Gemma 3 1B:<br>  Parameters:       1 billion<br>  Context:          32k tokens<br>  Modalities:       Text only<br>  VRAM (Q4):        ~0.7 GB  ← Raspberry Pi, mobile<br>  License:          Gemma Terms of Service (commercial OK)<br>  <br>Gemma 3 4B:<br>  Parameters:       4 billion<br>  Context:          128k tokens<br>  Modalities:       Text + Vision (multimodal!)<br>  VRAM (Q4):        ~2.5 GB  ← Any modern smartphone<br>  Notable:          First sub-5B model with strong vision capability<br>  <br>Gemma 3 12B:<br>  Parameters:       12 billion<br>  Context:          128k tokens<br>  Modalities:       Text + Vision<br>  VRAM (Q4):        ~7 GB   ← RTX 3080 / M2 Pro<br>  Quality:          Competitive with Llama 3.1 70B on many tasks<br>  <br>Gemma 3 27B:<br>  Parameters:       27 billion<br>  Context:          128k tokens<br>  Modalities:       Text + Vision<br>  VRAM (Q4):        ~15 GB  ← RTX 4090 / A100<br>  Quality:          Competitive with GPT-4o-mini</pre><h3>Gemma 3 Architecture Innovations</h3><pre>Key architectural improvements over Gemma 2:<br><br>1. Interleaved Sliding Window Attention (ISWA):<br>   - Alternates between local attention (4096 token window)<br>     and global attention (full 128k window)<br>   - Every 6th layer uses global attention; others use local<br>   - Dramatically reduces KV cache memory for long contexts<br>   <br>   Dense (every layer global):  O(n²) attention compute<br>   ISWA (1/6 layers global):   ~O(n × 4096) for most layers<br>   <br>2. Long RoPE position embeddings:<br>   - Original training: 8k context<br>   - Extended via RoPE scaling to 128k<br>   - No catastrophic forgetting during extension<br><br>3. Distillation from Gemini Ultra:<br>   - Smaller Gemma models are knowledge-distilled from Gemini<br>   - Teacher forcing with Gemini&#39;s logits during training<br>   - Transfers Gemini&#39;s reasoning patterns to smaller models</pre><h3>Gemma 3 Code Examples</h3><pre># ── Gemma 3 via Ollama ────────────────────────────────────────────────────────<br># ollama pull gemma3:4b        (2.5 GB)<br># ollama pull gemma3:12b       (7.4 GB)<br># ollama pull gemma3:27b       (15 GB)<br><br># ── Gemma 3 for Multimodal Tasks (4B with vision) ────────────────────────────<br>import google.generativeai as genai  # Gemma served via Vertex AI<br>from PIL import Image<br>import httpx<br>import base64<br>import asyncio<br><br>async def gemma3_vision_local(image_path: str, question: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    Use Gemma 3 4B vision capability locally via Ollama.<br>    Gemma 3 4B is one of the smallest models with genuine vision capability.<br>    &quot;&quot;&quot;<br>    <br>    # Encode image as base64<br>    with open(image_path, &quot;rb&quot;) as f:<br>        image_b64 = base64.b64encode(f.read()).decode(&quot;utf-8&quot;)<br>    <br>    ext = image_path.rsplit(&quot;.&quot;, 1)[-1].lower()<br>    mime_type = {&quot;jpg&quot;: &quot;image/jpeg&quot;, &quot;jpeg&quot;: &quot;image/jpeg&quot;,<br>                 &quot;png&quot;: &quot;image/png&quot;, &quot;webp&quot;: &quot;image/webp&quot;}.get(ext, &quot;image/jpeg&quot;)<br>    <br>    async with httpx.AsyncClient(timeout=120.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/v1/chat/completions&quot;,<br>            json={<br>                &quot;model&quot;: &quot;gemma3:4b&quot;,<br>                &quot;messages&quot;: [<br>                    {<br>                        &quot;role&quot;: &quot;user&quot;,<br>                        &quot;content&quot;: [<br>                            {<br>                                &quot;type&quot;: &quot;image_url&quot;,<br>                                &quot;image_url&quot;: {<br>                                    &quot;url&quot;: f&quot;data:{mime_type};base64,{image_b64}&quot;<br>                                },<br>                            },<br>                            {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: question},<br>                        ],<br>                    }<br>                ],<br>                &quot;temperature&quot;: 0.3,<br>                &quot;max_tokens&quot;: 512,<br>            },<br>        )<br>        return response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br><br><br># ── Gemma 3 for Long-Context Document Analysis (128k window) ─────────────────<br>async def gemma3_long_doc_analysis(document: str, query: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    Gemma 3 12B with 128k context — analyze an entire book chapter in one call.<br>    Much cheaper than cloud APIs for long documents processed at scale.<br>    &quot;&quot;&quot;<br>    async with httpx.AsyncClient(timeout=300.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/v1/chat/completions&quot;,<br>            json={<br>                &quot;model&quot;: &quot;gemma3:12b&quot;,<br>                &quot;messages&quot;: [<br>                    {<br>                        &quot;role&quot;: &quot;system&quot;,<br>                        &quot;content&quot;: &quot;You are a precise document analyst. Answer based only on the provided document.&quot;,<br>                    },<br>                    {<br>                        &quot;role&quot;: &quot;user&quot;,<br>                        &quot;content&quot;: f&quot;Document:\n\n{document}\n\n---\n\nQuery: {query}&quot;,<br>                    },<br>                ],<br>                &quot;temperature&quot;: 0.1,<br>                &quot;max_tokens&quot;: 1024,<br>                &quot;options&quot;: {&quot;num_ctx&quot;: 131072},  # Enable full 128k context<br>            },<br>        )<br>        return response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]</pre><h2>Part 4: Apple Intelligence — On-Device AI Architecture</h2><h3>The Apple Intelligence Stack</h3><p>Apple Intelligence is the most sophisticated production deployment of on-device AI at consumer scale. Understanding its architecture reveals the state of the art for edge AI:</p><pre>Apple Intelligence Architecture (2025):<br>─────────────────────────────────────────────────────────────────<br>On-Device Model (Primary):<br>  Parameters:      ~3 billion (estimated)<br>  Quantization:    4-bit with mixed precision (critical layers in 8-bit)<br>  Hardware:        Apple Neural Engine (ANE) — dedicated AI accelerator<br>  Latency:         ~3-5ms per token on M-series chips<br>  Privacy:         100% on-device — no data transmission<br>  Use cases:       Text rewrite, summarization, quick suggestions,<br>                   Writing Tools, notification summaries<br><br>Server Model (Private Cloud Compute):<br>  Parameters:      ~70+ billion (estimated, on Apple Silicon servers)<br>  Privacy:         Private Cloud Compute — Apple-audited, no data logging,<br>                   verifiable by third parties via cryptographic attestation<br>  Use cases:       Complex requests that exceed on-device model capability<br>  <br>OpenAI Integration:<br>  Model:           GPT-4o (user consent required explicitly)<br>  Use cases:       &quot;More capable&quot; requests forwarded with user permission<br>  Data policy:     OpenAI&#39;s standard terms apply<br><br>Routing Logic:<br>  Simple task → On-device model (instant, private)<br>  Medium task → Private Cloud Compute (private, more capable)<br>  Complex task → GPT-4o (consent required, most capable)</pre><h3>The Neural Engine — Why Apple&#39;s On-Device AI Is Fast</h3><pre>Apple Neural Engine (ANE) — A18 Pro chip (iPhone 16 Pro):<br>─────────────────────────────────────────────────────────────────<br>ANE Compute:         38 TOPS (tera operations per second)<br>Memory bandwidth:    ~68 GB/s (shared with CPU/GPU via unified memory)<br>Optimal operations:  INT8, INT4 matrix multiplications<br>Model size limit:    ~8-10 GB (total unified memory budget)<br>Power consumption:   ~1-2W during inference (vs 20W+ for GPU)<br>Framework:           Core ML (Apple&#39;s inference runtime)<br><br>Why ANE beats GPU for SLMs:<br>  • Purpose-built for matrix multiply (the dominant LLM operation)<br>  • Energy efficiency: 10-20× more efficient per TOPS than mobile GPU<br>  • Latency: Avoids PCIe/memory bandwidth bottlenecks of discrete GPUs<br>  • Thermal: Sustained performance without throttling on mobile</pre><h3>Deploying Models to Apple Neural Engine with Core ML</h3><pre># Step 1: Export a HuggingFace model to Core ML format<br># Requires: pip install coremltools transformers torch<br><br>import coremltools as ct<br>import torch<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br><br>def export_to_coreml(<br>    model_id: str = &quot;microsoft/phi-3-mini-4k-instruct&quot;,<br>    output_path: str = &quot;phi3_mini.mlpackage&quot;,<br>    sequence_length: int = 512,<br>):<br>    &quot;&quot;&quot;<br>    Export a HuggingFace model to Core ML format for Apple Neural Engine.<br>    <br>    The exported model can be used in iOS/macOS apps via Core ML framework.<br>    &quot;&quot;&quot;<br>    <br>    print(f&quot;Loading {model_id}...&quot;)<br>    model = AutoModelForCausalLM.from_pretrained(<br>        model_id,<br>        torch_dtype=torch.float32,  # Core ML converter needs float32<br>        trust_remote_code=True,<br>    )<br>    tokenizer = AutoTokenizer.from_pretrained(model_id)<br>    model.eval()<br>    <br>    # Create example input for tracing<br>    example_input_ids = torch.randint(0, 32000, (1, sequence_length))<br>    example_attention_mask = torch.ones(1, sequence_length, dtype=torch.long)<br>    <br>    # Trace the model with TorchScript<br>    print(&quot;Tracing model...&quot;)<br>    with torch.no_grad():<br>        traced_model = torch.jit.trace(<br>            model,<br>            (example_input_ids, example_attention_mask),<br>            strict=False,<br>        )<br>    <br>    # Convert to Core ML<br>    print(&quot;Converting to Core ML...&quot;)<br>    mlmodel = ct.convert(<br>        traced_model,<br>        inputs=[<br>            ct.TensorType(<br>                name=&quot;input_ids&quot;,<br>                shape=(1, ct.RangeDim(1, sequence_length)),<br>                dtype=int,<br>            ),<br>            ct.TensorType(<br>                name=&quot;attention_mask&quot;,<br>                shape=(1, ct.RangeDim(1, sequence_length)),<br>                dtype=int,<br>            ),<br>        ],<br>        # Compute units: ANE preferred, falls back to CPU if needed<br>        compute_units=ct.ComputeUnit.ALL,<br>        # 4-bit weight quantization for maximum efficiency<br>        pass_pipeline=ct.PassPipeline.DEFAULT,<br>        minimum_deployment_target=ct.target.iOS17,<br>    )<br>    <br>    # Apply 4-bit weight quantization (critical for performance)<br>    from coremltools.optimize.coreml import OpLinearQuantizerConfig, OptimizationConfig<br>    <br>    config = OptimizationConfig(<br>        global_config=OpLinearQuantizerConfig(<br>            mode=&quot;linear_symmetric&quot;,<br>            dtype=&quot;int4&quot;,<br>            granularity=&quot;per_block&quot;,<br>            block_size=32,<br>        )<br>    )<br>    <br>    compressed_model = ct.optimize.coreml.linear_quantize_weights(mlmodel, config)<br>    compressed_model.save(output_path)<br>    <br>    print(f&quot;✅ Saved to {output_path}&quot;)<br>    print(f&quot;   Model size: {os.path.getsize(output_path) / 1e9:.1f} GB&quot;)<br>    return compressed_model<br><br><br># Step 2: Use in Swift (iOS/macOS app)<br>swift_inference_code = &#39;&#39;&#39;<br>import CoreML<br>import Foundation<br><br>class PhiInferenceEngine {<br>    private let model: MLModel<br>    private let tokenizer: Tokenizer  // Use swift-transformers package<br>    <br>    init() throws {<br>        // Load the .mlpackage<br>        let modelURL = Bundle.main.url(/service/forresource: &quot;phi3_mini&quot;, withExtension: &quot;mlpackage&quot;)!<br>        let config = MLModelConfiguration()<br>        config.computeUnits = .all  // Use ANE + GPU + CPU<br>        self.model = try MLModel(contentsOf: modelURL, configuration: config)<br>        self.tokenizer = try Tokenizer(pretrained: &quot;microsoft/phi-3-mini-4k-instruct&quot;)<br>    }<br>    <br>    func generate(prompt: String, maxTokens: Int = 256) throws -&gt; String {<br>        var inputIds = tokenizer.encode(prompt)<br>        var generatedTokens: [Int] = []<br>        <br>        for _ in 0..&lt;maxTokens {<br>            // Prepare Core ML input<br>            let inputArray = try MLMultiArray(shape: [1, inputIds.count as NSNumber], dataType: .int32)<br>            for (i, id) in inputIds.enumerated() {<br>                inputArray[i] = NSNumber(value: id)<br>            }<br>            <br>            let input = try MLDictionaryFeatureProvider(dictionary: [<br>                &quot;input_ids&quot;: MLFeatureValue(multiArray: inputArray)<br>            ])<br>            <br>            // Run inference on ANE<br>            let output = try model.prediction(from: input)<br>            <br>            // Sample next token (greedy)<br>            let logits = output.featureValue(for: &quot;logits&quot;)!.multiArrayValue!<br>            let nextToken = argmax(logits, lastPosition: inputIds.count - 1)<br>            <br>            if nextToken == tokenizer.eosTokenId { break }<br>            <br>            generatedTokens.append(nextToken)<br>            inputIds.append(nextToken)<br>        }<br>        <br>        return tokenizer.decode(generatedTokens)<br>    }<br>}<br>&#39;&#39;&#39;<br>print(&quot;Swift inference code ready for iOS integration&quot;)</pre><h2>Part 5: The Full SLM Landscape in 2026</h2><h3>Comprehensive Model Comparison</h3><pre>Model               Params   Context   VRAM(Q4)  Notable For           License<br>────────────────────────────────────────────────────────────────────────────────────<br>Microsoft<br>  Phi-4             14B      16k       ~8 GB     STEM/math reasoning    MS Research<br>  Phi-3.5-mini      3.8B     128k      ~2.5 GB   Long context, fast     MIT<br>  Phi-3.5-MoE       16×3.8B  128k      ~7 GB     MoE efficiency         MIT<br>  Phi-3-mini        3.8B     4k/128k   ~2.5 GB   On-device, multilingual MIT<br><br>Google<br>  Gemma 3 27B       27B      128k      ~15 GB    GPT-4o-mini quality    Gemma ToS<br>  Gemma 3 12B       12B      128k      ~7 GB     Vision + long context  Gemma ToS<br>  Gemma 3 4B        4B       128k      ~2.5 GB   Best sub-5B vision     Gemma ToS<br>  Gemma 3 1B        1B       32k       ~0.7 GB   Embedded, IoT          Gemma ToS<br><br>Meta<br>  Llama 3.2 1B      1B       128k      ~0.7 GB   Smallest usable LLM    Llama 3.2<br>  Llama 3.2 3B      3B       128k      ~2 GB     Solid generalist       Llama 3.2<br>  Llama 3.1 8B      8B       128k      ~5 GB     Most-used open model   Llama 3.1<br>  Llama 3.3 70B     70B      128k      ~40 GB    Near-GPT-4 quality     Llama 3.3<br><br>Alibaba<br>  Qwen 2.5 0.5B     0.5B     32k       ~0.4 GB   Extreme edge           Apache 2.0<br>  Qwen 2.5 1.5B     1.5B     32k       ~1 GB     Strong for size        Apache 2.0<br>  Qwen 2.5 7B       7B       128k      ~4.5 GB   Best-in-class 7B       Apache 2.0<br>  Qwen 2.5-Coder 7B 7B       128k      ~4.5 GB   Specialized for code   Apache 2.0<br><br>Mistral<br>  Mistral 7B v0.3   7B       32k       ~4.5 GB   Fast, reliable         Apache 2.0<br>  Mistral Nemo 12B  12B      128k      ~7 GB     Multilingual           Apache 2.0<br><br>Apple (not public weights)<br>  Apple Intelligence ~3B     N/A       On-device Integrated iOS/macOS   Proprietary</pre><h3>Benchmarks: Small Models Punching Above Their Weight</h3><pre>MMLU (academic reasoning, 57 subjects):<br>  GPT-4o:          87.2%<br>  Phi-4 (14B):     84.8%  ← 14B beats most 70B models<br>  Llama 3.1 70B:   86.0%<br>  Gemma 3 27B:     83.2%<br>  Qwen 2.5 7B:     74.2%<br>  Phi-3.5-mini 3.8B: 69.0%<br><br>MATH (competition mathematics):<br>  GPT-4o:          74.6%<br>  Phi-4 (14B):     80.4%  ← Phi-4 BEATS GPT-4o on math!<br>  Llama 3.1 70B:   68.0%<br>  Gemma 3 27B:     71.5%<br>  Qwen 2.5 7B:     52.9%<br><br>HumanEval (coding):<br>  GPT-4o:          90.2%<br>  Phi-4 (14B):     82.6%<br>  Qwen 2.5-Coder 7B: 87.3%  ← Specialized model beats general 14B<br>  Llama 3.1 70B:   80.5%<br>  Gemma 3 12B:     72.0%<br><br>The lesson: For specialized tasks, a smaller specialized model<br>often beats a larger general-purpose model.</pre><h2>Part 6: Quantization Formats for On-Device Deployment</h2><p>Quantization is the technique that makes billion-parameter models fit on phones, laptops, and microcontrollers. Understanding the formats is essential for deployment engineering.</p><h3>The Quantization Landscape</h3><pre>Format    Runtime          Platforms               Typical Speedup   Quality Loss<br>──────────────────────────────────────────────────────────────────────────────────<br>GGUF      llama.cpp        CPU, Metal, CUDA         2-4×              Low (Q4)<br>          Ollama (uses<br>          llama.cpp)<br><br>ONNX      ONNX Runtime     Windows, Linux, macOS,   2-6×              Low-Medium<br>                           iOS, Android, Browser<br><br>Core ML   Core ML          iOS, macOS, watchOS      3-8× (ANE)        Low<br>          framework<br><br>ExecuTorch PyTorch Edge    iOS, Android, microcontr. 2-4×              Low<br>          (Meta)<br><br>TFLite    TF Lite          Android, microcontrollers 2-4×              Low<br>          (Google)<br><br>MLC       WebLLM           Browser (WebGPU/WASM),   3-6×              Low<br>(tvm)                      iOS, Android<br><br>QNN SDK   Qualcomm Neural  Snapdragon devices        5-10× (NPU)       Low<br>          Processing Units</pre><h3>GGUF Deep Dive — The Most Practical Format</h3><p>GGUF (GPT-Generated Unified Format) is the workhorse format for local LLM inference:</p><pre>import subprocess<br>from pathlib import Path<br><br>def understand_gguf_quantization_levels():<br>    &quot;&quot;&quot;<br>    GGUF quantization levels — the practical guide.<br>    <br>    Naming convention: Q{bits}_{variant}<br>    &quot;&quot;&quot;<br>    <br>    levels = {<br>        &quot;Q2_K&quot;: {<br>            &quot;bits_per_weight&quot;: 2.6,<br>            &quot;size_multiplier&quot;: 0.20,  # 20% of original FP16 size<br>            &quot;quality&quot;: &quot;Poor — noticeable degradation&quot;,<br>            &quot;use_case&quot;: &quot;Extreme memory constraints only&quot;,<br>        },<br>        &quot;Q3_K_M&quot;: {<br>            &quot;bits_per_weight&quot;: 3.35,<br>            &quot;size_multiplier&quot;: 0.26,<br>            &quot;quality&quot;: &quot;Acceptable — some capability loss&quot;,<br>            &quot;use_case&quot;: &quot;RAM-constrained devices (Raspberry Pi 4)&quot;,<br>        },<br>        &quot;Q4_K_M&quot;: {<br>            &quot;bits_per_weight&quot;: 4.83,<br>            &quot;size_multiplier&quot;: 0.37,<br>            &quot;quality&quot;: &quot;Good — minimal perceptible loss&quot;,<br>            &quot;use_case&quot;: &quot;Standard on-device deployment — RECOMMENDED&quot;,<br>        },<br>        &quot;Q5_K_M&quot;: {<br>            &quot;bits_per_weight&quot;: 5.68,<br>            &quot;size_multiplier&quot;: 0.44,<br>            &quot;quality&quot;: &quot;Very good — near indistinguishable from FP16&quot;,<br>            &quot;use_case&quot;: &quot;When you have enough RAM and want better quality&quot;,<br>        },<br>        &quot;Q6_K&quot;: {<br>            &quot;bits_per_weight&quot;: 6.57,<br>            &quot;size_multiplier&quot;: 0.50,<br>            &quot;quality&quot;: &quot;Excellent — essentially full quality&quot;,<br>            &quot;use_case&quot;: &quot;Laptop/desktop with 32+ GB unified memory&quot;,<br>        },<br>        &quot;Q8_0&quot;: {<br>            &quot;bits_per_weight&quot;: 8.0,<br>            &quot;size_multiplier&quot;: 0.61,<br>            &quot;quality&quot;: &quot;Near-perfect — essentially FP16 quality&quot;,<br>            &quot;use_case&quot;: &quot;When memory is plentiful, max quality needed&quot;,<br>        },<br>        &quot;F16&quot;: {<br>            &quot;bits_per_weight&quot;: 16.0,<br>            &quot;size_multiplier&quot;: 1.00,<br>            &quot;quality&quot;: &quot;Full (original model quality)&quot;,<br>            &quot;use_case&quot;: &quot;GPU fine-tuning or when size is no concern&quot;,<br>        },<br>    }<br>    <br>    # Example: Phi-4 (14B parameters) at different quantization levels<br>    base_size_gb = 14 * 2  # 14B params × 2 bytes/param for FP16 = ~28 GB<br>    <br>    print(f&quot;{&#39;Level&#39;:&lt;10} {&#39;Bits/W&#39;:&lt;8} {&#39;Size (GB)&#39;:&lt;12} {&#39;Quality&#39;:&lt;35} {&#39;Use Case&#39;}&quot;)<br>    print(&quot;─&quot; * 110)<br>    <br>    for level, info in levels.items():<br>        size = base_size_gb * info[&quot;size_multiplier&quot;]<br>        print(f&quot;{level:&lt;10} {info[&#39;bits_per_weight&#39;]:&lt;8.2f} {size:&lt;12.1f} &quot;<br>              f&quot;{info[&#39;quality&#39;]:&lt;35} {info[&#39;use_case&#39;]}&quot;)<br><br>understand_gguf_quantization_levels()</pre><p><strong>Output:</strong></p><pre>Level      Bits/W   Size (GB)    Quality                             Use Case<br>────────────────────────────────────────────────────────────────────────────────────────────────────────────────<br>Q2_K       2.60     5.6          Poor — noticeable degradation       Extreme memory constraints only<br>Q3_K_M     3.35     7.3          Acceptable — some capability loss   RAM-constrained devices (Raspberry Pi 4)<br>Q4_K_M     4.83     10.5         Good — minimal perceptible loss     Standard on-device deployment — RECOMMENDED<br>Q5_K_M     5.68     12.3         Very good — near indistinguishable  When you have enough RAM<br>Q6_K       6.57     14.2         Excellent — essentially full        Laptop/desktop with 32+ GB<br>Q8_0       8.00     17.3         Near-perfect                        Max quality, memory plentiful<br>F16        16.00    28.0         Full (original model quality)       GPU fine-tuning</pre><h3>Running Models in the Browser with WebLLM</h3><pre>// WebLLM — run LLMs in the browser via WebGPU<br>// No server, no API key, complete privacy<br><br>import * as webllm from &quot;@mlc-ai/web-llm&quot;;<br><br>class BrowserLLM {<br>    constructor() {<br>        this.engine = null;<br>        this.isLoading = false;<br>    }<br>    <br>    async initialize(<br>        modelId = &quot;Phi-3.5-mini-instruct-q4f16_1-MLC&quot;,<br>        onProgress = (report) =&gt; console.log(report)<br>    ) {<br>        this.isLoading = true;<br>        <br>        // Initialize engine — downloads and caches model in browser<br>        this.engine = await webllm.CreateMLCEngine(<br>            modelId,<br>            {<br>                initProgressCallback: onProgress,<br>                // Model is cached in browser&#39;s IndexedDB after first download<br>                // Subsequent loads are instant<br>            }<br>        );<br>        <br>        this.isLoading = false;<br>        console.log(&quot;✅ Model loaded in browser!&quot;);<br>        return this.engine;<br>    }<br>    <br>    async complete(messages, options = {}) {<br>        if (!this.engine) throw new Error(&quot;Engine not initialized&quot;);<br>        <br>        const response = await this.engine.chat.completions.create({<br>            messages,<br>            temperature: options.temperature ?? 0.7,<br>            max_tokens: options.maxTokens ?? 512,<br>            stream: options.stream ?? false,<br>        });<br>        <br>        return response.choices[0].message.content;<br>    }<br>    <br>    async *stream(messages, options = {}) {<br>        if (!this.engine) throw new Error(&quot;Engine not initialized&quot;);<br>        <br>        const chunks = await this.engine.chat.completions.create({<br>            messages,<br>            stream: true,<br>            temperature: options.temperature ?? 0.7,<br>            max_tokens: options.maxTokens ?? 512,<br>        });<br>        <br>        for await (const chunk of chunks) {<br>            const delta = chunk.choices[0]?.delta?.content;<br>            if (delta) yield delta;<br>        }<br>    }<br>}<br><br>// Available models for WebLLM (2026):<br>const WEB_LLM_MODELS = [<br>    &quot;Llama-3.2-1B-Instruct-q4f32_1-MLC&quot;,     // 0.8 GB — smallest usable<br>    &quot;Llama-3.2-3B-Instruct-q4f16_1-MLC&quot;,     // 1.8 GB — good quality<br>    &quot;Phi-3.5-mini-instruct-q4f16_1-MLC&quot;,     // 2.2 GB — excellent for size<br>    &quot;gemma-2-2b-it-q4f16_1-MLC&quot;,             // 1.5 GB — fast, mobile-grade<br>    &quot;Mistral-7B-Instruct-v0.3-q4f16_1-MLC&quot;,  // 4.0 GB — high quality<br>    &quot;Llama-3.1-8B-Instruct-q4f32_1-MLC&quot;,    // 5.0 GB — near cloud quality<br>];<br><br>// HTML integration:<br>const html_example = `<br>&lt;script type=&quot;module&quot;&gt;<br>    import { CreateMLCEngine } from &quot;https://esm.run/@mlc-ai/web-llm&quot;;<br>    <br>    const btn = document.getElementById(&quot;run-btn&quot;);<br>    const output = document.getElementById(&quot;output&quot;);<br>    <br>    let engine = null;<br>    <br>    async function loadModel() {<br>        engine = await CreateMLCEngine(<br>            &quot;Phi-3.5-mini-instruct-q4f16_1-MLC&quot;,<br>            { initProgressCallback: (p) =&gt; output.textContent = p.text }<br>        );<br>    }<br>    <br>    btn.addEventListener(&quot;click&quot;, async () =&gt; {<br>        const prompt = document.getElementById(&quot;prompt&quot;).value;<br>        const response = await engine.chat.completions.create({<br>            messages: [{ role: &quot;user&quot;, content: prompt }],<br>            stream: true,<br>        });<br>        <br>        output.textContent = &quot;&quot;;<br>        for await (const chunk of response) {<br>            output.textContent += chunk.choices[0]?.delta?.content ?? &quot;&quot;;<br>        }<br>    });<br>    <br>    loadModel();<br>&lt;/script&gt;<br>`;</pre><h2>Part 7: Choosing Between SLM and Cloud LLM</h2><h3>The Decision Framework</h3><pre>from dataclasses import dataclass<br>from enum import Enum<br><br>class Recommendation(Enum):<br>    CLOUD_LLM    = &quot;Use Cloud LLM (GPT-4o / Claude)&quot;<br>    SLM_LOCAL    = &quot;Use SLM Locally&quot;<br>    SLM_HYBRID   = &quot;Hybrid: SLM for simple, Cloud for complex&quot;<br>    SLM_EDGE     = &quot;Deploy SLM to Edge/Device&quot;<br><br>@dataclass<br>class WorkloadProfile:<br>    data_sensitivity: str      # &quot;public&quot; | &quot;internal&quot; | &quot;confidential&quot; | &quot;regulated&quot;<br>    task_complexity: str       # &quot;simple&quot; | &quot;moderate&quot; | &quot;complex&quot; | &quot;frontier&quot;<br>    latency_requirement_ms: int<br>    requests_per_day: int<br>    internet_available: bool<br>    gpu_available: bool<br>    target_device: str         # &quot;server&quot; | &quot;laptop&quot; | &quot;mobile&quot; | &quot;browser&quot; | &quot;edge&quot;<br>    budget_per_month_usd: float<br><br>def recommend_deployment(profile: WorkloadProfile) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Decision tree for SLM vs Cloud LLM deployment.<br>    &quot;&quot;&quot;<br>    <br>    reasons = []<br>    <br>    # Hard constraints that force SLM<br>    if not profile.internet_available:<br>        reasons.append(&quot;No internet — cloud LLM impossible&quot;)<br>        return {&quot;recommendation&quot;: Recommendation.SLM_LOCAL, &quot;reasons&quot;: reasons}<br>    <br>    if profile.data_sensitivity in (&quot;confidential&quot;, &quot;regulated&quot;):<br>        reasons.append(f&quot;Data sensitivity &#39;{profile.data_sensitivity}&#39; prohibits cloud&quot;)<br>        if profile.target_device in (&quot;mobile&quot;, &quot;browser&quot;, &quot;edge&quot;):<br>            return {&quot;recommendation&quot;: Recommendation.SLM_EDGE, &quot;reasons&quot;: reasons}<br>        return {&quot;recommendation&quot;: Recommendation.SLM_LOCAL, &quot;reasons&quot;: reasons}<br>    <br>    if profile.target_device in (&quot;mobile&quot;, &quot;browser&quot;, &quot;edge&quot;):<br>        reasons.append(f&quot;Target device &#39;{profile.target_device}&#39; requires on-device model&quot;)<br>        return {&quot;recommendation&quot;: Recommendation.SLM_EDGE, &quot;reasons&quot;: reasons}<br>    <br>    # Cost analysis<br>    # Cloud: assume gpt-4o-mini at $0.15/M input, $0.60/M output, ~500 tokens avg<br>    avg_tokens = 1000<br>    cloud_cost_per_request = (avg_tokens / 2_000_000) * 0.75  # blended<br>    cloud_monthly_cost = cloud_cost_per_request * profile.requests_per_day * 30<br>    <br>    if cloud_monthly_cost &gt; profile.budget_per_month_usd:<br>        reasons.append(<br>            f&quot;Cloud cost ${cloud_monthly_cost:.0f}/month exceeds budget &quot;<br>            f&quot;${profile.budget_per_month_usd:.0f}/month&quot;<br>        )<br>        return {&quot;recommendation&quot;: Recommendation.SLM_LOCAL, &quot;reasons&quot;: reasons}<br>    <br>    # Latency constraint<br>    if profile.latency_requirement_ms &lt; 200:<br>        reasons.append(f&quot;Latency requirement {profile.latency_requirement_ms}ms &quot;<br>                       &quot;may be hard to meet with cloud API (network round-trip ~100-300ms)&quot;)<br>        if profile.gpu_available:<br>            reasons.append(&quot;Local GPU inference can meet latency requirement&quot;)<br>            return {&quot;recommendation&quot;: Recommendation.SLM_LOCAL, &quot;reasons&quot;: reasons}<br>    <br>    # Task complexity<br>    if profile.task_complexity == &quot;frontier&quot;:<br>        reasons.append(&quot;Frontier task complexity requires cloud models (o3, Claude Opus)&quot;)<br>        return {&quot;recommendation&quot;: Recommendation.CLOUD_LLM, &quot;reasons&quot;: reasons}<br>    <br>    if profile.task_complexity == &quot;complex&quot;:<br>        reasons.append(&quot;Complex tasks benefit from cloud models for quality&quot;)<br>        return {&quot;recommendation&quot;: Recommendation.CLOUD_LLM, &quot;reasons&quot;: reasons}<br>    <br>    if profile.task_complexity in (&quot;simple&quot;, &quot;moderate&quot;):<br>        reasons.append(&quot;Simple/moderate tasks work well with SLM&quot;)<br>        reasons.append(&quot;Use hybrid: SLM handles routine, escalate complex to cloud&quot;)<br>        return {&quot;recommendation&quot;: Recommendation.SLM_HYBRID, &quot;reasons&quot;: reasons}<br>    <br>    # Default<br>    return {&quot;recommendation&quot;: Recommendation.CLOUD_LLM, &quot;reasons&quot;: [&quot;No specific reason for SLM&quot;]}<br><br><br># Examples:<br>profiles = [<br>    WorkloadProfile(<br>        data_sensitivity=&quot;regulated&quot;,  # Healthcare — PHI data<br>        task_complexity=&quot;moderate&quot;,<br>        latency_requirement_ms=2000,<br>        requests_per_day=5000,<br>        internet_available=True,<br>        gpu_available=True,<br>        target_device=&quot;server&quot;,<br>        budget_per_month_usd=5000,<br>    ),<br>    WorkloadProfile(<br>        data_sensitivity=&quot;public&quot;,<br>        task_complexity=&quot;simple&quot;,<br>        latency_requirement_ms=500,<br>        requests_per_day=1_000_000,  # High volume<br>        internet_available=True,<br>        gpu_available=False,<br>        target_device=&quot;server&quot;,<br>        budget_per_month_usd=500,<br>    ),<br>    WorkloadProfile(<br>        data_sensitivity=&quot;internal&quot;,<br>        task_complexity=&quot;simple&quot;,<br>        latency_requirement_ms=100,  # Real-time typing assistance<br>        requests_per_day=50000,<br>        internet_available=True,<br>        gpu_available=False,<br>        target_device=&quot;browser&quot;,<br>        budget_per_month_usd=200,<br>    ),<br>]<br><br>for p in profiles:<br>    result = recommend_deployment(p)<br>    print(f&quot;\nRecommendation: {result[&#39;recommendation&#39;].value}&quot;)<br>    for r in result[&quot;reasons&quot;]:<br>        print(f&quot;  • {r}&quot;)</pre><h2>Part 8: Building a Privacy-First Local AI Pipeline</h2><pre>import asyncio<br>import httpx<br>import os<br>from pathlib import Path<br><br>class LocalAIPipeline:<br>    &quot;&quot;&quot;<br>    A complete local AI pipeline — no data leaves your machine.<br>    <br>    Uses Ollama as the runtime with configurable models:<br>    - Gemma 3 4B for general tasks (vision + text, 128k context)<br>    - Phi-4 for reasoning/math tasks<br>    - Qwen 2.5-Coder for code tasks<br>    &quot;&quot;&quot;<br>    <br>    TASK_MODELS = {<br>        &quot;general&quot;:    &quot;gemma3:4b&quot;,<br>        &quot;reasoning&quot;:  &quot;phi4&quot;,<br>        &quot;code&quot;:       &quot;qwen2.5-coder:7b&quot;,<br>        &quot;vision&quot;:     &quot;gemma3:4b&quot;,<br>        &quot;fast&quot;:       &quot;gemma3:1b&quot;,<br>        &quot;long_doc&quot;:   &quot;gemma3:12b&quot;,<br>    }<br>    <br>    def __init__(self, ollama_host: str = &quot;http://localhost:11434&quot;):<br>        self.host = ollama_host<br>    <br>    async def _ensure_model_available(self, model: str) -&gt; bool:<br>        &quot;&quot;&quot;Check if model is pulled, pull if not.&quot;&quot;&quot;<br>        async with httpx.AsyncClient(timeout=10.0) as client:<br>            response = await client.get(f&quot;{self.host}/api/tags&quot;)<br>            tags = response.json()<br>            available = [m[&quot;name&quot;] for m in tags.get(&quot;models&quot;, [])]<br>            <br>            if model not in available:<br>                print(f&quot;Pulling {model}...&quot;)<br>                # Pull model<br>                async with client.stream(<br>                    &quot;POST&quot;,<br>                    f&quot;{self.host}/api/pull&quot;,<br>                    json={&quot;name&quot;: model},<br>                ) as stream:<br>                    async for line in stream.aiter_lines():<br>                        if line:<br>                            import json<br>                            data = json.loads(line)<br>                            if &quot;status&quot; in data:<br>                                print(f&quot;  {data[&#39;status&#39;]}&quot;, end=&quot;\r&quot;)<br>                print(f&quot;\n✅ {model} ready&quot;)<br>            <br>            return True<br>    <br>    async def complete(<br>        self,<br>        prompt: str,<br>        task: str = &quot;general&quot;,<br>        system_prompt: str | None = None,<br>        image_path: str | None = None,<br>        temperature: float = 0.7,<br>        max_tokens: int = 1024,<br>    ) -&gt; str:<br>        &quot;&quot;&quot;<br>        Complete a prompt with automatic model selection by task type.<br>        All processing stays on-device.<br>        &quot;&quot;&quot;<br>        <br>        model = self.TASK_MODELS.get(task, self.TASK_MODELS[&quot;general&quot;])<br>        await self._ensure_model_available(model)<br>        <br>        messages = []<br>        if system_prompt:<br>            messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt})<br>        <br>        # Build user message (with optional image)<br>        if image_path and task == &quot;vision&quot;:<br>            import base64<br>            with open(image_path, &quot;rb&quot;) as f:<br>                img_b64 = base64.b64encode(f.read()).decode()<br>            messages.append({<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: prompt,<br>                &quot;images&quot;: [img_b64],<br>            })<br>        else:<br>            messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br>        <br>        async with httpx.AsyncClient(timeout=300.0) as client:<br>            response = await client.post(<br>                f&quot;{self.host}/v1/chat/completions&quot;,<br>                json={<br>                    &quot;model&quot;: model,<br>                    &quot;messages&quot;: messages,<br>                    &quot;temperature&quot;: temperature,<br>                    &quot;max_tokens&quot;: max_tokens,<br>                },<br>            )<br>            return response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br>    <br>    async def classify_and_route(self, user_query: str) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Use a tiny model to classify the query, then route to the best local model.<br>        Everything stays on-device.<br>        &quot;&quot;&quot;<br>        <br>        # Fast classification with 1B model<br>        classification = await self.complete(<br>            prompt=f&quot;&quot;&quot;Classify this query into exactly one category:<br>            <br>Query: {user_query}<br><br>Categories:<br>- general: everyday questions, writing, explanation<br>- reasoning: math, logic, step-by-step problem solving<br>- code: programming, debugging, code review<br>- vision: questions about images (if image provided)<br>- fast: simple, 1-sentence answer questions<br><br>Reply with only the category name.&quot;&quot;&quot;,<br>            task=&quot;fast&quot;,<br>            temperature=0.0,<br>        )<br>        <br>        task = classification.strip().lower()<br>        if task not in self.TASK_MODELS:<br>            task = &quot;general&quot;<br>        <br>        # Route to appropriate model<br>        answer = await self.complete(user_query, task=task)<br>        <br>        return {<br>            &quot;answer&quot;: answer,<br>            &quot;task_type&quot;: task,<br>            &quot;model_used&quot;: self.TASK_MODELS[task],<br>            &quot;data_left_device&quot;: False,  # Privacy guarantee<br>        }<br><br><br># Usage<br>async def demo():<br>    pipeline = LocalAIPipeline()<br>    <br>    queries = [<br>        &quot;What is 2 + 2?&quot;,<br>        &quot;Prove that there are infinitely many prime numbers&quot;,<br>        &quot;Write a Python function to merge two sorted lists&quot;,<br>        &quot;Summarize the history of the Roman Empire&quot;,<br>    ]<br>    <br>    for query in queries:<br>        result = await pipeline.classify_and_route(query)<br>        print(f&quot;\nQuery: {query[:60]}&quot;)<br>        print(f&quot;Routed to: {result[&#39;model_used&#39;]} ({result[&#39;task_type&#39;]})&quot;)<br>        print(f&quot;Answer: {result[&#39;answer&#39;][:150]}...&quot;)<br>        print(f&quot;Privacy: {&#39;✅ On-device&#39; if not result[&#39;data_left_device&#39;] else &#39;☁️ Cloud&#39;}&quot;)<br><br>asyncio.run(demo())</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Choosing model size before benchmarking on your actual task</strong></p><p>“Bigger is better” intuition leads engineers to dismiss 7B models without testing. Qwen 2.5-Coder 7B outperforms many 70B models on coding tasks. Always benchmark on your specific task with your specific data before committing to a model.</p><p><strong>2. Using Q2/Q3 quantization for production</strong></p><p>Q2_K and Q3_K_M quantization levels produce noticeable quality degradation — inconsistent reasoning, introduced errors, degraded instruction following. In production, use Q4_K_M as the minimum. The size savings of Q2/Q3 are not worth the quality loss.</p><p><strong>3. Forgetting that on-device models need warm-up time</strong></p><p>The first inference call after loading a model is slow (model compilation, cache warming). For latency-sensitive applications, always make a dummy warm-up call after loading before serving real requests.</p><pre># ✅ Always warm up before serving<br>async def initialize_and_warmup(pipeline):<br>    await pipeline.complete(&quot;Hello&quot;, task=&quot;fast&quot;)  # Warm-up call<br>    print(&quot;Model warmed up — ready to serve&quot;)</pre><p><strong>4. Not accounting for context window in memory calculation</strong></p><p>The KV cache grows linearly with context length. At 128k context, a 7B model can require 2–4× more memory than at 4k context. If you’re using Gemma 3 12B at full 128k context, budget for significantly more VRAM than the base model size suggests.</p><p><strong>5. Using a generalist SLM when a specialized one exists</strong></p><p>For code tasks, Qwen 2.5-Coder or Phi-4 dramatically outperform same-size general models. For math, Phi-4 outperforms 70B generalists. For multilingual tasks, Qwen 2.5 multilingual variants outperform English-focused Phi. Always check if a domain-specialized SLM exists before defaulting to a larger generalist.</p><h2>💼 Quick Questions</h2><p><strong>Q: Why does Phi-4 (14B parameters) outperform models 5× its size on mathematics benchmarks?</strong></p><p>Phi-4 was trained on synthetically generated, pedagogically structured data — essentially high-quality textbook problems with worked solutions, generated by GPT-4o — rather than internet-scraped text. For mathematical reasoning, the quality of reasoning demonstrations in training data matters more than data quantity. A model trained on 100 carefully worked math problems learns better reasoning patterns than one trained on 1,000 problem statements without clear derivations. Phi-4 also benefits from Microsoft’s “near-miss” data synthesis, where the training data intentionally includes and corrects common reasoning errors, teaching the model to recognize and avoid them.</p><p><strong>Q: You need to build a medical transcription AI for a hospital. They cite HIPAA compliance — patient data cannot leave the hospital network. How do you architect this?</strong></p><p>Deploy an SLM on hospital servers (or edge devices in examination rooms). Specifically: (1) Evaluate Phi-4 or Gemma 3 12B for transcription quality on a labeled dataset of de-identified transcripts; (2) Run 4-bit quantized models via vLLM or Ollama on hospital-owned GPU servers; (3) Fine-tune on a small curated dataset of medical transcription examples using LoRA; (4) Serve via an internal REST API — no internet egress; (5) Add an audit log for all inferences (HIPAA requires audit trails). The quality gap vs GPT-4 is acceptable for transcription tasks — accuracy on well-defined domains like medical terminology is high at 7–14B scales.</p><p><strong>Q: What is the Core ML Neural Engine advantage for on-device inference vs using the GPU?</strong></p><p>The Apple Neural Engine is purpose-built for the dominant operation in transformer inference: INT8/INT4 matrix multiplication. At 38 TOPS for A18 Pro, it delivers 10–20× better energy efficiency per operation compared to using the mobile GPU for the same workload. This means sustained throughput without thermal throttling (critical for generation, which runs many sequential forward passes), dramatically longer battery life (1–2W vs 10–20W for GPU inference), and predictable latency without competing with graphics rendering. The tradeoff is that ANE requires quantized models compiled specifically for Core ML format — you can’t run arbitrary PyTorch models on the ANE directly.</p><h2>🏭 Production Considerations</h2><p><strong>Speculative decoding for SLM acceleration.</strong> Pair a tiny “draft model” (e.g., Llama-3.2–1B) with a larger “target model” (e.g., Llama-3.1–8B). The draft model generates tokens quickly; the target model verifies in parallel. 2–3× speedup with identical output quality to the target model alone. Available in llama.cpp, vLLM, and HuggingFace TGI.</p><p><strong>Model caching between requests is critical.</strong> Loading a 7B model takes 2–5 seconds. For production APIs, keep the model loaded in memory across requests (Ollama does this by default). Set a keep-alive timeout appropriate for your request volume.</p><p><strong>Build a quality regression test suite before deploying.</strong> SLMs have different failure modes than large models — they’re more sensitive to prompt format, more likely to truncate responses, and more variable across quantization levels. Build a 50–100 example eval set and run it before every model or quantization change.</p><p><strong>Layer caching with prefix caching.</strong> For systems where many requests share a system prompt (customer service, document analysis), enable prefix caching in your serving layer. vLLM supports this natively — the shared prefix’s KV cache is computed once and reused, dramatically reducing latency for repeated system prompts.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Data quality beats data quantity</strong> — Phi-4’s STEM performance at 14B parameters validates that curated, structured training data is more efficient than raw internet scale</li><li><strong>SLMs punch above their weight on specialized tasks</strong> — Qwen 2.5-Coder 7B beats many 70B models on code; Phi-4 beats GPT-4o on math</li><li><strong>The SLM landscape in 2026</strong> covers every size from 0.5B (Qwen 2.5) to 27B (Gemma 3 27B) with genuine frontier-class capability at the 12–14B range</li><li><strong>Apple Intelligence is the most sophisticated production SLM deployment</strong> — on-device 3B model with ANE acceleration, cloud tier with cryptographic privacy attestation</li><li><strong>Q4_K_M is the production default</strong> for GGUF quantization — minimal quality loss, 37% of original FP16 size</li><li><strong>Four deployment formats</strong> for different targets: GGUF (CPU/Ollama), ONNX (cross-platform), Core ML (Apple), ExecuTorch (iOS/Android/edge)</li><li><strong>WebLLM runs LLMs in the browser</strong> via WebGPU with zero server cost — real-time, private, serverless AI</li><li><strong>Privacy regulation is driving SLM adoption</strong> — HIPAA, GDPR, and enterprise data policies make on-device and self-hosted SLMs mandatory for growing categories of applications</li><li><strong>Hybrid routing is the production pattern</strong> — small fast SLM handles routine tasks, cloud LLM handles genuinely complex ones</li><li><strong>Speculative decoding</strong> with a tiny draft model accelerates larger SLMs by 2–3× with zero quality loss</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://arxiv.org/abs/2306.11644">Phi-1: Textbooks Are All You Need (2023)</a> — The original paper that started the data quality revolution</li><li><a href="https://arxiv.org/abs/2412.08905">Phi-4 Technical Report (Microsoft, 2024)</a> — Full architecture and training methodology</li><li><a href="https://storage.googleapis.com/deepmind-media/gemma/gemma-3-report.pdf">Gemma 3 Technical Report (Google, 2025)</a> — Architecture innovations including ISWA</li><li><a href="https://llm.mlc.ai/">MLC LLM / WebLLM Documentation</a> — Running LLMs in the browser</li><li><a href="https://machinelearning.apple.com/research/apple-intelligence-foundation-language-models">Apple Intelligence Foundation Model Technical Report</a> — On-device AI architecture details</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1794d9107b13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 12: Mixture-of-Experts (MoE) Architecture]]></title>
            <link>https://learncsdesigns.medium.com/day-12-mixture-of-experts-moe-architecture-72faa97748d0?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/72faa97748d0</guid>
            <category><![CDATA[llm-deployment]]></category>
            <category><![CDATA[ai-architecture]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[moe]]></category>
            <category><![CDATA[mixture-of-experts]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 08:11:34 GMT</pubDate>
            <atom:updated>2026-07-08T08:11:34.064Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Why activate the entire brain for every thought? MoE asks the right specialists and leaves the rest sleeping — that’s how you scale to a trillion parameters without going bankrupt on inference.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tHxf3Bs2JYLEc4uQUWTxhg.png" /></figure><h2>Why This Matters</h2><p>Every frontier model you interact with in 2026 — GPT-4o, Grok-3, DeepSeek-V3, Mixtral — almost certainly uses Mixture-of-Experts internally. It’s the architectural innovation that allowed the industry to scale model capacity to trillions of parameters while keeping inference costs manageable.</p><p>Understanding MoE isn’t academic. It directly impacts:</p><ul><li><strong>Which models you choose</strong> for a given task (MoE models have different latency/throughput tradeoffs than dense models)</li><li><strong>How you fine-tune</strong> (MoE fine-tuning has unique pitfalls around expert balancing)</li><li><strong>Deployment architecture</strong> (MoE models need expert-aware tensor parallelism)</li><li><strong>Cost modeling</strong> (active parameter count, not total parameter count, determines inference cost)</li></ul><p>If you’ve ever wondered why GPT-4 can be simultaneously huge <em>and</em> fast — MoE is the answer.</p><h2>Part 1: The Problem With Dense Transformers at Scale</h2><h3>The Dense Transformer Recap</h3><p>A standard (dense) transformer processes every input token through <em>every</em> neuron in every layer. In a 70B-parameter model, every single forward pass activates all 70 billion parameters. The entire model participates in every prediction.</p><pre>Dense Transformer Forward Pass:<br>Token → Attention Layer (all params active)<br>      → FFN Layer (all params active)<br>      → Attention Layer (all params active)<br>      → FFN Layer (all params active)<br>      → ... (32 layers)<br>      → Output<br><br>Active parameters per token: 70,000,000,000 (100%)</pre><p>This is conceptually simple and architecturally clean. But it creates a brutal scaling law:</p><pre>Compute cost (FLOPs) = 2 × N × T<br><br>Where:<br>  N = number of model parameters<br>  T = number of tokens<br><br>Doubling model size → doubles inference cost, every request, forever.</pre><p>By the time you reach 500B parameters, you need tens of A100s per inference request. By 1 trillion, the economics collapse for consumer-facing applications.</p><h3>The Scaling Tension</h3><p>AI labs face a fundamental tension:</p><pre>More Parameters         → More knowledge, better reasoning, higher quality<br>More Active Parameters  → Higher compute cost, higher latency, more expensive<br><br>The question: Can you have more parameters WITHOUT activating them all?</pre><p>The answer is yes. That’s Mixture-of-Experts.</p><h2>Part 2: The MoE Layer — Core Architecture</h2><h3>Intuition: Specialists, Not Generalists</h3><p>Imagine a hospital. When you arrive with chest pain, you don’t see every doctor on staff. A triage nurse (the router) assesses your case and sends you to the cardiologist (the expert). The orthopedic surgeon, dermatologist, and neurologist stay idle. You get specialist-quality care while only paying for one consultation.</p><p>MoE applies this exact pattern to neural network layers:</p><pre>Dense FFN Layer:          One FFN that handles everything<br>                          All neurons participate in every prediction<br><br>MoE Layer:                N expert FFNs (e.g., 8 experts)<br>                          A gating network selects K experts per token<br>                          Only K/N of the experts activate per token</pre><h3>Architecture Diagram</h3><pre>                     Input Token Embedding<br>                                │<br>                    ┌───────────┴───────────┐<br>                    │                       │<br>              Attention Layer           (shared across all tokens)<br>                    │<br>                    ├─── MoE Layer ──────────────────────────────┐<br>                    │                                            │<br>                    │    ┌─────────────────────────────────────┐ │<br>                    │    │         Gating Network (Router)     │ │<br>                    │    │                                     │ │<br>                    │    │   Input → Linear → Softmax → Top-K  │ │<br>                    │    └──────────────┬──────────────────────┘ │<br>                    │                   │                        │<br>                    │    Routes to K    │  experts (e.g., K=2)   │<br>                    │                   │                        │<br>                    │    ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐         │<br>                    │    │Exp 1│ │Exp 2│ │Exp 3│ │Exp 4│  ...    │<br>                    │    │(FFN)│ │(FFN)│ │(FFN)│ │(FFN)│         │<br>                    │    └──┬──┘ └──┬──┘ └─────┘ └─────┘         │<br>                    │       │ Active │ Active   (idle)  (idle)   │<br>                    │       └────────┘                           │<br>                    │           │  Weighted Sum                  │<br>                    │           ▼                                │<br>                    └────── Output (same shape as input) ────────┘<br>                                │<br>                         Next Attention Layer</pre><h3>The MoE Layer in Mathematics</h3><p>Let’s formalize what the diagram shows:</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>from typing import Tuple<br><br>class MoELayer(nn.Module):<br>    &quot;&quot;&quot;<br>    A single Mixture-of-Experts FFN layer.<br>    <br>    Replaces the standard FFN in a transformer block.<br>    Each expert is a full Feed-Forward Network (FFN).<br>    <br>    Architecture matches Mixtral 8x7B design.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(<br>        self,<br>        d_model: int,           # Hidden dimension (e.g., 4096)<br>        d_ffn: int,             # Expert FFN intermediate dim (e.g., 14336)<br>        num_experts: int = 8,   # Total number of experts (N)<br>        top_k: int = 2,         # Experts activated per token (K)<br>    ):<br>        super().__init__()<br>        <br>        self.num_experts = num_experts<br>        self.top_k = top_k<br>        self.d_model = d_model<br>        <br>        # ── Gating Network ───────────────────────────────────────────────────<br>        # A simple linear layer: input (d_model) → expert scores (num_experts)<br>        self.gate = nn.Linear(d_model, num_experts, bias=False)<br>        <br>        # ── Expert Networks ──────────────────────────────────────────────────<br>        # Each expert is a standard SwiGLU FFN (as used in Llama/Mixtral)<br>        # We store them as parameter lists for efficient dispatch<br>        self.experts = nn.ModuleList([<br>            FeedForwardNetwork(d_model, d_ffn)<br>            for _ in range(num_experts)<br>        ])<br>    <br>    def forward(self, x: torch.Tensor) -&gt; Tuple[torch.Tensor, torch.Tensor]:<br>        &quot;&quot;&quot;<br>        Forward pass with Top-K expert selection.<br>        <br>        Args:<br>            x: shape (batch_size, seq_len, d_model)<br>        <br>        Returns:<br>            output: shape (batch_size, seq_len, d_model)<br>            router_logits: shape (batch_size * seq_len, num_experts)<br>                           — returned for auxiliary loss computation<br>        &quot;&quot;&quot;<br>        batch_size, seq_len, d_model = x.shape<br>        <br>        # Reshape to (batch*seq, d_model) — treat each token independently<br>        x_flat = x.view(-1, d_model)    # (B*T, d_model)<br>        num_tokens = x_flat.shape[0]<br>        <br>        # ── Step 1: Compute routing scores ────────────────────────────────────<br>        router_logits = self.gate(x_flat)     # (B*T, num_experts)<br>        routing_weights = F.softmax(router_logits, dim=-1)  # (B*T, num_experts)<br>        <br>        # ── Step 2: Select Top-K experts per token ────────────────────────────<br>        # top_k_weights: (B*T, K)  — the softmax scores for chosen experts<br>        # top_k_indices: (B*T, K)  — which expert indices were chosen<br>        top_k_weights, top_k_indices = torch.topk(<br>            routing_weights,<br>            k=self.top_k,<br>            dim=-1,<br>        )<br>        <br>        # Re-normalize weights so they sum to 1.0 for the selected experts<br>        top_k_weights = top_k_weights / top_k_weights.sum(dim=-1, keepdim=True)<br>        <br>        # ── Step 3: Dispatch tokens to selected experts ───────────────────────<br>        output = torch.zeros_like(x_flat)   # (B*T, d_model)<br>        <br>        for k in range(self.top_k):<br>            # For each &quot;slot&quot; k, gather tokens assigned to each expert<br>            expert_idx = top_k_indices[:, k]      # (B*T,) — which expert for slot k<br>            weights_k  = top_k_weights[:, k]      # (B*T,) — weight for slot k<br>            <br>            # Process each expert<br>            for expert_id in range(self.num_experts):<br>                # Find which tokens are routed to this expert in slot k<br>                token_mask = (expert_idx == expert_id)<br>                <br>                if not token_mask.any():<br>                    continue  # No tokens for this expert this step<br>                <br>                # Extract the tokens for this expert<br>                expert_input = x_flat[token_mask]  # (n_tokens, d_model)<br>                <br>                # Run the expert FFN<br>                expert_output = self.experts[expert_id](expert_input)  # (n_tokens, d_model)<br>                <br>                # Weight and accumulate into output<br>                output[token_mask] += weights_k[token_mask].unsqueeze(-1) * expert_output<br>        <br>        # Reshape back to (batch, seq, d_model)<br>        output = output.view(batch_size, seq_len, d_model)<br>        <br>        return output, router_logits<br><br><br>class FeedForwardNetwork(nn.Module):<br>    &quot;&quot;&quot;Standard SwiGLU FFN — the building block for each expert.&quot;&quot;&quot;<br>    <br>    def __init__(self, d_model: int, d_ffn: int):<br>        super().__init__()<br>        self.w1 = nn.Linear(d_model, d_ffn, bias=False)   # gate projection<br>        self.w2 = nn.Linear(d_ffn, d_model, bias=False)   # down projection<br>        self.w3 = nn.Linear(d_model, d_ffn, bias=False)   # up projection<br>    <br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        # SwiGLU activation: SiLU(gate) * up, then project down<br>        return self.w2(F.silu(self.w1(x)) * self.w3(x))</pre><h3>The Key Insight: Active vs Total Parameters</h3><pre># Mixtral 8x7B concrete example:<br>total_parameters = 46_700_000_000    # ~47B total parameters<br>experts_per_layer = 8<br>top_k = 2                            # Only 2 of 8 experts activate per token<br><br># Active parameters during inference:<br># Attention layers are always active (shared)<br>attention_params = 14_000_000_000    # ~14B for all attention layers<br># Expert FFN layers: only 2/8 activate<br>ffn_total_params = 32_000_000_000    # ~32B split across 8 experts<br>ffn_active_params = ffn_total_params * (top_k / experts_per_layer)  # 8B<br><br>active_params = attention_params + ffn_active_params  # ~22B<br><br>print(f&quot;Total parameters:  {total_parameters / 1e9:.1f}B&quot;)<br>print(f&quot;Active parameters: {active_params / 1e9:.1f}B per token&quot;)<br>print(f&quot;Utilization:       {active_params / total_parameters * 100:.1f}%&quot;)<br><br># Output:<br># Total parameters:  46.7B<br># Active parameters: 22.0B per token<br># Utilization:       47.1%<br>#<br># Implication: Mixtral 8x7B has the KNOWLEDGE of a ~47B model<br># but the INFERENCE COST of a ~22B model</pre><h2>Part 3: The Expert Collapse Problem and Load Balancing</h2><h3>What Is Expert Collapse?</h3><p>Here’s the critical failure mode that plagued early MoE research. During training, if the gating network learns to strongly prefer 1–2 experts over the others, those experts receive more gradient updates, become better, and get chosen even more. The rest of the experts atrophy from disuse.</p><pre>Early training step:<br>  Expert 1: chosen 35% of tokens → gets more training → improves<br>  Expert 2: chosen 30% of tokens → gets training → improves<br>  Expert 3: chosen 20% of tokens → less training → less improvement<br>  Expert 4: chosen 10% of tokens → minimal training → plateaus<br>  Expert 5-8: chosen 5% combined → almost no training → useless<br><br>Converged (collapsed) state:<br>  Expert 1: chosen 70% of tokens<br>  Expert 2: chosen 28% of tokens<br>  Experts 3-8: chosen 2% combined → effectively dead</pre><p><strong>The result</strong>: an 8-expert model that behaves like a 2-expert model. You spent the memory budget of 8 experts but only got the quality of 2.</p><h3>The Auxiliary Load Balancing Loss</h3><p>The solution: add an <strong>auxiliary loss</strong> that penalizes uneven expert utilization, forcing all experts to receive roughly equal token assignments:</p><pre>def compute_auxiliary_loss(<br>    router_logits: torch.Tensor,    # (num_tokens, num_experts)<br>    top_k_indices: torch.Tensor,    # (num_tokens, K)<br>    num_experts: int,<br>    auxiliary_loss_coeff: float = 0.01,<br>) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;<br>    Compute the load balancing auxiliary loss from Switch Transformer / Mixtral.<br>    <br>    This loss encourages uniform expert utilization across the batch.<br>    It is ADDED to the main language modeling loss during training.<br>    <br>    L_total = L_LM + alpha * L_auxiliary<br>    <br>    The coefficient alpha (~0.01) is small — just enough to encourage<br>    balance without distorting the main training objective.<br>    &quot;&quot;&quot;<br>    num_tokens = router_logits.shape[0]<br>    <br>    # Fraction of tokens dispatched to each expert (f_i)<br>    # Shape: (num_experts,)<br>    expert_counts = torch.zeros(num_experts, device=router_logits.device)<br>    for k in range(top_k_indices.shape[1]):<br>        expert_counts.scatter_add_(<br>            0,<br>            top_k_indices[:, k],<br>            torch.ones(num_tokens, device=router_logits.device),<br>        )<br>    <br>    fraction_tokens_per_expert = expert_counts / (num_tokens * top_k_indices.shape[1])<br>    <br>    # Mean router probability per expert (p_i) — the average softmax score<br>    # Shape: (num_experts,)<br>    routing_probs = torch.softmax(router_logits, dim=-1)  # (num_tokens, num_experts)<br>    mean_routing_prob = routing_probs.mean(dim=0)          # (num_experts,)<br>    <br>    # Auxiliary loss: dot product of f_i and p_i, scaled by num_experts<br>    # This is minimized when all experts get equal fractions AND equal probabilities<br>    auxiliary_loss = num_experts * torch.dot(fraction_tokens_per_expert, mean_routing_prob)<br>    <br>    return auxiliary_loss_coeff * auxiliary_loss<br><br><br># In training loop:<br>def training_step(model, batch, optimizer, aux_loss_coeff=0.01):<br>    outputs, router_logits = model(batch[&quot;input_ids&quot;])<br>    <br>    # Main language modeling loss<br>    lm_loss = F.cross_entropy(<br>        outputs.view(-1, outputs.shape[-1]),<br>        batch[&quot;labels&quot;].view(-1),<br>    )<br>    <br>    # Auxiliary load balancing loss<br>    aux_loss = compute_auxiliary_loss(router_logits, ...)<br>    <br>    # Combined loss<br>    total_loss = lm_loss + aux_loss<br>    <br>    total_loss.backward()<br>    optimizer.step()<br>    <br>    return {<br>        &quot;lm_loss&quot;: lm_loss.item(),<br>        &quot;aux_loss&quot;: aux_loss.item(),<br>        &quot;total_loss&quot;: total_loss.item(),<br>    }<br></pre><h3>Expert Specialization: What Do Experts Learn?</h3><p>Research into trained MoE models shows that experts develop measurable specializations, though not as cleanly as you might expect:</p><pre>Observed expert specializations (from interpretability research):<br>─────────────────────────────────────────────────────────────────<br>Expert type         Description<br>─────────────────────────────────────────────────────────────────<br>Syntactic experts   Predominantly activate on function words,<br>                    punctuation, and grammatical markers<br><br>Domain experts      Preferentially route tokens in specific domains:<br>                    code tokens, mathematical notation, foreign language<br><br>Positional experts  Show strong sensitivity to token position —<br>                    beginning of sentence vs. end of sentence<br><br>Semantic experts    Cluster around semantic categories: verbs,<br>                    named entities, numerical expressions<br><br>&quot;Generalist&quot; exps   ~20-30% of experts handle everything equally<br>                    (more distributed specialization)</pre><p>This specialization is <em>emergent</em> — it’s not explicitly trained, it develops because routing that respects these distinctions reduces the LM loss.</p><h2>Part 4: Real-World MoE Models in 2026</h2><h3>Mixtral 8×7B — The Open-Source Pioneer</h3><p>Mistral AI’s Mixtral was the breakthrough that made MoE mainstream in open-source AI:</p><pre>Architecture Specifications:<br>─────────────────────────────────────────────────────────────────<br>Total parameters:       46.7B<br>Active parameters:      12.9B per token (attention + 2 of 8 expert FFNs)<br>Number of experts:      8 per MoE layer<br>Top-K:                  2 (2 experts per token)<br>Number of layers:       32<br>Hidden dimension:       4,096<br>Expert FFN dimension:   14,336<br>Attention heads:        32 heads / 8 KV heads (GQA)<br>Context window:         32,768 tokens<br>Architecture base:      Llama-style with SwiGLU<br><br>Memory Requirements:<br>  Full precision (BF16): ~93 GB VRAM<br>  4-bit quantized (Q4):  ~26 GB VRAM  ← practical for 2× A100 40GB<br>  <br>Key design choices:<br>  ✓ Sliding window attention (SWA) in some layers for efficiency<br>  ✓ Grouped Query Attention (GQA) for faster KV cache<br>  ✓ Fully open weights (Apache 2.0 license)</pre><h3>Mixtral 8×22B</h3><pre>Total parameters:       140.6B<br>Active parameters:      ~39B per token<br>Number of experts:      8<br>Top-K:                  2<br>Context window:         65,536 tokens<br>Quality:                Competitive with GPT-4 on many benchmarks<br>Memory (Q4):            ~80 GB  ← 2× A100 80GB or 4× A100 40GB</pre><h3>GPT-4 (Rumored Architecture)</h3><p>OpenAI has never officially confirmed GPT-4’s architecture, but leaks and inference from published research suggest:</p><pre>Rumored GPT-4 Architecture (not confirmed):<br>─────────────────────────────────────────────────────────────────<br>Architecture:           MoE Transformer<br>Total parameters:       ~1.8 trillion (rumored)<br>Number of experts:      ~16 experts per MoE layer (rumored)<br>Top-K:                  ~2 experts per token (rumored)<br>Active parameters:      ~220 billion per token (rumored)<br>Training:               Multiple phases including RLHF<br>Context window:         128,000 tokens (confirmed)<br><br>What&#39;s confirmed:<br>  ✓ GPT-4 is significantly larger than GPT-3.5 (175B)<br>  ✓ Has dramatically different capability profile than dense models<br>  ✓ Uses a mixture of model sizes internally (GPT-4o likely uses MoE)</pre><h3>Grok-1 (xAI — Open Source)</h3><p>xAI released Grok-1 with full weights in March 2024 — one of the largest open-source MoE models:</p><pre>Grok-1 Architecture:<br>─────────────────────────────────────────────────────────────────<br>Total parameters:       314.2B  (largest open-source MoE in 2024)<br>Active parameters:      ~87B per token<br>Number of experts:      8 per MoE layer<br>Top-K:                  2<br>Number of layers:       64<br>Hidden dimension:       6,144<br>Context window:         8,192 tokens<br>License:                Apache 2.0<br><br>Memory:<br>  Full precision (BF16): ~628 GB VRAM  ← 8× A100 80GB minimum<br>  4-bit quantized:       ~157 GB VRAM  ← 2× A100 80GB</pre><h3>DeepSeek-V3 — The 2024 Efficiency Breakthrough</h3><p>DeepSeek-V3 introduced several MoE innovations that dramatically improved training efficiency:</p><pre>DeepSeek-V3 Architecture:<br>─────────────────────────────────────────────────────────────────<br>Total parameters:       671B<br>Active parameters:      37B per token  ← remarkable efficiency<br>Total experts:          256 (routed) + 1 shared (always active)<br>Top-K:                  8 of 256 routed experts<br>Shared expert:          1 expert always activated (captures common patterns)<br>Context window:         128,000 tokens<br><br>DeepSeek MoE Innovations:<br>  1. Fine-grained experts: 256 small experts vs 8 large experts<br>     → more specialized routing, less token-expert mismatch<br>  <br>  2. Shared expert: One expert always active<br>     → handles universal patterns, reduces load on routed experts<br>  <br>  3. Expert-level load balancing: Per-sequence balancing<br>     → reduces communication overhead in distributed training<br>  <br>  4. Multi-head Latent Attention (MLA): Compresses KV cache<br>     → enables 128k context without memory explosion<br>     <br>  5. FP8 mixed precision training: First at this scale<br>     → 2.788 million H800 GPU-hours for 671B model<br>     → ~$5.5M training cost vs ~$100M for comparable dense models</pre><pre># DeepSeek&#39;s fine-grained expert insight:<br># Instead of 8 large experts (d_ffn=14336 each)<br># Use 64 small experts (d_ffn=1792 each) + Top-K=6<br># Total active compute is similar, but routing is more fine-grained<br><br>class DeepSeekMoELayer(nn.Module):<br>    &quot;&quot;&quot;<br>    DeepSeek-V3 style MoE with shared expert + fine-grained routing.<br>    &quot;&quot;&quot;<br>    <br>    def __init__(<br>        self,<br>        d_model: int,<br>        d_ffn: int,<br>        num_routed_experts: int = 256,  # Many small experts<br>        num_shared_experts: int = 1,    # Always active<br>        top_k: int = 8,                 # Select 8 from 256<br>    ):<br>        super().__init__()<br>        <br>        self.top_k = top_k<br>        <br>        # Shared expert — always runs (captures universal patterns)<br>        self.shared_experts = nn.ModuleList([<br>            FeedForwardNetwork(d_model, d_ffn)<br>            for _ in range(num_shared_experts)<br>        ])<br>        <br>        # Routed experts — only K/N activate per token<br>        self.routed_experts = nn.ModuleList([<br>            FeedForwardNetwork(d_model, d_ffn // (num_routed_experts // 8))<br>            for _ in range(num_routed_experts)<br>        ])<br>        <br>        # Gating over routed experts only<br>        self.gate = nn.Linear(d_model, num_routed_experts, bias=False)<br>    <br>    def forward(self, x):<br>        # Always run shared expert(s)<br>        shared_output = sum(expert(x) for expert in self.shared_experts)<br>        <br>        # Selectively run top-K routed experts<br>        routed_output = self._route_and_compute(x)<br>        <br>        return shared_output + routed_output<br>    <br>    def _route_and_compute(self, x):<br>        # [Same Top-K routing as before, on routed_experts]<br>        ...</pre><h3>MoE Model Comparison Table (2026)</h3><pre>Model              Total Params  Active Params  Experts  K    Open?   Quality<br>────────────────────────────────────────────────────────────────────────────────<br>Mixtral 8×7B       46.7B         12.9B          8        2    ✓       GPT-3.5+<br>Mixtral 8×22B      140.6B        39.1B          8        2    ✓       GPT-4-<br>Grok-1             314.2B        87B            8        2    ✓       GPT-4-<br>DeepSeek-V3        671B          37B            256+1    8    ✓       GPT-4+<br>DeepSeek-V2        236B          21B            160+2    6    ✓       GPT-4-<br>GPT-4 (rumored)    ~1.8T         ~220B          ~16      ~2   ✗       GPT-4<br>Grok-3 (rumored)   ~2T           unknown        MoE      -    ✗       GPT-4+</pre><h2>Part 5: Running MoE Models — Practical Code</h2><h3>Via Ollama (Local)</h3><pre># Pull Mixtral 8x7B (26 GB quantized)<br>ollama pull mixtral:8x7b<br><br># Pull Mixtral 8x7B Instruct (fine-tuned for chat)<br>ollama pull mixtral:8x7b-instruct<br><br># Larger: Mixtral 8x22B (80 GB quantized) — needs 2x A100 80GB<br>ollama pull mixtral:8x22b</pre><pre>import httpx<br>import asyncio<br>import json<br><br>async def mixtral_local(<br>    prompt: str,<br>    model: str = &quot;mixtral:8x7b-instruct&quot;,<br>    temperature: float = 0.7,<br>    stream: bool = True,<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Run Mixtral 8x7B locally via Ollama.<br>    <br>    Performance on consumer hardware:<br>      RTX 4090 (24GB VRAM):   ~15-20 tokens/sec (Q4 quantized)<br>      M2 Max (96GB unified):  ~8-12 tokens/sec<br>      CPU only (64GB RAM):    ~1-2 tokens/sec (slow but works)<br>    &quot;&quot;&quot;<br>    <br>    if stream:<br>        full_response = &quot;&quot;<br>        async with httpx.AsyncClient(timeout=300.0) as client:<br>            async with client.stream(<br>                &quot;POST&quot;,<br>                &quot;http://localhost:11434/api/generate&quot;,<br>                json={<br>                    &quot;model&quot;: model,<br>                    &quot;prompt&quot;: prompt,<br>                    &quot;options&quot;: {<br>                        &quot;temperature&quot;: temperature,<br>                        &quot;top_p&quot;: 0.9,<br>                        &quot;top_k&quot;: 40,<br>                        &quot;repeat_penalty&quot;: 1.1,<br>                        &quot;num_predict&quot;: 1024,<br>                        &quot;num_ctx&quot;: 32768,  # Mixtral&#39;s full context window<br>                    },<br>                    &quot;stream&quot;: True,<br>                },<br>            ) as response:<br>                async for line in response.aiter_lines():<br>                    if line:<br>                        data = json.loads(line)<br>                        if not data.get(&quot;done&quot;, False):<br>                            token = data.get(&quot;response&quot;, &quot;&quot;)<br>                            full_response += token<br>                            print(token, end=&quot;&quot;, flush=True)<br>        print()<br>        return full_response<br>    <br>    else:<br>        async with httpx.AsyncClient(timeout=300.0) as client:<br>            response = await client.post(<br>                &quot;http://localhost:11434/api/generate&quot;,<br>                json={<br>                    &quot;model&quot;: model,<br>                    &quot;prompt&quot;: prompt,<br>                    &quot;stream&quot;: False,<br>                    &quot;options&quot;: {&quot;temperature&quot;: temperature, &quot;num_predict&quot;: 1024},<br>                },<br>            )<br>            return response.json()[&quot;response&quot;]<br><br><br># Chat format (instruct model)<br>async def mixtral_chat(messages: list[dict], model: str = &quot;mixtral:8x7b-instruct&quot;) -&gt; str:<br>    &quot;&quot;&quot;Chat completion format via Ollama&#39;s OpenAI-compatible endpoint.&quot;&quot;&quot;<br>    <br>    async with httpx.AsyncClient(timeout=300.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/v1/chat/completions&quot;,  # OpenAI-compatible API<br>            json={<br>                &quot;model&quot;: model,<br>                &quot;messages&quot;: messages,<br>                &quot;temperature&quot;: 0.7,<br>                &quot;max_tokens&quot;: 1024,<br>            },<br>        )<br>        return response.json()[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br><br># Usage — drop-in replacement for OpenAI client calls<br>result = asyncio.run(mixtral_chat([<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful AI assistant.&quot;},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Explain the difference between TCP and UDP in simple terms&quot;},<br>]))</pre><h3>Via Mistral API (Cloud)</h3><pre>from mistralai import Mistral<br>import asyncio<br>import os<br><br>client = Mistral(api_key=os.environ[&quot;MISTRAL_API_KEY&quot;])<br><br>async def mistral_api_complete(<br>    prompt: str,<br>    model: str = &quot;mistral-large-latest&quot;,  # or &quot;open-mixtral-8x7b&quot;, &quot;open-mixtral-8x22b&quot;<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Mistral AI API — managed hosting for Mixtral models.<br>    <br>    Available models:<br>      open-mixtral-8x7b      → Mixtral 8x7B, cheapest<br>      open-mixtral-8x22b     → Mixtral 8x22B, balanced<br>      mistral-large-latest   → Mistral&#39;s frontier model<br>      mistral-medium-latest  → Balanced capability/cost<br>      mistral-small-latest   → Fast, efficient<br>    &quot;&quot;&quot;<br>    <br>    response = await client.chat.complete_async(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}<br>        ],<br>        temperature=0.7,<br>        max_tokens=1024,<br>        safe_prompt=False,  # Set True to enable safety filtering<br>    )<br>    <br>    return {<br>        &quot;text&quot;: response.choices[0].message.content,<br>        &quot;model&quot;: response.model,<br>        &quot;input_tokens&quot;: response.usage.prompt_tokens,<br>        &quot;output_tokens&quot;: response.usage.completion_tokens,<br>    }<br><br><br># ── Mixtral with function calling ──────────────────────────────────────────────<br>async def mixtral_function_calling():<br>    &quot;&quot;&quot;Mixtral 8x22B supports tool/function calling.&quot;&quot;&quot;<br>    <br>    tools = [<br>        {<br>            &quot;type&quot;: &quot;function&quot;,<br>            &quot;function&quot;: {<br>                &quot;name&quot;: &quot;get_weather&quot;,<br>                &quot;description&quot;: &quot;Get current weather for a city&quot;,<br>                &quot;parameters&quot;: {<br>                    &quot;type&quot;: &quot;object&quot;,<br>                    &quot;properties&quot;: {<br>                        &quot;city&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;description&quot;: &quot;City name&quot;},<br>                        &quot;unit&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;enum&quot;: [&quot;celsius&quot;, &quot;fahrenheit&quot;]},<br>                    },<br>                    &quot;required&quot;: [&quot;city&quot;],<br>                },<br>            },<br>        }<br>    ]<br>    <br>    response = await client.chat.complete_async(<br>        model=&quot;open-mixtral-8x22b&quot;,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What&#39;s the weather in Tokyo?&quot;}],<br>        tools=tools,<br>        tool_choice=&quot;auto&quot;,<br>    )<br>    <br>    # Check if the model wants to call a function<br>    message = response.choices[0].message<br>    if message.tool_calls:<br>        tool_call = message.tool_calls[0]<br>        print(f&quot;Function to call: {tool_call.function.name}&quot;)<br>        print(f&quot;Arguments: {tool_call.function.arguments}&quot;)<br>    <br>    return response</pre><h2>Part 6: MoE Deployment Architecture</h2><h3>Memory Layout for Distributed Inference</h3><p>MoE models require special deployment consideration because experts can be distributed across devices:</p><pre>Single Device (≤26 GB available):<br>  Mixtral 8x7B Q4 (~26 GB) — barely fits on 1× RTX 4090<br>  All 8 experts reside on same GPU<br>  Expert switching: in-memory (fast)<br><br>Multi-Device Expert Parallelism:<br>  Strategy: Each GPU hosts a subset of experts<br>  <br>  Example: Mixtral 8x7B across 4× RTX 3090 (24GB each):<br>  ┌──────────────────────────────────────────────────┐<br>  │ GPU 0: Experts 1, 2  + Attention layers (shared) │<br>  │ GPU 1: Experts 3, 4  + Attention layers (shared) │<br>  │ GPU 2: Experts 5, 6  + Attention layers (shared) │<br>  │ GPU 3: Experts 7, 8  + Attention layers (shared) │<br>  └──────────────────────────────────────────────────┘<br>  <br>  Token routing requires All-to-All communication between GPUs<br>  → Expert parallelism adds communication overhead<br>  → Only beneficial if experts don&#39;t fit on a single device</pre><pre># Using vLLM for production Mixtral deployment<br># vLLM handles MoE-aware tensor parallelism automatically<br><br>&quot;&quot;&quot;<br>bash command to serve Mixtral via vLLM:<br><br>python -m vllm.entrypoints.openai.api_server \<br>    --model mistralai/Mixtral-8x7B-Instruct-v0.1 \<br>    --tensor-parallel-size 2 \    # Use 2 GPUs<br>    --dtype bfloat16 \<br>    --max-model-len 32768 \<br>    --gpu-memory-utilization 0.9 \<br>    --port 8000<br><br>For 4-bit quantized:<br>python -m vllm.entrypoints.openai.api_server \<br>    --model mistralai/Mixtral-8x7B-Instruct-v0.1 \<br>    --quantization awq \          # AWQ 4-bit quantization<br>    --tensor-parallel-size 1 \    # Single GPU with quantization<br>    --dtype float16<br>&quot;&quot;&quot;<br><br># Then use the OpenAI-compatible endpoint:<br>from openai import AsyncOpenAI<br><br>vllm_client = AsyncOpenAI(<br>    base_url=&quot;http://localhost:8000/v1&quot;,<br>    api_key=&quot;not-needed&quot;,<br>)<br><br>response = await vllm_client.chat.completions.create(<br>    model=&quot;mistralai/Mixtral-8x7B-Instruct-v0.1&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Hello!&quot;}],<br>    temperature=0.7,<br>)</pre><h3>MoE vs Dense: Throughput and Latency Profile</h3><pre>Metric              Dense 13B        MoE 47B (22B active)    Difference<br>────────────────────────────────────────────────────────────────────────<br>Quality             Moderate         High (GPT-3.5 tier)     MoE wins<br>Memory (BF16)       ~26 GB           ~93 GB                  Dense wins<br>Memory (Q4)         ~7 GB            ~26 GB                  Dense wins<br>Tokens/sec (A100)   ~80 tok/s        ~25 tok/s               Dense wins<br>First token latency  ~50ms           ~120ms                  Dense wins<br>Batch throughput    High             Moderate                Dense wins<br>Per-token cost      Low              Low (same as 22B)       Roughly equal<br>Reasoning quality   Moderate         High                    MoE wins<br><br>Summary:<br>  Use Dense when:  Latency-critical, memory-constrained, high-traffic<br>  Use MoE when:    Quality matters, batch workloads, asynchronous tasks</pre><h2>Part 7: MoE in the Context of Scaling Laws</h2><h3>Why MoE Changes the Compute-Quality Tradeoff</h3><p>The Chinchilla scaling laws (Hoffmann et al., 2022) describe how to optimally allocate training compute between model size and training tokens for <em>dense</em> models. MoE breaks these laws favorably:</p><pre>Dense scaling:<br>  2× compute → ~1.3-1.4× capability improvement<br><br>MoE scaling (same training FLOPs):<br>  Train a 4× larger model (4× params) with MoE<br>  Active compute per token is the same as a 1× dense model<br>  Capability improvement: ~1.8-2.0× <br>  <br>  Why? Parameter count and compute cost are decoupled.<br>  You can increase the &quot;knowledge storage&quot; (parameters)<br>  without proportionally increasing inference compute.</pre><h3>The Three Scaling Axes</h3><p>Modern AI has three axes along which you can scale:</p><pre>1. Training-time compute (dense scaling):<br>   More params × more tokens = bigger, smarter model<br>   Cost paid: once during training<br><br>2. Test-time compute (reasoning models, Day 14):<br>   More thinking tokens = smarter answers on hard problems<br>   Cost paid: every inference call<br><br>3. Architectural efficiency (MoE):<br>   More total params at same inference cost = better knowledge, quality<br>   Cost paid: memory (you need more VRAM to store all experts)</pre><p>Real frontier models exploit all three simultaneously.</p><h2>Part 8: Fine-Tuning MoE Models</h2><p>Fine-tuning MoE models has unique considerations compared to dense models:</p><pre># Fine-tuning Mixtral with LoRA — targeting specific expert layers<br>from peft import LoraConfig, get_peft_model, TaskType<br>from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig<br>import torch<br><br>def load_mixtral_for_finetuning(<br>    model_id: str = &quot;mistralai/Mixtral-8x7B-Instruct-v0.1&quot;,<br>):<br>    &quot;&quot;&quot;Load Mixtral 8x7B for LoRA fine-tuning in 4-bit precision.&quot;&quot;&quot;<br>    <br>    # 4-bit quantization config (QLoRA approach)<br>    bnb_config = BitsAndBytesConfig(<br>        load_in_4bit=True,<br>        bnb_4bit_use_double_quant=True,<br>        bnb_4bit_quant_type=&quot;nf4&quot;,<br>        bnb_4bit_compute_dtype=torch.bfloat16,<br>    )<br>    <br>    model = AutoModelForCausalLM.from_pretrained(<br>        model_id,<br>        quantization_config=bnb_config,<br>        device_map=&quot;auto&quot;,<br>        trust_remote_code=True,<br>    )<br>    <br>    # LoRA config — for MoE models, target BOTH expert layers and attention<br>    lora_config = LoraConfig(<br>        task_type=TaskType.CAUSAL_LM,<br>        r=16,                    # LoRA rank<br>        lora_alpha=32,<br>        lora_dropout=0.05,<br>        <br>        # For Mixtral, target both attention and MoE expert weights<br>        target_modules=[<br>            &quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;,  # Attention<br>            &quot;w1&quot;, &quot;w2&quot;, &quot;w3&quot;,                          # Expert FFN layers<br>            # Note: Targeting gate/router weights is generally NOT recommended<br>            # — it can disrupt the routing balance learned during pretraining<br>        ],<br>        <br>        bias=&quot;none&quot;,<br>    )<br>    <br>    model = get_peft_model(model, lora_config)<br>    model.print_trainable_parameters()<br>    # Output: trainable params: ~85M || all params: 46.7B || trainable%: 0.18%<br>    <br>    return model<br><br><br># ⚠️ Critical MoE Fine-tuning Warning:<br># <br># If you fine-tune only SOME experts (e.g., by freezing others), you may<br># disrupt the load balance the model learned during pretraining. The gating<br># network was trained to route tokens to all experts. If some experts are <br># frozen (outdated), the model may route to them inappropriately.<br>#<br># Best practice: Use LoRA on ALL expert layers simultaneously,<br># or freeze ALL experts and only fine-tune attention + gate.</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Confusing total parameters with active parameters</strong></p><p>When evaluating a model’s inference cost, <strong>active parameters</strong> (not total) determine FLOPs, latency, and GPU memory <em>during inference</em>. Mixtral 8×7B (47B total, 13B active) has similar inference speed to a 13B dense model, not a 47B dense model. Benchmark actual throughput — don’t estimate from parameter counts alone.</p><p><strong>2. Underestimating VRAM requirements</strong></p><p>You need VRAM to hold ALL experts simultaneously, even if only 2 are active per token. Mixtral 8×7B requires 26 GB in Q4 — significantly more than a 7B dense model (4 GB). Always calculate total model size, not active compute, for memory planning.</p><p><strong>3. Fine-tuning gate/router weights</strong></p><p>The routing network is carefully trained during pretraining to balance load and specialization. Fine-tuning it on a small dataset typically destroys the balance — you end up with expert collapse on your fine-tuned domain. Target FFN expert weights and attention, not the gating network.</p><p><strong>4. Ignoring expert utilization during inference monitoring</strong></p><p>In production, monitor which experts are being activated. If one expert is handling 60%+ of tokens (indicating collapse), your model quality has degraded from the baseline. Add expert utilization tracking to your observability pipeline.</p><p><strong>5. Batch size = 1 with MoE in production</strong></p><p>MoE models are particularly inefficient at batch size 1 because expert utilization is low (you might only need 1–2 experts per token). At higher batch sizes, multiple tokens are processed simultaneously, spreading load more evenly across experts and improving hardware utilization. Always batch requests in production.</p><h2>💼 Quick Questions</h2><p><strong>Q: What is the fundamental tradeoff that MoE architecture makes?</strong></p><p>MoE trades <strong>memory</strong> for <strong>compute efficiency</strong>. A MoE model stores more parameters (more expert FFNs) than a comparably-performing dense model, requiring more VRAM to hold all experts in memory. However, only a small fraction of experts activate per token, so the inference FLOPs are much lower than a dense model of the same total parameter count. The payoff: you get the <em>knowledge capacity</em> of a large model at the <em>inference cost</em> of a small model — provided you have enough memory to store all the experts.</p><p><strong>Q: What is expert collapse and how do modern MoE models prevent it?</strong></p><p>Expert collapse is when the gating network converges to routing the majority of tokens to just 1–2 experts, leaving the rest underutilized. This defeats the purpose of MoE — you pay the memory cost of N experts but only benefit from 2. Modern prevention techniques include: (1) Auxiliary load balancing loss that penalizes uneven expert utilization during training; (2) Expert capacity limits that hard-cap how many tokens can go to any single expert per batch; (3) Random expert initialization that breaks symmetry early in training; (4) Noise injection into routing logits during training to encourage exploration.</p><p><strong>Q: Why did DeepSeek-V3 use 256 small experts instead of 8 large experts?</strong></p><p>Fine-grained experts allow more precise routing — a token about quantum mechanics goes to a “physics subtopic” expert rather than a coarse “science” expert. The granularity improves expert specialization, reduces token-expert mismatch (where a token ends up in a suboptimal expert because there’s no better choice), and enables better load distribution across a larger expert pool. The shared expert (always active) handles universal linguistic patterns so the 256 routed experts can specialize more aggressively without degrading quality on common tokens.</p><h2>🏭 Production Considerations</h2><p><strong>Profile your workload before choosing MoE.</strong> If your use case is latency-critical (real-time chat with &lt;500ms response), a dense 13B model will outperform Mixtral 8×7B. If your use case is batch processing (nightly document analysis), MoE’s quality advantage at similar inference cost makes it the better choice.</p><p><strong>Use vLLM for production MoE serving.</strong> vLLM has native MoE-aware paged attention and expert parallelism. It handles token dispatch, expert routing, and KV cache management automatically — far more efficient than naïve HuggingFace inference for MoE models.</p><p><strong>Monitor expert utilization.</strong> Instrument your production model to track per-expert activation rates. An imbalance (&gt;2× deviation from uniform) indicates routing issues that degrade quality. This is especially important after fine-tuning.</p><p><strong>Expert parallelism adds communication overhead.</strong> When splitting experts across multiple GPUs, All-to-All communication between routing and expert execution adds latency. This overhead is worth it when a model doesn’t fit on a single GPU, but for models that fit on one device, single-GPU inference is faster than multi-GPU expert parallelism.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>MoE decouples parameter count from inference compute</strong> — more total parameters for the same FLOPs by only activating a fraction of experts per token</li><li><strong>The gating network</strong> is a small linear layer that routes each token to the Top-K experts by computing softmax scores over all experts</li><li><strong>Expert collapse</strong> is the training failure mode — auxiliary load balancing loss prevents it by penalizing uneven utilization</li><li><strong>Active parameters, not total parameters</strong>, determine inference FLOPs and speed — Mixtral 8×7B runs at ~13B dense model speeds</li><li><strong>Memory cost is the tradeoff</strong> — you need VRAM for ALL experts, even idle ones</li><li><strong>Mixtral 8×7B is the practical open-source MoE benchmark</strong> — runs on 2× A100 40GB in BF16, or single RTX 4090 in Q4</li><li><strong>DeepSeek-V3’s fine-grained 256-expert design</strong> is the 2024/25 state-of-the-art, achieving frontier quality at 37B active parameters</li><li><strong>Don’t fine-tune the gating network</strong> — it disrupts pretraining’s carefully learned load balance</li><li><strong>MoE scales better than dense</strong> along the parameter axis — the capacity scaling laws are more favorable</li><li><strong>All three scaling axes</strong> (training compute, test-time compute, architectural efficiency) are used simultaneously in frontier 2026 models</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://arxiv.org/abs/2401.04088">Mixtral of Experts Paper (Mistral AI, 2024)</a> — The original Mixtral paper with full architecture details</li><li><a href="https://arxiv.org/abs/2101.03961">Switch Transformer (Google, 2021)</a> — The paper that made MoE practical at scale</li><li><a href="https://arxiv.org/abs/2412.19437">DeepSeek-V3 Technical Report (2024)</a> — Fine-grained expert design and training innovations</li><li><a href="https://arxiv.org/abs/1701.06538">Outrageously Large Neural Networks (Shazeer et al., 2017)</a> — The original modern MoE paper</li><li><a href="https://arxiv.org/abs/2308.00951">From Sparse to Soft Mixtures of Experts (2023)</a> — Google’s Soft MoE variant that removes token dropping</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72faa97748d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 11: AI Reasoning Models: o3, Claude Sonnet Thinking, Gemini Thinking]]></title>
            <link>https://learncsdesigns.medium.com/day-11-ai-reasoning-models-o3-claude-sonnet-thinking-gemini-thinking-34f0f84f83d4?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f0f84f83d4</guid>
            <category><![CDATA[reasoning-model]]></category>
            <category><![CDATA[test-time-compute]]></category>
            <category><![CDATA[chain-of-thought]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[extended-thinking]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 05:58:11 GMT</pubDate>
            <atom:updated>2026-07-08T05:58:11.951Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Standard models answer. Reasoning models think before they answer. The difference isn’t vocabulary — it’s the willingness to be wrong mid-thought.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0B_zzzmGOkWXjRNsPnQpQ.png" /></figure><h2>Why This Matters</h2><p>From 2020 to 2023, AI progress was dominated by <strong>training-time scaling</strong>: bigger models, more data, more compute during training. GPT-3 → GPT-4 → Claude 3 followed this path. Results were impressive, but hitting a wall of diminishing returns.</p><p>In late 2024, OpenAI released o1 and changed the paradigm. The breakthrough insight: <strong>you can trade inference compute for reasoning quality.</strong> Give a model more time to “think” before answering — with internal chain-of-thought — and you unlock capabilities that scale with that thinking budget, independent of model size.</p><p>By 2026, every frontier lab has reasoning models:</p><ul><li>OpenAI: o3, o4-mini (successor to o1/o1-mini)</li><li>Anthropic: Extended Thinking in Claude Opus 4 and Sonnet 4</li><li>Google: Gemini 2.5 Pro/Flash with thinking mode</li><li>Open-source: DeepSeek-R1, QwQ-32B, Llama reasoning variants</li></ul><p>The ability to choose correctly between standard and reasoning models — and configure them well — is now a core AI engineering skill. Use o3 when you need it and you build a superpower. Use it when you don’t and you burn money on unnecessary latency.</p><h2>Part 1: What Makes a Reasoning Model Different</h2><h3>The Core Architecture: Internal Chain-of-Thought</h3><p>Standard models (GPT-4o, Claude Sonnet, Gemini Flash) process your prompt and generate a response in a single autoregressive pass. They may produce good outputs, but there’s no structured deliberation — the “thinking” and the “answer” are the same forward pass.</p><p>Reasoning models introduce a <strong>private scratchpad</strong> — an internal chain-of-thought that runs before the final response:</p><pre>Standard model:<br>  Input → [single forward pass] → Output<br><br>Reasoning model:<br>  Input → [think: &quot;Let me approach this...&quot;]<br>        → [think: &quot;Step 1: identify constraints...&quot;]<br>        → [think: &quot;Wait, that approach has a flaw...&quot;]<br>        → [think: &quot;Alternative: try from the other direction...&quot;]<br>        → [think: &quot;Yes, this works. Final answer:&quot;]<br>        → Output</pre><p>The internal thinking tokens are generated by the model for itself, not the user. They are typically not shown in the final output (though Anthropic’s extended thinking optionally surfaces them).</p><h3>Test-Time Compute Scaling</h3><p>The pivotal idea behind reasoning models is <strong>test-time compute scaling</strong>: performance improves as you give the model more compute during inference (not just during training).</p><p>The key papers established this empirically:</p><pre>Finding: For hard math/coding/reasoning problems, accuracy follows a<br>power-law relationship with the number of &quot;thinking tokens&quot; used.<br><br>More thinking tokens → Higher accuracy<br>(up to a point — there are diminishing returns)</pre><p>This is fundamentally different from standard scaling:</p><pre>Training-time scaling:     10× more training compute → ~modest improvement<br>Test-time scaling:         10× more thinking tokens  → dramatic improvement on hard problems</pre><p>The implication: for a hard reasoning task, a smaller model with a large thinking budget can outperform a much larger model without thinking.</p><h3>Two Implementation Approaches</h3><p><strong>OpenAI’s approach (o-series):</strong> The thinking is fully hidden. You never see the internal chain-of-thought. You control thinking via reasoning_effort (low/medium/high) which adjusts the thinking budget. The model decides internally how to use those thinking tokens.</p><p><strong>Anthropic’s approach (Extended Thinking):</strong> The internal thinking blocks are optionally returned to you. You can read the model’s reasoning process — useful for debugging, auditing, and trust-building. You control thinking via a budget_tokens parameter.</p><p><strong>Google’s approach (Gemini Thinking):</strong> Similar to Anthropic — thinking is returned as a separate part of the response. Controlled via thinking_config with a thinking_budget in tokens.</p><h2>Part 2: OpenAI o3 and o4-mini</h2><h3>Model Overview (2026)</h3><pre>Model       Context    Thinking      Input $/M    Output $/M   Best for<br>────────────────────────────────────────────────────────────────────────────<br>o3          200k       Hidden        $10.00       $40.00       Frontier reasoning,<br>                                                               research, hard math<br>o4-mini     200k       Hidden        $1.10        $4.40        Cost-efficient reasoning,<br>                                                               coding, everyday logic<br>o1          200k       Hidden        $15.00       $60.00       Legacy — prefer o3<br>o1-mini     128k       Hidden        $3.00        $12.00       Legacy — prefer o4-mini</pre><h3>o3 API Usage</h3><pre>from openai import AsyncOpenAI<br>import asyncio<br>import time<br><br>client = AsyncOpenAI()<br><br>async def reason_with_o3(<br>    problem: str,<br>    effort: str = &quot;medium&quot;,  # &quot;low&quot; | &quot;medium&quot; | &quot;high&quot;<br>    system_prompt: str | None = None,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Use o3 for complex reasoning tasks.<br>    <br>    effort=&quot;low&quot;    → faster, cheaper, fewer thinking tokens<br>    effort=&quot;medium&quot; → balanced (default recommendation)<br>    effort=&quot;high&quot;   → maximum thinking, slowest, most expensive<br>    &quot;&quot;&quot;<br>    <br>    messages = []<br>    if system_prompt:<br>        messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt})<br>    messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem})<br>    <br>    start_time = time.time()<br>    <br>    response = await client.chat.completions.create(<br>        model=&quot;o3&quot;,<br>        messages=messages,<br>        reasoning_effort=effort,     # Key parameter for o-series<br>        max_completion_tokens=8_000, # Includes thinking + output tokens<br>        # Note: temperature is NOT supported on o-series (always 1)<br>        # Note: top_p is NOT supported on o-series<br>    )<br>    <br>    latency_ms = (time.time() - start_time) * 1000<br>    <br>    usage = response.usage<br>    <br>    return {<br>        &quot;answer&quot;: response.choices[0].message.content,<br>        &quot;model&quot;: &quot;o3&quot;,<br>        &quot;input_tokens&quot;: usage.prompt_tokens,<br>        &quot;output_tokens&quot;: usage.completion_tokens,<br>        # reasoning_tokens shows how much was spent on thinking<br>        &quot;reasoning_tokens&quot;: getattr(usage.completion_tokens_details, &quot;reasoning_tokens&quot;, 0),<br>        &quot;cost_usd&quot;: (usage.prompt_tokens / 1_000_000 * 10.00) +<br>                    (usage.completion_tokens / 1_000_000 * 40.00),<br>        &quot;latency_ms&quot;: latency_ms,<br>    }<br><br><br>async def reason_with_o4_mini(<br>    problem: str,<br>    effort: str = &quot;medium&quot;,<br>) -&gt; dict:<br>    &quot;&quot;&quot;o4-mini: Same API pattern, 9× cheaper than o3.&quot;&quot;&quot;<br>    <br>    start_time = time.time()<br>    <br>    response = await client.chat.completions.create(<br>        model=&quot;o4-mini&quot;,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem}],<br>        reasoning_effort=effort,<br>        max_completion_tokens=8_000,<br>    )<br>    <br>    latency_ms = (time.time() - start_time) * 1000<br>    usage = response.usage<br>    <br>    return {<br>        &quot;answer&quot;: response.choices[0].message.content,<br>        &quot;model&quot;: &quot;o4-mini&quot;,<br>        &quot;input_tokens&quot;: usage.prompt_tokens,<br>        &quot;output_tokens&quot;: usage.completion_tokens,<br>        &quot;reasoning_tokens&quot;: getattr(usage.completion_tokens_details, &quot;reasoning_tokens&quot;, 0),<br>        &quot;cost_usd&quot;: (usage.prompt_tokens / 1_000_000 * 1.10) +<br>                    (usage.completion_tokens / 1_000_000 * 4.40),<br>        &quot;latency_ms&quot;: latency_ms,<br>    }<br><br><br># ── Example usage ────────────────────────────────────────────────────────────<br>async def demo_o3():<br>    # Hard math problem<br>    result = await reason_with_o3(<br>        problem=&quot;&quot;&quot;<br>        A train leaves Station A at 9:00 AM traveling at 80 mph toward Station B.<br>        Another train leaves Station B at 9:30 AM traveling at 100 mph toward Station A.<br>        The stations are 350 miles apart. <br>        <br>        At what time do the trains meet? At what distance from Station A?<br>        Show your full reasoning.<br>        &quot;&quot;&quot;,<br>        effort=&quot;medium&quot;,<br>    )<br>    <br>    print(f&quot;Answer: {result[&#39;answer&#39;]}&quot;)<br>    print(f&quot;Reasoning tokens used: {result[&#39;reasoning_tokens&#39;]:,}&quot;)<br>    print(f&quot;Total cost: ${result[&#39;cost_usd&#39;]:.4f}&quot;)<br>    print(f&quot;Latency: {result[&#39;latency_ms&#39;]:.0f}ms&quot;)<br><br>asyncio.run(demo_o3())</pre><h3>Reasoning Effort vs Performance vs Cost</h3><pre>import asyncio<br>from openai import AsyncOpenAI<br><br>client = AsyncOpenAI()<br><br>async def benchmark_reasoning_effort(problem: str):<br>    &quot;&quot;&quot;Compare all three effort levels on the same problem.&quot;&quot;&quot;<br>    <br>    efforts = [&quot;low&quot;, &quot;medium&quot;, &quot;high&quot;]<br>    results = []<br>    <br>    for effort in efforts:<br>        response = await client.chat.completions.create(<br>            model=&quot;o3&quot;,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem}],<br>            reasoning_effort=effort,<br>            max_completion_tokens=8_000,<br>        )<br>        <br>        usage = response.usage<br>        reasoning_tokens = getattr(usage.completion_tokens_details, &quot;reasoning_tokens&quot;, 0)<br>        cost = (usage.prompt_tokens * 10 + usage.completion_tokens * 40) / 1_000_000<br>        <br>        results.append({<br>            &quot;effort&quot;: effort,<br>            &quot;reasoning_tokens&quot;: reasoning_tokens,<br>            &quot;output_tokens&quot;: usage.completion_tokens - reasoning_tokens,<br>            &quot;cost_usd&quot;: cost,<br>            &quot;answer_preview&quot;: response.choices[0].message.content[:100],<br>        })<br>    <br>    # Print comparison table<br>    print(f&quot;{&#39;Effort&#39;:&lt;8} {&#39;Reasoning Tok&#39;:&lt;16} {&#39;Output Tok&#39;:&lt;12} {&#39;Cost&#39;:&gt;8}&quot;)<br>    print(&quot;─&quot; * 50)<br>    for r in results:<br>        print(<br>            f&quot;{r[&#39;effort&#39;]:&lt;8} {r[&#39;reasoning_tokens&#39;]:&lt;16,} &quot;<br>            f&quot;{r[&#39;output_tokens&#39;]:&lt;12,} ${r[&#39;cost_usd&#39;]:&gt;7.4f}&quot;<br>        )<br>    <br>    return results<br><br># Typical output on a hard coding problem:<br># Effort   Reasoning Tok    Output Tok   Cost<br># ──────────────────────────────────────────────<br># low      1,247            312          $0.0661<br># medium   4,891            425          $0.2160<br># high     18,334           512          $0.7516</pre><h3>Important o-Series Constraints</h3><pre># ⚠️ Parameters NOT supported on o-series models (will raise errors):<br># - temperature (always 1.0 internally)<br># - top_p<br># - presence_penalty<br># - frequency_penalty<br># - logprobs<br><br># ✅ Parameters that ARE supported:<br># - reasoning_effort (&quot;low&quot; | &quot;medium&quot; | &quot;high&quot;)<br># - max_completion_tokens (includes both reasoning + output tokens)<br># - tools / function calling<br># - response_format (including JSON mode)<br># - system messages (supported in o3/o4-mini, not in original o1)<br><br># ✅ o3 supports vision (images in messages)<br># ✅ o3 supports function/tool calling with reasoning<br># ✅ o3 supports structured output (JSON mode)</pre><h2>Part 3: Anthropic Claude Extended Thinking</h2><h3>How Extended Thinking Works</h3><p>Anthropic’s implementation makes thinking <em>visible</em>. The API returns thinking blocks as separate content items, letting you inspect the model’s reasoning chain:</p><pre>import anthropic<br>import asyncio<br>import time<br><br>client = anthropic.AsyncAnthropic()<br><br>async def claude_extended_thinking(<br>    problem: str,<br>    budget_tokens: int = 8_000,     # Max tokens for internal thinking<br>    model: str = &quot;claude-sonnet-4-20250514&quot;,<br>    show_thinking: bool = True,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Use Claude with extended thinking enabled.<br>    <br>    budget_tokens: How many tokens to allocate for internal reasoning<br>                  Minimum: 1,024<br>                  Maximum: 32,000 (Sonnet 4), varies by model<br>                  <br>    The thinking content is returned as separate blocks you can inspect.<br>    &quot;&quot;&quot;<br>    <br>    start_time = time.time()<br>    <br>    response = await client.messages.create(<br>        model=model,<br>        max_tokens=budget_tokens + 2_000,  # budget + room for output<br>        thinking={<br>            &quot;type&quot;: &quot;enabled&quot;,<br>            &quot;budget_tokens&quot;: budget_tokens,<br>        },<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem}<br>        ],<br>        # Note: temperature must be 1.0 when extended thinking is enabled<br>        # (Anthropic enforces this)<br>    )<br>    <br>    latency_ms = (time.time() - start_time) * 1000<br>    <br>    # Parse the response — thinking and text are separate content blocks<br>    thinking_content = []<br>    text_content = []<br>    <br>    for block in response.content:<br>        if block.type == &quot;thinking&quot;:<br>            thinking_content.append(block.thinking)<br>        elif block.type == &quot;text&quot;:<br>            text_content.append(block.text)<br>    <br>    final_answer = &quot;\n&quot;.join(text_content)<br>    full_thinking = &quot;\n\n&quot;.join(thinking_content)<br>    <br>    # Cost calculation<br>    cost = (<br>        response.usage.input_tokens / 1_000_000 * 3.00 +     # Sonnet input<br>        response.usage.output_tokens / 1_000_000 * 15.00     # Sonnet output<br>    )<br>    <br>    if show_thinking:<br>        print(&quot;─&quot; * 50)<br>        print(&quot;🧠 INTERNAL THINKING:&quot;)<br>        print(&quot;─&quot; * 50)<br>        print(full_thinking[:1000] + (&quot;...&quot; if len(full_thinking) &gt; 1000 else &quot;&quot;))<br>        print(&quot;─&quot; * 50)<br>    <br>    return {<br>        &quot;answer&quot;: final_answer,<br>        &quot;thinking&quot;: full_thinking,<br>        &quot;thinking_length&quot;: len(full_thinking),<br>        &quot;model&quot;: model,<br>        &quot;input_tokens&quot;: response.usage.input_tokens,<br>        &quot;output_tokens&quot;: response.usage.output_tokens,<br>        &quot;cost_usd&quot;: cost,<br>        &quot;latency_ms&quot;: latency_ms,<br>    }<br><br><br>async def demo_extended_thinking():<br>    <br>    # Complex logical reasoning problem<br>    result = await claude_extended_thinking(<br>        problem=&quot;&quot;&quot;<br>        There are 5 houses in a row, each painted a different color.<br>        Each house is occupied by a person of different nationality.<br>        Each person drinks a different beverage, smokes a different brand, and owns a different pet.<br>        <br>        Clues:<br>        1. The Brit lives in the red house.<br>        2. The Swede keeps dogs.<br>        3. The Dane drinks tea.<br>        4. The green house is on the left of the white house.<br>        5. The green house&#39;s owner drinks coffee.<br>        6. The person who smokes Pall Mall rears birds.<br>        7. The owner of the yellow house smokes Dunhill.<br>        8. The man living in the center house drinks milk.<br>        9. The Norwegian lives in the first house.<br>        10. The man who smokes Blends lives next to the one who keeps cats.<br>        11. The man who keeps horses lives next to the man who smokes Dunhill.<br>        12. The owner who smokes BlueMaster drinks beer.<br>        13. The German smokes Prince.<br>        14. The Norwegian lives next to the blue house.<br>        15. The man who smokes Blends has a neighbor who drinks water.<br>        <br>        Who owns the fish?<br>        &quot;&quot;&quot;,<br>        budget_tokens=10_000,<br>        show_thinking=True,<br>    )<br>    <br>    print(f&quot;\n✅ FINAL ANSWER:\n{result[&#39;answer&#39;]}&quot;)<br>    print(f&quot;\nThinking length: {result[&#39;thinking_length&#39;]:,} characters&quot;)<br>    print(f&quot;Cost: ${result[&#39;cost_usd&#39;]:.4f}&quot;)<br>    print(f&quot;Latency: {result[&#39;latency_ms&#39;]:.0f}ms&quot;)<br><br>asyncio.run(demo_extended_thinking())</pre><h3>Streaming Extended Thinking</h3><p>For production applications where latency matters, stream extended thinking to show progress:</p><pre>import anthropic<br>import asyncio<br><br>client = anthropic.AsyncAnthropic()<br><br>async def stream_extended_thinking(problem: str, budget_tokens: int = 6_000):<br>    &quot;&quot;&quot;<br>    Stream extended thinking — shows progress in real-time.<br>    Especially useful for long-running reasoning tasks.<br>    &quot;&quot;&quot;<br>    <br>    print(&quot;🧠 Thinking&quot;, end=&quot;&quot;, flush=True)<br>    <br>    thinking_buffer = []<br>    answer_buffer = []<br>    current_block_type = None<br>    <br>    async with client.messages.stream(<br>        model=&quot;claude-sonnet-4-20250514&quot;,<br>        max_tokens=budget_tokens + 2_000,<br>        thinking={<br>            &quot;type&quot;: &quot;enabled&quot;,<br>            &quot;budget_tokens&quot;: budget_tokens,<br>        },<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: problem}],<br>    ) as stream:<br>        <br>        async for event in stream:<br>            <br>            # Track block type transitions<br>            if hasattr(event, &quot;type&quot;):<br>                <br>                if event.type == &quot;content_block_start&quot;:<br>                    block = event.content_block<br>                    if block.type == &quot;thinking&quot;:<br>                        current_block_type = &quot;thinking&quot;<br>                        print(&quot;\n[Thinking started...]&quot;, flush=True)<br>                    elif block.type == &quot;text&quot;:<br>                        current_block_type = &quot;text&quot;<br>                        print(&quot;\n[Generating answer...]\n&quot;, flush=True)<br>                <br>                elif event.type == &quot;content_block_delta&quot;:<br>                    delta = event.delta<br>                    <br>                    if current_block_type == &quot;thinking&quot; and hasattr(delta, &quot;thinking&quot;):<br>                        thinking_buffer.append(delta.thinking)<br>                        # Show a progress indicator during thinking<br>                        if len(&quot;&quot;.join(thinking_buffer)) % 500 == 0:<br>                            print(&quot;.&quot;, end=&quot;&quot;, flush=True)<br>                    <br>                    elif current_block_type == &quot;text&quot; and hasattr(delta, &quot;text&quot;):<br>                        answer_buffer.append(delta.text)<br>                        print(delta.text, end=&quot;&quot;, flush=True)<br>    <br>    print()  # Final newline<br>    <br>    return {<br>        &quot;thinking&quot;: &quot;&quot;.join(thinking_buffer),<br>        &quot;answer&quot;: &quot;&quot;.join(answer_buffer),<br>    }</pre><h3>Thinking Budget Tuning Guide</h3><pre># Budget recommendations by task complexity<br><br>THINKING_BUDGETS = {<br>    <br>    # Simple multi-step problems<br>    # (multi-digit arithmetic, basic logic, simple code bugs)<br>    &quot;simple&quot;: 1_024,         # Minimum — quick verification<br>    <br>    # Medium complexity<br>    # (algorithm design, moderate math proofs, debugging complex code)<br>    &quot;medium&quot;: 5_000,<br>    <br>    # Hard problems<br>    # (competition math, complex system design, multi-constraint optimization)<br>    &quot;hard&quot;: 10_000,<br>    <br>    # Research-grade<br>    # (novel proofs, complex security analysis, advanced reasoning chains)<br>    &quot;research&quot;: 20_000,<br>    <br>    # Maximum (Sonnet 4 limit)<br>    &quot;maximum&quot;: 32_000,<br>}<br><br># Rule of thumb:<br># budget_tokens ≈ 2-3× the length of your expected reasoning chain<br># For most production tasks: 5,000-8,000 is the sweet spot</pre><h2>Part 4: Google Gemini Thinking Mode</h2><h3>Gemini 2.5 Pro and Flash with Thinking</h3><pre>import google.generativeai as genai<br>import os<br>import time<br><br>genai.configure(api_key=os.environ[&quot;GEMINI_API_KEY&quot;])<br><br>def gemini_thinking(<br>    problem: str,<br>    model_name: str = &quot;gemini-2.5-pro&quot;,<br>    thinking_budget: int = 8_192,   # Max tokens for thinking<br>    show_thinking: bool = True,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Use Gemini with thinking mode enabled.<br>    <br>    gemini-2.5-pro:   Higher quality, more expensive<br>    gemini-2.5-flash: Faster, cheaper, good for most reasoning tasks<br>    <br>    thinking_budget: 0 disables thinking, -1 = dynamic (model decides)<br>    &quot;&quot;&quot;<br>    <br>    model = genai.GenerativeModel(<br>        model_name=model_name,<br>        generation_config=genai.GenerationConfig(<br>            max_output_tokens=thinking_budget + 2_048,<br>        ),<br>    )<br>    <br>    # Enable thinking via generation config<br>    # Note: Gemini API for thinking is through GenerationConfig<br>    thinking_config = genai.types.ThinkingConfig(<br>        thinking_budget=thinking_budget,<br>        # include_thoughts=True returns the thinking in response<br>    )<br>    <br>    start_time = time.time()<br>    <br>    response = model.generate_content(<br>        problem,<br>        generation_config=genai.GenerationConfig(<br>            thinking_config=thinking_config,<br>        ),<br>    )<br>    <br>    latency_ms = (time.time() - start_time) * 1000<br>    <br>    # Parse response parts — Gemini returns thought parts separately<br>    thinking_text = &quot;&quot;<br>    answer_text = &quot;&quot;<br>    <br>    for part in response.candidates[0].content.parts:<br>        if hasattr(part, &quot;thought&quot;) and part.thought:<br>            thinking_text += part.text<br>        else:<br>            answer_text += part.text<br>    <br>    if show_thinking and thinking_text:<br>        print(&quot;─&quot; * 50)<br>        print(&quot;🧠 GEMINI THINKING:&quot;)<br>        print(&quot;─&quot; * 50)<br>        print(thinking_text[:800] + &quot;...&quot; if len(thinking_text) &gt; 800 else thinking_text)<br>        print(&quot;─&quot; * 50)<br>    <br>    # Usage metadata<br>    usage = response.usage_metadata<br>    input_tokens = usage.prompt_token_count<br>    output_tokens = usage.candidates_token_count<br>    thoughts_tokens = usage.thoughts_token_count if hasattr(usage, &quot;thoughts_token_count&quot;) else 0<br>    <br>    # Cost (Gemini 2.5 Pro, ≤200k context)<br>    cost = (input_tokens / 1_000_000 * 1.25) + (output_tokens / 1_000_000 * 10.00)<br>    <br>    return {<br>        &quot;answer&quot;: answer_text,<br>        &quot;thinking&quot;: thinking_text,<br>        &quot;input_tokens&quot;: input_tokens,<br>        &quot;output_tokens&quot;: output_tokens,<br>        &quot;thoughts_tokens&quot;: thoughts_tokens,<br>        &quot;cost_usd&quot;: cost,<br>        &quot;latency_ms&quot;: latency_ms,<br>    }<br><br><br># Gemini Flash thinking — great cost/performance ratio<br>def gemini_flash_thinking(problem: str, thinking_budget: int = 5_000) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Gemini 2.5 Flash with thinking — the most cost-efficient reasoning model.<br>    <br>    Pricing: $0.075/M input, $0.30/M output<br>    Use for: everyday reasoning tasks that don&#39;t need o3-level compute<br>    &quot;&quot;&quot;<br>    return gemini_thinking(<br>        problem=problem,<br>        model_name=&quot;gemini-2.5-flash&quot;,<br>        thinking_budget=thinking_budget,<br>    )</pre><h2>Part 5: Reasoning vs Standard — The Decision Framework</h2><h3>Performance Comparison by Task Category</h3><pre>Task Category            Standard Model    Reasoning Model    Reasoning Advantage<br>──────────────────────────────────────────────────────────────────────────────────<br>Creative writing         ★★★★★             ★★★★☆              None — often worse<br>Simple Q&amp;A               ★★★★★             ★★★★☆              None — wastes budget<br>Summarization            ★★★★★             ★★★★☆              None<br>Translation              ★★★★★             ★★★★☆              None<br>Multi-step math          ★★☆☆☆             ★★★★★              Dramatic (+30-50%)<br>Logic puzzles            ★★★☆☆             ★★★★★              Large (+25-40%)<br>Competition coding       ★★★☆☆             ★★★★★              Large (+20-35%)<br>Debugging complex code   ★★★☆☆             ★★★★★              Significant<br>Security analysis        ★★★☆☆             ★★★★★              Significant<br>Scientific reasoning     ★★★☆☆             ★★★★★              Large<br>Planning &amp; scheduling    ★★★★☆             ★★★★★              Moderate (+10-20%)<br>RAG Q&amp;A (factual)        ★★★★★             ★★★★☆              None — overkill<br>JSON extraction          ★★★★★             ★★★★☆              None — overkill</pre><h3>The Cost-Benefit Calculator</h3><pre>from dataclasses import dataclass<br>from enum import Enum<br><br>class TaskComplexity(Enum):<br>    TRIVIAL = &quot;trivial&quot;       # Q&amp;A, classification, translation<br>    MODERATE = &quot;moderate&quot;     # Multi-step, analysis, planning<br>    HARD = &quot;hard&quot;             # Math proofs, complex debugging, multi-constraint<br>    RESEARCH = &quot;research&quot;     # Novel problems, frontier challenges<br><br>@dataclass<br>class ModelOption:<br>    name: str<br>    input_cost_per_million: float<br>    output_cost_per_million: float<br>    avg_latency_seconds: float<br>    accuracy_by_complexity: dict  # complexity → relative accuracy (0-1)<br><br>MODEL_OPTIONS = {<br>    &quot;gpt-4o&quot;: ModelOption(<br>        name=&quot;gpt-4o&quot;,<br>        input_cost_per_million=2.50,<br>        output_cost_per_million=10.00,<br>        avg_latency_seconds=3.5,<br>        accuracy_by_complexity={<br>            TaskComplexity.TRIVIAL:  0.98,<br>            TaskComplexity.MODERATE: 0.87,<br>            TaskComplexity.HARD:     0.65,<br>            TaskComplexity.RESEARCH: 0.40,<br>        },<br>    ),<br>    &quot;o4-mini&quot;: ModelOption(<br>        name=&quot;o4-mini&quot;,<br>        input_cost_per_million=1.10,<br>        output_cost_per_million=4.40,<br>        avg_latency_seconds=12.0,<br>        accuracy_by_complexity={<br>            TaskComplexity.TRIVIAL:  0.97,<br>            TaskComplexity.MODERATE: 0.93,<br>            TaskComplexity.HARD:     0.84,<br>            TaskComplexity.RESEARCH: 0.70,<br>        },<br>    ),<br>    &quot;o3&quot;: ModelOption(<br>        name=&quot;o3&quot;,<br>        input_cost_per_million=10.00,<br>        output_cost_per_million=40.00,<br>        avg_latency_seconds=45.0,<br>        accuracy_by_complexity={<br>            TaskComplexity.TRIVIAL:  0.98,<br>            TaskComplexity.MODERATE: 0.96,<br>            TaskComplexity.HARD:     0.92,<br>            TaskComplexity.RESEARCH: 0.85,<br>        },<br>    ),<br>    &quot;claude-sonnet-4&quot;: ModelOption(<br>        name=&quot;claude-sonnet-4&quot;,<br>        input_cost_per_million=3.00,<br>        output_cost_per_million=15.00,<br>        avg_latency_seconds=4.0,<br>        accuracy_by_complexity={<br>            TaskComplexity.TRIVIAL:  0.98,<br>            TaskComplexity.MODERATE: 0.90,<br>            TaskComplexity.HARD:     0.72,<br>            TaskComplexity.RESEARCH: 0.50,<br>        },<br>    ),<br>    &quot;claude-sonnet-4-thinking&quot;: ModelOption(<br>        name=&quot;claude-sonnet-4 + thinking&quot;,<br>        input_cost_per_million=3.00,<br>        output_cost_per_million=15.00,  # Thinking tokens also charged at output rate<br>        avg_latency_seconds=20.0,<br>        accuracy_by_complexity={<br>            TaskComplexity.TRIVIAL:  0.98,<br>            TaskComplexity.MODERATE: 0.94,<br>            TaskComplexity.HARD:     0.87,<br>            TaskComplexity.RESEARCH: 0.73,<br>        },<br>    ),<br>}<br><br>def recommend_model(<br>    task_complexity: TaskComplexity,<br>    avg_input_tokens: int = 500,<br>    avg_output_tokens: int = 500,<br>    latency_budget_seconds: float = 30.0,<br>    accuracy_threshold: float = 0.85,<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;<br>    Recommend the best model for given constraints.<br>    <br>    Returns ranked list of models meeting accuracy_threshold<br>    within latency_budget, sorted by cost.<br>    &quot;&quot;&quot;<br>    <br>    results = []<br>    for model_name, model in MODEL_OPTIONS.items():<br>        accuracy = model.accuracy_by_complexity[task_complexity]<br>        <br>        # Skip models that don&#39;t meet accuracy threshold<br>        if accuracy &lt; accuracy_threshold:<br>            continue<br>        <br>        # Skip models too slow for latency budget<br>        if model.avg_latency_seconds &gt; latency_budget_seconds:<br>            continue<br>        <br>        # Calculate cost<br>        cost = (<br>            (avg_input_tokens / 1_000_000) * model.input_cost_per_million +<br>            (avg_output_tokens / 1_000_000) * model.output_cost_per_million<br>        )<br>        <br>        results.append({<br>            &quot;model&quot;: model_name,<br>            &quot;accuracy&quot;: accuracy,<br>            &quot;cost_per_request&quot;: round(cost, 6),<br>            &quot;latency_seconds&quot;: model.avg_latency_seconds,<br>        })<br>    <br>    # Sort by cost (cheapest first)<br>    results.sort(key=lambda x: x[&quot;cost_per_request&quot;])<br>    return results<br><br><br># Usage<br>print(&quot;=== Task: Hard coding problem (accuracy ≥ 85%) ===&quot;)<br>recommendations = recommend_model(<br>    task_complexity=TaskComplexity.HARD,<br>    accuracy_threshold=0.85,<br>    latency_budget_seconds=60.0,<br>)<br>for r in recommendations:<br>    print(f&quot;  {r[&#39;model&#39;]:&lt;30} accuracy={r[&#39;accuracy&#39;]:.0%}  &quot;<br>          f&quot;cost=${r[&#39;cost_per_request&#39;]:.5f}  latency={r[&#39;latency_seconds&#39;]}s&quot;)<br><br># Output:<br># === Task: Hard coding problem (accuracy ≥ 85%) ===<br>#   claude-sonnet-4 + thinking     accuracy=87%  cost=$0.00900  latency=20.0s<br>#   o4-mini                        accuracy=84%  cost=$0.00275  latency=12.0s<br>#   o3                             accuracy=92%  cost=$0.02500  latency=45.0s</pre><h2>Part 6: Dynamic Model Router</h2><p>Build a production router that automatically selects standard vs reasoning model based on task classification:</p><pre>from openai import AsyncOpenAI<br>import anthropic<br>import asyncio<br><br>openai_client = AsyncOpenAI()<br>anthropic_client = anthropic.AsyncAnthropic()<br><br># Task classifier — asks a fast cheap model to classify complexity<br>async def classify_task_complexity(user_query: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    Use a cheap fast model to classify whether a reasoning model is needed.<br>    Returns: &quot;simple&quot; | &quot;moderate&quot; | &quot;complex&quot;<br>    &quot;&quot;&quot;<br>    <br>    response = await openai_client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,  # Fast and cheap for classification<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: &quot;&quot;&quot;Classify the complexity of this task. Reply with exactly one word:<br>                <br>&quot;simple&quot; — factual lookup, translation, simple Q&amp;A, summarization, extraction<br>&quot;moderate&quot; — analysis, planning, multi-step problem, code review<br>&quot;complex&quot; — math proofs, competition coding, multi-constraint reasoning, logical puzzles, <br>             debugging tricky bugs, security analysis, algorithm design from scratch<br><br>Reply with only the classification word.&quot;&quot;&quot;,<br>            },<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_query},<br>        ],<br>        temperature=0,<br>        max_tokens=10,<br>    )<br>    <br>    classification = response.choices[0].message.content.strip().lower()<br>    return classification if classification in (&quot;simple&quot;, &quot;moderate&quot;, &quot;complex&quot;) else &quot;moderate&quot;<br><br><br>async def smart_complete(<br>    query: str,<br>    system_prompt: str = &quot;You are a helpful AI assistant.&quot;,<br>    force_reasoning: bool = False,<br>    verbose: bool = True,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Intelligently routes between standard and reasoning models.<br>    <br>    - Simple tasks → gpt-4o (fast, cheap)<br>    - Moderate tasks → gpt-4o or o4-mini depending on query signals<br>    - Complex tasks → o3 or claude extended thinking<br>    &quot;&quot;&quot;<br>    <br>    if force_reasoning:<br>        complexity = &quot;complex&quot;<br>    else:<br>        complexity = await classify_task_complexity(query)<br>    <br>    if verbose:<br>        print(f&quot;[Router] Classified as: {complexity}&quot;)<br>    <br>    if complexity == &quot;simple&quot;:<br>        # Standard model — fast and cheap<br>        response = await openai_client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query},<br>            ],<br>            temperature=0.7,<br>            max_tokens=1_000,<br>        )<br>        return {<br>            &quot;answer&quot;: response.choices[0].message.content,<br>            &quot;model_used&quot;: &quot;gpt-4o&quot;,<br>            &quot;reasoning_used&quot;: False,<br>            &quot;cost_usd&quot;: (<br>                response.usage.prompt_tokens / 1_000_000 * 2.50 +<br>                response.usage.completion_tokens / 1_000_000 * 10.00<br>            ),<br>        }<br>    <br>    elif complexity == &quot;moderate&quot;:<br>        # o4-mini — reasoning without frontier cost<br>        response = await openai_client.chat.completions.create(<br>            model=&quot;o4-mini&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query},<br>            ],<br>            reasoning_effort=&quot;medium&quot;,<br>            max_completion_tokens=5_000,<br>        )<br>        usage = response.usage<br>        return {<br>            &quot;answer&quot;: response.choices[0].message.content,<br>            &quot;model_used&quot;: &quot;o4-mini&quot;,<br>            &quot;reasoning_used&quot;: True,<br>            &quot;reasoning_tokens&quot;: getattr(usage.completion_tokens_details, &quot;reasoning_tokens&quot;, 0),<br>            &quot;cost_usd&quot;: (<br>                usage.prompt_tokens / 1_000_000 * 1.10 +<br>                usage.completion_tokens / 1_000_000 * 4.40<br>            ),<br>        }<br>    <br>    else:  # complex<br>        # Claude extended thinking — surfaces the reasoning chain<br>        response = await anthropic_client.messages.create(<br>            model=&quot;claude-sonnet-4-20250514&quot;,<br>            max_tokens=12_000,<br>            thinking={&quot;type&quot;: &quot;enabled&quot;, &quot;budget_tokens&quot;: 8_000},<br>            system=system_prompt,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}],<br>        )<br>        <br>        answer = &quot;&quot;<br>        for block in response.content:<br>            if block.type == &quot;text&quot;:<br>                answer += block.text<br>        <br>        cost = (<br>            response.usage.input_tokens / 1_000_000 * 3.00 +<br>            response.usage.output_tokens / 1_000_000 * 15.00<br>        )<br>        <br>        return {<br>            &quot;answer&quot;: answer,<br>            &quot;model_used&quot;: &quot;claude-sonnet-4-thinking&quot;,<br>            &quot;reasoning_used&quot;: True,<br>            &quot;cost_usd&quot;: cost,<br>        }<br><br><br># Demo<br>async def demo_smart_router():<br>    queries = [<br>        (&quot;What is the capital of Japan?&quot;, False),<br>        (&quot;Write a Python function to reverse a linked list&quot;, False),<br>        (<br>            &quot;Prove that the square root of 2 is irrational using proof by contradiction. &quot;<br>            &quot;Then explain what this implies about the density of irrational numbers on the real line.&quot;,<br>            False,<br>        ),<br>    ]<br>    <br>    for query, force in queries:<br>        print(f&quot;\nQuery: {query[:70]}...&quot;)<br>        result = await smart_complete(query, force_reasoning=force)<br>        print(f&quot;Model: {result[&#39;model_used&#39;]}&quot;)<br>        print(f&quot;Cost:  ${result[&#39;cost_usd&#39;]:.5f}&quot;)<br>        print(f&quot;Answer preview: {result[&#39;answer&#39;][:150]}...&quot;)<br><br>asyncio.run(demo_smart_router())</pre><h2>Part 7: Open-Source Reasoning Models</h2><p>The reasoning paradigm has been democratized through open-source:</p><pre># ── DeepSeek-R1 via Ollama (local, free) ─────────────────────────────────────<br>import httpx<br>import json<br>import asyncio<br><br>async def deepseek_r1_local(problem: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    Run DeepSeek-R1 locally via Ollama.<br>    <br>    DeepSeek-R1 uses &lt;think&gt;...&lt;/think&gt; tags for internal reasoning.<br>    This is visible in the output — you can parse it.<br>    <br>    Models available:<br>    - deepseek-r1:7b    → 5 GB, decent reasoning, runs on 8GB RAM<br>    - deepseek-r1:14b   → 10 GB, better reasoning, needs 12GB RAM<br>    - deepseek-r1:32b   → 20 GB, strong reasoning, needs 24GB RAM<br>    - deepseek-r1:70b   → 40 GB, near-frontier, needs 48GB RAM<br>    &quot;&quot;&quot;<br>    <br>    async with httpx.AsyncClient(timeout=300.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/api/generate&quot;,<br>            json={<br>                &quot;model&quot;: &quot;deepseek-r1:7b&quot;,<br>                &quot;prompt&quot;: problem,<br>                &quot;options&quot;: {<br>                    &quot;temperature&quot;: 0.6,     # DeepSeek-R1 recommends ~0.6<br>                    &quot;top_p&quot;: 0.95,<br>                    &quot;num_predict&quot;: 4096,<br>                },<br>                &quot;stream&quot;: False,<br>            },<br>        )<br>        <br>        result = response.json()[&quot;response&quot;]<br>        <br>        # Parse out thinking and answer<br>        if &quot;&lt;think&gt;&quot; in result and &quot;&lt;/think&gt;&quot; in result:<br>            think_start = result.index(&quot;&lt;think&gt;&quot;) + len(&quot;&lt;think&gt;&quot;)<br>            think_end = result.index(&quot;&lt;/think&gt;&quot;)<br>            thinking = result[think_start:think_end].strip()<br>            answer = result[result.index(&quot;&lt;/think&gt;&quot;) + len(&quot;&lt;/think&gt;&quot;):].strip()<br>        else:<br>            thinking = &quot;&quot;<br>            answer = result<br>        <br>        return {<br>            &quot;thinking&quot;: thinking,<br>            &quot;answer&quot;: answer,<br>        }<br><br><br># ── QwQ-32B via Ollama ────────────────────────────────────────────────────────<br>async def qwq_local(problem: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    QwQ-32B: Alibaba&#39;s open reasoning model.<br>    Comparable to o1-mini on math and coding tasks.<br>    Requires ~20 GB VRAM for full precision, ~12 GB for Q4.<br>    &quot;&quot;&quot;<br>    async with httpx.AsyncClient(timeout=300.0) as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/api/generate&quot;,<br>            json={<br>                &quot;model&quot;: &quot;qwq:32b&quot;,<br>                &quot;prompt&quot;: f&quot;&lt;|im_start|&gt;user\n{problem}&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n&quot;,<br>                &quot;options&quot;: {&quot;temperature&quot;: 0.7, &quot;num_predict&quot;: 8192},<br>                &quot;stream&quot;: False,<br>            },<br>        )<br>        return {&quot;answer&quot;: response.json()[&quot;response&quot;]}</pre><h3>Open-Source Reasoning Model Comparison</h3><pre>Model              Size    VRAM needed    Quality vs o1-mini    License<br>──────────────────────────────────────────────────────────────────────────────<br>DeepSeek-R1:7b     5 GB    8 GB           ~60%                  MIT<br>DeepSeek-R1:14b    10 GB   12 GB          ~75%                  MIT<br>DeepSeek-R1:32b    20 GB   24 GB          ~88%                  MIT<br>DeepSeek-R1:70b    40 GB   48 GB          ~95%                  MIT<br>QwQ-32B            20 GB   24 GB          ~85%                  Apache 2.0<br>Llama 3.3:70b      40 GB   48 GB          ~70%                  Llama license<br>Sky-T1:32b         20 GB   24 GB          ~82%                  Apache 2.0</pre><h2>Part 8: Common Failure Modes of Reasoning Models</h2><h3>1. Overthinking Simple Problems</h3><pre># ❌ Anti-pattern: Using o3 for trivial tasks<br>response = await openai_client.chat.completions.create(<br>    model=&quot;o3&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is 15% of 80?&quot;}],<br>    reasoning_effort=&quot;high&quot;,<br>)<br># o3 will spend thousands of tokens thinking about<br># basic arithmetic. Correct answer, but $0.05+ wasted.<br><br># ✅ Correct: Use fast model for simple math<br>response = await openai_client.chat.completions.create(<br>    model=&quot;gpt-4o-mini&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is 15% of 80?&quot;}],<br>    temperature=0,<br>)<br># $0.000001 — 50,000× cheaper. Same answer.</pre><h3>2. Assuming Thinking = Grounding</h3><p>Reasoning models think more carefully, but they don’t have access to real-time information. A long thinking chain applied to a factual question about recent events just produces a more elaborate hallucination.</p><pre># ❌ Wrong: Using o3 for real-time information<br>result = await reason_with_o3(<br>    &quot;What is the current stock price of NVIDIA?&quot;,<br>    effort=&quot;high&quot;,<br>)<br># o3 will confidently reason its way to a fabricated<br># stock price from its training data<br><br># ✅ Correct: Retrieve first, then reason<br># 1. Use web search / tool call to get current price<br># 2. Then use o3 to analyze/reason about it</pre><h3>3. Ignoring Latency in User-Facing Flows</h3><pre>Reasoning effort &quot;high&quot; on o3: 30-90 seconds<br>Standard gpt-4o: 2-5 seconds<br><br>If a user is waiting for a response, even &quot;medium&quot; reasoning (15-30s)<br>may be unacceptable. Always measure and set expectations.</pre><pre># Production pattern: Show progress for long reasoning<br>async def reason_with_progress_indicator(problem: str):<br>    &quot;&quot;&quot;<br>    For UI-facing applications: show the user reasoning is in progress.<br>    &quot;&quot;&quot;<br>    import asyncio<br>    <br>    # Start reasoning in background<br>    reasoning_task = asyncio.create_task(<br>        reason_with_o3(problem, effort=&quot;medium&quot;)<br>    )<br>    <br>    # Show progress indicator while waiting<br>    start = asyncio.get_event_loop().time()<br>    while not reasoning_task.done():<br>        elapsed = asyncio.get_event_loop().time() - start<br>        print(f&quot;\r⏳ Reasoning... {elapsed:.0f}s&quot;, end=&quot;&quot;, flush=True)<br>        await asyncio.sleep(0.5)<br>    <br>    print()<br>    return await reasoning_task</pre><h3>4. Not Validating Reasoning Chain Conclusions</h3><p>Extended thinking is visible but not infallible. The model can reason incorrectly. Treat the thinking chain as debug output, not ground truth.</p><pre>async def verify_reasoning_conclusion(<br>    problem: str,<br>    reasoning_result: dict,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Use a fast model to verify the conclusion from extended thinking.<br>    Catches cases where the reasoning chain arrived at a wrong answer.<br>    &quot;&quot;&quot;<br>    <br>    verification_prompt = f&quot;&quot;&quot;<br>Problem: {problem}<br><br>Proposed answer: {reasoning_result[&#39;answer&#39;]}<br><br>Is this answer correct? Reply with:<br>- &quot;CORRECT&quot; if the answer is right<br>- &quot;INCORRECT: [brief explanation]&quot; if the answer is wrong<br>&quot;&quot;&quot;<br>    <br>    response = await openai_client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: verification_prompt}],<br>        temperature=0,<br>        max_tokens=100,<br>    )<br>    <br>    verdict = response.choices[0].message.content<br>    reasoning_result[&quot;verification&quot;] = verdict<br>    reasoning_result[&quot;verified&quot;] = verdict.startswith(&quot;CORRECT&quot;)<br>    return reasoning_result</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Using </strong><strong>temperature with o-series models</strong> OpenAI&#39;s o3/o4-mini don&#39;t support temperature — they&#39;ll raise an API error. The model&#39;s internal reasoning uses its own temperature schedule. Remove temperature from your API calls when switching to o-series.</p><p><strong>2. Forgetting that thinking tokens cost money (Anthropic)</strong> Claude’s thinking tokens are charged at the output token rate. A 10,000-token thinking budget at $15/M output costs $0.15 <em>just for thinking</em>, before any output. Budget accordingly.</p><p><strong>3. Using reasoning models for structured output without JSON mode</strong> Reasoning models may include their thought process as preamble before the JSON, breaking your parser. Always use response_format={&quot;type&quot;: &quot;json_object&quot;} or Anthropic&#39;s structured output when you need parseable output from a reasoning model.</p><p><strong>4. Setting </strong><strong>max_completion_tokens too low on o-series</strong> The max_completion_tokens parameter includes <em>both</em> thinking tokens and output tokens. If you set max_completion_tokens=1000 and the model needs 800 thinking tokens, you only have 200 left for the actual answer — which gets truncated. Set it to budget + expected_output_length.</p><p><strong>5. Treating reasoning models as silver bullets</strong> On tasks where standard models already perform near-perfectly (factual Q&amp;A, translation, simple summarization), reasoning models add cost and latency with no quality improvement. Profile your tasks before committing to reasoning models in production.</p><h2>💼 Quick Questions</h2><p><strong>Q: Explain the difference between training-time scaling and test-time compute scaling. Why does test-time scaling work?</strong></p><p>Training-time scaling improves model capability by increasing model parameters, training data, or training compute during the training phase — a one-time investment. Test-time compute scaling improves output quality at inference time by allocating more compute to “thinking” before generating the final answer. It works because complex problems benefit from iterative refinement: the model can propose a solution, check it for errors, revise it, and verify again — all within the thinking budget. The empirical observation is that for hard reasoning tasks, accuracy follows a power-law with thinking compute, independent of the base model size.</p><p><strong>Q: When would you choose o4-mini over o3, and when would you choose o3?</strong></p><p>o4-mini is the right choice when: the task is moderately complex but not frontier-level hard; cost efficiency matters; latency is a concern (o4-mini is ~4× faster); and you’re processing high volumes. o3 is justified when: the task is genuinely at the frontier of difficulty (competition-level math, deep security analysis, complex multi-constraint problems); accuracy is critical and cost is secondary; or you’re building a research/analysis tool rather than a high-volume consumer product. As a rule: try o4-mini first and benchmark accuracy — only escalate to o3 if o4-mini’s accuracy is insufficient.</p><p><strong>Q: How would you design a production system that uses reasoning models cost-effectively for a document Q&amp;A product?</strong></p><p>Three-layer architecture: (1) Retrieval layer — use embeddings + vector search to retrieve the top-3 relevant passages (no reasoning needed here, use a cheap embedding model); (2) Classification layer — use gpt-4o-mini (fast, cheap) to classify the question as simple/moderate/complex based on whether it requires multi-step reasoning over the passages; (3) Completion layer — route simple questions to gpt-4o ($0.003/request), moderate to o4-mini ($0.01/request), complex to o3 ($0.05/request). This achieves frontier reasoning quality on hard questions while keeping average cost ~80% lower than using o3 for everything. Instrument the classification layer carefully — miscategorizing complex questions as simple is the main failure mode.</p><h2>🏭 Production Considerations</h2><p><strong>Reasoning models have variable latency.</strong> o3 at reasoning_effort=&quot;high&quot; can take 30–90 seconds. Build async patterns, streaming progress indicators, or asynchronous job queues for any user-facing flow. Never call reasoning models synchronously on a web request thread.</p><p><strong>Implement per-user and per-session reasoning budgets.</strong> Reasoning models can be 100× more expensive than standard models. Without budget controls, a single malicious or confused user can incur thousands of dollars in API costs. Implement token budgets at the user session level.</p><p><strong>Log thinking chains for debugging.</strong> Anthropic and Gemini’s visible thinking is invaluable for understanding why a model produced a wrong answer. Store thinking chains alongside inputs/outputs in your logging system, but filter them from end-user display unless explicitly surfacing them adds value.</p><p><strong>Benchmark your specific tasks before committing.</strong> The general accuracy comparisons above are approximations. Your domain may have different relative performance. Before deploying reasoning models in production, run an offline eval on 100+ representative examples and measure accuracy, latency, and cost for each model tier.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Reasoning models use test-time compute scaling</strong> — more thinking tokens = better answers on hard problems, independent of model size</li><li><strong>Internal chain-of-thought is the mechanism</strong> — the model generates a private scratchpad before producing the final answer</li><li><strong>Three provider implementations</strong>: OpenAI (hidden thinking, effort levels), Anthropic (visible thinking, budget tokens), Gemini (visible thinking, budget tokens)</li><li><strong>o4-mini is the sweet spot for most production reasoning tasks</strong> — 9× cheaper than o3 with 90% of the capability improvement</li><li><strong>Reasoning models are NOT improvements on all tasks</strong> — for simple Q&amp;A, translation, and extraction, they cost more with no accuracy benefit</li><li><strong>Open-source reasoning is viable</strong> — DeepSeek-R1 and QwQ-32B run locally for free and compete with o1-mini</li><li><strong>Thinking tokens cost money</strong> — Claude extended thinking charges thinking at the output token rate; budget accordingly</li><li><strong>Always measure latency</strong> — “high effort” o3 can take 30–90 seconds; never block a user-facing UI thread</li><li><strong>Dynamic routing is the production pattern</strong> — classify task complexity cheaply, then route to the appropriate model tier</li><li><strong>Visible thinking chains are debugging gold</strong> — they reveal reasoning errors that output alone hides</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://openai.com/index/openai-o1-system-card/">OpenAI o1 System Card (2024)</a> — The original technical documentation for o1 reasoning</li><li><a href="https://arxiv.org/abs/2501.12948">DeepSeek-R1 Paper</a> — How DeepSeek trained an open-source reasoning model rivaling o1</li><li><a href="https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking">Anthropic Extended Thinking Guide</a> — Official docs with budget recommendations</li><li><a href="https://arxiv.org/abs/2408.03314">Scaling LLM Test-Time Compute (Google, 2024)</a> — Empirical study of test-time compute scaling laws</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f0f84f83d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 10: Context Windows — The Key Architectural Constraint]]></title>
            <link>https://learncsdesigns.medium.com/day-10-context-windows-the-key-architectural-constraint-374501ed7ba7?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/374501ed7ba7</guid>
            <category><![CDATA[context-window]]></category>
            <category><![CDATA[long-context]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Thu, 18 Jun 2026 16:03:23 GMT</pubDate>
            <atom:updated>2026-06-18T16:03:23.418Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Context is not just memory. It is the entire universe the model can see. Everything outside the window does not exist.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ri0KRdEo9jYxG1YrmJz1Ww.png" /></figure><h2>Why This Matters</h2><p>Context windows are the single most important architectural constraint in LLM application design. Almost every major failure mode in production AI — forgotten instructions, hallucinated facts, ignored documents, broken multi-turn conversations — traces back to context window mismanagement.</p><p>In 2026, context windows have grown dramatically:</p><ul><li>GPT-4o: 128,000 tokens</li><li>Claude Opus 4: 200,000 tokens</li><li>Gemini 2.5 Pro: <strong>1,048,576 tokens</strong> (1 million+)</li></ul><p>Bigger is better, right? Not quite. Long context comes with compounding costs, latency penalties, the lost-in-the-middle attention problem, and architectural tradeoffs that make it wrong for many applications. Understanding <em>when</em> to use long context — and <em>when not to</em> — is what separates junior AI engineers from senior ones.</p><h2>Part 1: What a Context Window Actually Is</h2><h3>The Formal Definition</h3><p>A context window is the maximum number of <strong>tokens</strong> the model can process in a single forward pass. This includes:</p><pre>Total context = System prompt + Conversation history + Retrieved documents + Current user message + Model response</pre><p>Every single token across all these sources competes for space in the same fixed-size window. The model has no awareness of anything outside the window — not your database, not previous conversations from last week, not the other 900 pages of the document you fed it 500 pages of.</p><h3>Tokens ≠ Words</h3><p>A crucial nuance: tokens are not words. The tokenization ratio varies by content type:</p><pre>import tiktoken<br><br>encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br><br># Approximate token ratios by content type<br>examples = {<br>    &quot;English prose&quot;:      &quot;The quick brown fox jumps over the lazy dog&quot;,<br>    &quot;Python code&quot;:        &quot;def fibonacci(n):\n    if n &lt;= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)&quot;,<br>    &quot;JSON data&quot;:          &#39;{&quot;name&quot;: &quot;Alice&quot;, &quot;age&quot;: 30, &quot;city&quot;: &quot;New York&quot;}&#39;,<br>    &quot;Technical jargon&quot;:   &quot;Kullback-Leibler divergence Transformer self-attention&quot;,<br>    &quot;Chinese text&quot;:       &quot;你好，世界。这是一个测试句子。&quot;,<br>}<br><br>for label, text in examples.items():<br>    tokens = encoder.encode(text)<br>    ratio = len(tokens) / len(text.split())<br>    print(f&quot;{label:25} | Words: {len(text.split()):4} | Tokens: {len(tokens):4} | Tokens/word: {ratio:.2f}&quot;)</pre><p><strong>Output:</strong></p><pre>English prose             | Words:    9 | Tokens:    9 | Tokens/word: 1.00<br>Python code               | Words:   12 | Tokens:   28 | Tokens/word: 2.33<br>JSON data                 | Words:    7 | Tokens:   19 | Tokens/word: 2.71<br>Technical jargon          | Words:    4 | Tokens:   11 | Tokens/word: 2.75<br>Chinese text              | Words:    1 | Tokens:   10 | Tokens/word: 10.00</pre><p><strong>Rule of thumb for capacity planning:</strong></p><pre>Content type          Tokens per page (A4, ~400 words)<br>────────────────────────────────────────────────────────<br>English prose         ~500 tokens<br>Code                  ~800 tokens  (more tokens per line)<br>JSON/XML data         ~1,000 tokens (structure overhead)<br>Mixed technical doc   ~600 tokens</pre><p>So a 128k token window holds roughly:</p><ul><li>~250 pages of English prose</li><li>~160 pages of code</li><li>~128 pages of structured data</li></ul><h3>The Hard Truncation Problem</h3><p>What happens when you exceed the context limit? The SDK raises an error — or worse, silently truncates your input without telling you:</p><pre>from openai import AsyncOpenAI, BadRequestError<br>import tiktoken<br><br>client = AsyncOpenAI()<br>encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br><br>async def safe_complete(messages: list[dict], model: str = &quot;gpt-4o&quot;) -&gt; str:<br>    &quot;&quot;&quot;<br>    Context-aware completion that counts tokens BEFORE sending.<br>    Raises an explicit error instead of silent truncation.<br>    &quot;&quot;&quot;<br>    <br>    MODEL_LIMITS = {<br>        &quot;gpt-4o&quot;:           128_000,<br>        &quot;gpt-4o-mini&quot;:      128_000,<br>        &quot;o3&quot;:               200_000,<br>        &quot;claude-opus-4&quot;:    200_000,<br>        &quot;claude-sonnet-4&quot;:  200_000,<br>        &quot;gemini-2.5-pro&quot;:   1_048_576,<br>        &quot;gemini-2.5-flash&quot;: 1_048_576,<br>    }<br>    <br>    RESERVE_FOR_OUTPUT = 4_096  # Reserve tokens for the model&#39;s response<br>    <br>    # Count tokens in all messages<br>    total_tokens = 0<br>    for message in messages:<br>        # +4 per message for role/formatting overhead<br>        total_tokens += 4 + len(encoder.encode(message.get(&quot;content&quot;, &quot;&quot;)))<br>    total_tokens += 2  # priming tokens<br>    <br>    limit = MODEL_LIMITS.get(model, 128_000)<br>    available = limit - RESERVE_FOR_OUTPUT<br>    <br>    if total_tokens &gt; available:<br>        raise ValueError(<br>            f&quot;Input too long: {total_tokens:,} tokens exceeds &quot;<br>            f&quot;{available:,} available ({limit:,} limit − {RESERVE_FOR_OUTPUT:,} output reserve). &quot;<br>            f&quot;Reduce input by {total_tokens - available:,} tokens.&quot;<br>        )<br>    <br>    response = await client.chat.completions.create(<br>        model=model,<br>        messages=messages,<br>        max_tokens=RESERVE_FOR_OUTPUT,<br>    )<br>    <br>    return response.choices[0].message.content</pre><h2>Part 2: The Physics of Long Context — KV Cache</h2><h3>Why Long Context Is Expensive</h3><p>To understand why long context costs money and latency, you need to understand the <strong>Key-Value (KV) Cache</strong> — the memory structure that makes transformer inference work.</p><p>During a forward pass, every token in the context produces two vectors: a <strong>Key</strong> and a <strong>Value</strong>. These are used by the self-attention mechanism so that later tokens can “look back” at earlier ones.</p><pre>For each transformer layer:<br>  KV cache size = context_length × num_heads × head_dimension × 2 × dtype_bytes<br><br>Example: GPT-4o, 128k context, 96 layers, 128 heads, 128 dim, float16:<br>  = 128,000 × 128 × 128 × 2 × 2 bytes<br>  = ~84 GB of KV cache for one request at full contextExample: GPT-4o, 128k context, 96 layers, 128 heads, 128 dim, float16:<br>  = 128,000 × 128 × 128 × 2 × 2 bytes<br>  = ~84 GB of KV cache for one request at full context</pre><p>This means:</p><pre>Context length    KV cache (approximate)    Latency impact<br>────────────────────────────────────────────────────────────<br>4,096 tokens      ~2.5 GB                   Baseline<br>32,000 tokens     ~20 GB                    4-6× slower<br>128,000 tokens    ~84 GB                    20-40× slower<br>1,000,000 tokens  ~640 GB                   100×+ slower, GPU cluster needed</pre><p><strong>Cost implications</strong> (OpenAI GPT-4o pricing, 2026):</p><pre>Context length    Input cost (per request)<br>──────────────────────────────────────────<br>1,000 tokens      $0.0000025<br>10,000 tokens     $0.000025<br>100,000 tokens    $0.00025<br>1,000,000 tokens  $0.0025</pre><p>This might seem cheap per request, but at scale:</p><pre># Cost calculator for long-context at scale<br>def context_cost_analysis(<br>    avg_context_tokens: int,<br>    requests_per_day: int,<br>    input_cost_per_million: float = 2.50,<br>    output_cost_per_million: float = 10.00,<br>    avg_output_tokens: int = 500,<br>) -&gt; dict:<br>    <br>    daily_input_tokens = avg_context_tokens * requests_per_day<br>    daily_output_tokens = avg_output_tokens * requests_per_day<br>    <br>    daily_input_cost = (daily_input_tokens / 1_000_000) * input_cost_per_million<br>    daily_output_cost = (daily_output_tokens / 1_000_000) * output_cost_per_million<br>    daily_total = daily_input_cost + daily_output_cost<br>    <br>    return {<br>        &quot;avg_context_tokens&quot;: f&quot;{avg_context_tokens:,}&quot;,<br>        &quot;requests_per_day&quot;: f&quot;{requests_per_day:,}&quot;,<br>        &quot;daily_input_cost&quot;: f&quot;${daily_input_cost:.2f}&quot;,<br>        &quot;daily_output_cost&quot;: f&quot;${daily_output_cost:.2f}&quot;,<br>        &quot;daily_total&quot;: f&quot;${daily_total:.2f}&quot;,<br>        &quot;monthly_total&quot;: f&quot;${daily_total * 30:.2f}&quot;,<br>        &quot;annual_total&quot;: f&quot;${daily_total * 365:.2f}&quot;,<br>    }<br><br># A document Q&amp;A app: 50k avg context, 10,000 requests/day<br>print(context_cost_analysis(50_000, 10_000))<br># → monthly: ~$3,750<br><br># Same app with RAG (3k context): massive savings<br>print(context_cost_analysis(3_000, 10_000))<br># → monthly: ~$225<br><br><br>Output:<br><br>{&#39;avg_context_tokens&#39;: &#39;50,000&#39;, &#39;requests_per_day&#39;: &#39;10,000&#39;, &#39;daily_input_cost&#39;: &#39;$1250.00&#39;, &#39;daily_output_cost&#39;: &#39;$50.00&#39;, &#39;daily_total&#39;: &#39;$1300.00&#39;, &#39;monthly_total&#39;: &#39;$39000.00&#39;, &#39;annual_total&#39;: &#39;$474500.00&#39;}<br>{&#39;avg_context_tokens&#39;: &#39;3,000&#39;, &#39;requests_per_day&#39;: &#39;10,000&#39;, &#39;daily_input_cost&#39;: &#39;$75.00&#39;, &#39;daily_output_cost&#39;: &#39;$50.00&#39;, &#39;daily_total&#39;: &#39;$125.00&#39;, &#39;monthly_total&#39;: &#39;$3750.00&#39;, &#39;annual_total&#39;: &#39;$45625.00&#39;}</pre><p><strong>This is why RAG exists.</strong> Stuffing 50k tokens into every request is 16× more expensive than retrieving 3k relevant tokens.</p><h2>Part 3: The Long-Context Model Landscape (2026)</h2><pre>Model                    Context      Input cost/M    Best for<br>────────────────────────────────────────────────────────────────────────────<br>GPT-4o                   128k         $2.50           Standard production<br>GPT-4o-mini              128k         $0.15           Cost-sensitive, high volume<br>o3                       200k         $10.00          Complex reasoning tasks<br>Claude Opus 4            200k         $15.00          Long document analysis<br>Claude Sonnet 4          200k         $3.00           Balanced long-context<br>Claude Haiku 3.5         200k         $0.80           Fast, cheap long-context<br>Gemini 2.5 Pro           1M           $1.25 (≤200k)   Entire codebases, books<br>                                      $2.50 (&gt;200k)<br>Gemini 2.5 Flash         1M           $0.075          Highest volume, long docs<br>Gemini 1.5 Pro           2M           $1.25 (≤128k)   Legacy very long context</pre><h3>When to Choose Each Tier</h3><pre>Context need                           Recommendation<br>──────────────────────────────────────────────────────────────────────────<br>&lt; 10k tokens (chat, short Q&amp;A)         Any model — optimize for cost/quality<br>10k – 50k (medium documents)           GPT-4o, Claude Sonnet, Gemini Flash<br>50k – 200k (books, codebases)          Claude Sonnet/Opus, Gemini Pro<br>200k – 1M (entire repos, research)     Gemini 2.5 Pro/Flash exclusively<br>Repeating same context (caching)       Claude (prompt caching), Gemini (implicit)</pre><h3>Prompt Caching: The Cost Multiplier</h3><p>Anthropic and Google offer <strong>prompt caching</strong> — if you send the same prefix (system prompt + documents) repeatedly, the KV cache is computed once and reused:</p><pre>import anthropic<br><br>client = anthropic.AsyncAnthropic()<br><br># With prompt caching — pay full price first time, ~10% on cache hits<br>response = await client.messages.create(<br>    model=&quot;claude-sonnet-4-20250514&quot;,<br>    max_tokens=1024,<br>    system=[<br>        {<br>            &quot;type&quot;: &quot;text&quot;,<br>            &quot;text&quot;: &quot;You are a legal analyst. Here are the relevant statutes:&quot;,<br>        },<br>        {<br>            &quot;type&quot;: &quot;text&quot;,<br>            &quot;text&quot;: very_long_legal_document,  # e.g., 100k tokens<br>            &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;},  # Cache this prefix<br>        }<br>    ],<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What does Section 4.2 say about liability?&quot;}<br>    ],<br>)<br><br># cache_creation_input_tokens — paid at full price (first time)<br># cache_read_input_tokens — paid at 10% of normal price (subsequent calls)<br>print(response.usage.cache_creation_input_tokens)<br>print(response.usage.cache_read_input_tokens)</pre><p><strong>Cache savings at scale:</strong></p><pre>Scenario: 100k token document, 1000 queries/day<br>  Without caching: 1000 × 100k × $3/M = $300/day<br>  With caching:    1× $30 creation + 999× $3 cache read = $33/day<br>  Savings: 89%</pre><h2>Part 4: The Lost-in-the-Middle Problem</h2><h3>What the Research Found</h3><p>In 2023, researchers at Stanford published a landmark paper: “Lost in the Middle: How Language Models Use Long Contexts.” The finding was striking and has been confirmed across multiple frontier models:</p><p><strong>LLMs perform significantly worse at retrieving information placed in the middle of long contexts compared to information at the beginning or end.</strong></p><pre>Information position     Retrieval accuracy (multi-document QA)<br>────────────────────────────────────────────────────────────────<br>Beginning of context      ~75-80%<br>End of context            ~70-75%<br>Middle of context         ~50-60%  ← significant degradation</pre><p>This U-shaped performance curve has major implications for system design.</p><h3>Why This Happens</h3><p>The attention mechanism in transformers has recency bias and primacy bias baked in through how positional encodings and attention patterns develop during training. The model “remembers” the beginning (system prompt) and the end (most recent message) most reliably. Middle content — especially in very long contexts — receives diffuse, diluted attention.</p><h3>Designing Around It</h3><pre>from enum import Enum<br><br>class DocumentPlacementStrategy(Enum):<br>    RECENCY_FIRST = &quot;recency_first&quot;       # Most relevant docs at the end<br>    PRIMACY_FIRST = &quot;primacy_first&quot;       # Most relevant docs at the start<br>    SANDWICH = &quot;sandwich&quot;                 # Most relevant at start AND end<br><br>def build_context_with_placement(<br>    system_prompt: str,<br>    documents: list[dict],      # Each: {&quot;content&quot;: str, &quot;relevance_score&quot;: float}<br>    user_query: str,<br>    strategy: DocumentPlacementStrategy = DocumentPlacementStrategy.RECENCY_FIRST,<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;<br>    Arrange documents in context to exploit primacy/recency bias.<br>    <br>    Relevance scores should be pre-computed (e.g., from a retriever).<br>    Higher score = more relevant.<br>    &quot;&quot;&quot;<br>    <br>    # Sort documents by relevance<br>    sorted_docs = sorted(documents, key=lambda d: d[&quot;relevance_score&quot;])<br>    <br>    if strategy == DocumentPlacementStrategy.RECENCY_FIRST:<br>        # Most relevant at the end (recency bias favors retrieval)<br>        ordered = sorted_docs  # Least relevant first, most relevant last<br>        <br>    elif strategy == DocumentPlacementStrategy.PRIMACY_FIRST:<br>        # Most relevant at the start<br>        ordered = sorted_docs[::-1]  # Most relevant first<br>        <br>    elif strategy == DocumentPlacementStrategy.SANDWICH:<br>        # Top-1 at start, rest in middle, top-2 at end<br>        if len(sorted_docs) &gt;= 2:<br>            most_relevant = sorted_docs[-1]<br>            second_most   = sorted_docs[-2]<br>            middle        = sorted_docs[:-2]<br>            ordered = [most_relevant] + middle + [second_most]<br>        else:<br>            ordered = sorted_docs[::-1]<br>    <br>    # Build context string<br>    doc_context = &quot;\n\n&quot;.join([<br>        f&quot;[Document {i+1} | Relevance: {doc[&#39;relevance_score&#39;]:.2f}]\n{doc[&#39;content&#39;]}&quot;<br>        for i, doc in enumerate(ordered)<br>    ])<br>    <br>    return [<br>        {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: f&quot;{doc_context}\n\n---\n\nQuestion: {user_query}&quot;<br>        }<br>    ]<br><br># Usage<br>messages = build_context_with_placement(<br>    system_prompt=&quot;You are a helpful research assistant. Answer based on the documents provided.&quot;,<br>    documents=[<br>        {&quot;content&quot;: &quot;Climate change refers to...&quot;, &quot;relevance_score&quot;: 0.45},<br>        {&quot;content&quot;: &quot;The Paris Agreement of 2015...&quot;, &quot;relevance_score&quot;: 0.91},<br>        {&quot;content&quot;: &quot;Carbon dioxide levels have...&quot;, &quot;relevance_score&quot;: 0.73},<br>        {&quot;content&quot;: &quot;Renewable energy adoption...&quot;, &quot;relevance_score&quot;: 0.58},<br>    ],<br>    user_query=&quot;What international agreements address climate change?&quot;,<br>    strategy=DocumentPlacementStrategy.RECENCY_FIRST,<br>)</pre><p><strong>Practical rule:</strong> For RAG systems, place your highest-relevance retrieved chunks at the <strong>end</strong> of the context (just before the user query). This exploits the recency bias and measurably improves retrieval accuracy.</p><h2>Part 5: Four Context Management Strategies</h2><p>When documents exceed your context window (or cost constraints), you have four strategies. Each has different tradeoffs:</p><h3>Strategy 1: Sliding Window</h3><p>Process the document in overlapping chunks, maintaining a window of context:</p><pre>import asyncio<br>from openai import AsyncOpenAI<br><br>client = AsyncOpenAI()<br><br>async def sliding_window_summarize(<br>    long_text: str,<br>    window_tokens: int = 3_000,<br>    overlap_tokens: int = 500,<br>    model: str = &quot;gpt-4o-mini&quot;,<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Summarize a document too long for a single context window.<br>    Uses overlapping windows to prevent losing information at boundaries.<br>    &quot;&quot;&quot;<br>    import tiktoken<br>    encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br>    <br>    # Tokenize the full text<br>    all_tokens = encoder.encode(long_text)<br>    total_tokens = len(all_tokens)<br>    <br>    print(f&quot;Document: {total_tokens:,} tokens → processing in sliding windows&quot;)<br>    <br>    chunks = []<br>    step = window_tokens - overlap_tokens<br>    <br>    for start in range(0, total_tokens, step):<br>        end = min(start + window_tokens, total_tokens)<br>        chunk_tokens = all_tokens[start:end]<br>        chunk_text = encoder.decode(chunk_tokens)<br>        chunks.append(chunk_text)<br>        <br>        if end &gt;= total_tokens:<br>            break<br>    <br>    print(f&quot;Created {len(chunks)} overlapping windows&quot;)<br>    <br>    # Summarize each window<br>    window_summaries = []<br>    for i, chunk in enumerate(chunks):<br>        response = await client.chat.completions.create(<br>            model=model,<br>            messages=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: &quot;Summarize the key information in this text section. &quot;<br>                               &quot;Be concise but preserve all important facts, names, and decisions.&quot;<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: chunk}<br>            ],<br>            temperature=0.1,<br>            max_tokens=500,<br>        )<br>        summary = response.choices[0].message.content<br>        window_summaries.append(f&quot;[Window {i+1}/{len(chunks)}]\n{summary}&quot;)<br>        print(f&quot;  ✓ Window {i+1}/{len(chunks)} summarized&quot;)<br>    <br>    # Final synthesis<br>    combined = &quot;\n\n&quot;.join(window_summaries)<br>    final_response = await client.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: &quot;You have summaries of consecutive sections of a document. &quot;<br>                           &quot;Synthesize them into a single coherent summary.&quot;<br>            },<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: combined}<br>        ],<br>        temperature=0.2,<br>        max_tokens=1_000,<br>    )<br>    <br>    return final_response.choices[0].message.content</pre><p><strong>Best for:</strong> Long document summarization, processing books, legal document review. <strong>Weakness:</strong> Multiple API calls = higher latency and cost. Information that spans chunk boundaries may be lost.</p><h3>Strategy 2: Map-Reduce</h3><p>Parallel processing of independent chunks, then aggregation:</p><pre>import asyncio<br>from openai import AsyncOpenAI<br>import tiktoken<br><br>client = AsyncOpenAI()<br>encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br><br>def chunk_text(text: str, chunk_size: int = 4_000, overlap: int = 200) -&gt; list[str]:<br>    &quot;&quot;&quot;Split text into overlapping token chunks.&quot;&quot;&quot;<br>    tokens = encoder.encode(text)<br>    chunks = []<br>    step = chunk_size - overlap<br>    <br>    for start in range(0, len(tokens), step):<br>        end = min(start + chunk_size, len(tokens))<br>        chunks.append(encoder.decode(tokens[start:end]))<br>        if end &gt;= len(tokens):<br>            break<br>    <br>    return chunks<br><br>async def map_chunk(chunk: str, query: str, chunk_id: int) -&gt; str:<br>    &quot;&quot;&quot;MAP phase: extract relevant info from one chunk.&quot;&quot;&quot;<br>    response = await client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,  # Use cheaper model for map phase<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: f&quot;Extract information relevant to this query: &#39;{query}&#39;\n&quot;<br>                           &quot;Return only relevant excerpts and facts. &quot;<br>                           &quot;If nothing is relevant, say &#39;No relevant information.&#39;&quot;<br>            },<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: chunk}<br>        ],<br>        temperature=0,<br>        max_tokens=400,<br>    )<br>    result = response.choices[0].message.content<br>    if &quot;No relevant information&quot; not in result:<br>        return f&quot;[Chunk {chunk_id}]\n{result}&quot;<br>    return &quot;&quot;<br><br>async def reduce_results(mapped_results: list[str], query: str) -&gt; str:<br>    &quot;&quot;&quot;REDUCE phase: synthesize all relevant extractions.&quot;&quot;&quot;<br>    relevant = [r for r in mapped_results if r.strip()]<br>    <br>    if not relevant:<br>        return &quot;No relevant information found in the document.&quot;<br>    <br>    combined = &quot;\n\n&quot;.join(relevant)<br>    <br>    response = await client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,  # Use better model for final synthesis<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: &quot;Synthesize these extractions into a comprehensive answer. &quot;<br>                           &quot;Eliminate redundancy. Be direct and accurate.&quot;<br>            },<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: f&quot;Query: {query}\n\nExtractions:\n{combined}&quot;<br>            }<br>        ],<br>        temperature=0.1,<br>        max_tokens=1_000,<br>    )<br>    return response.choices[0].message.content<br><br>async def map_reduce_qa(document: str, query: str, concurrency: int = 10) -&gt; str:<br>    &quot;&quot;&quot;<br>    Answer a query over a long document using MapReduce.<br>    <br>    Processes chunks in parallel — much faster than sequential sliding window<br>    for Q&amp;A tasks.<br>    &quot;&quot;&quot;<br>    chunks = chunk_text(document, chunk_size=4_000)<br>    print(f&quot;MapReduce: {len(chunks)} chunks, query=&#39;{query[:50]}...&#39;&quot;)<br>    <br>    # MAP phase — parallel execution with concurrency limit<br>    semaphore = asyncio.Semaphore(concurrency)<br>    <br>    async def bounded_map(chunk, idx):<br>        async with semaphore:<br>            return await map_chunk(chunk, query, idx)<br>    <br>    mapped = await asyncio.gather(<br>        *[bounded_map(chunk, i+1) for i, chunk in enumerate(chunks)]<br>    )<br>    <br>    relevant_count = sum(1 for r in mapped if r.strip())<br>    print(f&quot;MAP complete: {relevant_count}/{len(chunks)} chunks had relevant info&quot;)<br>    <br>    # REDUCE phase<br>    result = await reduce_results(list(mapped), query)<br>    return result<br><br># Usage<br>answer = asyncio.run(map_reduce_qa(<br>    document=very_long_document,  # Could be 500k+ words<br>    query=&quot;What were the main causes of the 2008 financial crisis?&quot;<br>))</pre><p><strong>Best for:</strong> Q&amp;A over long documents, information extraction, parallel processing where chunks are independent. <strong>Weakness:</strong> Aggregation quality depends on the REDUCE step. Doesn’t maintain narrative continuity.</p><h3>Strategy 3: Hierarchical Summarization</h3><p>Build a tree of summaries — summarize chunks, then summarize summaries:</p><pre>import asyncio<br>from openai import AsyncOpenAI<br>import tiktoken<br>import math<br><br>client = AsyncOpenAI()<br>encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br><br>async def summarize_chunk(text: str, level: int = 0) -&gt; str:<br>    &quot;&quot;&quot;Summarize a text chunk at a given hierarchy level.&quot;&quot;&quot;<br>    instructions = {<br>        0: &quot;Summarize this text section, preserving key facts, decisions, and named entities.&quot;,<br>        1: &quot;Summarize these section summaries into a chapter-level overview.&quot;,<br>        2: &quot;Create an executive summary from these chapter overviews.&quot;,<br>    }<br>    <br>    response = await client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: instructions.get(level, instructions[2])},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: text}<br>        ],<br>        temperature=0.1,<br>        max_tokens=600,<br>    )<br>    return response.choices[0].message.content<br><br>async def hierarchical_summarize(<br>    document: str,<br>    chunk_tokens: int = 3_000,<br>    max_levels: int = 3,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Build a summary tree from bottom up.<br>    <br>    Returns a dict with summaries at each level:<br>    {<br>        &quot;level_0&quot;: [...chunk summaries...],<br>        &quot;level_1&quot;: [...summaries of summaries...],<br>        &quot;final&quot;: &quot;executive summary&quot;<br>    }<br>    &quot;&quot;&quot;<br>    <br>    # Level 0: Chunk the document<br>    tokens = encoder.encode(document)<br>    chunk_token_groups = [<br>        tokens[i:i+chunk_tokens]<br>        for i in range(0, len(tokens), chunk_tokens)<br>    ]<br>    chunks = [encoder.decode(t) for t in chunk_token_groups]<br>    <br>    print(f&quot;Hierarchical summarization: {len(chunks)} chunks, {len(tokens):,} tokens&quot;)<br>    <br>    result = {}<br>    current_summaries = chunks<br>    <br>    for level in range(max_levels):<br>        print(f&quot;Level {level}: Summarizing {len(current_summaries)} items...&quot;)<br>        <br>        # Summarize all items at this level in parallel<br>        summaries = await asyncio.gather(<br>            *[summarize_chunk(text, level) for text in current_summaries]<br>        )<br>        <br>        result[f&quot;level_{level}&quot;] = list(summaries)<br>        <br>        # Check if we&#39;re done (few enough summaries to fit in one context)<br>        total_tokens = sum(len(encoder.encode(s)) for s in summaries)<br>        if total_tokens &lt; 4_000 or len(summaries) &lt;= 3:<br>            # Final synthesis<br>            combined = &quot;\n\n&quot;.join(f&quot;[Section {i+1}]\n{s}&quot; for i, s in enumerate(summaries))<br>            final_summary = await summarize_chunk(combined, level=2)<br>            result[&quot;final&quot;] = final_summary<br>            print(f&quot;Final summary created at level {level}&quot;)<br>            break<br>        <br>        # Group summaries into larger chunks for next level<br>        group_size = math.ceil(math.sqrt(len(summaries)))<br>        grouped = []<br>        for i in range(0, len(summaries), group_size):<br>            group = summaries[i:i+group_size]<br>            grouped.append(&quot;\n\n&quot;.join(group))<br>        <br>        current_summaries = grouped<br>    <br>    return result</pre><p><strong>Best for:</strong> Book summarization, research paper digestion, building document hierarchies for navigation. <strong>Weakness:</strong> Loses fine-grained details at higher levels. Computationally expensive.</p><h3>Strategy 4: Retrieval-Augmented Generation (RAG)</h3><p>Rather than fitting all content in context, <em>retrieve</em> only what’s relevant:</p><pre>from openai import AsyncOpenAI<br>import numpy as np<br><br>client = AsyncOpenAI()<br><br># Simplified RAG context builder<br>async def build_rag_context(<br>    query: str,<br>    document_chunks: list[dict],    # pre-chunked and embedded documents<br>    max_context_tokens: int = 6_000,<br>    top_k: int = 10,<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Build a context from retrieved chunks that fits within token budget.<br>    <br>    document_chunks: [{&quot;text&quot;: str, &quot;embedding&quot;: list[float], &quot;source&quot;: str}]<br>    &quot;&quot;&quot;<br>    <br>    import tiktoken<br>    encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br>    <br>    # 1. Embed the query<br>    query_embedding_response = await client.embeddings.create(<br>        model=&quot;text-embedding-3-small&quot;,<br>        input=query,<br>    )<br>    query_embedding = np.array(query_embedding_response.data[0].embedding)<br>    <br>    # 2. Score all chunks by cosine similarity<br>    def cosine_similarity(a: np.ndarray, b: np.ndarray) -&gt; float:<br>        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))<br>    <br>    scored_chunks = []<br>    for chunk in document_chunks:<br>        chunk_embedding = np.array(chunk[&quot;embedding&quot;])<br>        score = cosine_similarity(query_embedding, chunk_embedding)<br>        scored_chunks.append({**chunk, &quot;score&quot;: score})<br>    <br>    # 3. Sort by relevance, take top_k<br>    scored_chunks.sort(key=lambda x: x[&quot;score&quot;], reverse=True)<br>    top_chunks = scored_chunks[:top_k]<br>    <br>    # 4. Fill context up to token budget (most relevant chunks first)<br>    context_parts = []<br>    used_tokens = 0<br>    <br>    for chunk in top_chunks:<br>        chunk_text = f&quot;[Source: {chunk[&#39;source&#39;]} | Relevance: {chunk[&#39;score&#39;]:.2f}]\n{chunk[&#39;text&#39;]}&quot;<br>        chunk_tokens = len(encoder.encode(chunk_text))<br>        <br>        if used_tokens + chunk_tokens &gt; max_context_tokens:<br>            break<br>        <br>        context_parts.append(chunk_text)<br>        used_tokens += chunk_tokens<br>    <br>    print(f&quot;RAG: Selected {len(context_parts)}/{len(top_chunks)} chunks, &quot;<br>          f&quot;{used_tokens:,}/{max_context_tokens:,} tokens used&quot;)<br>    <br>    # 5. Order by source position (not relevance) to maintain narrative flow<br>    # This counteracts lost-in-the-middle — most relevant was already at position 1<br>    return &quot;\n\n---\n\n&quot;.join(context_parts)<br><br><br># Full RAG pipeline<br>async def rag_answer(query: str, document_chunks: list[dict]) -&gt; str:<br>    context = await build_rag_context(query, document_chunks)<br>    <br>    response = await client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=[<br>            {<br>                &quot;role&quot;: &quot;system&quot;,<br>                &quot;content&quot;: &quot;Answer the question using only the provided context. &quot;<br>                           &quot;If the answer isn&#39;t in the context, say so explicitly.&quot;<br>            },<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: f&quot;Context:\n{context}\n\n---\n\nQuestion: {query}&quot;<br>            }<br>        ],<br>        temperature=0.1,<br>        max_tokens=1_000,<br>    )<br>    return response.choices[0].message.content</pre><p><strong>Best for:</strong> Document Q&amp;A, knowledge base search, enterprise information retrieval. <strong>Weakness:</strong> Only works when the question can be answered by a few relevant passages. Fails for cross-document reasoning.</p><h2>Part 6: Context Management in Multi-Turn Conversations</h2><p>Conversations accumulate context automatically. Without management, a long chat session eventually hits the context limit:</p><pre>from openai import AsyncOpenAI<br>from collections import deque<br>import tiktoken<br><br>client = AsyncOpenAI()<br>encoder = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br><br>class ManagedConversation:<br>    &quot;&quot;&quot;<br>    A multi-turn conversation that automatically manages context length.<br>    <br>    Strategies:<br>    - SLIDING: Drop oldest messages when over budget<br>    - SUMMARIZE: Compress old history into a rolling summary<br>    &quot;&quot;&quot;<br>    <br>    def __init__(<br>        self,<br>        system_prompt: str,<br>        max_context_tokens: int = 100_000,<br>        output_reserve: int = 4_000,<br>        strategy: str = &quot;summarize&quot;,  # &quot;sliding&quot; or &quot;summarize&quot;<br>        model: str = &quot;gpt-4o&quot;,<br>    ):<br>        self.system_prompt = system_prompt<br>        self.max_context_tokens = max_context_tokens<br>        self.output_reserve = output_reserve<br>        self.strategy = strategy<br>        self.model = model<br>        <br>        self.messages: list[dict] = []<br>        self.summary: str = &quot;&quot;  # Rolling summary of compressed history<br>    <br>    def _count_tokens(self, messages: list[dict]) -&gt; int:<br>        total = 4  # Base overhead<br>        for msg in messages:<br>            total += 4 + len(encoder.encode(msg.get(&quot;content&quot;, &quot;&quot;)))<br>        return total<br>    <br>    def _system_messages(self) -&gt; list[dict]:<br>        &quot;&quot;&quot;Build system context including rolling summary if it exists.&quot;&quot;&quot;<br>        content = self.system_prompt<br>        if self.summary:<br>            content += f&quot;\n\n[Conversation summary so far]\n{self.summary}&quot;<br>        return [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: content}]<br>    <br>    async def _compress_history(self) -&gt; None:<br>        &quot;&quot;&quot;Summarize the oldest half of the conversation history.&quot;&quot;&quot;<br>        <br>        half = len(self.messages) // 2<br>        to_compress = self.messages[:half]<br>        self.messages = self.messages[half:]<br>        <br>        # Build a conversation transcript to summarize<br>        transcript = &quot;\n&quot;.join([<br>            f&quot;{m[&#39;role&#39;].upper()}: {m[&#39;content&#39;]}&quot;<br>            for m in to_compress<br>        ])<br>        <br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o-mini&quot;,<br>            messages=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: &quot;Summarize this conversation excerpt concisely. &quot;<br>                               &quot;Preserve key facts, decisions, and user preferences.&quot;<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: transcript}<br>            ],<br>            temperature=0,<br>            max_tokens=500,<br>        )<br>        <br>        new_summary = response.choices[0].message.content<br>        <br>        # Append to existing summary<br>        if self.summary:<br>            self.summary = f&quot;{self.summary}\n\n[Later:]\n{new_summary}&quot;<br>        else:<br>            self.summary = new_summary<br>        <br>        print(f&quot;  [Context manager] Compressed {half} messages into summary&quot;)<br>    <br>    async def chat(self, user_message: str) -&gt; str:<br>        &quot;&quot;&quot;Send a message and get a response, with automatic context management.&quot;&quot;&quot;<br>        <br>        self.messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_message})<br>        <br>        # Check if we&#39;re over budget<br>        all_messages = self._system_messages() + self.messages<br>        current_tokens = self._count_tokens(all_messages)<br>        budget = self.max_context_tokens - self.output_reserve<br>        <br>        if current_tokens &gt; budget:<br>            print(f&quot;  [Context manager] {current_tokens:,} tokens &gt; {budget:,} budget&quot;)<br>            <br>            if self.strategy == &quot;sliding&quot;:<br>                # Drop oldest messages (keep system + last N messages)<br>                while self._count_tokens(self._system_messages() + self.messages) &gt; budget:<br>                    if len(self.messages) &gt; 2:<br>                        dropped = self.messages.pop(0)<br>                        print(f&quot;  [Context manager] Dropped: {dropped[&#39;content&#39;][:50]}...&quot;)<br>                    else:<br>                        break<br>            <br>            elif self.strategy == &quot;summarize&quot;:<br>                await self._compress_history()<br>        <br>        # Make the API call<br>        response = await client.chat.completions.create(<br>            model=self.model,<br>            messages=self._system_messages() + self.messages,<br>            temperature=0.7,<br>            max_tokens=self.output_reserve,<br>        )<br>        <br>        assistant_message = response.choices[0].message.content<br>        self.messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: assistant_message})<br>        <br>        return assistant_message<br>    <br>    def context_report(self) -&gt; dict:<br>        all_messages = self._system_messages() + self.messages<br>        current_tokens = self._count_tokens(all_messages)<br>        return {<br>            &quot;current_tokens&quot;: current_tokens,<br>            &quot;max_tokens&quot;: self.max_context_tokens,<br>            &quot;utilization&quot;: f&quot;{current_tokens / self.max_context_tokens * 100:.1f}%&quot;,<br>            &quot;message_count&quot;: len(self.messages),<br>            &quot;has_summary&quot;: bool(self.summary),<br>            &quot;summary_preview&quot;: self.summary[:100] + &quot;...&quot; if self.summary else None,<br>        }<br><br><br># Usage<br>async def demo_conversation():<br>    convo = ManagedConversation(<br>        system_prompt=&quot;You are a helpful AI assistant.&quot;,<br>        max_context_tokens=50_000,<br>        strategy=&quot;summarize&quot;,<br>    )<br>    <br>    # Simulate a long conversation<br>    exchanges = [<br>        &quot;My name is Alex and I&#39;m building a fintech startup.&quot;,<br>        &quot;We&#39;re focused on B2B payment infrastructure for Southeast Asia.&quot;,<br>        &quot;Our main challenge is regulatory compliance across 6 different markets.&quot;,<br>        &quot;Can you explain the key regulatory frameworks I need to know?&quot;,<br>        # ... many more messages<br>    ]<br>    <br>    for message in exchanges:<br>        response = await convo.chat(message)<br>        print(f&quot;User: {message}&quot;)<br>        print(f&quot;Assistant: {response[:100]}...\n&quot;)<br>    <br>    print(&quot;Context report:&quot;, convo.context_report())</pre><h2>Part 7: Choosing the Right Strategy</h2><pre>Document size        Query type              Recommended strategy<br>──────────────────────────────────────────────────────────────────────────<br>&lt; 50k tokens         Any                     Direct context stuffing<br>                                             (fits in Claude/Gemini window)<br><br>50k – 200k tokens    Q&amp;A, fact lookup        RAG (retrieve top-k chunks)<br>                     Summarization           Hierarchical or Map-Reduce<br><br>200k – 1M tokens     Q&amp;A, extraction         Gemini 2.5 Pro (1M context)<br>                                             or RAG<br><br>&gt; 1M tokens          Any                     RAG required — no model<br>                     (e.g., full codebases)  fits this in context window<br><br>Multi-turn chat      Conversation            Sliding window or summarization<br>                     management              (depending on coherence needs)<br><br>Repeated same doc    Q&amp;A                     Prompt caching (Claude/Gemini)</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Assuming the model read the whole document</strong></p><p>When you send a 100k-token document, the model processes all tokens — but attention is not uniform. Critical information buried in the middle may be de-emphasized. Always test retrieval quality across document positions.</p><p><strong>2. Silent truncation by SDKs</strong></p><p>Some older integrations silently truncate input rather than raising an error. Always count tokens before sending large contexts. Never assume “it fits.”</p><pre># ✅ Always check before sending<br>assert count_tokens(messages) &lt; MODEL_LIMIT - OUTPUT_RESERVE, &quot;Context too large!&quot;</pre><p><strong>3. Forgetting output tokens count against the limit</strong></p><p>The context window includes <em>both</em> input and output. If you send 127,000 tokens to a 128k model and ask for a 2,000-token response, you’ll hit the limit mid-generation.</p><p><strong>4. Using a 1M-context model for everything</strong></p><p>Gemini 2.5 Pro’s 1M context window is impressive — but at 100k+ tokens, costs compound and latency grows. Use it when you genuinely need it. For most Q&amp;A tasks, RAG at 3k-10k tokens is 10–100× cheaper and faster.</p><p><strong>5. Not handling the summarize strategy’s information loss</strong></p><p>Rolling summarization loses details. If a user asks “What did I say about X in our third message?” after 50 compressed messages, the answer may be gone. For applications where conversation history fidelity matters, use a database + retrieval, not summarization.</p><h2>💼 Quick Questions</h2><p><strong>Q: A user complains that your chatbot “forgot” something they mentioned 20 messages ago. What’s the likely cause and how do you fix it?</strong></p><p>The conversation exceeded the context window and oldest messages were dropped (sliding window) or compressed (summarization). The fix depends on the use case: if exact recall is required, store conversation history in a database and retrieve relevant past messages using semantic search before each turn. If approximate recall is acceptable, tune the summarization strategy to preserve key user facts (name, preferences, stated goals) in a structured memory store separate from the rolling summary.</p><p><strong>Q: When would you use long-context stuffing vs RAG?</strong></p><p>Long-context stuffing is appropriate when: (1) the entire document must be read holistically, not by passage retrieval (e.g., understanding the overall argument of a book); (2) cross-document reasoning is required (RAG can only retrieve isolated passages); (3) the document is small enough that cost and latency are acceptable. RAG is superior when: (1) you need cost efficiency at scale; (2) you need low latency; (3) the question can be answered by a few relevant passages; (4) documents are larger than any available context window.</p><p><strong>Q: Explain the lost-in-the-middle problem and how you would design around it in a production RAG system.</strong></p><p>The model’s attention is stronger at the beginning and end of the context. Middle content is retrieved less reliably. Design mitigations: (1) Place highest-relevance chunks at the end of the context (recency bias); (2) Use a “sandwich” layout with the most critical chunk first and second-most-critical last; (3) Limit retrieved chunks to the absolute minimum needed — padding the context with low-relevance filler amplifies the problem; (4) Use reranking to aggressively filter retrieved chunks before building context.</p><h2>🏭 Production Considerations</h2><p><strong>Monitor context utilization in your observability pipeline.</strong> Log prompt_tokens, completion_tokens, and total_tokens from every API response. Alert when average utilization exceeds 80% of the window — approaching the limit causes non-linear quality degradation.</p><p><strong>Token counting is cheap; overruns are expensive.</strong> Adding a tiktoken token count before every API call costs microseconds but prevents context overflow errors in production. Always count before sending.</p><p><strong>Prompt caching is a free optimization for high-volume applications.</strong> If your system prompt or reference documents are shared across many requests (which they almost always are), enabling Anthropic’s prompt caching or Gemini’s implicit caching can reduce costs by 70–90% on those tokens.</p><p><strong>Test your context management strategy against real edge cases.</strong> The message “What did we discuss at the beginning of this conversation?” is a standard test for sliding window failure. Build an eval suite that exercises your context management code before deploying.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>The context window is the entire universe</strong> the model can see — everything outside it does not exist</li><li><strong>Tokens ≠ words</strong> — code, JSON, and technical content use significantly more tokens per “word”</li><li><strong>Long context is expensive</strong> — KV cache size scales linearly; at 1M tokens, you need a GPU cluster just for KV</li><li><strong>Lost in the middle is real</strong> — place your highest-relevance content at the end or beginning of long contexts</li><li><strong>Prompt caching is a free win</strong> — if the same documents appear across many requests, cache them</li><li><strong>Four strategies</strong> for long documents: sliding window, map-reduce, hierarchical summarization, RAG</li><li><strong>Multi-turn conversations need management</strong> — implement sliding window or summarization before you hit the limit, not after</li><li><strong>Count tokens before every large context call</strong> — never rely on the API to tell you when you’ve overflowed</li><li><strong>Match strategy to use case</strong> — 1M context models exist, but cost and latency make them wrong for most tasks</li><li><strong>RAG is the production default for document Q&amp;A</strong> — long context stuffing is for edge cases that require holistic reading</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://arxiv.org/abs/2307.03172">“Lost in the Middle” Paper (Stanford, 2023)</a> — The landmark study on attention degradation in long contexts</li><li><a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching">Anthropic Prompt Caching Guide</a> — How to enable and measure cache savings</li><li><a href="https://github.com/openai/tiktoken">Tiktoken</a> — OpenAI’s tokenizer for accurate token counting</li><li><a href="https://github.com/google-gemini/cookbook">Gemini Long Context Cookbook</a> — Official examples for 1M-context use cases</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=374501ed7ba7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 9: LLM Inference Fundamentals — Temperature, Top-P, and Sampling]]></title>
            <link>https://learncsdesigns.medium.com/day-9-llm-inference-fundamentals-temperature-top-p-and-sampling-92c7f16e9969?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/92c7f16e9969</guid>
            <category><![CDATA[topps]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ai-foundation]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 16:16:22 GMT</pubDate>
            <atom:updated>2026-06-14T16:16:22.325Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“The difference between a poet and a fact-checker isn’t intelligence — it’s entropy. Temperature is how you dial between the two.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XLH1xLiFmqJyTVv0zx8xjw.jpeg" /></figure><h2>Why This Matters</h2><p>You’ve learned to call the API. You’ve sent prompts. You’ve gotten responses. But most developers use the default parameters and wonder why the output is sometimes too random, sometimes too repetitive, sometimes too robotic.</p><p>Sampling parameters are the dials on your mixing board. Get them wrong and your AI application produces outputs that feel wrong to users — even if they can’t articulate why. Get them right and the output feels exactly tailored to the task.</p><p>These parameters also directly affect:</p><ul><li><strong>Output quality</strong>: wrong temperature → incoherent or boring responses</li><li><strong>Reliability</strong>: high randomness in production → unpredictable behavior</li><li><strong>Cost</strong>: verbose outputs cost more tokens</li><li><strong>User trust</strong>: repetitive or hallucinated outputs erode confidence</li></ul><p>Understanding what happens inside the model during generation is the difference between guessing at parameters and making informed engineering decisions.</p><h2>Part 1: How LLMs Actually Generate Text</h2><h3>The Sampling Loop</h3><p>Before you can understand temperature, you need to understand what the model is actually doing when it generates a response. Here’s the process at each step:</p><pre>Input tokens → Transformer layers → Logits vector → Softmax → Probability distribution → Sample → Next token → Repeat</pre><p>Let’s break each step down:</p><p><strong>Step 1: The model produces logits</strong></p><p>After processing your input through all transformer layers, the model outputs a vector of raw scores called <strong>logits</strong> — one score per token in the vocabulary. GPT-4 has a vocabulary of ~100,000 tokens. So for each generation step, the model produces 100,000 numbers.</p><pre># Conceptually (you don&#39;t call this directly in production):<br># logits shape: [vocab_size]  e.g., [100,277]<br><br>logits = [<br>    2.3,   # token &quot; the&quot;<br>    1.8,   # token &quot; a&quot;<br>    4.7,   # token &quot; Paris&quot;<br>    0.1,   # token &quot; London&quot;<br>    -2.1,  # token &quot; banana&quot;<br>    # ... 100,272 more scores<br>]</pre><p><strong>Step 2: Softmax converts logits to probabilities</strong></p><p>The softmax function converts the raw scores into a valid probability distribution that sums to 1.0:</p><pre>import numpy as np<br><br>def softmax(logits):<br>    exp_logits = np.exp(logits - np.max(logits))  # numerical stability<br>    return exp_logits / exp_logits.sum()<br><br># Input logits<br>logits = np.array([2.3, 1.8, 4.7, 0.1, -2.1])<br><br># Output probabilities<br>probs = softmax(logits)<br># [0.038, 0.023, 0.922, 0.004, 0.000]<br># &quot; Paris&quot; has 92.2% probability — very confident</pre><p><strong>Step 3: Sample from the distribution</strong></p><p>The model samples one token from this probability distribution. This is <strong>not argmax</strong> (always picking the highest probability token) — it’s random sampling weighted by probability. This is why running the same prompt twice can give different outputs.</p><pre>import random<br><br>tokens = [&quot; the&quot;, &quot; a&quot;, &quot; Paris&quot;, &quot; London&quot;, &quot; banana&quot;]<br>probs = [0.038, 0.023, 0.922, 0.004, 0.000]<br><br># Weighted random sample<br>next_token = random.choices(tokens, weights=probs, k=1)[0]<br># &quot; Paris&quot; most of the time, but occasionally &quot; the&quot; or &quot; a&quot;</pre><p><strong>Step 4: Append and repeat</strong></p><p>The sampled token is appended to the sequence, and the entire process repeats until the model generates an end-of-sequence token or hits max_tokens.</p><p>This sampling loop is where every parameter you’re about to learn exerts its influence.</p><h2>Part 2: Temperature — The Creativity Dial</h2><h3>What Temperature Does</h3><p>Temperature modifies the logits <em>before</em> the softmax step. It divides each logit by the temperature value:</p><pre>def softmax_with_temperature(logits, temperature):<br>    # Divide logits by temperature BEFORE softmax<br>    scaled_logits = logits / temperature<br>    exp_logits = np.exp(scaled_logits - np.max(scaled_logits))<br>    return exp_logits / exp_logits.sum()</pre><p>The effect is profound:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>logits = np.array([4.7, 2.3, 1.8, 0.1, -2.1])  # Simulated logits<br>tokens = [&quot;Paris&quot;, &quot;Rome&quot;, &quot;London&quot;, &quot;Tokyo&quot;, &quot;Banana&quot;]<br><br>temperatures = [0.1, 0.5, 1.0, 1.5, 2.0]<br><br>for temp in temperatures:<br>    probs = softmax_with_temperature(logits, temp)<br>    formatted = &quot;, &quot;.join(f&quot;{t}:{p:.3f}&quot; for t, p in zip(tokens, probs))<br>    print(f&quot;T={temp}: {formatted}&quot;)</pre><p><strong>Output:</strong></p><pre>T=0.1:  Paris:1.000, Rome:0.000, London:0.000, Tokyo:0.000, Banana:0.000<br>T=0.5:  Paris:0.985, Rome:0.013, London:0.002, Tokyo:0.000, Banana:0.000<br>T=1.0:  Paris:0.834, Rome:0.124, London:0.077, Tokyo:0.011, Banana:0.001<br>T=1.5:  Paris:0.640, Rome:0.182, London:0.148, Tokyo:0.029, Banana:0.002<br>T=2.0:  Paris:0.463, Rome:0.211, London:0.192, Tokyo:0.094, Banana:0.019</pre><p><strong>The intuition:</strong></p><ul><li><strong>Low temperature (→ 0):</strong> Distribution becomes peaky — the highest-probability token dominates. Behavior approaches greedy (deterministic) decoding. Confident, focused, repetitive.</li><li><strong>High temperature (→ ∞):</strong> Distribution flattens — all tokens become roughly equally likely. Behavior becomes random, creative, incoherent at extremes.</li><li><strong>Temperature = 1.0:</strong> The model’s learned distribution, unmodified.</li></ul><h3>Temperature as Entropy</h3><p>If you want the physics framing: temperature in LLMs is directly borrowed from thermodynamics. In statistical mechanics, temperature measures the entropy (disorder) of a system. Higher temperature = more disorder = more randomness in particle behavior. The analogy is exact.</p><p>Low entropy = predictable = low temperature. High entropy = unpredictable = high temperature.</p><h3>Temperature Reference Table</h3><pre>Temperature    Behavior                          Best Use Cases<br>───────────────────────────────────────────────────────────────────<br>0.0 – 0.1    Near-deterministic, greedy         Fact retrieval, structured output,<br>                                                 routing decisions, classification<br>                                                 <br>0.2 – 0.4    Focused, minimal variation         Code generation, SQL, function calls,<br>                                                 math, data extraction, Q&amp;A with facts<br>                                                 <br>0.5 – 0.7    Balanced (recommended default)     General chat, summarization,<br>                                                 analysis, question answering<br>                                                 <br>0.7 – 1.0    Creative, some surprise            Blog writing, marketing copy,<br>                                                 product descriptions, emails<br>                                                 <br>1.0 – 1.3    High creativity, occasional drift  Brainstorming, poetry, story starters,<br>                                                 creative ideation<br>                                                 <br>1.5+         Unpredictable, often incoherent    Experimental use only —<br>                                                 rarely appropriate in production</pre><h3>Code: Temperature in Practice</h3><pre>from openai import AsyncOpenAI<br>import asyncio<br><br>client = AsyncOpenAI()<br><br>async def compare_temperatures(prompt: str, temperatures: list[float]):<br>    &quot;&quot;&quot;Send the same prompt at different temperatures and compare outputs.&quot;&quot;&quot;<br>    <br>    print(f&quot;Prompt: {prompt}\n{&#39;=&#39;*60}&quot;)<br>    <br>    for temp in temperatures:<br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>            temperature=temp,<br>            max_tokens=100,<br>        )<br>        print(f&quot;\nTemperature {temp}:&quot;)<br>        print(response.choices[0].message.content)<br>        print(&quot;-&quot; * 40)<br><br>async def main():<br>    # Creative prompt — notice how outputs diverge at higher temps<br>    await compare_temperatures(<br>        prompt=&quot;Complete this sentence: The old lighthouse keeper noticed something strange in the fog —&quot;,<br>        temperatures=[0.1, 0.7, 1.2],<br>    )<br>    <br>    # Factual prompt — higher temp introduces errors<br>    await compare_temperatures(<br>        prompt=&quot;What is the capital of Australia?&quot;,<br>        temperatures=[0.0, 0.7, 1.5],<br>    )<br><br>asyncio.run(main())</pre><p><strong>Key observation:</strong> On the factual question, temperature 0.0 reliably answers “Canberra.” At 1.5, you may occasionally see “Sydney” (common misconception) or other errors as lower-probability tokens get sampled.</p><h2>Part 3: Top-P (Nucleus Sampling)</h2><h3>What Top-P Does</h3><p>Top-P, also called <strong>nucleus sampling</strong>, provides an alternative to temperature for controlling randomness. Instead of scaling the entire distribution, it dynamically selects the smallest set of tokens whose cumulative probability exceeds P, then samples only from that set.</p><pre>def top_p_sampling(probs: np.ndarray, p: float) -&gt; np.ndarray:<br>    &quot;&quot;&quot;<br>    Zero out probabilities for tokens outside the nucleus.<br>    <br>    The nucleus = smallest set of tokens whose cumulative probability &gt;= p<br>    &quot;&quot;&quot;<br>    # Sort tokens by probability (descending)<br>    sorted_indices = np.argsort(probs)[::-1]<br>    sorted_probs = probs[sorted_indices]<br>    <br>    # Find the nucleus — tokens whose cumulative prob reaches p<br>    cumulative_probs = np.cumsum(sorted_probs)<br>    <br>    # Find the cutoff point<br>    cutoff_idx = np.searchsorted(cumulative_probs, p) + 1<br>    <br>    # Zero out tokens outside the nucleus<br>    filtered_probs = np.zeros_like(probs)<br>    nucleus_indices = sorted_indices[:cutoff_idx]<br>    filtered_probs[nucleus_indices] = probs[nucleus_indices]<br>    <br>    # Renormalize<br>    filtered_probs /= filtered_probs.sum()<br>    <br>    return filtered_probs</pre><p><strong>Walkthrough with numbers:</strong></p><pre>tokens = [&quot;Paris&quot;, &quot;Rome&quot;, &quot;London&quot;, &quot;Berlin&quot;, &quot;Osaka&quot;, &quot;Banana&quot;, ...]<br>probs  = [0.55,   0.20,   0.12,    0.06,    0.04,   0.002,  ...]<br>cumsum = [0.55,   0.75,   0.87,    0.93,    0.97,   0.972,  ...]<br><br># top_p = 0.90<br># Nucleus stops at &quot;London&quot; (cumsum = 0.87 → 0.93 passes 0.90 at &quot;Berlin&quot;)<br># Actually at Berlin: cumsum = 0.93 ≥ 0.90, so nucleus = {Paris, Rome, London, Berlin}<br># &quot;Osaka&quot;, &quot;Banana&quot; and everything after: excluded, probability set to 0<br># Sample only from the nucleus, renormalized</pre><h3>The Key Difference: Top-P Adapts</h3><p>The critical insight about top-P: <strong>the nucleus size adapts dynamically to the model’s confidence.</strong></p><p>When the model is very confident (one token has 95% probability), the nucleus is tiny — just 1–2 tokens. The model stays focused.</p><p>When the model is uncertain (many tokens each with 5–10% probability), the nucleus is large — perhaps 20–30 tokens. The model is given more freedom.</p><p>This is the core advantage over temperature alone: top-P adjusts to the model’s own uncertainty.</p><pre>Model state          top_p=0.9 behavior<br>───────────────────────────────────────<br>Very confident:      Nucleus = 2-3 tokens  → stays focused<br>Somewhat confident:  Nucleus = 5-10 tokens → some variation<br>Uncertain:           Nucleus = 15-20 tokens → explores options<br>Very uncertain:      Nucleus = 30-50 tokens → broad exploration</pre><h3>Top-P Values Guide</h3><pre>Top-P    Effect                            Use Cases<br>──────────────────────────────────────────────────────────────────<br>1.0      No filtering (sample all tokens)  Rarely useful in production<br>0.95     Very light filtering              Creative writing, brainstorming<br>0.9      Standard — recommended default   General purpose, chat<br>0.85     Moderate focus                   Summarization, analysis  <br>0.7      Focused                          Factual Q&amp;A, technical docs<br>0.5      Highly focused                   Structured outputs, classification<br>&lt;0.3     Very narrow nucleus              Near-deterministic (use temp=0 instead)</pre><h2>Part 4: Top-K Sampling</h2><p>Top-K is simpler: keep only the top K highest-probability tokens, zero out the rest, and sample from those K tokens.</p><pre>def top_k_sampling(probs: np.ndarray, k: int) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Keep only the top-k probability tokens.&quot;&quot;&quot;<br>    top_k_indices = np.argsort(probs)[-k:]  # indices of top k tokens<br>    <br>    filtered_probs = np.zeros_like(probs)<br>    filtered_probs[top_k_indices] = probs[top_k_indices]<br>    filtered_probs /= filtered_probs.sum()  # renormalize<br>    <br>    return filtered_probs</pre><p><strong>Top-K vs Top-P comparison:</strong></p><pre>Feature            Top-K                      Top-P<br>────────────────────────────────────────────────────────────<br>Nucleus size       Fixed (always K tokens)    Adaptive (varies by confidence)<br>Vocabulary         Cuts at a fixed count      Cuts at a probability threshold<br>Behavior when      K might still include      P automatically focuses to<br>model is           low-prob noise             just the confident tokens<br>confident<br>Behavior when      K might miss many          P includes all reasonable<br>model is           reasonable tokens          tokens dynamically<br>uncertain<br>Used in            Older generation, Ollama   GPT-4, Claude, Gemini (default)<br>                   LM Studio local models     modern production APIs</pre><p><strong>Top-K with OpenAI-compatible APIs (Ollama):</strong></p><pre>import httpx<br>import json<br>import asyncio<br><br>async def ollama_with_top_k(prompt: str, k: int = 40):<br>    &quot;&quot;&quot;Ollama supports top_k — useful for local model inference.&quot;&quot;&quot;<br>    <br>    async with httpx.AsyncClient() as client:<br>        response = await client.post(<br>            &quot;http://localhost:11434/api/generate&quot;,<br>            json={<br>                &quot;model&quot;: &quot;llama3.2&quot;,<br>                &quot;prompt&quot;: prompt,<br>                &quot;options&quot;: {<br>                    &quot;top_k&quot;: k,          # Keep top-K tokens<br>                    &quot;top_p&quot;: 0.9,        # Then apply nucleus filtering<br>                    &quot;temperature&quot;: 0.7,  # Scale before both filters<br>                    &quot;num_predict&quot;: 200,<br>                },<br>                &quot;stream&quot;: False,<br>            },<br>            timeout=60.0,<br>        )<br>        return response.json()[&quot;response&quot;]<br><br><br>async def main():<br>    response = await ollama_with_top_k(<br>        &quot;Write a short story about AI&quot;,<br>        k=50<br>    )<br>    print(response)<br><br>asyncio.run(main())</pre><h2>Part 5: Frequency and Presence Penalties</h2><p>These OpenAI-specific parameters (also supported by many open-source APIs) modify the logits based on what tokens have <em>already appeared</em> in the generated text.</p><h3>Frequency Penalty</h3><p><strong>Reduces probability of tokens in proportion to how often they’ve already appeared.</strong></p><pre>New logit = Original logit − (frequency_penalty × count_of_token_in_output)</pre><p>Effect: tokens that appear frequently get their probability reduced more severely.</p><pre># Example: if &quot;the&quot; has appeared 5 times so far and frequency_penalty = 0.5:<br># &quot;the&quot; logit is reduced by 0.5 × 5 = 2.5<br># This significantly reduces the probability of &quot;the&quot; appearing again</pre><pre>frequency_penalty    Effect<br>──────────────────────────────────────────────────────────<br>0.0                  No penalty (default)<br>0.3                  Light reduction of repetition<br>0.6                  Moderate — good for longer texts<br>1.0                  Strong — model avoids most repeated tokens<br>2.0                  Very strong — output becomes highly varied<br>                     but may sacrifice coherence</pre><h3>Presence Penalty</h3><p><strong>Reduces probability of tokens that have appeared at all in the output, regardless of how many times.</strong></p><pre>New logit = Original logit − presence_penalty × (1 if token has appeared else 0)</pre><p>Effect: any token that’s appeared at all gets a flat penalty. Encourages the model to introduce new topics and vocabulary.</p><pre>response = await client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Write a blog post about machine learning&quot;}],<br>    temperature=0.7,<br>    frequency_penalty=0.3,  # Reduce repeated words<br>    presence_penalty=0.2,   # Encourage topic diversity<br>    max_tokens=800,<br>)</pre><h3>Penalty Decision Guide</h3><pre>Situation                              Recommendation<br>──────────────────────────────────────────────────────────────────────<br>Short responses (&lt; 100 tokens)         penalties = 0 (negligible benefit)<br>Long-form content (blog posts, docs)   frequency_penalty = 0.3–0.5<br>Repetitive output complaints           frequency_penalty = 0.5–0.7<br>Output stays on one topic too long     presence_penalty = 0.2–0.4<br>Structured output / JSON               penalties = 0 (can corrupt structure)<br>Code generation                        penalties = 0 (syntax must repeat)</pre><h2>Part 6: Repetition Penalty (Open-Source / Ollama)</h2><p>Open-source models (Llama, Mistral, Qwen) use a different parameter called repetition_penalty, which works multiplicatively rather than additively.</p><pre># Ollama / llama.cpp repetition_penalty<br># Values &gt; 1.0 penalize repetition; 1.0 = no penalty<br>async with httpx.AsyncClient() as http_client:<br>    response = await http_client.post(<br>        &quot;http://localhost:11434/api/generate&quot;,<br>        json={<br>            &quot;model&quot;: &quot;llama3.2&quot;,<br>            &quot;prompt&quot;: &quot;Write a story about a robot&quot;,<br>            &quot;options&quot;: {<br>                &quot;temperature&quot;: 0.8,<br>                &quot;top_p&quot;: 0.9,<br>                &quot;top_k&quot;: 40,<br>                &quot;repeat_penalty&quot;: 1.1,      # 1.0 = off, 1.1 = light, 1.3 = strong<br>                &quot;repeat_last_n&quot;: 64,        # How many past tokens to check<br>                &quot;num_predict&quot;: 300,<br>            },<br>            &quot;stream&quot;: False,<br>        }<br>    )</pre><pre>repeat_penalty    Effect<br>──────────────────────────────────────────<br>1.0               No penalty (neutral)<br>1.05 – 1.1        Light — subtle reduction<br>1.1 – 1.2         Moderate — good default<br>1.2 – 1.3         Strong — visible variety<br>1.4+              Very strong — can cause incoherence</pre><h2>Part 7: How Parameters Interact</h2><p>The parameters don’t operate independently — they execute in a specific order on the logit/probability distribution:</p><pre>Raw logits<br>    │<br>    ▼<br>Apply temperature (divide logits by T)<br>    │<br>    ▼<br>Apply frequency/presence penalties (subtract from logits)<br>    │<br>    ▼<br>Convert to probabilities via softmax<br>    │<br>    ▼<br>Apply Top-K filtering (zero out all but top K)<br>    │<br>    ▼<br>Apply Top-P filtering (zero out below nucleus threshold)<br>    │<br>    ▼<br>Renormalize probabilities<br>    │<br>    ▼<br>Sample one token</pre><p><strong>Critical interaction: Temperature + Top-P</strong></p><p>OpenAI recommends: <strong>change either temperature OR top_p, not both simultaneously.</strong> Modifying both creates compounding effects that are hard to reason about.</p><pre># ✅ Correct — change one<br>{&quot;temperature&quot;: 0.3, &quot;top_p&quot;: 1.0}    # Cool temperature, no top-p filtering<br>{&quot;temperature&quot;: 1.0, &quot;top_p&quot;: 0.8}    # Default temperature, nucleus filtering<br><br># ⚠️ Technically works, but harder to reason about<br>{&quot;temperature&quot;: 0.7, &quot;top_p&quot;: 0.9}    # Both active — interactions compound<br><br># In practice, most production teams use temperature alone<br># and only add top_p when they need adaptive nucleus behavior# ⚠️ Technically works, but harder to reason about<br>{&quot;temperature&quot;: 0.7, &quot;top_p&quot;: 0.9}    # Both active — interactions compound</pre><h2>Part 8: Production Parameter Presets</h2><p>Here’s a tested set of parameter presets covering the most common production use cases. Save this as a reference:</p><pre># parameter_presets.py<br>from dataclasses import dataclass<br>from typing import Optional<br><br>@dataclass<br>class SamplingParams:<br>    temperature: float<br>    top_p: float<br>    frequency_penalty: float<br>    presence_penalty: float<br>    max_tokens: Optional[int] = None<br>    description: str = &quot;&quot;<br><br><br>PRESETS = {<br>    <br>    # ── Deterministic / Structured ──────────────────────────────────────────────<br>    &quot;deterministic&quot;: SamplingParams(<br>        temperature=0.0,<br>        top_p=1.0,<br>        frequency_penalty=0.0,<br>        presence_penalty=0.0,<br>        description=&quot;Greedy decoding — same input always gives same output. &quot;<br>                    &quot;Use for: classification, routing, structured JSON, math.&quot;,<br>    ),<br>    <br>    &quot;extraction&quot;: SamplingParams(<br>        temperature=0.1,<br>        top_p=1.0,<br>        frequency_penalty=0.0,<br>        presence_penalty=0.0,<br>        description=&quot;Near-deterministic. Use for: data extraction, NER, &quot;<br>                    &quot;schema filling, fact retrieval.&quot;,<br>    ),<br>    <br>    # ── Code ────────────────────────────────────────────────────────────────────<br>    &quot;code&quot;: SamplingParams(<br>        temperature=0.2,<br>        top_p=0.95,<br>        frequency_penalty=0.0,<br>        presence_penalty=0.0,<br>        description=&quot;Low temperature preserves syntax. Slight top-p prevents &quot;<br>                    &quot;hallucinating unusual APIs. Use for: code generation, SQL, &quot;<br>                    &quot;function calls.&quot;,<br>    ),<br>    <br>    # ── Balanced / General Purpose ───────────────────────────────────────────────<br>    &quot;default&quot;: SamplingParams(<br>        temperature=0.7,<br>        top_p=1.0,<br>        frequency_penalty=0.0,<br>        presence_penalty=0.0,<br>        description=&quot;General purpose. Good starting point for most tasks.&quot;,<br>    ),<br>    <br>    &quot;chat&quot;: SamplingParams(<br>        temperature=0.7,<br>        top_p=0.95,<br>        frequency_penalty=0.1,<br>        presence_penalty=0.0,<br>        description=&quot;Conversational assistant. Light frequency penalty prevents &quot;<br>                    &quot;repetitive phrasing in longer conversations.&quot;,<br>    ),<br>    <br>    # ── Analysis / Writing ───────────────────────────────────────────────────────<br>    &quot;analysis&quot;: SamplingParams(<br>        temperature=0.5,<br>        top_p=0.9,<br>        frequency_penalty=0.2,<br>        presence_penalty=0.0,<br>        description=&quot;Analytical tasks: summarization, document analysis, &quot;<br>                    &quot;report generation. Focused but not robotic.&quot;,<br>    ),<br>    <br>    &quot;writing&quot;: SamplingParams(<br>        temperature=0.8,<br>        top_p=0.95,<br>        frequency_penalty=0.4,<br>        presence_penalty=0.1,<br>        description=&quot;Long-form writing: blog posts, articles, marketing copy. &quot;<br>                    &quot;Frequency penalty prevents word repetition over long outputs.&quot;,<br>    ),<br>    <br>    # ── Creative ────────────────────────────────────────────────────────────────<br>    &quot;creative&quot;: SamplingParams(<br>        temperature=1.0,<br>        top_p=0.95,<br>        frequency_penalty=0.5,<br>        presence_penalty=0.2,<br>        description=&quot;Creative writing: stories, poetry, brainstorming. &quot;<br>                    &quot;High temperature + penalties encourage diverse vocabulary.&quot;,<br>    ),<br>    <br>    &quot;brainstorm&quot;: SamplingParams(<br>        temperature=1.1,<br>        top_p=0.98,<br>        frequency_penalty=0.5,<br>        presence_penalty=0.4,<br>        description=&quot;Ideation: generating diverse options, exploring possibilities. &quot;<br>                    &quot;High presence penalty pushes toward covering new angles.&quot;,<br>    ),<br>}<br><br><br>def apply_preset(preset_name: str, **overrides) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Get parameter dict for a preset, with optional overrides.<br>    <br>    Usage:<br>        params = apply_preset(&quot;code&quot;, max_tokens=500)<br>        response = await client.chat.completions.create(**params, ...)<br>    &quot;&quot;&quot;<br>    if preset_name not in PRESETS:<br>        raise ValueError(f&quot;Unknown preset: {preset_name}. &quot;<br>                         f&quot;Available: {list(PRESETS.keys())}&quot;)<br>    <br>    preset = PRESETS[preset_name]<br>    result = {<br>        &quot;temperature&quot;: preset.temperature,<br>        &quot;top_p&quot;: preset.top_p,<br>        &quot;frequency_penalty&quot;: preset.frequency_penalty,<br>        &quot;presence_penalty&quot;: preset.presence_penalty,<br>    }<br>    <br>    if preset.max_tokens:<br>        result[&quot;max_tokens&quot;] = preset.max_tokens<br>    <br>    result.update(overrides)<br>    return result<br><br><br># ── Usage examples ─────────────────────────────────────────────────────────────<br>async def example_usage():<br>    from openai import AsyncOpenAI<br>    client = AsyncOpenAI()<br>    <br>    # Code generation — low temperature, no penalties<br>    code_params = apply_preset(&quot;code&quot;, max_tokens=500)<br>    response = await client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=[{<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;Write a Python function to parse a JWT token&quot;<br>        }],<br>        **code_params,<br>    )<br>    <br>    # Creative story — high temperature, strong penalties<br>    creative_params = apply_preset(&quot;creative&quot;, max_tokens=600)<br>    story = await client.chat.completions.create(<br>        model=&quot;gpt-4o&quot;,<br>        messages=[{<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;Write the opening paragraph of a noir detective story&quot;<br>        }],<br>        **creative_params,<br>    )</pre><h2>Part 9: Parameters Across Providers</h2><p>Each provider exposes these controls differently. Here’s the unified reference:</p><pre># ── OpenAI ──────────────────────────────────────────────────────────────────────<br>await openai_client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[...],<br>    temperature=0.7,           # 0.0 – 2.0<br>    top_p=0.95,                # 0.0 – 1.0<br>    frequency_penalty=0.3,     # -2.0 – 2.0<br>    presence_penalty=0.1,      # -2.0 – 2.0<br>    max_tokens=500,<br>    n=1,                       # Number of completions to generate<br>    seed=42,                   # For reproducibility (not strict guarantee)<br>)<br><br># ── Anthropic ───────────────────────────────────────────────────────────────────<br>await anthropic_client.messages.create(<br>    model=&quot;claude-sonnet-4-20250514&quot;,<br>    messages=[...],<br>    temperature=0.7,           # 0.0 – 1.0 (note: max is 1.0, not 2.0)<br>    top_p=0.95,                # 0.0 – 1.0<br>    top_k=40,                  # Top-K (Anthropic supports this too)<br>    max_tokens=500,<br>    # Note: No frequency/presence penalties in Anthropic API<br>)<br><br># ── Gemini ───────────────────────────────────────────────────────────────────────<br>model = genai.GenerativeModel(<br>    model_name=&quot;gemini-2.5-flash&quot;,<br>    generation_config=genai.GenerationConfig(<br>        temperature=0.7,           # 0.0 – 2.0<br>        top_p=0.95,                # 0.0 – 1.0<br>        top_k=40,                  # Integer &gt; 0<br>        max_output_tokens=500,<br>        candidate_count=1,         # Number of candidate completions<br>        stop_sequences=[&quot;END&quot;],    # Custom stop strings<br>    )<br>)<br><br># ── Ollama (local models) ─────────────────────────────────────────────────────────<br>{<br>    &quot;model&quot;: &quot;llama3.2&quot;,<br>    &quot;options&quot;: {<br>        &quot;temperature&quot;: 0.7,        # 0.0 – 2.0<br>        &quot;top_p&quot;: 0.9,              # 0.0 – 1.0<br>        &quot;top_k&quot;: 40,               # Positive integer<br>        &quot;repeat_penalty&quot;: 1.1,     # &gt; 1.0 penalizes repetition<br>        &quot;repeat_last_n&quot;: 64,       # Context window for repeat penalty<br>        &quot;num_predict&quot;: 500,        # max_tokens equivalent<br>        &quot;seed&quot;: -1,                # -1 = random, positive = reproducible<br>        &quot;num_ctx&quot;: 4096,           # Context window size<br>        &quot;mirostat&quot;: 0,             # Advanced: Mirostat sampling (0=off, 1, 2)<br>        &quot;mirostat_tau&quot;: 5.0,       # Target entropy (Mirostat only)<br>        &quot;mirostat_eta&quot;: 0.1,       # Learning rate (Mirostat only)<br>    }<br>}</pre><h3>Provider Quirks Worth Knowing</h3><pre>Provider      Temperature max    Penalties           Top-K     Notes<br>─────────────────────────────────────────────────────────────────────────<br>OpenAI        2.0               freq + presence ✓   No        seed for reproducibility<br>Anthropic     1.0               None                Yes       Stricter than OpenAI range<br>Gemini        2.0               None                Yes       candidate_count for beam<br>Ollama        2.0               repeat_penalty ✓    Yes       Mirostat available</pre><h2>Part 10: Determinism and Reproducibility</h2><h3>The Seed Parameter</h3><p>OpenAI provides a seed parameter for reproducibility, but it&#39;s not a hard guarantee:</p><pre># Same seed → usually same output (but not guaranteed across model updates)<br>response1 = await client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Pick a random number from 1-10&quot;}],<br>    temperature=0.7,<br>    seed=42,<br>)<br><br>response2 = await client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Pick a random number from 1-10&quot;}],<br>    temperature=0.7,<br>    seed=42,<br>)<br><br># Check system_fingerprint — if different, model weights changed between calls<br>print(response1.system_fingerprint)  # fp_a2c3b1...<br>print(response2.system_fingerprint)  # fp_a2c3b1... (same = same model)</pre><h3>True Determinism: Temperature = 0</h3><p>If you need genuine reproducibility (evaluation pipelines, unit tests, regression testing), use temperature=0:</p><pre># For evaluation and testing — ALWAYS use temperature=0<br>async def evaluate_model(test_cases: list[dict]) -&gt; list[dict]:<br>    results = []<br>    for test in test_cases:<br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: test[&quot;input&quot;]}],<br>            temperature=0,      # Deterministic — critical for fair evaluation<br>            top_p=1.0,<br>            seed=0,             # Belt and suspenders<br>        )<br>        results.append({<br>            &quot;input&quot;: test[&quot;input&quot;],<br>            &quot;expected&quot;: test[&quot;expected&quot;],<br>            &quot;actual&quot;: response.choices[0].message.content,<br>        })<br>    return results</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Using temperature=1.0 for factual questions</strong></p><p>Temperature 1.0 is the model’s raw, unmodified distribution — built for language modeling, not factual accuracy. For Q&amp;A, use 0.1–0.3. A common symptom: correct answers most of the time, wrong answers occasionally in production.</p><p><strong>2. Using temperature=0 for creative tasks</strong></p><p>At temperature 0, the model produces the single most probable completion. This works great for facts, but creative writing becomes robotic and formulaic. Users notice.</p><p><strong>3. Setting both frequency and presence penalties high</strong></p><p>High frequency_penalty + high presence_penalty together create outputs that desperately avoid any familiar phrasing. The result sounds alien and stilted. Use one at a time, at moderate values.</p><p><strong>4. Not adjusting temperature for structured output (JSON)</strong></p><p>When you need JSON output, use temperature=0 and the response_format={&quot;type&quot;: &quot;json_object&quot;} parameter. Any randomness in structured output causes parsing failures.</p><pre># ✅ Correct — deterministic for structured output<br>response = await client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Extract name and age from: John is 32&quot;}],<br>    temperature=0,                              # Critical<br>    response_format={&quot;type&quot;: &quot;json_object&quot;},   # Ensure valid JSON<br>)</pre><p><strong>5. Ignoring context length when applying penalties</strong></p><p>frequency_penalty and presence_penalty apply to the <em>output so far</em> during generation. For very short outputs (1-3 sentences), penalties have almost no visible effect. Don&#39;t apply penalties to short Q&amp;A responses expecting any difference.</p><p><strong>6. Mixing temperature and top_p both to non-default values</strong></p><p>This creates compound filtering effects that are hard to interpret. Pick one primary control. OpenAI’s own documentation says: “We generally recommend altering this or temperature but not both.”</p><h2>💼 Quick Questions</h2><p><strong>Q: What’s the difference between temperature and top-P at a mechanistic level?</strong></p><p>Temperature scales all logits before softmax, uniformly adjusting the sharpness of the entire distribution. Top-P filters the already-converted probability distribution, removing low-probability tokens entirely. Temperature changes the <em>shape</em> of the distribution; top-P changes its <em>support</em> (which tokens are possible at all). They operate on different stages of the sampling pipeline.</p><p><strong>Q: Why would you use top-P instead of just using a lower temperature?</strong></p><p>Top-P adapts to the model’s confidence dynamically. When the model is very confident (one token has 90% probability), top-P=0.9 samples from just 1–2 tokens. When the model is uncertain (50 tokens each at 2%), top-P=0.9 includes all 50. A fixed temperature can’t adapt this way — it applies the same amount of sharpening regardless of how confident the model already is.</p><p><strong>Q: How would you configure parameters for an AI system that classifies customer support tickets into categories?</strong></p><p>temperature=0, top_p=1.0, no penalties. Classification is deterministic by nature — you want the model&#39;s highest-confidence prediction every time. Any randomness introduces inconsistency across repeated runs on identical tickets. Also add response_format={&quot;type&quot;: &quot;json_object&quot;} and specify the exact category names in the prompt.</p><p><strong>Q: A client complains that their LLM-generated product descriptions are repetitive. What do you investigate?</strong></p><p>First check frequency_penalty — it&#39;s likely 0.0 (default). Increase to 0.3–0.5. Also check if the prompt template is over-constraining the model (tight word-by-word instructions force repetition regardless of penalties). Check output length — short outputs can&#39;t avoid repetition even with penalties active. If using Claude, note there are no frequency penalties; instead craft the prompt to explicitly ask for varied vocabulary.</p><h2>🏭 Production Considerations</h2><p><strong>Evaluation first, parameters second.</strong> Before tuning parameters in production, build an offline evaluation dataset with 50–100 representative examples and a scoring rubric. Tune parameters against that dataset, not live traffic. Parameter changes that feel like improvements in casual testing can silently degrade quality for edge cases.</p><p><strong>Temperature drift in fine-tuned models.</strong> If you fine-tune a model, the learned distribution changes. A temperature that worked well for the base model may produce different behavior on the fine-tuned version. Always re-evaluate parameters after fine-tuning.</p><p><strong>Separate parameter profiles per use case.</strong> Large production applications have multiple AI-powered features: chat, summarization, code generation, classification. Each needs its own parameter preset. Maintain a central params_config.yaml and load it at startup — never hardcode parameters scattered across the codebase.</p><p><strong>Log parameters with every request.</strong> When debugging quality issues, you need to know exactly what parameters produced the output. Log model, temperature, top_p, max_tokens, and any penalties alongside the input/output in your observability pipeline.</p><p><strong>Watch for </strong><strong>finish_reason: length.</strong> If a response ends with finish_reason=length, the model was cut off by max_tokens. Increase max_tokens or the output may be incomplete in ways that are hard to detect automatically.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>LLMs generate text by sampling</strong> from a probability distribution — understanding this makes all other parameters intuitive</li><li><strong>Temperature controls distribution sharpness</strong> — lower = more confident/deterministic, higher = more creative/random</li><li><strong>Top-P adapts the nucleus dynamically</strong> to the model’s own confidence level</li><li><strong>Top-K provides fixed-size filtering</strong> — simpler but less adaptive than top-P</li><li><strong>Frequency penalty reduces repeated tokens</strong> proportionally to how often they’ve appeared</li><li><strong>Presence penalty encourages topic diversity</strong> by penalizing any token that’s appeared at all</li><li><strong>Use temperature=0 for evaluation, classification, and structured output</strong> — never introduce randomness where you need consistency</li><li><strong>Don’t tune both temperature and top-P simultaneously</strong> — pick one primary control</li><li><strong>Different use cases need different presets</strong> — code, creative writing, factual Q&amp;A, and structured output each have optimal settings</li><li><strong>Provider APIs differ in their ranges and supported parameters</strong> — Anthropic’s max temperature is 1.0, not 2.0</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://platform.openai.com/docs/api-reference/chat">OpenAI API Reference — Chat Completions</a> — canonical parameter documentation</li><li><a href="https://arxiv.org/abs/1904.09751">The Nucleus Sampling Paper (2020)</a> — Holtzman et al., original top-P research</li><li><a href="https://huggingface.co/blog/how-to-generate">Hugging Face — How to Generate</a> — excellent visual explanation of sampling strategies</li><li><a href="https://arxiv.org/abs/2306.01708">Temperature and Creativity in LLMs</a> — empirical study of temperature effects on factuality vs creativity</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92c7f16e9969" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 8: Running LLMs Locally with Ollama & LM Studio]]></title>
            <link>https://learncsdesigns.medium.com/day-8-running-llms-locally-with-ollama-lm-studio-f5d0ba562135?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5d0ba562135</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[lm-studio]]></category>
            <category><![CDATA[localai]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 06:59:35 GMT</pubDate>
            <atom:updated>2026-06-07T06:59:35.061Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“The most powerful AI model is the one you fully control — running on your own hardware, your own network, with your own data never leaving the room.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsdHFEx4iLTG4z7VkbEbFA.jpeg" /></figure><h2>Why This Matters</h2><p>For the first three years of the LLM era, running a capable AI model required API keys, internet connectivity, and per-token billing. In 2026, that assumption is broken.</p><p>You can now run models on your laptop that match or exceed GPT-3.5 performance — completely offline, completely free after hardware costs, with zero data leaving your machine.</p><p>This matters for several concrete reasons:</p><p><strong>Privacy.</strong> Medical records, legal documents, internal code, personal journals. Many of the most valuable use cases for AI are also the ones where sending data to a third-party cloud is non-negotiable. Local inference solves this entirely.</p><p><strong>Cost.</strong> At scale, API costs dominate AI product economics. A startup processing millions of documents per month can spend $50,000+ on API fees that could instead run on $10,000 of on-premises hardware.</p><p><strong>Latency.</strong> Cloud API round trips add 200ms–2s of network latency. A local model running on a GPU server in the same datacenter responds in milliseconds.</p><p><strong>Control.</strong> No rate limits. No model deprecation forcing you to migrate. No terms of service updates. No vendor lock-in.</p><p>The tools that made local inference practical for individual developers — Ollama and LM Studio — are the focus of today’s article.</p><h2>Part 1: Understanding Model Quantization</h2><p>Before installing anything, you need to understand the concept that makes local LLMs possible on consumer hardware: <strong>quantization</strong>.</p><h3>What is Quantization?</h3><p>A standard LLM stores each parameter (weight) as a 32-bit floating point number. A 7 billion parameter model therefore requires:</p><pre>7,000,000,000 × 4 bytes (FP32) = 28 GB</pre><p>That’s more than most consumer GPUs hold. But we don’t need FP32 precision for inference. Research shows you can represent weights with much lower precision with minimal quality loss:</p><pre>Precision    Bits per weight   7B model size   Quality loss<br>──────────────────────────────────────────────────────────<br>FP32         32 bits           28 GB           0% (reference)<br>FP16         16 bits           14 GB           ~0%<br>GGUF Q8_0    8 bits             7 GB           ~0.1%<br>GGUF Q6_K    6 bits             5.5 GB         ~0.2%<br>GGUF Q5_K_M  5 bits             4.8 GB         ~0.5%<br>GGUF Q4_K_M  4 bits             4.1 GB         ~1%        ← Sweet spot<br>GGUF Q3_K_M  3 bits             3.3 GB         ~2-3%<br>GGUF Q2_K    2 bits             2.7 GB         ~5%+       ← Avoid</pre><p><strong>GGUF</strong> (GPT-Generated Unified Format) is the file format used by llama.cpp and all tools built on it (Ollama, LM Studio). It packages the quantized weights and metadata into a single file.</p><h3>The K-Quant System</h3><p>The K in Q4_K_M means the quantization uses a per-block method that&#39;s more accurate than simple per-tensor quantization. The letter after _K_ indicates size variant:</p><ul><li>S = Small (more aggressive quantization, smaller)</li><li>M = Medium ← <strong>Most common recommendation</strong></li><li>L = Large (less aggressive, slightly larger but higher quality)</li></ul><h3>Practical Recommendation</h3><p>For most use cases: <strong>Q4_K_M</strong>. It reduces model size by ~7x vs FP16 while maintaining ~99% of the original quality. This is the format Ollama uses by default for most models.</p><h2>Part 2: Hardware Requirements</h2><h3>The RAM Rule</h3><p>The dominant constraint for local LLMs is memory. You need enough <strong>RAM + VRAM</strong> (combined) to hold the model. As a rule of thumb:</p><pre>Model Size     Q4_K_M Required    Min GPU VRAM    Min RAM (CPU only)<br>──────────────────────────────────────────────────────────────────<br>1B params      ~0.8 GB            4 GB            8 GB RAM<br>3B params      ~2 GB              4 GB            8 GB RAM<br>7B params      ~4.1 GB            6 GB            16 GB RAM<br>13B params     ~7.4 GB            8 GB            32 GB RAM<br>32B params     ~19 GB             24 GB           64 GB RAM<br>70B params     ~40 GB             48 GB (2× 24GB) 64 GB RAM</pre><p><strong>Key principle:</strong> If the model fits in VRAM, inference is GPU-accelerated (fast). If it doesn’t, it falls back to CPU/RAM (slow but functional).</p><h3>GPU Recommendations by Budget</h3><pre>Budget       GPU               VRAM    Best Local Models<br>───────────────────────────────────────────────────────────────────────<br>$0            MacBook M-series  Unified  Llama 3.2 3B, Phi-3 Mini<br>$200          RTX 3060          12 GB    Llama 3.1 8B, Mistral 7B<br>$400          RTX 3080          10 GB    Llama 3.1 8B, Qwen 2.5 7B<br>$800          RTX 4070          12 GB    Llama 3.1 8B, Codestral 7B<br>$1500         RTX 4090          24 GB    Llama 3.1 70B Q4, Qwen 72B Q4<br>$3000+        RTX 6000 Ada      48 GB    Full Llama 70B FP16<br>$8000+        2× A6000          96 GB    Llama 3.1 405B Q4</pre><h3>Apple Silicon Advantage</h3><p>Apple M-series chips use unified memory (shared between CPU and GPU), giving them a unique advantage for local LLMs:</p><ul><li>M1 16GB: Runs 7B models well, 13B models slowly</li><li>M2/M3 32GB: Runs 13B models excellently, 34B models adequately</li><li>M3 Max 128GB: Runs 70B models well</li></ul><p>Ollama has native Apple Silicon support with Metal GPU acceleration. LM Studio also has native macOS builds. For Mac users, local LLMs are effectively zero-configuration.</p><h2>Part 3: Ollama</h2><p>Ollama is the dominant tool for local LLM inference as of 2026. It provides:</p><ul><li>A simple CLI to pull, run, and manage models</li><li>An HTTP server with an <strong>OpenAI-compatible API</strong> (drop-in replacement)</li><li>Automatic GPU detection and acceleration</li><li>A curated model library with 100+ models</li></ul><h3>Installation</h3><pre># macOS / Linux<br>curl -fsSL https://ollama.com/install.sh | sh<br><br># Windows<br># Download installer from: https://ollama.com/download<br><br># Verify installation<br>ollama --version<br># ollama version 0.30.x<br><br># Ollama starts as a background service automatically<br># Check it&#39;s running:<br>curl http://localhost:11434/api/version</pre><h3>Pulling and Running Models</h3><pre># Pull a model (downloads to ~/.ollama/models/)<br>ollama pull llama3.2          # 3B — very fast, good for simple tasks<br>ollama pull llama3.1:8b       # 8B — excellent general purpose<br>ollama pull llama3.1:70b      # 70B — near-GPT-4 quality (needs 48GB+ VRAM)<br>ollama pull mistral           # Mistral 7B — fast and capable<br>ollama pull qwen2.5:7b        # Alibaba Qwen 2.5 7B — strong on code<br>ollama pull deepseek-r1:7b    # DeepSeek R1 7B — reasoning model<br>ollama pull phi4              # Microsoft Phi-4 — 14B, punches above weight<br>ollama pull nomic-embed-text  # Embedding model for RAG<br>ollama pull codellama:13b     # Meta&#39;s code-specialized model<br><br># Run interactively<br>ollama run llama3.1:8b<br><br># Single-turn from CLI<br>ollama run llama3.1:8b &quot;What is the difference between TCP and UDP?&quot;<br><br># List downloaded models<br>ollama list<br><br># Remove a model<br>ollama rm llama3.2<br><br># Show model info (parameters, quantization, etc.)<br>ollama show llama3.1:8b</pre><h3>The Ollama API</h3><p>Once Ollama is running, it exposes an HTTP API on localhost:11434. There are two API styles:</p><p><strong>1. Native Ollama API:</strong></p><pre>import requests<br>import json<br><br>OLLAMA_BASE = &quot;http://localhost:11434&quot;<br><br>def chat_ollama(<br>    prompt: str,<br>    model: str = &quot;llama3.1:8b&quot;,<br>    system: str = None,<br>    stream: bool = False,<br>) -&gt; str:<br>    &quot;&quot;&quot;Call Ollama native chat API.&quot;&quot;&quot;<br>    messages = []<br>    if system:<br>        messages.append({&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system})<br>    messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br><br>    response = requests.post(<br>        f&quot;{OLLAMA_BASE}/api/chat&quot;,<br>        json={<br>            &quot;model&quot;: model,<br>            &quot;messages&quot;: messages,<br>            &quot;stream&quot;: stream,<br>            &quot;options&quot;: {<br>                &quot;temperature&quot;: 0.7,<br>                &quot;top_p&quot;: 0.9,<br>                &quot;num_ctx&quot;: 8192,        # Context window size<br>                &quot;num_predict&quot;: 2048,    # Max tokens to generate<br>            },<br>        },<br>        stream=stream,<br>        timeout=120,<br>    )<br>    response.raise_for_status()<br><br>    if stream:<br>        full_response = []<br>        for line in response.iter_lines():<br>            if line:<br>                chunk = json.loads(line)<br>                if chunk.get(&quot;message&quot;, {}).get(&quot;content&quot;):<br>                    content = chunk[&quot;message&quot;][&quot;content&quot;]<br>                    print(content, end=&quot;&quot;, flush=True)<br>                    full_response.append(content)<br>                if chunk.get(&quot;done&quot;):<br>                    print()  # newline<br>                    break<br>        return &quot;&quot;.join(full_response)<br>    else:<br>        return response.json()[&quot;message&quot;][&quot;content&quot;]<br><br><br># Usage<br>response = chat_ollama(<br>    &quot;Explain quicksort in Python with code.&quot;,<br>    model=&quot;llama3.1:8b&quot;,<br>    system=&quot;You are a concise technical educator. Always include runnable code.&quot;,<br>)<br>print(response)<br><br># Streaming<br>chat_ollama(<br>    &quot;Write a haiku about machine learning&quot;,<br>    model=&quot;llama3.1:8b&quot;,<br>    stream=True,<br>)</pre><p><strong>2. OpenAI-Compatible API (recommended for production):</strong></p><p>Ollama exposes an OpenAI-compatible endpoint at /v1/. This means you can use the official openai Python library with zero code changes — just point it at localhost:</p><pre>from openai import OpenAI<br><br># Drop-in replacement: just change base_url<br>client = OpenAI(<br>    base_url=&quot;http://localhost:11434/v1&quot;,<br>    api_key=&quot;ollama&quot;,  # Required by the library but ignored by Ollama<br>)<br><br>def chat_local(<br>    prompt: str,<br>    model: str = &quot;llama3.1:8b&quot;,<br>    system: str = &quot;You are a helpful assistant.&quot;,<br>    temperature: float = 0.7,<br>    max_tokens: int = 2048,<br>) -&gt; str:<br>    &quot;&quot;&quot;Chat with a local Ollama model using the OpenAI SDK.&quot;&quot;&quot;<br>    response = client.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>        ],<br>        temperature=temperature,<br>        max_tokens=max_tokens,<br>    )<br>    return response.choices[0].message.content<br><br><br># Works exactly like the OpenAI API<br>result = chat_local(<br>    &quot;What are the SOLID principles in software engineering?&quot;,<br>    model=&quot;llama3.1:8b&quot;,<br>)<br>print(result)<br><br># Streaming also works identically<br>stream = client.chat.completions.create(<br>    model=&quot;llama3.1:8b&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;List 5 Python best practices.&quot;}],<br>    stream=True,<br>)<br><br>for chunk in stream:<br>    if chunk.choices[0].delta.content:<br>        print(chunk.choices[0].delta.content, end=&quot;&quot;, flush=True)<br>print()</pre><h3>Generating Embeddings with Ollama</h3><pre>from openai import OpenAI<br>from typing import List<br>import numpy as np<br><br>embedding_client = OpenAI(<br>    base_url=&quot;http://localhost:11434/v1&quot;,<br>    api_key=&quot;ollama&quot;,<br>)<br><br>def embed_text(texts: List[str], model: str = &quot;nomic-embed-text&quot;) -&gt; List[List[float]]:<br>    &quot;&quot;&quot;<br>    Generate embeddings locally using Ollama.<br>    <br>    nomic-embed-text: 768 dimensions, good general-purpose embeddings<br>    mxbai-embed-large: 1024 dimensions, higher quality<br>    all-minilm: 384 dimensions, very fast<br>    &quot;&quot;&quot;<br>    response = embedding_client.embeddings.create(<br>        model=model,<br>        input=texts,<br>    )<br>    return [item.embedding for item in response.data]<br><br><br># Example: similarity search without cloud<br>texts = [<br>    &quot;Python is a high-level programming language&quot;,<br>    &quot;Machine learning uses statistical techniques&quot;,<br>    &quot;Snakes are reptiles found worldwide&quot;,<br>    &quot;Neural networks are inspired by the brain&quot;,<br>]<br><br>embeddings = embed_text(texts)<br><br># Cosine similarity<br>def cosine_similarity(a: List[float], b: List[float]) -&gt; float:<br>    a_np, b_np = np.array(a), np.array(b)<br>    return float(np.dot(a_np, b_np) / (np.linalg.norm(a_np) * np.linalg.norm(b_np)))<br><br>query = &quot;deep learning algorithms&quot;<br>query_embedding = embed_text([query])[0]<br><br>scores = [(texts[i], cosine_similarity(query_embedding, embeddings[i])) for i in range(len(texts))]<br>scores.sort(key=lambda x: x[1], reverse=True)<br><br>for text, score in scores:<br>    print(f&quot;{score:.3f}: {text}&quot;)<br># 0.724: Neural networks are inspired by the brain<br># 0.689: Machine learning uses statistical techniques<br># 0.423: Python is a high-level programming language<br># 0.198: Snakes are reptiles found worldwide</pre><h3>Custom Modelfiles: Customizing Behavior</h3><p>Ollama’s Modelfile is like a Dockerfile for LLMs — it lets you bake in system prompts, custom parameters, and even your own GGUF files:</p><pre># Create a custom Modelfile<br>cat &gt; Modelfile &lt;&lt; &#39;EOF&#39;<br># Base model<br>FROM llama3.1:8b<br><br># Baked-in system prompt — this applies to every conversation<br>SYSTEM &quot;&quot;&quot;<br>You are Aria, a senior Python engineer at a top tech company.<br>Your responses are:<br>- Concise but complete<br>- Always include working code examples<br>- Focused on production-quality patterns, not toy examples<br>- You flag potential issues (performance, security, edge cases)<br>Never pad responses or explain basic concepts unless asked.<br>&quot;&quot;&quot;<br><br># Generation parameters<br>PARAMETER temperature 0.3<br>PARAMETER top_p 0.9<br>PARAMETER num_ctx 16384<br>PARAMETER num_predict 4096<br>PARAMETER repeat_penalty 1.1<br>EOF<br><br># Build the custom model<br>ollama create aria-engineer -f Modelfile<br><br># Run your custom model<br>ollama run aria-engineer &quot;How do I implement connection pooling in PostgreSQL?&quot;<br><br># List it alongside other models<br>ollama list</pre><pre># Modelfile with a custom GGUF (bring your own model)<br>cat &gt; Modelfile.custom &lt;&lt; &#39;EOF&#39;<br># Use a GGUF file you downloaded from Hugging Face<br>FROM ./models/custom-model-Q4_K_M.gguf<br><br>SYSTEM &quot;You are a domain expert in financial analysis.&quot;<br><br>PARAMETER temperature 0.1<br>PARAMETER num_ctx 32768<br>EOF<br><br>ollama create finance-expert -f Modelfile.custom</pre><h3>Running Ollama as a Server</h3><p>For production deployments, you often want Ollama accessible over a network:</p><pre># Run Ollama server bound to all interfaces (network accessible)<br>OLLAMA_HOST=0.0.0.0:11434 ollama serve<br><br># With GPU configuration<br>CUDA_VISIBLE_DEVICES=0,1 OLLAMA_HOST=0.0.0.0:11434 ollama serve<br><br># Environment variables<br>OLLAMA_MODELS=/mnt/ssd/models      # Custom model storage path<br>OLLAMA_NUM_PARALLEL=4              # Concurrent requests per model<br>OLLAMA_MAX_LOADED_MODELS=3         # Models to keep warm in memory<br>OLLAMA_KEEP_ALIVE=10m              # How long to keep model in memory</pre><pre>import httpx<br>import json<br>import re<br>from typing import Optional, List, Dict<br>from tenacity import retry, stop_after_attempt, wait_exponential<br><br><br>class OllamaClient:<br>    &quot;&quot;&quot;<br>    Production-ready Ollama client.<br><br>    Features:<br>    - Connection pooling<br>    - Retry logic<br>    - Health checks<br>    - Model management<br>    - Token usage tracking<br>    - Markdown/code fence cleanup<br>    - JSON output support<br>    - Context manager support<br>    &quot;&quot;&quot;<br><br>    def __init__(<br>        self,<br>        base_url: str = &quot;http://localhost:11434&quot;,<br>        default_model: str = &quot;llama3.1:8b&quot;,<br>        timeout: float = 120.0,<br>    ):<br>        self.base_url = base_url<br>        self.default_model = default_model<br><br>        self._client = httpx.Client(<br>            base_url=base_url,<br>            timeout=timeout,<br>            limits=httpx.Limits(<br>                max_connections=20,<br>                max_keepalive_connections=10,<br>            ),<br>        )<br><br>        self._total_completion_tokens = 0<br>        self._total_prompt_tokens = 0<br>        self._request_count = 0<br><br>    # ----------------------------------------------------<br>    # Context manager support<br>    # ----------------------------------------------------<br><br>    def __enter__(self):<br>        return self<br><br>    def __exit__(self, exc_type, exc_val, exc_tb):<br>        self.close()<br><br>    # ----------------------------------------------------<br>    # Utilities<br>    # ----------------------------------------------------<br><br>    def _clean_response(self, text: str) -&gt; str:<br>        &quot;&quot;&quot;<br>        Remove markdown code fences.<br>        &quot;&quot;&quot;<br><br>        text = text.strip()<br><br>        text = re.sub(<br>            r&quot;^```[a-zA-Z0-9_+-]*\n&quot;,<br>            &quot;&quot;,<br>            text,<br>        )<br><br>        text = re.sub(<br>            r&quot;\n```$&quot;,<br>            &quot;&quot;,<br>            text,<br>        )<br><br>        return text.strip()<br><br>    # ----------------------------------------------------<br>    # Health<br>    # ----------------------------------------------------<br><br>    def is_healthy(self) -&gt; bool:<br>        &quot;&quot;&quot;<br>        Check whether Ollama server is running.<br>        &quot;&quot;&quot;<br><br>        try:<br>            resp = self._client.get(&quot;/api/version&quot;)<br>            return resp.status_code == 200<br>        except Exception:<br>            return False<br><br>    # ----------------------------------------------------<br>    # Models<br>    # ----------------------------------------------------<br><br>    def list_models(self) -&gt; List[str]:<br>        &quot;&quot;&quot;<br>        List installed models.<br>        &quot;&quot;&quot;<br><br>        resp = self._client.get(&quot;/api/tags&quot;)<br>        resp.raise_for_status()<br><br>        return [<br>            model[&quot;name&quot;]<br>            for model in resp.json().get(&quot;models&quot;, [])<br>        ]<br><br>    def pull_model(self, model: str) -&gt; None:<br>        &quot;&quot;&quot;<br>        Download model if not already installed.<br>        &quot;&quot;&quot;<br><br>        if model in self.list_models():<br>            print(f&quot;✓ {model} already installed&quot;)<br>            return<br><br>        print(f&quot;Pulling {model}...\n&quot;)<br><br>        with self._client.stream(<br>            &quot;POST&quot;,<br>            &quot;/api/pull&quot;,<br>            json={&quot;name&quot;: model},<br>        ) as response:<br><br>            response.raise_for_status()<br><br>            for line in response.iter_lines():<br>                if not line:<br>                    continue<br><br>                data = json.loads(line)<br><br>                if &quot;status&quot; in data:<br>                    print(<br>                        f&quot;\r{data[&#39;status&#39;]}&quot;,<br>                        end=&quot;&quot;,<br>                        flush=True,<br>                    )<br><br>        print(f&quot;\n✓ {model} ready&quot;)<br><br>    # ----------------------------------------------------<br>    # Chat<br>    # ----------------------------------------------------<br><br>    @retry(<br>        stop=stop_after_attempt(3),<br>        wait=wait_exponential(<br>            multiplier=1,<br>            min=1,<br>            max=10,<br>        ),<br>    )<br>    def chat(<br>        self,<br>        prompt: str,<br>        model: Optional[str] = None,<br>        system: Optional[str] = None,<br>        context_messages: Optional[List[Dict]] = None,<br>        temperature: float = 0.7,<br>        clean_output: bool = True,<br>    ) -&gt; str:<br>        &quot;&quot;&quot;<br>        Standard chat completion.<br>        &quot;&quot;&quot;<br><br>        model = model or self.default_model<br><br>        messages = []<br><br>        if system:<br>            messages.append(<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: system,<br>                }<br>            )<br><br>        if context_messages:<br>            messages.extend(context_messages)<br><br>        messages.append(<br>            {<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: prompt,<br>            }<br>        )<br><br>        resp = self._client.post(<br>            &quot;/api/chat&quot;,<br>            json={<br>                &quot;model&quot;: model,<br>                &quot;messages&quot;: messages,<br>                &quot;stream&quot;: False,<br>                &quot;options&quot;: {<br>                    &quot;temperature&quot;: temperature,<br>                },<br>            },<br>        )<br><br>        resp.raise_for_status()<br><br>        data = resp.json()<br><br>        self._request_count += 1<br>        self._total_completion_tokens += data.get(<br>            &quot;eval_count&quot;,<br>            0,<br>        )<br><br>        self._total_prompt_tokens += data.get(<br>            &quot;prompt_eval_count&quot;,<br>            0,<br>        )<br><br>        content = data[&quot;message&quot;][&quot;content&quot;]<br><br>        if clean_output:<br>            content = self._clean_response(content)<br><br>        return content<br><br>    # ----------------------------------------------------<br>    # Structured JSON output<br>    # ----------------------------------------------------<br><br>    def chat_json(<br>        self,<br>        prompt: str,<br>        schema_prompt: str,<br>        model: Optional[str] = None,<br>    ) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Force JSON response.<br>        &quot;&quot;&quot;<br><br>        model = model or self.default_model<br><br>        resp = self._client.post(<br>            &quot;/api/chat&quot;,<br>            json={<br>                &quot;model&quot;: model,<br>                &quot;messages&quot;: [<br>                    {<br>                        &quot;role&quot;: &quot;system&quot;,<br>                        &quot;content&quot;: schema_prompt,<br>                    },<br>                    {<br>                        &quot;role&quot;: &quot;user&quot;,<br>                        &quot;content&quot;: prompt,<br>                    },<br>                ],<br>                &quot;stream&quot;: False,<br>                &quot;format&quot;: &quot;json&quot;,<br>            },<br>        )<br><br>        resp.raise_for_status()<br><br>        data = resp.json()<br><br>        return json.loads(<br>            data[&quot;message&quot;][&quot;content&quot;]<br>        )<br><br>    # ----------------------------------------------------<br>    # Embeddings<br>    # ----------------------------------------------------<br><br>    def embed(<br>        self,<br>        text: str,<br>        model: str = &quot;nomic-embed-text&quot;,<br>    ):<br>        &quot;&quot;&quot;<br>        Generate embeddings.<br>        &quot;&quot;&quot;<br><br>        resp = self._client.post(<br>            &quot;/api/embeddings&quot;,<br>            json={<br>                &quot;model&quot;: model,<br>                &quot;prompt&quot;: text,<br>            },<br>        )<br><br>        resp.raise_for_status()<br><br>        return resp.json()[&quot;embedding&quot;]<br><br>    # ----------------------------------------------------<br>    # Usage stats<br>    # ----------------------------------------------------<br><br>    @property<br>    def total_tokens_generated(self) -&gt; int:<br>        return self._total_completion_tokens<br><br>    @property<br>    def total_prompt_tokens(self) -&gt; int:<br>        return self._total_prompt_tokens<br><br>    @property<br>    def request_count(self) -&gt; int:<br>        return self._request_count<br><br>    # ----------------------------------------------------<br>    # Cleanup<br>    # ----------------------------------------------------<br><br>    def close(self):<br>        self._client.close()<br><br><br># ==========================================================<br># Example Usage<br># ==========================================================<br><br>if __name__ == &quot;__main__&quot;:<br><br>    with OllamaClient(<br>        default_model=&quot;llama3.1:8b&quot;<br>    ) as ollama:<br><br>        if not ollama.is_healthy():<br>            raise RuntimeError(<br>                &quot;Ollama is not running. Start it using:\n\nollama serve&quot;<br>            )<br><br>        print(<br>            &quot;Available models:&quot;,<br>            ollama.list_models(),<br>        )<br><br>        response = ollama.chat(<br>            prompt=&quot;Write a Python function to validate email addresses.&quot;,<br>            system=&quot;&quot;&quot;<br>You are a Python expert.<br><br>Rules:<br>- Return raw Python code only.<br>- No markdown.<br>- No triple backticks.<br>- No explanations.<br>- First character must be Python code.<br>&quot;&quot;&quot;,<br>            temperature=0,<br>        )<br><br>        print(&quot;\nResponse:\n&quot;)<br>        print(response)<br><br>        print(<br>            f&quot;\nCompletion Tokens: {ollama.total_tokens_generated:,}&quot;<br>        )<br><br>        print(<br>            f&quot;Prompt Tokens: {ollama.total_prompt_tokens:,}&quot;<br>        )<br><br>        print(<br>            f&quot;Requests Made: {ollama.request_count:,}&quot;<br>        )</pre><h2>Part 4: LM Studio</h2><p>LM Studio is a GUI application for downloading, managing, and serving local models. It’s ideal for:</p><ul><li>Non-CLI users who want a visual interface</li><li>Quickly comparing multiple models without writing code</li><li>Teams that need a shared local inference server</li><li>Exploring model capabilities interactively before integrating into code</li></ul><h3>Installation</h3><p>Download from <a href="https://lmstudio.ai/">lmstudio.ai</a> — available for macOS, Windows, and Linux.</p><h3>Key Features</h3><p><strong>Model Browser:</strong> LM Studio has a built-in Hugging Face model browser. Search, filter by VRAM requirement, and download GGUF models directly from the app.</p><p><strong>Chat Interface:</strong> Test models with a full chat UI, adjustable parameters (temperature, context window, system prompt), and response timing metrics.</p><p><strong>Local Server:</strong> LM Studio runs an OpenAI-compatible server on localhost:1234. Any code that works with the OpenAI API works with LM Studio.</p><h3>Using LM Studio’s Server via Python</h3><pre>from openai import OpenAI<br><br># LM Studio&#39;s OpenAI-compatible server<br>lm_client = OpenAI(<br>    base_url=&quot;http://localhost:1234/v1&quot;,<br>    api_key=&quot;lm-studio&quot;,  # Placeholder — not validated<br>)<br><br>def chat_lm_studio(<br>    prompt: str,<br>    system: str = &quot;You are a helpful assistant.&quot;,<br>    model: str = &quot;local-model&quot;,  # LM Studio uses &quot;local-model&quot; as default<br>    temperature: float = 0.7,<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Chat with whatever model is loaded in LM Studio.<br>    <br>    Note: The model parameter is ignored by LM Studio — it always uses<br>    the currently loaded model. Pass it for API compatibility only.<br>    &quot;&quot;&quot;<br>    response = lm_client.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>        ],<br>        temperature=temperature,<br>        max_tokens=2048,<br>    )<br>    return response.choices[0].message.content<br><br><br># Get list of loaded models<br>models = lm_client.models.list()<br>print(&quot;Models available in LM Studio:&quot;)<br>for model in models.data:<br>    print(f&quot;  - {model.id}&quot;)<br><br><br># Example: technical writing assistant<br>result = chat_lm_studio(<br>    &quot;Explain REST vs GraphQL vs gRPC for a senior engineer.&quot;,<br>    system=&quot;You are a senior backend architect. Be concise and opinionated.&quot;,<br>)<br>print(result)</pre><h3>LM Studio Configuration Tips</h3><p>In the <strong>Server Settings</strong> panel:</p><ul><li><strong>Context Length</strong>: Set to match your model’s supported context (8192, 16384, 32768)</li><li><strong>GPU Layers</strong>: Set to maximum — this offloads as many transformer layers as possible to GPU</li><li><strong>CPU Threads</strong>: Set to physical core count (not hyperthreads)</li><li><strong>Flash Attention</strong>: Enable if your hardware supports it (M2/M3, RTX 30xx+) — significant speedup</li></ul><h2>Part 5: Unified Local + Cloud Client</h2><p>The most production-ready pattern is a unified client that tries local inference first and falls back to a cloud API if the local model isn’t available or the task requires higher capability.</p><pre>&quot;&quot;&quot;<br>Unified LLM Client<br>==================<br><br>Features:<br>- Local-first routing (Ollama -&gt; LM Studio -&gt; OpenAI -&gt; Anthropic)<br>- Connection pooling<br>- Retry with exponential backoff<br>- Health checks<br>- Usage statistics<br>- Environment variable configuration via .env<br><br>Requirements:<br>pip install openai anthropic httpx python-dotenv tenacity<br><br>Example .env:<br><br>OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxx<br>ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxx<br><br>OLLAMA_BASE_URL=http://localhost:11434/v1<br>OLLAMA_API_KEY=ollama<br><br>LM_STUDIO_BASE_URL=http://localhost:1234/v1<br>LM_STUDIO_API_KEY=lm-studio<br><br>LOCAL_MODEL=llama3.1:8b<br>CLOUD_MODEL=gpt-4o-mini<br><br>LOCAL_TIMEOUT=60<br>CLOUD_TIMEOUT=30<br><br>PREFER_LOCAL=true<br>&quot;&quot;&quot;<br><br>from __future__ import annotations<br><br>import logging<br>import os<br>import time<br>from dataclasses import dataclass<br>from enum import Enum<br>from typing import Dict, List, Optional, Tuple<br><br>import httpx<br>from anthropic import Anthropic<br>from dotenv import load_dotenv<br>from openai import OpenAI<br>from tenacity import retry, stop_after_attempt, wait_exponential<br><br># ------------------------------------------------------------------------------<br># Load Environment Variables<br># ------------------------------------------------------------------------------<br><br>load_dotenv()<br><br># ------------------------------------------------------------------------------<br># Logging<br># ------------------------------------------------------------------------------<br><br>logging.basicConfig(<br>    level=logging.INFO,<br>    format=&quot;%(asctime)s | %(levelname)s | %(message)s&quot;,<br>)<br><br>logger = logging.getLogger(__name__)<br><br># ------------------------------------------------------------------------------<br># Enums<br># ------------------------------------------------------------------------------<br><br><br>class InferenceBackend(Enum):<br>    OLLAMA = &quot;ollama&quot;<br>    LM_STUDIO = &quot;lm_studio&quot;<br>    OPENAI = &quot;openai&quot;<br>    ANTHROPIC = &quot;anthropic&quot;<br><br><br># ------------------------------------------------------------------------------<br># Config<br># ------------------------------------------------------------------------------<br><br><br>@dataclass<br>class ModelConfig:<br>    local_model: str = os.getenv(&quot;LOCAL_MODEL&quot;, &quot;llama3.1:8b&quot;)<br>    cloud_model: str = os.getenv(&quot;CLOUD_MODEL&quot;, &quot;gpt-4o-mini&quot;)<br>    prefer_local: bool = (<br>        os.getenv(&quot;PREFER_LOCAL&quot;, &quot;true&quot;).lower() == &quot;true&quot;<br>    )<br>    local_timeout: float = float(os.getenv(&quot;LOCAL_TIMEOUT&quot;, &quot;60&quot;))<br>    cloud_timeout: float = float(os.getenv(&quot;CLOUD_TIMEOUT&quot;, &quot;30&quot;))<br><br><br># ------------------------------------------------------------------------------<br># Client<br># ------------------------------------------------------------------------------<br><br><br>class UnifiedLLMClient:<br>    &quot;&quot;&quot;<br>    Production-ready LLM Router<br><br>    Routing Strategy:<br>        Ollama<br>          ↓<br>        LM Studio<br>          ↓<br>        OpenAI<br>          ↓<br>        Anthropic<br>    &quot;&quot;&quot;<br><br>    def __init__(self, config: Optional[ModelConfig] = None):<br>        self.config = config or ModelConfig()<br><br>        # Shared HTTP client for pooling<br>        self.http_client = httpx.Client(<br>            timeout=self.config.local_timeout,<br>            limits=httpx.Limits(<br>                max_connections=50,<br>                max_keepalive_connections=20,<br>            ),<br>        )<br><br>        # ------------------------------------------------------------------<br>        # Local Clients<br>        # ------------------------------------------------------------------<br><br>        self._ollama = OpenAI(<br>            base_url=os.getenv(<br>                &quot;OLLAMA_BASE_URL&quot;,<br>                &quot;http://localhost:11434/v1&quot;,<br>            ),<br>            api_key=os.getenv(<br>                &quot;OLLAMA_API_KEY&quot;,<br>                &quot;ollama&quot;,<br>            ),<br>            timeout=self.config.local_timeout,<br>            http_client=self.http_client,<br>        )<br><br>        self._lm_studio = OpenAI(<br>            base_url=os.getenv(<br>                &quot;LM_STUDIO_BASE_URL&quot;,<br>                &quot;http://localhost:1234/v1&quot;,<br>            ),<br>            api_key=os.getenv(<br>                &quot;LM_STUDIO_API_KEY&quot;,<br>                &quot;lm-studio&quot;,<br>            ),<br>            timeout=self.config.local_timeout,<br>            http_client=self.http_client,<br>        )<br><br>        # ------------------------------------------------------------------<br>        # Lazy Cloud Clients<br>        # ------------------------------------------------------------------<br><br>        self._openai: Optional[OpenAI] = None<br>        self._anthropic: Optional[Anthropic] = None<br><br>        # ------------------------------------------------------------------<br>        # Stats<br>        # ------------------------------------------------------------------<br><br>        self._stats = {<br>            &quot;ollama_calls&quot;: 0,<br>            &quot;lm_studio_calls&quot;: 0,<br>            &quot;openai_calls&quot;: 0,<br>            &quot;anthropic_calls&quot;: 0,<br>            &quot;fallbacks&quot;: 0,<br>        }<br><br>    # --------------------------------------------------------------------------<br>    # Cloud Clients<br>    # --------------------------------------------------------------------------<br><br>    def _get_openai(self) -&gt; OpenAI:<br>        if self._openai is None:<br>            api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br><br>            if not api_key:<br>                raise ValueError(<br>                    &quot;OPENAI_API_KEY not found in environment&quot;<br>                )<br><br>            self._openai = OpenAI(<br>                api_key=api_key,<br>                timeout=self.config.cloud_timeout,<br>            )<br><br>        return self._openai<br><br>    def _get_anthropic(self) -&gt; Anthropic:<br>        if self._anthropic is None:<br>            api_key = os.getenv(&quot;ANTHROPIC_API_KEY&quot;)<br><br>            if not api_key:<br>                raise ValueError(<br>                    &quot;ANTHROPIC_API_KEY not found in environment&quot;<br>                )<br><br>            self._anthropic = Anthropic(<br>                api_key=api_key,<br>                timeout=self.config.cloud_timeout,<br>            )<br><br>        return self._anthropic<br><br>    # --------------------------------------------------------------------------<br>    # Health Checks<br>    # --------------------------------------------------------------------------<br><br>    def _is_ollama_running(self) -&gt; bool:<br>        try:<br>            url = os.getenv(<br>                &quot;OLLAMA_BASE_URL&quot;,<br>                &quot;http://localhost:11434/v1&quot;,<br>            ).replace(&quot;/v1&quot;, &quot;/api/version&quot;)<br><br>            response = httpx.get(url, timeout=2.0)<br>            return response.status_code == 200<br><br>        except Exception:<br>            return False<br><br>    def _is_lm_studio_running(self) -&gt; bool:<br>        try:<br>            url = os.getenv(<br>                &quot;LM_STUDIO_BASE_URL&quot;,<br>                &quot;http://localhost:1234/v1&quot;,<br>            ) + &quot;/models&quot;<br><br>            response = httpx.get(url, timeout=2.0)<br>            return response.status_code == 200<br><br>        except Exception:<br>            return False<br><br>    # --------------------------------------------------------------------------<br>    # Retry Wrapper<br>    # --------------------------------------------------------------------------<br><br>    @retry(<br>        stop=stop_after_attempt(3),<br>        wait=wait_exponential(multiplier=1, min=1, max=8),<br>        reraise=True,<br>    )<br>    def _chat_with_backend(<br>        self,<br>        backend: InferenceBackend,<br>        messages: List[Dict],<br>        temperature: float,<br>        max_tokens: int,<br>    ) -&gt; Tuple[str, InferenceBackend]:<br><br>        # --------------------------------------------------------------<br>        # Ollama<br>        # --------------------------------------------------------------<br><br>        if backend == InferenceBackend.OLLAMA:<br>            response = self._ollama.chat.completions.create(<br>                model=self.config.local_model,<br>                messages=messages,<br>                temperature=temperature,<br>                max_tokens=max_tokens,<br>            )<br><br>            self._stats[&quot;ollama_calls&quot;] += 1<br><br>            return (<br>                response.choices[0].message.content,<br>                backend,<br>            )<br><br>        # --------------------------------------------------------------<br>        # LM Studio<br>        # --------------------------------------------------------------<br><br>        elif backend == InferenceBackend.LM_STUDIO:<br>            response = self._lm_studio.chat.completions.create(<br>                model=&quot;local-model&quot;,<br>                messages=messages,<br>                temperature=temperature,<br>                max_tokens=max_tokens,<br>            )<br><br>            self._stats[&quot;lm_studio_calls&quot;] += 1<br><br>            return (<br>                response.choices[0].message.content,<br>                backend,<br>            )<br><br>        # --------------------------------------------------------------<br>        # OpenAI<br>        # --------------------------------------------------------------<br><br>        elif backend == InferenceBackend.OPENAI:<br>            response = (<br>                self._get_openai()<br>                .chat.completions.create(<br>                    model=self.config.cloud_model,<br>                    messages=messages,<br>                    temperature=temperature,<br>                    max_tokens=max_tokens,<br>                )<br>            )<br><br>            self._stats[&quot;openai_calls&quot;] += 1<br><br>            return (<br>                response.choices[0].message.content,<br>                backend,<br>            )<br><br>        # --------------------------------------------------------------<br>        # Anthropic<br>        # --------------------------------------------------------------<br><br>        elif backend == InferenceBackend.ANTHROPIC:<br>            response = self._get_anthropic().messages.create(<br>                model=&quot;claude-3-5-sonnet-latest&quot;,<br>                max_tokens=max_tokens,<br>                temperature=temperature,<br>                messages=[<br>                    {<br>                        &quot;role&quot;: &quot;user&quot;,<br>                        &quot;content&quot;: messages[-1][&quot;content&quot;],<br>                    }<br>                ],<br>            )<br><br>            self._stats[&quot;anthropic_calls&quot;] += 1<br><br>            return (<br>                response.content[0].text,<br>                backend,<br>            )<br><br>        raise ValueError(f&quot;Unsupported backend: {backend}&quot;)<br><br>    # --------------------------------------------------------------------------<br>    # Main Chat API<br>    # --------------------------------------------------------------------------<br><br>    def chat(<br>        self,<br>        prompt: str,<br>        system: str = &quot;You are a helpful assistant.&quot;,<br>        temperature: float = 0.7,<br>        max_tokens: int = 2048,<br>        force_cloud: bool = False,<br>        force_local: bool = False,<br>    ) -&gt; Dict:<br><br>        messages = [<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>        ]<br><br>        if force_cloud:<br>            backends_to_try = [<br>                InferenceBackend.OPENAI,<br>                InferenceBackend.ANTHROPIC,<br>            ]<br><br>        elif force_local:<br>            backends_to_try = [<br>                InferenceBackend.OLLAMA,<br>                InferenceBackend.LM_STUDIO,<br>            ]<br><br>        elif self.config.prefer_local:<br>            backends_to_try = [<br>                InferenceBackend.OLLAMA,<br>                InferenceBackend.LM_STUDIO,<br>                InferenceBackend.OPENAI,<br>                InferenceBackend.ANTHROPIC,<br>            ]<br><br>        else:<br>            backends_to_try = [<br>                InferenceBackend.OPENAI,<br>                InferenceBackend.ANTHROPIC,<br>            ]<br><br>        last_error = None<br>        first_attempt = True<br><br>        for backend in backends_to_try:<br><br>            if (<br>                backend == InferenceBackend.OLLAMA<br>                and not self._is_ollama_running()<br>            ):<br>                logger.info(<br>                    &quot;Ollama not running. Skipping.&quot;<br>                )<br>                continue<br><br>            if (<br>                backend == InferenceBackend.LM_STUDIO<br>                and not self._is_lm_studio_running()<br>            ):<br>                logger.info(<br>                    &quot;LM Studio not running. Skipping.&quot;<br>                )<br>                continue<br><br>            try:<br>                if not first_attempt:<br>                    self._stats[&quot;fallbacks&quot;] += 1<br>                    logger.warning(<br>                        f&quot;Falling back to {backend.value}&quot;<br>                    )<br><br>                first_attempt = False<br><br>                start_time = time.monotonic()<br><br>                response, used_backend = (<br>                    self._chat_with_backend(<br>                        backend=backend,<br>                        messages=messages,<br>                        temperature=temperature,<br>                        max_tokens=max_tokens,<br>                    )<br>                )<br><br>                latency_ms = (<br>                    time.monotonic() - start_time<br>                ) * 1000<br><br>                return {<br>                    &quot;response&quot;: response,<br>                    &quot;backend&quot;: used_backend.value,<br>                    &quot;latency_ms&quot;: round(latency_ms),<br>                    &quot;model&quot;: (<br>                        self.config.local_model<br>                        if used_backend<br>                        in (<br>                            InferenceBackend.OLLAMA,<br>                            InferenceBackend.LM_STUDIO,<br>                        )<br>                        else self.config.cloud_model<br>                    ),<br>                }<br><br>            except Exception as e:<br>                logger.exception(<br>                    f&quot;{backend.value} failed&quot;<br>                )<br><br>                last_error = e<br>                continue<br><br>        raise RuntimeError(<br>            f&quot;All inference backends failed.\n&quot;<br>            f&quot;Last error: {last_error}&quot;<br>        )<br><br>    # --------------------------------------------------------------------------<br>    # Stats<br>    # --------------------------------------------------------------------------<br><br>    @property<br>    def stats(self):<br><br>        total = (<br>            self._stats[&quot;ollama_calls&quot;]<br>            + self._stats[&quot;lm_studio_calls&quot;]<br>            + self._stats[&quot;openai_calls&quot;]<br>            + self._stats[&quot;anthropic_calls&quot;]<br>        )<br><br>        local_calls = (<br>            self._stats[&quot;ollama_calls&quot;]<br>            + self._stats[&quot;lm_studio_calls&quot;]<br>        )<br><br>        cloud_calls = (<br>            self._stats[&quot;openai_calls&quot;]<br>            + self._stats[&quot;anthropic_calls&quot;]<br>        )<br><br>        return {<br>            **self._stats,<br>            &quot;total_calls&quot;: total,<br>            &quot;local_percentage&quot;: round(<br>                local_calls / max(total, 1) * 100,<br>                1,<br>            ),<br>            &quot;cloud_percentage&quot;: round(<br>                cloud_calls / max(total, 1) * 100,<br>                1,<br>            ),<br>        }<br><br>    # --------------------------------------------------------------------------<br>    # Cleanup<br>    # --------------------------------------------------------------------------<br><br>    def close(self):<br>        try:<br>            self.http_client.close()<br>        except Exception:<br>            pass<br><br><br># ------------------------------------------------------------------------------<br># Utility<br># ------------------------------------------------------------------------------<br><br><br>def validate_config():<br>    print(&quot;\n=== Configuration ===&quot;)<br>    print(<br>        &quot;LOCAL_MODEL:&quot;,<br>        os.getenv(&quot;LOCAL_MODEL&quot;),<br>    )<br>    print(<br>        &quot;CLOUD_MODEL:&quot;,<br>        os.getenv(&quot;CLOUD_MODEL&quot;),<br>    )<br>    print(<br>        &quot;PREFER_LOCAL:&quot;,<br>        os.getenv(&quot;PREFER_LOCAL&quot;),<br>    )<br><br>    print(<br>        &quot;OPENAI_API_KEY:&quot;,<br>        &quot;SET&quot;<br>        if os.getenv(&quot;OPENAI_API_KEY&quot;)<br>        else &quot;MISSING&quot;,<br>    )<br><br>    print(<br>        &quot;ANTHROPIC_API_KEY:&quot;,<br>        &quot;SET&quot;<br>        if os.getenv(&quot;ANTHROPIC_API_KEY&quot;)<br>        else &quot;MISSING&quot;,<br>    )<br><br>    print(&quot;=====================\n&quot;)<br><br><br># ------------------------------------------------------------------------------<br># Example Usage<br># ------------------------------------------------------------------------------<br><br>if __name__ == &quot;__main__&quot;:<br><br>    validate_config()<br><br>    client = UnifiedLLMClient()<br><br>    try:<br>        # Automatic routing<br>        result = client.chat(<br>            prompt=&quot;Explain database indexing strategies.&quot;,<br>            system=&quot;You are a senior database architect.&quot;,<br>        )<br><br>        print(<br>            f&quot;\nBackend: {result[&#39;backend&#39;]}&quot;<br>        )<br>        print(<br>            f&quot;Latency: {result[&#39;latency_ms&#39;]}ms&quot;<br>        )<br>        print(<br>            f&quot;Model: {result[&#39;model&#39;]}\n&quot;<br>        )<br><br>        print(result[&quot;response&quot;])<br><br>        # Force cloud<br>        cloud_result = client.chat(<br>            prompt=&quot;Compare monolithic and microservice architectures.&quot;,<br>            force_cloud=True,<br>        )<br><br>        print(<br>            f&quot;\nCloud Backend: {cloud_result[&#39;backend&#39;]}&quot;<br>        )<br><br>        # Force local<br>        local_result = client.chat(<br>            prompt=&quot;Summarize confidential HR notes.&quot;,<br>            force_local=True,<br>        )<br><br>        print(<br>            f&quot;\nLocal Backend: {local_result[&#39;backend&#39;]}&quot;<br>        )<br><br>        print(&quot;\nStats:&quot;)<br>        print(client.stats)<br><br>    finally:<br>        client.close()</pre><h2>Part 6: Choosing the Right Local Model</h2><p>Not all local models are equal. Here’s a practical guide for 2026:</p><pre>Task                     Recommended Model           Why<br>─────────────────────────────────────────────────────────────────────────────<br>General chat/writing     Llama 3.1 8B               Best balance, well-tested<br>Code generation          Qwen 2.5 Coder 7B          Trained on code, fast<br>Python specifically      DeepSeek-Coder-V2 7B       Top Python benchmark scores<br>Reasoning/math           DeepSeek-R1 7B             CoT reasoning, offline<br>Instruction following    Mistral 7B Instruct        Very reliable formatting<br>Embeddings               nomic-embed-text           Fast, good quality<br>Document summarization   Llama 3.1 8B (16K ctx)     Long context support<br>Privacy-sensitive any    Phi-4 14B                  Microsoft, no telemetry<br>Resource-constrained     Llama 3.2 3B / Phi-3 Mini  Runs on 4GB RAM<br>Maximum quality local    Llama 3.1 70B Q4           Near-frontier, 40GB VRAM</pre><h2>Part 7: Performance Benchmarking</h2><p>Before deploying locally, benchmark your setup:</p><pre>import time<br>import statistics<br>from typing import Callable<br><br>def benchmark_model(<br>    chat_fn: Callable[[str], str],<br>    model_name: str,<br>    prompts: list[str] = None,<br>    runs: int = 5,<br>) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Benchmark a local model&#39;s performance.<br>    <br>    Measures:<br>    - Time to first token (TTFT) — perceived latency<br>    - Tokens per second — throughput<br>    - Total latency per request<br>    &quot;&quot;&quot;<br>    if prompts is None:<br>        prompts = [<br>            &quot;What is 47 * 89?&quot;,<br>            &quot;List 5 Python built-in functions.&quot;,<br>            &quot;What&#39;s the capital of France?&quot;,<br>            &quot;Write a one-line function to reverse a string in Python.&quot;,<br>            &quot;Explain what HTTP means in one sentence.&quot;,<br>        ]<br>    <br>    latencies = []<br>    <br>    print(f&quot;\n{&#39;─&#39;*50}&quot;)<br>    print(f&quot;Benchmarking: {model_name}&quot;)<br>    print(f&quot;Runs: {runs} prompts × 1 = {runs} total requests&quot;)<br>    print(f&quot;{&#39;─&#39;*50}&quot;)<br>    <br>    for i, prompt in enumerate(prompts[:runs]):<br>        start = time.monotonic()<br>        response = chat_fn(prompt)<br>        elapsed = time.monotonic() - start<br>        <br>        words = len(response.split())<br>        approx_tokens = int(words * 1.3)<br>        tokens_per_second = approx_tokens / elapsed if elapsed &gt; 0 else 0<br>        <br>        latencies.append(elapsed)<br>        print(f&quot;Run {i+1}: {elapsed:.2f}s | ~{tokens_per_second:.0f} tok/s | {len(response)} chars&quot;)<br>    <br>    return {<br>        &quot;model&quot;: model_name,<br>        &quot;mean_latency_s&quot;: round(statistics.mean(latencies), 2),<br>        &quot;p50_latency_s&quot;: round(statistics.median(latencies), 2),<br>        &quot;p95_latency_s&quot;: round(sorted(latencies)[int(len(latencies) * 0.95)], 2) if len(latencies) &gt;= 2 else latencies[-1],<br>        &quot;min_latency_s&quot;: round(min(latencies), 2),<br>        &quot;max_latency_s&quot;: round(max(latencies), 2),<br>    }<br><br><br># Run benchmark<br>from openai import OpenAI<br><br>ollama = OpenAI(base_url=&quot;http://localhost:11434/v1&quot;, api_key=&quot;ollama&quot;)<br><br>def make_chat_fn(model: str) -&gt; Callable:<br>    def chat(prompt: str) -&gt; str:<br>        response = ollama.chat.completions.create(<br>            model=model,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>            max_tokens=256,<br>        )<br>        return response.choices[0].message.content<br>    return chat<br><br># Compare models<br>models_to_benchmark = [&quot;llama3.2&quot;, &quot;llama3.1:8b&quot;, &quot;mistral&quot;]<br><br>results = []<br>for model in models_to_benchmark:<br>    result = benchmark_model(make_chat_fn(model), model)<br>    results.append(result)<br><br>print(&quot;\n\nSummary:&quot;)<br>print(f&quot;{&#39;Model&#39;:&lt;25} {&#39;Mean (s)&#39;:&lt;12} {&#39;P50 (s)&#39;:&lt;12} {&#39;P95 (s)&#39;:&lt;12}&quot;)<br>print(&quot;─&quot; * 60)<br>for r in results:<br>    print(f&quot;{r[&#39;model&#39;]:&lt;25} {r[&#39;mean_latency_s&#39;]:&lt;12} {r[&#39;p50_latency_s&#39;]:&lt;12} {r[&#39;p95_latency_s&#39;]:&lt;12}&quot;)</pre><h2>🔍 Common Mistakes</h2><p><strong>1. Running a 7B model on 8GB RAM without GPU</strong> Without GPU offloading, a Q4 7B model uses all your RAM and inference drops to 1–2 tokens/second. Either use a smaller model (3B) or add a GPU. Check ollama ps to see GPU layer counts.</p><p><strong>2. Ignoring </strong><strong>num_ctx (context window)</strong> Ollama defaults to a 2048-token context for many models. For longer documents or multi-turn conversations, set num_ctx to 8192 or higher. Larger context = more RAM usage, so balance accordingly.</p><p><strong>3. Not keeping models warm</strong> Ollama unloads models from memory after 5 minutes by default. If your app has bursty traffic, the first request after idle will be slow (model reload). Set OLLAMA_KEEP_ALIVE=60m to keep models warm longer.</p><p><strong>4. Using LM Studio and Ollama on the same machine simultaneously</strong> Both grab GPU resources. Running them simultaneously causes memory conflicts. Pick one per session, or configure them for different GPUs if you have multiple.</p><p><strong>5. Pulling models without checking VRAM first</strong> ollama pull llama3.1:70b downloads 40GB. If you only have 12GB VRAM, it&#39;ll fall back to CPU inference at 1–3 tokens/second. Always check your VRAM before pulling large models.</p><p><strong>6. Forgetting temperature matters more for local models</strong> Local models are generally more literal and less “creative” than frontier cloud models. A temperature of 0.7 that works well on GPT-4o may feel flat on Llama 3.1 8B. Experiment — 0.8–0.9 often works better locally.</p><h2>💼 Quick Questions</h2><p><strong>Q: What is model quantization and what are the tradeoffs?</strong> Quantization reduces the bit-width of model weights from FP32/FP16 to INT8, INT4, or lower. This reduces memory footprint and increases inference speed at the cost of slight quality degradation. Q4_K_M is the sweet spot: ~7x smaller than FP16 with ~1% quality loss. Lower than Q4 (Q3, Q2) shows noticeable quality degradation.</p><p><strong>Q: How does Ollama’s GPU offloading work?</strong> Ollama uses llama.cpp under the hood, which splits transformer layers between GPU VRAM (fast) and CPU RAM (slow). You can set --gpu-layers N to specify exactly how many layers go to GPU. If your model fits entirely in VRAM, all layers are GPU-accelerated. For partial GPU offloading, inference speed is proportional to what percentage is on GPU.</p><p><strong>Q: When would you choose a local LLM over a cloud API in production?</strong> Local LLMs are preferred when: (1) data cannot leave the network (HIPAA, legal, financial PII), (2) latency requirements are sub-100ms, (3) usage volume makes API costs prohibitive, (4) offline/edge deployment is required, or (5) custom fine-tuned models need to be served. Cloud APIs are preferred when: maximum quality is needed, compute infrastructure isn’t available, or usage is sporadic.</p><p><strong>Q: What is GGUF and why does it exist?</strong> GGUF (GPT-Generated Unified Format) replaced the older GGML format in 2023. It’s a binary format that packages model weights, tokenizer, metadata, and quantization information into a single file. It supports memory mapping, enabling fast loading, and multiple quantization types in one format. It’s the standard format for CPU/hybrid inference in the llama.cpp ecosystem.</p><h2>🏭 Production Considerations</h2><p><strong>Model serving infrastructure:</strong> For production teams, consider running Ollama behind a load balancer with multiple GPU nodes. Ollama’s API is stateless — you can load-balance across instances running the same model.</p><p><strong>Model selection by task type:</strong> Build a routing layer that sends different task types to different models. Use Llama 3.2 3B for simple classification/extraction tasks (fast, cheap), Llama 3.1 8B for general generation, and cloud APIs for complex reasoning.</p><p><strong>Data governance:</strong> The primary enterprise value of local LLMs is data residency. Document your inference stack architecture clearly — CISOs and legal teams need to verify that customer data never touches external APIs.</p><p><strong>Monitoring:</strong> Ollama doesn’t expose Prometheus metrics natively. Build a wrapper that logs model name, latency, token counts, and error rates to your observability stack (Prometheus + Grafana, Datadog, etc.).</p><p><strong>Docker deployment:</strong></p><pre># Dockerfile for Ollama-based AI service<br>FROM ollama/ollama:latest<br><br># Pre-pull models during build (bakes model into image — large but self-contained)<br>RUN ollama serve &amp; sleep 5 &amp;&amp; ollama pull llama3.1:8b &amp;&amp; kill %1<br><br>ENV OLLAMA_HOST=0.0.0.0:11434<br>ENV OLLAMA_KEEP_ALIVE=30m<br>EXPOSE 11434<br><br>CMD [&quot;ollama&quot;, &quot;serve&quot;]</pre><pre># docker-compose.yml<br>version: &#39;3.8&#39;<br>services:<br>  ollama:<br>    image: ollama/ollama:latest<br>    ports:<br>      - &quot;11434:11434&quot;<br>    volumes:<br>      - ollama_data:/root/.ollama<br>    environment:<br>      - OLLAMA_KEEP_ALIVE=30m<br>      - OLLAMA_NUM_PARALLEL=2<br>    deploy:<br>      resources:<br>        reservations:<br>          devices:<br>            - driver: nvidia<br>              count: 1<br>              capabilities: [gpu]<br>  <br>  ai-service:<br>    build: .<br>    ports:<br>      - &quot;8000:8000&quot;<br>    depends_on:<br>      - ollama<br>    environment:<br>      - OLLAMA_URL=http://ollama:11434<br><br>volumes:<br>  ollama_data:</pre><h2>⚡ Performance &amp; Scalability</h2><p><strong>GPU layer optimization:</strong> In Ollama, run ollama ps while a model is loaded to see how many layers are on GPU vs. CPU. More GPU layers = faster inference. If you have spare VRAM, increase GPU layer count via Modelfile: PARAMETER num_gpu 99 (offload all possible layers).</p><p><strong>Parallel requests:</strong> By default, Ollama handles one request at a time per model. For concurrent workloads, set OLLAMA_NUM_PARALLEL=4 (or higher) and ensure you have enough VRAM for parallel KV-cache. Each parallel request needs its own KV cache allocation.</p><p><strong>KV cache:</strong> The KV (Key-Value) cache stores intermediate attention computations for the context. Longer contexts and more parallel requests require proportionally more VRAM. A rough formula: KV cache size ≈ 2 × num_layers × num_heads × head_dim × context_length × precision_bytes.</p><p><strong>Batching:</strong> For batch workloads (processing many documents), structure your workload to send requests in parallel rather than sequentially. Python asyncio + aiohttp with Ollama&#39;s async endpoint significantly increases throughput.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Quantization makes local inference practical</strong> — Q4_K_M reduces a 7B model to ~4GB with only ~1% quality loss</li><li><strong>Ollama is the standard CLI for local LLMs</strong> — pull any model in one command, OpenAI-compatible API out of the box</li><li><strong>LM Studio is the GUI alternative</strong> — visual model manager, built-in chat, OpenAI-compatible server</li><li><strong>Apple Silicon has a structural advantage</strong> — unified memory means no VRAM bottleneck for M2/M3 Max users</li><li><strong>Local ≠ worse</strong> — Llama 3.1 8B is competitive with GPT-3.5 on most tasks, running completely offline</li><li><strong>The right architecture</strong> — local-first with cloud fallback gives you privacy, cost efficiency, and reliability simultaneously</li><li><strong>Same OpenAI SDK, different base_url</strong> — swap between local and cloud with a single config change</li></ol><h2>📚 Further Reading</h2><ul><li><a href="https://github.com/ollama/ollama">Ollama GitHub</a> — Source, issues, and model library</li><li><a href="https://lmstudio.ai/docs">LM Studio Documentation</a> — Official LM Studio docs</li><li><a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> — The inference engine under Ollama</li><li><a href="https://github.com/ggerganov/ggml/blob/master/docs/gguf.md">GGUF Format Specification</a> — Deep dive into the file format</li><li><a href="https://huggingface.co/models?library=gguf">Hugging Face GGUF Models</a> — Browse thousands of quantized models</li><li><a href="https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard">Open LLM Leaderboard</a> — Compare local model quality benchmarks</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5d0ba562135" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 7 — Setting Up Your AI Engineering Environment]]></title>
            <link>https://learncsdesigns.medium.com/day-7-setting-up-your-ai-engineering-environment-4d637d83fd96?source=rss-2918beb827f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d637d83fd96</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Neeraj Kushwaha]]></dc:creator>
            <pubDate>Sat, 30 May 2026 11:33:53 GMT</pubDate>
            <atom:updated>2026-05-30T11:33:53.037Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“A craftsman is only as good as their tools — and their discipline in maintaining them. Your development environment is the foundation of everything you build. Get it right once, and it pays dividends for the entire journey.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnR6hKx518Wj9i4gOmKvQQ.jpeg" /></figure><h2>Why This Day Matters</h2><p>Most tutorials show you how to write AI code. Very few show you how to set up the environment that lets you write AI code safely, efficiently, and in a way that scales from prototype to production.</p><p>This is the day that prevents the common disasters: leaked API keys, broken dependency versions, environment conflicts, and “works on my machine” debugging sessions. Done right once, your environment serves you for years.</p><h2>Part 1: Python Environment Management</h2><h3>Why Not Just Use System Python?</h3><p>System Python (the Python that comes with your OS) is a trap:</p><ul><li>Version is often outdated</li><li>No isolation between projects</li><li>pip install pollutes global packages</li><li>Breaking one project can break all projects</li></ul><p>The correct approach: version-managed Python with isolated virtual environments per project.</p><h3>pyenv: Python Version Management</h3><p>pyenv lets you install and switch between multiple Python versions without touching your system Python.</p><pre># Install pyenv (macOS)<br>brew install pyenv</pre><pre># Add to shell config (~/.zshrc or ~/.bashrc)<br>echo &#39;export PYENV_ROOT=&quot;$HOME/.pyenv&quot;&#39; &gt;&gt; ~/.zshrc<br>echo &#39;command -v pyenv &gt;/dev/null || export PATH=&quot;$PYENV_ROOT/bin:$PATH&quot;&#39; &gt;&gt; ~/.zshrc<br>echo &#39;eval &quot;$(pyenv init -)&quot;&#39; &gt;&gt; ~/.zshrc</pre><pre># Reload shell<br>source ~/.zshrc</pre><pre># Install Python versions<br>pyenv install 3.12.3<br>pyenv install 3.11.9</pre><pre># Set global default<br>pyenv global 3.12.3</pre><pre># Set project-specific version (creates .python-version file)<br>cd my-project<br>pyenv local 3.12.3</pre><pre># Verify<br>python --version  # Python 3.12.3</pre><h3>uv: The Modern Python Package Manager</h3><p>uv is a drop-in replacement for pip that is 10-100× faster. Written in Rust by Astral (the team behind ruff). In 2026, it&#39;s rapidly becoming the standard.</p><pre># Install uv<br>curl -LsSf https://astral.sh/uv/install.sh | sh</pre><pre># Create a project with a specific Python version<br>uv init my-ai-project --python 3.12</pre><pre>cd my-ai-project</pre><pre># Add dependencies (creates pyproject.toml and uv.lock)<br>uv add openai anthropic langchain langchain-openai<br>uv add python-dotenv pydantic fastapi uvicorn<br>uv add --dev pytest ruff mypy pre-commit</pre><pre># Create and activate virtual environment<br>uv venv<br>source .venv/bin/activate  # macOS/Linux<br># or .venv\Scripts\activate  # Windows</pre><pre># Run a script (auto-activates venv)<br>uv run python script.py</pre><pre># Sync all dependencies (from uv.lock — reproducible)<br>uv sync</pre><p>Why uv over conda/pip for AI projects:</p><ul><li>100× faster than pip (no more 5-minute install sessions)</li><li>Deterministic installs via lockfile</li><li>Works with existing pyproject.toml standard</li><li>Better dependency resolution (avoids conflicts)</li></ul><h3>conda: When You Still Need It</h3><p>Use conda when you need CUDA toolkit management or packages not available on PyPI:</p><pre># Install Miniforge (conda without commercial restrictions)<br>brew install miniforge  # macOS<br># or download from conda-forge.org</pre><pre># Create environment for GPU-based training<br>conda create -n ai-training python=3.12 -y<br>conda activate ai-training</pre><pre># Install CUDA-dependent packages via conda<br>conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia<br>pip install transformers accelerate bitsandbytes peft trl</pre><pre># Export environment for reproducibility<br>conda env export &gt; environment.yml<br>conda env create -f environment.yml  # Recreate anywhere</pre><h2>Part 2: Project Structure for AI Applications</h2><p>A well-structured AI project is readable, maintainable, and deployable. Here’s the structure I use for every production AI application:</p><pre>my-ai-app/<br>├── .env                        # Local secrets (never commit)<br>├── .env.example                # Template with dummy values (commit this)<br>├── .gitignore                  # Include .env, __pycache__, .venv, etc.<br>├── .python-version             # pyenv Python version pin<br>├── pyproject.toml              # Dependencies and tool config<br>├── uv.lock                     # Locked dependency versions<br>├── README.md                   # Project overview and setup instructions<br>│<br>├── src/<br>│   └── my_ai_app/<br>│       ├── __init__.py<br>│       ├── config.py           # Configuration management (Pydantic Settings)<br>│       ├── models/             # Pydantic models for data structures<br>│       │   ├── __init__.py<br>│       │   └── schemas.py<br>│       ├── services/           # Business logic and AI integrations<br>│       │   ├── __init__.py<br>│       │   ├── llm.py          # LLM client wrapper<br>│       │   ├── embeddings.py   # Embedding service<br>│       │   └── rag.py          # RAG pipeline<br>│       ├── prompts/            # Prompt templates (versioned)<br>│       │   ├── __init__.py<br>│       │   ├── system_prompts.py<br>│       │   └── templates/<br>│       │       ├── qa.txt<br>│       │       └── summarize.txt<br>│       └── api/                # API layer (if building an API)<br>│           ├── __init__.py<br>│           ├── main.py<br>│           └── routes/<br>│<br>├── tests/<br>│   ├── conftest.py             # Test fixtures<br>│   ├── unit/<br>│   │   └── test_llm.py<br>│   └── integration/<br>│       └── test_rag_pipeline.py<br>│<br>├── notebooks/                  # Exploration and prototyping<br>│   ├── 01_data_exploration.ipynb<br>│   ├── 02_embedding_experiments.ipynb<br>│   └── 03_prompt_tuning.ipynb<br>│<br>├── scripts/                    # One-off scripts (data loading, indexing)<br>│   ├── ingest_documents.py<br>│   └── evaluate_model.py<br>│<br>├── data/<br>│   ├── raw/                    # Original, immutable data<br>│   ├── processed/              # Cleaned and processed data<br>│   └── eval/                   # Evaluation datasets<br>│<br>└── docker/<br>    ├── Dockerfile<br>    └── docker-compose.yml</pre><h2>Part 3: Configuration and Secrets Management</h2><p>The most common and most dangerous mistake in AI development: hardcoding API keys.</p><h3>The Right Pattern: Pydantic Settings</h3><pre># src/my_ai_app/config.py<br>from pydantic_settings import BaseSettings, SettingsConfigDict<br>from pydantic import SecretStr, field_validator<br>from functools import lru_cache<br><br>class Settings(BaseSettings):<br>    &quot;&quot;&quot;<br>    Application configuration loaded from environment variables.<br>    <br>    Pydantic automatically reads from:<br>    1. Environment variables<br>    2. .env file (if load_dotenv is configured)<br>    3. .env.{environment} files<br>    <br>    SecretStr types are masked in logs and repr output.<br>    &quot;&quot;&quot;<br>    model_config = SettingsConfigDict(<br>        env_file=&quot;.env&quot;,<br>        env_file_encoding=&quot;utf-8&quot;,<br>        case_sensitive=False<br>    )<br>    <br>    # API Keys - always use SecretStr<br>    openai_api_key: SecretStr<br>    anthropic_api_key: SecretStr<br>    google_api_key: SecretStr | None = None<br>    <br>    # Model defaults<br>    default_model: str = &quot;gpt-4o&quot;<br>    embedding_model: str = &quot;text-embedding-3-small&quot;<br>    <br>    # Infrastructure<br>    redis_url: str = &quot;redis://localhost:6379&quot;<br>    vector_db_url: str = &quot;http://localhost:6333&quot;<br>    <br>    # Application settings<br>    max_tokens: int = 2048<br>    temperature: float = 0.7<br>    debug: bool = False<br>    environment: str = &quot;development&quot;  # development, staging, production<br>    <br>    @field_validator(&quot;environment&quot;)<br>    @classmethod<br>    def validate_environment(cls, v):<br>        allowed = {&quot;development&quot;, &quot;staging&quot;, &quot;production&quot;}<br>        if v not in allowed:<br>            raise ValueError(f&quot;environment must be one of {allowed}&quot;)<br>        return v<br>    <br>    @property<br>    def is_production(self) -&gt; bool:<br>        return self.environment == &quot;production&quot;<br>@lru_cache<br>def get_settings() -&gt; Settings:<br>    &quot;&quot;&quot;<br>    Cache settings to avoid reading .env file on every request.<br>    Use lru_cache so settings are loaded once and reused.<br>    &quot;&quot;&quot;<br>    return Settings()<br># Usage pattern - import get_settings, not Settings directly<br># This enables dependency injection in FastAPI<br>settings = get_settings()<br># API keys accessed via get_secret_value() - never stored as plain string<br>api_key = settings.openai_api_key.get_secret_value()</pre><h3>The .env File</h3><pre># .env (NEVER commit this file)<br>OPENAI_API_KEY=sk-proj-abc123...<br>ANTHROPIC_API_KEY=sk-ant-def456...<br>GOOGLE_API_KEY=AIza...<br><br># Model settings<br>DEFAULT_MODEL=gpt-4o<br>EMBEDDING_MODEL=text-embedding-3-small<br># Infrastructure<br>REDIS_URL=redis://localhost:6379<br>VECTOR_DB_URL=http://localhost:6333<br># App settings<br>DEBUG=true<br>ENVIRONMENT=development<br>MAX_TOKENS=2048<br>TEMPERATURE=0.7</pre><pre># .env.example (COMMIT this file — real values replaced with placeholders)<br>OPENAI_API_KEY=sk-proj-your-key-here<br>ANTHROPIC_API_KEY=sk-ant-your-key-here<br>GOOGLE_API_KEY=your-google-api-key<br><br>DEFAULT_MODEL=gpt-4o<br>EMBEDDING_MODEL=text-embedding-3-small<br>REDIS_URL=redis://localhost:6379<br>VECTOR_DB_URL=http://localhost:6333<br>DEBUG=false<br>ENVIRONMENT=development<br>MAX_TOKENS=2048<br>TEMPERATURE=0.7</pre><h3>Multi-Environment Configuration</h3><pre>.env                # Local development (gitignored)<br>.env.example        # Template (committed)<br>.env.staging        # Staging overrides (gitignored)<br>.env.production     # Production overrides (use secrets manager instead)</pre><p><strong>For production:</strong> Never use .env files in production containers. Use:</p><ul><li>AWS Secrets Manager / Parameter Store</li><li>GCP Secret Manager</li><li>Azure Key Vault</li><li>HashiCorp Vault</li><li>Kubernetes Secrets</li></ul><h2>Part 4: The .gitignore for AI Projects</h2><p>This is one of the most important files in your project. A single missed API key pushed to GitHub can result in thousands of dollars in charges within hours.</p><pre># .gitignore for AI engineering projects<br><br># Environment files (CRITICAL - API keys live here)<br>.env<br>.env.*<br>!.env.example       # Exception: commit the template<br># Python<br>__pycache__/<br>*.py[cod]<br>*$py.class<br>*.pyc<br>.Python<br>.venv/<br>venv/<br>env/<br>ENV/<br># Jupyter<br>.ipynb_checkpoints/<br>*/.ipynb_checkpoints/*<br># Data (often large - use DVC or cloud storage instead)<br>data/raw/<br>data/processed/<br>*.csv<br>*.parquet<br>*.jsonl<br># Model weights (too large for git - use Hugging Face Hub)<br>*.bin<br>*.safetensors<br>*.gguf<br>*.pt<br>*.pth<br>models/<br># Vector database data<br>chroma_db/<br>qdrant_storage/<br>pinecone_cache/<br># IDE<br>.vscode/settings.json   # Exclude personal settings<br>.idea/<br>*.swp<br># OS<br>.DS_Store<br>Thumbs.db<br># Build<br>dist/<br>build/<br>*.egg-info/<br># Testing<br>.pytest_cache/<br>.coverage<br>htmlcov/<br># Logs (don&#39;t commit logs - use structured logging to a log service)<br>*.log<br>logs/</pre><h3>Pre-commit Hook: Your Last Line of Defense</h3><pre># Install pre-commit<br>pip install pre-commit<br><br># Create .pre-commit-config.yaml</pre><pre># .pre-commit-config.yaml<br>repos:<br>  # Detect secrets before they&#39;re committed<br>  - repo: https://github.com/gitleaks/gitleaks<br>    rev: v8.18.0<br>    hooks:<br>      - id: gitleaks<br>  <br>  # Detect hardcoded secrets (alternative/complement)<br>  - repo: https://github.com/Yelp/detect-secrets<br>    rev: v1.4.0<br>    hooks:<br>      - id: detect-secrets<br>        args: [&#39;--baseline&#39;, &#39;.secrets.baseline&#39;]<br>  <br>  # Code quality<br>  - repo: https://github.com/astral-sh/ruff-pre-commit<br>    rev: v0.4.0<br>    hooks:<br>      - id: ruff           # Linting<br>      - id: ruff-format    # Formatting (replaces black)<br>  <br>  # Type checking<br>  - repo: https://github.com/pre-commit/mirrors-mypy<br>    rev: v1.10.0<br>    hooks:<br>      - id: mypy<br>        additional_dependencies: [types-requests]<br>  <br>  # Standard file hygiene<br>  - repo: https://github.com/pre-commit/pre-commit-hooks<br>    rev: v4.6.0<br>    hooks:<br>      - id: trailing-whitespace<br>      - id: end-of-file-fixer<br>      - id: check-yaml<br>      - id: check-json<br>      - id: check-added-large-files<br>        args: [&#39;--maxkb=500&#39;]  # Block files larger than 500KB<br>      - id: no-commit-to-branch<br>        args: [&#39;--branch&#39;, &#39;main&#39;, &#39;--branch&#39;, &#39;production&#39;]</pre><pre># Initialize pre-commit<br>pre-commit install  # Installs hooks in .git/hooks/<br>pre-commit run --all-files  # Run on all files once</pre><p>Now every git commit automatically runs these checks. Secrets are caught before they ever reach GitHub.</p><h2>Part 5: VS Code Configuration for AI Development</h2><h3>Essential Extensions</h3><pre>// Install via VS Code Extensions panel or CLI:<br>// code --install-extension &lt;extension-id&gt;<br><br>{<br>  &quot;ai_engineering_extensions&quot;: [<br>    &quot;ms-python.python&quot;,              // Python language support<br>    &quot;ms-python.vscode-pylance&quot;,      // Fast Python type checking<br>    &quot;charliermarsh.ruff&quot;,            // Fast Python linting (replaces pylint)<br>    &quot;ms-toolsai.jupyter&quot;,            // Jupyter notebook support<br>    &quot;ms-toolsai.jupyter-keymap&quot;,     // Jupyter keybindings<br>    &quot;GitHub.copilot&quot;,                // AI pair programming<br>    &quot;GitHub.copilot-chat&quot;,           // AI chat for code<br>    &quot;donjayamanne.githistory&quot;,       // Git log visualization<br>    &quot;eamodio.gitlens&quot;,               // Enhanced Git visualization<br>    &quot;mikestead.dotenv&quot;,              // .env file highlighting<br>    &quot;redhat.vscode-yaml&quot;,            // YAML support<br>    &quot;ms-azuretools.vscode-docker&quot;,   // Docker support<br>    &quot;humao.rest-client&quot;,             // HTTP request testing (like Postman)<br>    &quot;mechatroner.rainbow-csv&quot;        // CSV visualization<br>  ]<br>}</pre><h3>VS Code Settings for AI Development</h3><pre>// .vscode/settings.json (commit this — shared team settings)<br>{<br>  &quot;editor.formatOnSave&quot;: true,<br>  &quot;editor.codeActionsOnSave&quot;: {<br>    &quot;source.fixAll.ruff&quot;: &quot;explicit&quot;,<br>    &quot;source.organizeImports.ruff&quot;: &quot;explicit&quot;<br>  },<br>  &quot;[python]&quot;: {<br>    &quot;editor.defaultFormatter&quot;: &quot;charliermarsh.ruff&quot;<br>  },<br>  &quot;python.defaultInterpreterPath&quot;: &quot;${workspaceFolder}/.venv/bin/python&quot;,<br>  &quot;python.terminal.activateEnvironment&quot;: true,<br>  <br>  // Jupyter settings<br>  &quot;jupyter.defaultKernel&quot;: &quot;Python 3 (.venv)&quot;,<br>  &quot;jupyter.askForKernelRestart&quot;: false,<br>  <br>  // Hide clutter in explorer<br>  &quot;files.exclude&quot;: {<br>    &quot;**/__pycache__&quot;: true,<br>    &quot;**/*.pyc&quot;: true,<br>    &quot;**/.pytest_cache&quot;: true,<br>    &quot;**/.ruff_cache&quot;: true,<br>    &quot;**/.mypy_cache&quot;: true,<br>    &quot;.venv&quot;: true<br>  },<br>  <br>  // Terminal<br>  &quot;terminal.integrated.defaultProfile.osx&quot;: &quot;zsh&quot;,<br>  &quot;terminal.integrated.env.osx&quot;: {<br>    &quot;PYTHONPATH&quot;: &quot;${workspaceFolder}/src&quot;<br>  }<br>}</pre><h2>Part 6: The Complete LLM Client Wrapper</h2><p>Every AI project should have a centralized LLM client that handles retries, logging, cost tracking, and error handling. This is the component most tutorials skip — and most production bugs originate from.</p><pre># src/my_ai_app/services/llm.py<br>import time<br>import logging<br>from typing import Iterator<br>from openai import OpenAI, APIError, RateLimitError, APIConnectionError<br>from anthropic import Anthropic<br>from tenacity import (<br>    retry,<br>    stop_after_attempt,<br>    wait_exponential,<br>    retry_if_exception_type<br>)<br>from my_ai_app.config import get_settings<br><br>logger = logging.getLogger(__name__)<br># Cost per 1M tokens (update as pricing changes)<br>MODEL_COSTS = {<br>    &quot;gpt-4o&quot;: {&quot;input&quot;: 2.50, &quot;output&quot;: 10.00},<br>    &quot;gpt-4o-mini&quot;: {&quot;input&quot;: 0.15, &quot;output&quot;: 0.60},<br>    &quot;gpt-4o-2024-11-20&quot;: {&quot;input&quot;: 2.50, &quot;output&quot;: 10.00},<br>    &quot;claude-sonnet-4-6&quot;: {&quot;input&quot;: 3.00, &quot;output&quot;: 15.00},<br>    &quot;claude-haiku-4-5-20251001&quot;: {&quot;input&quot;: 0.25, &quot;output&quot;: 1.25},<br>}<br>class LLMClient:<br>    &quot;&quot;&quot;<br>    Production-grade LLM client with:<br>    - Automatic retries with exponential backoff<br>    - Cost tracking per call<br>    - Structured logging<br>    - Error handling and categorization<br>    - Streaming support<br>    &quot;&quot;&quot;<br>    <br>    def __init__(self):<br>        settings = get_settings()<br>        self.openai = OpenAI(<br>            api_key=settings.openai_api_key.get_secret_value()<br>        )<br>        self.anthropic = Anthropic(<br>            api_key=settings.anthropic_api_key.get_secret_value()<br>        )<br>        self.settings = settings<br>        self._total_cost = 0.0<br>        self._total_calls = 0<br>    <br>    def _calculate_cost(<br>        self,<br>        model: str,<br>        input_tokens: int,<br>        output_tokens: int<br>    ) -&gt; float:<br>        &quot;&quot;&quot;Calculate cost in USD for a model call.&quot;&quot;&quot;<br>        if model not in MODEL_COSTS:<br>            return 0.0<br>        costs = MODEL_COSTS[model]<br>        return (<br>            (input_tokens / 1_000_000) * costs[&quot;input&quot;] +<br>            (output_tokens / 1_000_000) * costs[&quot;output&quot;]<br>        )<br>    <br>    @retry(<br>        retry=retry_if_exception_type((RateLimitError, APIConnectionError)),<br>        wait=wait_exponential(multiplier=1, min=4, max=60),<br>        stop=stop_after_attempt(3)<br>    )<br>    def chat(<br>        self,<br>        messages: list[dict],<br>        model: str | None = None,<br>        system: str | None = None,<br>        temperature: float | None = None,<br>        max_tokens: int | None = None,<br>        **kwargs<br>    ) -&gt; dict:<br>        &quot;&quot;&quot;<br>        Unified chat interface for OpenAI and Anthropic models.<br>        <br>        Returns dict with: content, model, input_tokens, output_tokens, cost_usd<br>        &quot;&quot;&quot;<br>        model = model or self.settings.default_model<br>        temperature = temperature if temperature is not None else self.settings.temperature<br>        max_tokens = max_tokens or self.settings.max_tokens<br>        <br>        start_time = time.time()<br>        <br>        try:<br>            if &quot;claude&quot; in model:<br>                result = self._call_anthropic(<br>                    messages, model, system, temperature, max_tokens, **kwargs<br>                )<br>            else:<br>                result = self._call_openai(<br>                    messages, model, system, temperature, max_tokens, **kwargs<br>                )<br>            <br>            latency_ms = round((time.time() - start_time) * 1000)<br>            cost = self._calculate_cost(<br>                model,<br>                result[&quot;input_tokens&quot;],<br>                result[&quot;output_tokens&quot;]<br>            )<br>            <br>            self._total_cost += cost<br>            self._total_calls += 1<br>            <br>            logger.info(<br>                &quot;LLM call&quot;,<br>                extra={<br>                    &quot;model&quot;: model,<br>                    &quot;input_tokens&quot;: result[&quot;input_tokens&quot;],<br>                    &quot;output_tokens&quot;: result[&quot;output_tokens&quot;],<br>                    &quot;latency_ms&quot;: latency_ms,<br>                    &quot;cost_usd&quot;: round(cost, 6)<br>                }<br>            )<br>            <br>            return {**result, &quot;cost_usd&quot;: round(cost, 6), &quot;latency_ms&quot;: latency_ms}<br>        <br>        except RateLimitError as e:<br>            logger.warning(f&quot;Rate limit hit for {model}: {e}&quot;)<br>            raise<br>        except APIError as e:<br>            logger.error(f&quot;API error for {model}: {e}&quot;)<br>            raise<br>    <br>    def _call_openai(<br>        self, messages, model, system, temperature, max_tokens, **kwargs<br>    ) -&gt; dict:<br>        if system:<br>            messages = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system}] + messages<br>        <br>        response = self.openai.chat.completions.create(<br>            model=model,<br>            messages=messages,<br>            temperature=temperature,<br>            max_tokens=max_tokens,<br>            **kwargs<br>        )<br>        <br>        return {<br>            &quot;content&quot;: response.choices[0].message.content,<br>            &quot;model&quot;: model,<br>            &quot;input_tokens&quot;: response.usage.prompt_tokens,<br>            &quot;output_tokens&quot;: response.usage.completion_tokens<br>        }<br>    <br>    def _call_anthropic(<br>        self, messages, model, system, temperature, max_tokens, **kwargs<br>    ) -&gt; dict:<br>        create_kwargs = {<br>            &quot;model&quot;: model,<br>            &quot;messages&quot;: messages,<br>            &quot;temperature&quot;: temperature,<br>            &quot;max_tokens&quot;: max_tokens,<br>            **kwargs<br>        }<br>        if system:<br>            create_kwargs[&quot;system&quot;] = system<br>        <br>        response = self.anthropic.messages.create(**create_kwargs)<br>        <br>        return {<br>            &quot;content&quot;: response.content[0].text,<br>            &quot;model&quot;: model,<br>            &quot;input_tokens&quot;: response.usage.input_tokens,<br>            &quot;output_tokens&quot;: response.usage.output_tokens<br>        }<br>    <br>    def stream(<br>        self,<br>        messages: list[dict],<br>        model: str | None = None,<br>        system: str | None = None,<br>        **kwargs<br>    ) -&gt; Iterator[str]:<br>        &quot;&quot;&quot;Stream tokens from OpenAI models.&quot;&quot;&quot;<br>        model = model or self.settings.default_model<br>        <br>        if system:<br>            messages = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system}] + messages<br>        <br>        stream = self.openai.chat.completions.create(<br>            model=model,<br>            messages=messages,<br>            stream=True,<br>            **kwargs<br>        )<br>        <br>        for chunk in stream:<br>            if chunk.choices[0].delta.content:<br>                yield chunk.choices[0].delta.content<br>    <br>    @property<br>    def session_cost(self) -&gt; float:<br>        &quot;&quot;&quot;Total cost for this session in USD.&quot;&quot;&quot;<br>        return round(self._total_cost, 4)<br>    <br>    @property<br>    def session_calls(self) -&gt; int:<br>        &quot;&quot;&quot;Total API calls in this session.&quot;&quot;&quot;<br>        return self._total_calls</pre><h3>Usage Pattern</h3><pre># In your application<br>from my_ai_app.services.llm import LLMClient<br><br>client = LLMClient()<br># Simple chat<br>result = client.chat(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Summarize quantum computing in 3 sentences.&quot;}],<br>    model=&quot;gpt-4o-mini&quot;,  # Use cheap model for simple tasks<br>    temperature=0.3<br>)<br>print(result[&quot;content&quot;])<br>print(f&quot;Cost: ${result[&#39;cost_usd&#39;]}&quot;)<br># With system prompt<br>result = client.chat(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Review this code: def foo(): pass&quot;}],<br>    system=&quot;You are an expert Python code reviewer. Be concise and specific.&quot;,<br>    model=&quot;gpt-4o&quot;,<br>    temperature=0.1<br>)<br># Streaming<br>for token in client.stream(<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Write a poem about Python.&quot;}]<br>):<br>    print(token, end=&quot;&quot;, flush=True)<br>print(f&quot;\nSession cost: ${client.session_cost}&quot;)</pre><h2>Part 7: Jupyter Best Practices for AI Development</h2><p>Notebooks are indispensable for exploration and prototyping. But they have pitfalls. Here’s how to use them productively.</p><pre># Standard AI Jupyter notebook header<br># Put this in your first cell<br><br># %% [markdown]<br># # Experiment: Prompt Strategy Comparison<br># **Date:** 2026-05-16<br># **Author:** Neeraj<br># **Goal:** Compare chain-of-thought vs. direct prompting for math problems<br># **Status:** In Progress<br># %%<br># Standard imports<br>import os<br>import json<br>import time<br>from pathlib import Path<br>from dotenv import load_dotenv<br># Load environment variables<br>load_dotenv()<br># Project imports<br>import sys<br>sys.path.insert(0, str(Path(&quot;..&quot;).resolve()))<br>from src.my_ai_app.services.llm import LLMClient<br>from src.my_ai_app.config import get_settings<br># Initialize<br>client = LLMClient()<br>settings = get_settings()<br>print(f&quot;Environment: {settings.environment}&quot;)<br>print(f&quot;Default model: {settings.default_model}&quot;)</pre><h3>nbstripout: Never Commit Notebook Outputs</h3><p>Notebook outputs can contain PII, API responses, and model outputs that shouldn’t be in git. Use nbstripout to automatically strip outputs before committing.</p><pre>pip install nbstripout<br>nbstripout --install  # Installs git filter — outputs stripped automatically on commit</pre><h2>🔍 Common Mistakes to Avoid</h2><h3>Mistake 1: Checking .env into Git</h3><p>The single most dangerous mistake. Add .env to .gitignore before your first commit. Use .env.example as the committed template. Set up gitleaks pre-commit hook as a safety net.</p><h3>Mistake 2: Using Mutable Default Arguments</h3><p>A classic Python pitfall that’s common in AI code:</p><pre># ❌ Wrong: mutable default argument<br>def build_messages(user_msg: str, history: list = []) -&gt; list:<br>    history.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_msg})<br>    return history<br># The same [] is shared across all calls!<br><br># ✅ Correct<br>def build_messages(user_msg: str, history: list | None = None) -&gt; list:<br>    if history is None:<br>        history = []<br>    history.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_msg})<br>    return history</pre><h3>Mistake 3: No Retry Logic on LLM Calls</h3><p>API calls fail. Rate limits hit. Networks time out. Always wrap LLM calls with retry logic using tenacity or similar. The LLMClient above handles this, but raw API calls without retries are fragile in production.</p><h3>Mistake 4: Hardcoding Model Names in Business Logic</h3><p>Model names change (provider deprecations are common). Centralize model names in configuration:</p><pre># ❌ Bad: model hardcoded throughout codebase<br>response = client.chat.completions.create(model=&quot;gpt-4o&quot;, ...)<br><br># ✅ Good: model from configuration, easily changed<br>response = client.chat.completions.create(model=settings.default_model, ...)</pre><h3>Mistake 5: Not Logging Token Usage</h3><p>Without logging token usage per request, you can’t debug cost spikes, attribute costs to features, or optimize expensive prompts. The LLMClient above logs this automatically.</p><h2>💼 Quick Questions</h2><p><strong>Q1:</strong> How do you manage secrets in an AI application across development, staging, and production?</p><blockquote><strong>Answer:</strong> Development: .env files loaded by python-dotenv, excluded from git via .gitignore, with .env.example committed as documentation. Pre-commit hooks (gitleaks) prevent accidental commits. Staging: Same pattern with environment-specific values, never in version control. Production: Cloud secrets managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) injected as environment variables at runtime. Never .env files in production containers.</blockquote><p><strong>Q2:</strong> Why should you wrap LLM API calls in a custom client class rather than calling them directly?</p><blockquote><strong>Answer:</strong> A custom client centralizes: (1) retry logic with exponential backoff — essential since API calls fail intermittently; (2) cost tracking — you need to know what each call costs; (3) structured logging — trace every request with model, tokens, latency, and cost; (4) error categorization — different exceptions (rate limit vs. API error vs. network error) need different handling; (5) model routing — swap models without changing calling code; (6) configuration management — one place to update model names and defaults.</blockquote><p><strong>Q3:</strong> What is the purpose of a uv.lock or requirements.txt file, and why does it matter for AI projects?</p><blockquote><strong>Answer:</strong> Lock files record the exact version of every installed package (including transitive dependencies). This ensures that uv sync or pip install -r requirements.txt produces bit-for-bit identical environments everywhere — your laptop, CI, staging, production. For AI projects this is critical because library versions (transformers, langchain, openai SDK) often have breaking changes, and a model&#39;s behavior can change subtly with library updates. Without lockfiles, &quot;works on my machine&quot; is a constant source of friction.</blockquote><p><strong>Q4:</strong> What is pyenv and why use it instead of system Python?</p><blockquote><strong>Answer:</strong> pyenv manages multiple Python versions on a single machine. System Python is often outdated, shared across all projects, and can&#39;t be easily upgraded without risk to the OS. With pyenv: each project can pin its own Python version (via .python-version); upgrading Python for one project doesn&#39;t affect others; you can test compatibility across versions. This is essential for AI projects because many frameworks (PyTorch, JAX, etc.) have specific Python version requirements.</blockquote><p><strong>Q5:</strong> How would you structure an AI project to make it testable?</p><blockquote><strong>Answer:</strong> Key patterns: (1) Separate configuration from code — use Pydantic Settings with environment variables; (2) Dependency injection — pass LLM clients as parameters rather than importing globals; (3) Abstract LLM calls behind interfaces — mock the interface in tests, not the actual API; (4) Keep prompts out of business logic — store in separate files for easier testing and iteration; (5) Write integration tests against real APIs but with small, cheap requests; (6) Use recorded API responses (VCR cassettes) for fast, cost-free unit testing.</blockquote><h2>🏭 Production Considerations</h2><p><strong>Container-Friendly Configuration:</strong> In containerized production environments, configuration comes from environment variables injected at runtime — not from files. Your Pydantic Settings class handles this seamlessly: it reads from both .env files (development) and actual environment variables (production) with the same code.</p><p><strong>Secret Rotation:</strong> API keys should be rotatable without downtime. Using a secrets manager (not environment variables baked into container images) allows you to rotate keys by updating the secret and restarting pods — without rebuilding containers.</p><p><strong>PYTHONPATH Management:</strong> In Python projects, import paths can be a source of subtle bugs. Set PYTHONPATH explicitly in your VS Code settings, Makefile, and Docker configuration to ensure consistent behavior.</p><h2>⚡ Performance &amp; Scalability Insights</h2><p><strong>uv over pip in CI:</strong> CI pipelines that install Python dependencies are often bottlenecked by pip&#39;s slow resolution. Switching from pip install -r requirements.txt to uv sync typically reduces CI install time from 2-5 minutes to 10-30 seconds. At scale (hundreds of CI runs per day), this compounds.</p><p><strong>Lazy Imports for Fast Startup:</strong> AI libraries (torch, transformers) are slow to import. In production APIs, import heavy libraries lazily (inside functions) or in background threads to keep startup time fast and health checks responsive.</p><h2>🔑 Key Takeaways</h2><ol><li><strong>Your environment is infrastructure.</strong> Treat it with the same care as production infrastructure — version everything, document it, automate it.</li><li><strong>Secrets management is non-negotiable.</strong> .gitignore, .env.example, and pre-commit hooks form the minimum defense. For production, use a secrets manager.</li><li><strong>Centralize LLM calls in a custom client.</strong> Retries, logging, cost tracking, and error handling belong in one place — not scattered throughout your application.</li><li><strong>Use </strong><strong>pyproject.toml + </strong><strong>uv.lock for reproducibility.</strong> The combination of pyenv (Python version), uv (package management), and lock files produces environments that are identical everywhere.</li><li><strong>Structure projects for the long term, not just the prototype.</strong> The patterns here scale from a solo prototype to a 10-engineer team. Apply them from Day 1.</li></ol><h2>📚 Further Reading &amp; Resources</h2><ul><li><a href="https://docs.astral.sh/uv/">uv Documentation</a> — The modern Python package manager</li><li><a href="https://docs.pydantic.dev/latest/concepts/pydantic_settings/">Pydantic Settings Documentation</a> — Configuration management</li><li><a href="https://pre-commit.com/">pre-commit Documentation</a> — Git hook framework</li><li><a href="https://12factor.net/">The Twelve-Factor App</a> — Methodology for building robust applications (especially Factor III: Config)</li><li><a href="https://tenacity.readthedocs.io/">tenacity Documentation</a> — Retry library for Python</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d637d83fd96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>