<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vansh Sharma on Medium]]></title>
        <description><![CDATA[Stories by Vansh Sharma on Medium]]></description>
        <link>https://medium.com/@vanshsharma9354?source=rss-cf70bfe7604e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SW3K0fELdluMxzh-5trO3w.jpeg</url>
            <title>Stories by Vansh Sharma on Medium</title>
            <link>https://medium.com/@vanshsharma9354?source=rss-cf70bfe7604e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:50:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vanshsharma9354/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Context Window: The Hard Limit on What an LLM Can See]]></title>
            <link>https://medium.com/@vanshsharma9354/context-window-the-hard-limit-on-what-an-llm-can-see-1d6a013bbd18?source=rss-cf70bfe7604e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d6a013bbd18</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Vansh Sharma]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 19:19:57 GMT</pubDate>
            <atom:updated>2026-04-10T19:19:57.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VF2IUTmTYu3Wk8_U7EOF9A.jpeg" /></figure><p><em>Series: An Engineer’s Explorations in LLM Concepts</em></p><p>We’ve now covered what tokens are, how they’re billed, and what parameters do with them. But there’s one more physical constraint that shapes everything you build with LLMs: <strong>the context window</strong>.</p><h3>The Simple Answer</h3><p>The context window is the <strong>maximum number of tokens an LLM can process in a single call</strong> — reading and writing combined.</p><p>Think of it like a whiteboard. The model can only see and work with what’s written on the whiteboard right now. It cannot look at yesterday’s whiteboard. It cannot open a drawer. Everything it needs must fit on that one board.</p><p>When the board is full, old content gets erased to make room. The model doesn’t know it was erased. It just works with what it can see.</p><h3>Why Does a Context Limit Even Exist?</h3><p>This is not an arbitrary product decision. It’s a mathematical constraint baked into the transformer architecture itself.</p><p>The mechanism responsible is called <strong>attention</strong>. And attention is expensive.</p><p>Here’s why:</p><p>In a transformer, every token must mathematically “attend to” — consider its relationship with — every other token in the context. This is what gives LLMs their ability to connect information across a long document.</p><p>Context Length Relative Compute Relative Memory 4K tokens 1× 1× 16K tokens 16× 16× 64K tokens 256× 256× 200K tokens 2,500× 2,500×</p><p>This is why 200K context windows required genuine engineering breakthroughs — not just more GPUs.</p><h3>The Engineering That Made Long Contexts Possible</h3><p>Getting from 4K to 200K+ required solving three major problems:</p><h3>Problem 1: Positional Encoding — How Does the Model Know Token Order?</h3><p>Tokens arrive as vectors with no inherent order. The model needs to know that token 1 came before token 2, and token 5,000 came much later. Early models used fixed sinusoidal position encodings — essentially a hardcoded table of positions. This broke down badly beyond the lengths seen in training.</p><p><strong>Solution: RoPE (Rotary Position Embedding)</strong></p><pre>flowchart LR<br>    A[&quot;Token at position 1\n[0.8, 0.3, ...]&quot;] --&gt;|&quot;Rotate vector\nby angle θ × 1&quot;| C[&quot;Position-encoded\nvector&quot;]<br>    B[&quot;Token at position 500\n[0.8, 0.3, ...]&quot;] --&gt;|&quot;Rotate vector\nby angle θ × 500&quot;| D[&quot;Position-encoded\nvector&quot;]<br>    C --&gt; E[&quot;Attention can now compute\nrelative distance between\nany two tokens accurately\neven at 200K positions&quot;]<br>    D --&gt; E</pre><p>Instead of a lookup table, RoPE encodes position by <em>rotating</em> the token’s vector by an angle proportional to its position. Because of how rotation math works, the dot product between two tokens naturally encodes their <em>relative distance</em> — and this generalizes well to positions never seen during training.</p><h3>Problem 2: Memory — The KV Cache Explosion</h3><p>During generation, the model needs to store intermediate computations for every token in the context (called the <strong>Key-Value cache</strong> or KV cache). At 200K tokens with full precision:</p><pre>200,000 tokens × 96 layers × 2 (K and V) × 128 dimensions × 2 bytes<br>= ~10 GB just for the KV cache</pre><p>That’s before the model weights themselves (14–28 GB for a mid-size model).</p><p><strong>Solution: Flash Attention</strong></p><p>Flash Attention is a rewrite of the attention computation that tiles the calculation into GPU SRAM-friendly chunks instead of materializing the full N×N attention matrix in HBM (slow memory). It produces identical results but uses memory proportional to N rather than N².</p><pre>flowchart LR<br>    subgraph &quot;Standard Attention&quot;<br>        A[&quot;Load all Q, K, V\ninto GPU HBM memory\n→ O(N²) memory&quot;] --&gt; B[&quot;Compute full\nN×N attention matrix&quot;]<br>        B --&gt; C[&quot;Write result\nback to HBM&quot;]<br>    end</pre><p>Flash Attention is one of those rare algorithmic improvements that was both theoretically elegant and practically transformative — making 100K+ contexts feasible on hardware that would have choked under the naive implementation.</p><h3>Problem 3: Attention Head Redundancy — GQA</h3><p>In the original transformer, every attention head (there are typically 32–96 of them) maintained its own separate Key and Value matrices. At long contexts this multiplied memory requirements linearly with head count.</p><p><strong>Solution: Grouped Query Attention (GQA)</strong></p><pre>flowchart TD<br>    subgraph &quot;Multi-Head Attention (original)&quot;<br>        A1[&quot;Head 1: Q, K, V&quot;]<br>        A2[&quot;Head 2: Q, K, V&quot;]<br>        A3[&quot;Head 3: Q, K, V&quot;]<br>        A4[&quot;Head 4: Q, K, V&quot;]<br>    end</pre><p>By sharing K and V matrices across groups of query heads, GQA reduces the KV cache size by 4–8× with minimal impact on model quality. It’s now standard in all modern long-context models (LLaMA 3, Claude, Gemini, GPT-4).</p><h3>What Actually Happens When You Exceed the Context Window?</h3><p>Two behaviors, depending on the implementation</p><p>The silent truncation case is the dangerous one. The model doesn’t say “I’ve forgotten the beginning.” It just answers with whatever it can see, confidently. This is exactly what happened in the PDF summarizer story at the top — the model summarized what fit in its window and said nothing about the rest.</p><h3>The “Lost in the Middle” Problem</h3><p>Even within a context that fits, there’s a performance issue worth knowing: <strong>models attend better to the beginning and end of a long context than to the middle</strong>.</p><p>This is known as the <strong>“Lost in the Middle”</strong> effect (from a 2023 Stanford paper). If you’re doing RAG or document analysis and you stuff 10 retrieved chunks into the context, the chunks in positions 3–7 are recalled less reliably than chunks at the beginning or end.</p><p><strong>Practical implication:</strong> When injecting retrieved content, put the most important chunks at the top or bottom of the context, not buried in the middle.</p><h3>Context Window vs. Model Memory — A Critical Distinction</h3><p>This confusion comes up constantly:</p><p>When you paste a document into the chat, that knowledge lives only in the context window — for this conversation only. It does not go into the model’s weights. The model will not “remember” it next time.</p><p>This is why products like Claude’s Memory feature are architecturally interesting — they solve this by storing summaries externally and injecting them back into the context window on the next call. The parameters never change; the illusion of memory is created by context injection.</p><h3>Practical Guidelines: Designing Around Context Limits</h3><p>Scenario Recommended Approach Long documents (&gt;50K tokens) Chunk + RAG, don’t stuff the full doc Long conversations Sliding window + summarization (see Part 2) Critical information Put it at start or end of context (avoid middle burial) System prompt Keep it tight — it costs tokens on every call Multi-document analysis Hierarchical summarization: summarize each doc, then synthesize Code repositories Index + retrieve relevant files, don’t dump the whole repo</p><h3>Context Windows Across Current Models (2025)</h3><p>Model Context Window Notes Claude Sonnet 4 200K tokens ~150,000 words GPT-4o 128K tokens ~96,000 words Gemini 1.5 Pro 1M tokens Experimental, quality degrades Gemini 1.5 Flash 1M tokens Faster, lower quality at extremes LLaMA 3.1 70B 128K tokens Open weights Mistral Large 128K tokens Strong mid-size option</p><p>One important caveat on Gemini’s 1M context: raw context length and <em>effective</em> context length are different. Recall and reasoning quality degrades meaningfully toward the middle of very long contexts on current models. The “Lost in the Middle” effect compounds as the window grows.</p><h3>The Mental Model to Keep</h3><blockquote><strong><em>The context window is the model’s working whiteboard — everything must fit on it for a single call, nothing persists after the call ends, and the model has no awareness of what’s been erased. Bigger whiteboards are harder to build because attention scales quadratically, but engineering advances (RoPE, Flash Attention, GQA) have pushed that limit from 4K to 1M+ tokens in just three years.</em></strong></blockquote><p>The architectural decisions that follow from this:</p><ul><li>RAG is a response to context limits — inject only what’s relevant</li><li>Summarization is a response to context limits — compress what must persist</li><li>Model tiering is a response to context limits — use cheaper models for retrieval, expensive ones for synthesis</li><li>Every system prompt is a permanent tax on your context budget</li></ul><h3>What’s Next</h3><p>We’ve covered the physical machinery: tokens, billing, parameters, context. Now we get to something more behavioral — a feature that changes how the model reasons before it answers.</p><p>In <strong>Part 5 — Thinking Tokens: How Modern LLMs Reason Before They Respond</strong>, we’ll look at what “thinking” actually means mechanically, how it emerged from reinforcement learning (not programming), and when it’s worth paying the extra cost.</p><p><strong>Tags:</strong> #LLM #AIEngineering #ContextWindow #Transformers #FlashAttention #MachineLearning #RAG #SoftwareArchitecture #DeepLearning #MLEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d6a013bbd18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parameters: The Frozen Knowledge Inside a 7B or 70B Model]]></title>
            <link>https://medium.com/@vanshsharma9354/parameters-the-frozen-knowledge-inside-a-7b-or-70b-model-2150dd8127f8?source=rss-cf70bfe7604e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2150dd8127f8</guid>
            <category><![CDATA[llm-parameters]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Vansh Sharma]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 05:25:30 GMT</pubDate>
            <atom:updated>2026-03-28T05:25:30.763Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CqZC4nGo4sYjdCwDuqjrvg.jpeg" /></figure><p><em>Series: An Engineer’s Explorations in LLM Concepts</em></p><blockquote><em>“7 billion parameters. 70 billion parameters. You see these numbers on every model card, every benchmark, every research paper. But what does the number actually mean?”</em></blockquote><p>In Part 1, we learned that your text becomes a stream of tokens. In Part 2, we learned you’re billed for every token that flows through the API. Now comes the natural next question: <strong>what happens to those tokens once the model receives them?</strong></p><p>The answer is: they flow through billions of parameters. And understanding what a parameter is — really is — changes how you think about everything LLMs can and cannot do.</p><h3>The Simple Explanation</h3><p>A <strong>parameter</strong> is just a number.</p><p>Not a rule. Not a fact. Not a piece of code. Just a decimal number — something like 0.73 or -1.24 or 0.0031.</p><p>A 7B model has <strong>7,000,000,000</strong> of these numbers. A 70B model has 70 billion. These numbers are the model’s entire brain — every language pattern, every fact, every reasoning ability the model has is encoded somewhere across these billions of decimal values.</p><p>They’re called “frozen” because after training ends, they never change again. When you call the API, the parameters just sit there — static — while your tokens flow through them.</p><h3>What Is a Parameter, Technically?</h3><p>Parameters are the <strong>weights on the connections between neurons</strong> in a neural network.</p><p>Think of a neuron as a small calculator. It takes in some numbers, multiplies each by a weight (parameter), adds them up, and passes the result forward. Do this across billions of neurons, organized into layers, and you have a transformer.</p><h3>How Do Parameters Get Their Values? (Training)</h3><p>Parameters don’t start out meaningful. They begin as <strong>random numbers</strong>.</p><p>Say the training text is: &quot;The capital of France is ___&quot;</p><ul><li>The model predicts &quot;Berlin&quot; (wrong)</li><li>Loss is calculated: prediction was bad</li><li>All 7 billion parameters are nudged — some up, some down — to make &quot;Paris&quot; slightly more likely next time</li><li>This happens for trillions of examples over weeks of compute time</li></ul><p>After enough nudges, the knowledge that &quot;Paris&quot; follows &quot;capital of France is&quot; is encoded — somehow, distributed across millions of those decimal numbers. Not in any single parameter. Spread across all of them.</p><p>This is what makes LLMs fascinating and a little mysterious: there’s no row in a database that says France → Paris. That knowledge is <em>distributed</em> across billions of weights.</p><h3>Why Does Model Size (7B vs 70B) Matter?</h3><p>More parameters = more capacity to store subtle patterns.</p><p>But here’s the important nuance:</p><blockquote><strong><em>Parameters ≠ Quality. Parameters = Capacity.</em></strong></blockquote><p>A 70B model trained on garbage data will underperform a well-trained 7B. The training data quality, the training process (RLHF, Constitutional AI, etc.), and the fine-tuning approach matter just as much as raw size.</p><p>This is why Mistral’s 7B punched well above its weight class when it launched — exceptional training on a smaller architecture.</p><h3>Parameters Are Frozen — What That Means in Practice</h3><p>This is the root cause of:</p><ul><li><strong>Knowledge cutoffs</strong> — the model doesn’t know about events after training ended</li><li><strong>Hallucinations</strong> — the model sometimes fills gaps with confident-sounding guesses rather than saying “I don’t know” (more on this in Part 6)</li><li><strong>Why RAG works</strong> — by injecting fresh information into the context window, you bypass the frozen knowledge limitation entirely</li></ul><h3>The Mental Model to Keep</h3><blockquote><em>Parameters are the frozen, distributed memory of everything the model learned during training. They’re not a lookup table — knowledge is spread across billions of decimal numbers as geometric patterns. The model size tells you the capacity of that memory; the training process determines the quality of what’s stored in it.</em></blockquote><p>Once you see parameters this way, several things click:</p><ul><li>Why models have knowledge cutoffs (training ended, parameters froze)</li><li>Why bigger models know more subtle things (more capacity)</li><li>Why you can’t “teach” a model by chatting with it (parameters don’t change at inference)</li><li>Why the same architecture trained on better data beats a larger model on worse data</li></ul><h3>What’s Next</h3><p>We’ve now traced the full journey: tokens come in, flow through embeddings, pass through transformer layers with billions of parameter multiplications, and produce output tokens one at a time.</p><p>But there’s a physical constraint on all of this: <strong>how many tokens can the model process at once?</strong> That limit has a name, and it affects every architectural decision you’ll make when building with LLMs.</p><p>In <strong>Part 4 — Context Window: The Hard Limit on What an LLM Can See</strong>, we’ll explore why this constraint exists, what engineering tricks are used to push it further, and what really happens when you exceed it.</p><p><em>Series: An Engineer’s Explorations in LLM Concepts Part 3 of 6</em></p><p><strong>Tags:</strong> #LLM #MachineLearning #NeuralNetworks #AIEngineering #DeepLearning #Transformers #SoftwareEngineering #Parameters #ModelArchitecture</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2150dd8127f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Billing: How Input Tokens, Output Tokens and History Add Up]]></title>
            <link>https://medium.com/@vanshsharma9354/llm-billing-how-input-tokens-output-tokens-and-history-add-up-c343301b8368?source=rss-cf70bfe7604e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c343301b8368</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Vansh Sharma]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 17:13:18 GMT</pubDate>
            <atom:updated>2026-03-25T17:13:18.926Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dqw6YLMGUIVUXS-AIQxICw.jpeg" /></figure><p><em>Series: An Engineer’s Explorations in LLM Concepts</em></p><blockquote><em>“I thought I understood the pricing. It’s just input tokens plus output tokens, right? . The bill wasn’t for my last message. It was for every message. Every single time.”</em></blockquote><p>If you’ve read Part 1, you know what tokens are. Now comes the part that directly hits your bill, your product margins, and your architecture decisions: <strong>how does LLM pricing actually work?</strong></p><h3>The Simple Answer (Plain English First)</h3><p>Every time you call an LLM API, you are charged for:</p><ol><li><strong>Every token you send in</strong> (your prompt, history, system instructions)</li><li><strong>Every token the model sends back</strong> (the response)</li></ol><p>That’s it. But the difference is entirely in what “every token you send in” actually means.</p><h3>The Billing Formula</h3><pre>Total Cost = (Input Tokens × Input Price) + (Output Tokens × Output Price)</pre><p>For reference, here are approximate prices as of mid-2025 (always check the official docs — prices change):</p><p>Model Input (per 1M tokens) Output (per 1M tokens) Claude Sonnet 4 ~$3.00 ~$15.00 GPT-4o ~$2.50 ~$10.00 Gemini 1.5 Pro ~$1.25 ~$5.00 Claude Haiku 4 ~$0.80 ~$4.00</p><p>Two things jump out immediately:</p><ol><li><strong>Output tokens are 3–5x more expensive than input tokens.</strong> This is because generating a token requires a full forward pass through the model, while reading an input token is cheaper computationally.</li><li><strong>The price per million sounds small — until you’re at scale.</strong></li></ol><h3>The Hidden Cost: You’re Paying for History Too</h3><p>Here’s the part that surprises:</p><p>LLMs are <strong>stateless</strong>. The model has no memory between calls. Every single API call is a fresh start.</p><p>So how does a chatbot “remember” what you said three messages ago?</p><p><strong>You send the entire conversation history on every call.</strong> User-&gt;&gt;Your App: Message 1: &quot;Hi, my name is Arjun&quot;<br> Your App-&gt;&gt;LLM API: [Message 1]<br> LLM API--&gt;&gt;Your App: &quot;Hello Arjun! How can I help?&quot;</p><pre>    User-&gt;&gt;Your App: Message 1: &quot;Hi, my name is Arjun&quot;<br>    Your App-&gt;&gt;LLM API: [Message 1]<br>    LLM API--&gt;&gt;Your App: &quot;Hello Arjun! How can I help?&quot;</pre><pre>    User-&gt;&gt;Your App: Message 3: &quot;We build fintech tools&quot;<br>    Your App-&gt;&gt;LLM API: [Message 1 + Reply 1 + Message 2 + Reply 2 + Message 3]<br>    LLM API--&gt;&gt;Your App: &quot;Fintech is a great space...&quot;</pre><p>Notice what’s happening: by Message 3, you’re sending Messages 1 and 2 <em>and their replies</em> just to provide context. The model doesn’t ask for this — <strong>your application code is responsible for bundling and sending all of it.</strong></p><h3>What “Input Tokens” Actually Includes</h3><p>On every single API call, input tokens = everything in this stack:</p><pre>    A --&gt; E[&quot;📦 Full Input\nSent to the API\non every call&quot;]<br>    B --&gt; E<br>    C --&gt; E<br>    D --&gt; E</pre><pre>    E --&gt; F[&quot;💰 Billed as\nInput Tokens&quot;]</pre><p>Your actual new question might be 20 tokens. But if you have a 500-token system prompt and 2000 tokens of history, you’re paying for 2520 tokens of input — just to ask a 20-token question.</p><h3>The Compounding Cost Problem</h3><p>Let’s make this concrete with numbers.</p><p>Assume:</p><ul><li>System prompt: <strong>200 tokens</strong> (constant)</li><li>Each user message: ~<strong>50 tokens</strong> average</li><li>Each model reply: ~<strong>150 tokens</strong> average</li><li>So each exchange adds ~<strong>200 tokens</strong> to the history</li></ul><p>Here’s how costs compound across a conversation (using Claude Sonnet at $3/1M input, $15/1M output):</p><p>Message # Input Tokens Sent Output Tokens Input Cost Output Cost Call Total 1 250 (system + msg) 150 $0.00075 $0.00225 $0.003 3 650 150 $0.00195 $0.00225 $0.0042 5 1,050 150 $0.00315 $0.00225 $0.0054 10 2,050 150 $0.00615 $0.00225 $0.0084 20 4,050 150 $0.01215 $0.00225 $0.0144</p><p>A single conversation of 20 messages costs roughly <strong>$0.15–0.20</strong> in API calls alone — and the cost per message keeps growing because the history keeps growing.</p><p>Now multiply by 10,000 users a day.</p><h3>The True Shape of Your API Cost</h3><p>The cost isn’t flat. It grows linearly with every exchange. A 20-message conversation costs roughly <strong>8x more per call</strong> than the first message — even if the user’s messages stay the same length.</p><h3>Developer Strategies to Manage History Costs</h3><p>Knowing this, experienced engineers use several patterns to keep costs under control:</p><h3>1. Sliding Window — Keep Only the Last N Turns</h3><pre>MAX_HISTORY_TURNS = 10  # keep last 10 exchanges</pre><p><strong>Trade-off:</strong> The model loses early context. Fine for most support/Q&amp;A bots, problematic for tasks that require remembering the beginning of a long session.</p><h3>2. Summarization — Compress Old History</h3><p>Instead of dropping old messages, summarize them:text\nSummary + Messages 11–15&quot;]</p><p><strong>Trade-off:</strong> You spend tokens on summarization, but the compressed summary is much cheaper to carry forward than the raw history.</p><h3>3. Selective Memory — Store and Retrieve, Don’t Carry</h3><p>For long-running applications (days/weeks of history), don’t stuff history into the context at all. Use a vector database:</p><p>This is the foundation of <strong>RAG (Retrieval Augmented Generation)</strong> — and it’s why RAG reduces costs while also improving relevance.</p><h3>4. Model Tiering — Use Cheap Models Where You Can</h3><p>Not every step in your pipeline needs the most capable model:<br> B -- Code generation --&gt; D</p><p>Classification itself is cheap. Routing to the right model based on complexity saves significant costs at scale.</p><h3>The Platform Layer: Who Actually Pays?</h3><p>This is a question worth understanding if you’re building on top of LLM-powered tools.</p><p>Key insight: <strong>Platforms like Perplexity, Cursor, and GitHub Copilot are Anthropic/OpenAI API customers.</strong> They pay per token on the back end and recover costs (and margin) through subscriptions on the front end.</p><p>When Perplexity gives you “unlimited” searches, they’re betting that the average user’s token consumption costs less than their subscription fee. It’s an actuarial calculation — exactly like insurance.</p><p><strong>Claude.ai is different</strong> — Anthropic owns both the platform and the model, so the cost is internal. There’s no markup to a third-party API. This is why usage limits on Claude.ai are set by business logic, not a hard cost ceiling per user.</p><h3>The Mental Model to Keep</h3><blockquote><strong><em>Every LLM API call is a fresh conversation.</em></strong><em> Your application is responsible for feeding the model its memory on every call. You pay for every token of that memory, every time. The cost doesn’t just grow with usage — it grows with conversation length too.</em></blockquote><h3>What’s Next</h3><p>We’ve covered what tokens are and how they’re billed. But there’s a deeper question we’ve been dancing around: <em>what does the model actually do with those tokens once it receives them?</em></p><p>The answer involves 7 billion (or 70 billion) numbers, a process called forward propagation, and a surprisingly elegant idea called an embedding vector.</p><p>In <strong>Part 3 — Parameters: The Frozen Knowledge Inside a 7B or 70B Model</strong>, we’ll read about this.</p><p><em>Series: An Engineer’s Explorations in LLM Concepts ( Part 2 of 6 )</em></p><p><strong>Tags:</strong> #LLM #AIEngineering #APIDesign #MachineLearning #SoftwareEngineering #CostOptimization #GPT #Claude #Tokens #ProductEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c343301b8368" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tokens: The Atom of Everything in Large Language Models]]></title>
            <link>https://pub.towardsai.net/tokens-the-atom-of-everything-in-large-language-models-212b459b0e96?source=rss-cf70bfe7604e------2</link>
            <guid isPermaLink="false">https://medium.com/p/212b459b0e96</guid>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Vansh Sharma]]></dc:creator>
            <pubDate>Sun, 22 Mar 2026 04:22:00 GMT</pubDate>
            <atom:updated>2026-03-22T05:30:22.291Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YfM1N7YE75oTuzUtoAOFOQ.png" /></figure><h4><em>Series: An Engineer’s Explorations in LLM Concepts</em></h4><blockquote><em>“It started with a simple question: </em>What exactly is a token?<em> Three hours later, I had filled six browser tabs, youtube videos and had somehow ended up reading about some 5–6 concepts of LLM.”</em></blockquote><p>If you’ve worked with any LLM API — OpenAI, Anthropic, Gemini — you’ve seen the word <em>token</em> everywhere. In the docs, in the pricing, in the error messages. And at some point you’ve probably thought: <em>okay but what actually is a token?</em></p><h3>The Simple Answer (Plain English First)</h3><p>Think of a token as a <strong>chunk of text</strong> — not necessarily a full word, not a single character, but something in between.</p><p>The sentence:</p><pre>&quot;I love tokenization&quot;</pre><p>…doesn’t become 3 tokens (one per word). It becomes something like:</p><pre>[&quot;I&quot;, &quot; love&quot;, &quot; token&quot;, &quot;ization&quot;]</pre><p>Four tokens. The word “tokenization” split in two because the tokenizer decided “token” and “ization” are more useful chunks to know separately.</p><p>That’s the intuition. Now let’s go deeper.</p><h3>What Is a Token, Technically?</h3><p>A token is <strong>the basic unit of text</strong> that an LLM reads, processes, and generates. Everything that flows through a language model — your prompt, the system instructions, the model’s reply — is processed as a sequence of tokens.</p><p>There are roughly <strong>three types</strong> of tokens:</p><p>Type Example Notes Full word &quot;the&quot;, &quot;is&quot;, &quot;and&quot; Very common words are often single tokens Sub-word &quot;token&quot; + &quot;ization&quot; Rare or long words get split Character-level &quot;x&quot;, &quot;Z&quot; Very rare in modern tokenizers</p><blockquote><strong><em>Rule of thumb:</em></strong><em> 1 token ≈ 4 characters of English text, or roughly ¾ of a word. So 100 tokens ≈ 75 words</em></blockquote><h3>How Does the Tokenizer Decide Where to Split?</h3><p>This is where it gets interesting. The tokenizer doesn’t follow grammar rules or dictionary definitions. It uses an algorithm called <strong>Byte Pair Encoding (BPE)</strong>.</p><p>Here’s how BPE works:</p><pre>flowchart TD<br>    A[&quot;Start: Raw Training Text\n(billions of words)&quot;] --&gt; B[&quot;Step 1: Split everything\ninto individual characters&quot;]<br>    B --&gt; C[&quot;Step 2: Count all adjacent\ncharacter pairs&quot;]<br>    C --&gt; D[&quot;Step 3: Merge the most\nfrequent pair into one token&quot;]<br>    D --&gt; E{&quot;Vocabulary\nsize reached?&quot;}<br>    E -- No --&gt; C<br>    E -- Yes --&gt; F[&quot;Final Vocabulary\n~50,000 tokens&quot;]<br>    F --&gt; G[&quot;Tokenizer is frozen\nand shipped with the model&quot;]</pre><p><strong>Concrete example:</strong></p><p>Imagine the word &quot;low&quot; appears 5 times and &quot;lower&quot; appears 40 times in training data.</p><ul><li>BPE sees l, o, w, e, r as individual characters first</li><li>It notices l+o appears very frequently → merges into lo</li><li>Then lo+w is frequent → merges into low</li><li>Then low+e → lowe</li><li>Then lowe+r → lower</li></ul><p>After enough merges, &quot;lower&quot; becomes a single token because it appears so often it&#39;s worth having a dedicated slot for.</p><h3>The Tokenization Pipeline — Step by Step</h3><p>Here’s the full journey from your text to tokens inside the model:</p><pre>flowchart LR<br>    A[&quot;Your Text\n&#39;My name is Arjun&#39;&quot;] --&gt; B[&quot;Tokenizer\n(BPE algorithm)&quot;]<br>    B --&gt; C[&quot;Token Strings\n[&#39;My&#39;, &#39; name&#39;, &#39; is&#39;, &#39; Ar&#39;, &#39;jun&#39;]&quot;]<br>    C --&gt; D[&quot;Token IDs\n[3421, 1438, 318, 1274, 18150]&quot;]<br>    D --&gt; E[&quot;Embedding Layer\nEach ID → vector of ~4000 numbers&quot;]<br>    E --&gt; F[&quot;Transformer Layers\n(the actual LLM)&quot;]<br>    F --&gt; G[&quot;Output Token IDs&quot;]<br>    G --&gt; H[&quot;Decoded back to text\nby the tokenizer&quot;]</pre><p>Notice step 4 — token IDs get converted into <strong>embedding vectors</strong> before the model ever sees them. The model doesn’t work with text or even token IDs directly. It works with lists of numbers. But that’s a story for the Parameters blog (Part 3).</p><h3>Why Do Tokens Vary Between Models?</h3><p>This is a nuance that trips up a lot of developers.</p><p><strong>GPT-4, Claude, and Gemini do NOT share the same tokenizer.</strong></p><p>Each model trains its own tokenizer on its own dataset. This means:</p><ul><li>The same sentence can produce a <strong>different number of tokens</strong> across models</li><li>Token IDs for the same word can be <strong>completely different</strong> between GPT-4 and Claude</li><li>A prompt that costs X tokens on OpenAI might cost Y tokens on Anthropic</li></ul><pre>flowchart TD<br>    A[&quot;Input: &#39;Tokenization is fascinating&#39;&quot;]<br>    A --&gt; B[&quot;GPT-4 Tokenizer\n→ 4 tokens&quot;]<br>    A --&gt; C[&quot;Claude Tokenizer\n→ 5 tokens&quot;]<br>    A --&gt; D[&quot;Gemini Tokenizer\n→ 4 tokens&quot;]</pre><pre>    B --&gt; E[&quot;Different IDs\nDifferent splits\nDifferent costs&quot;]<br>    C --&gt; E<br>    D --&gt; E</pre><p>The practical implication: <strong>always test token counts on the model you’re actually using.</strong> Don’t assume they’re the same.</p><h3>Edge Cases That Will Surprise You</h3><h3>1. Numbers are expensive</h3><pre>&quot;1234567&quot; → [&quot;123&quot;, &quot;456&quot;, &quot;7&quot;]   (3 tokens)</pre><p>Numbers don’t compress as well as words because they’re more varied in training data.</p><h3>2. Emojis are very expensive</h3><pre>&quot;🔥&quot; → [&quot;\xf0\x9f&quot;, &quot;\x94\xa5&quot;]   (2 tokens, sometimes more)</pre><p>Emojis are multi-byte Unicode characters. The tokenizer often has to use multiple tokens to represent a single emoji.</p><h3>3. Code has its own tokenization behavior</h3><pre># &quot;def calculate_total_price(items):&quot; <br># → [&quot;def&quot;, &quot; calculate&quot;, &quot;_total&quot;, &quot;_price&quot;, &quot;(&quot;, &quot;items&quot;, &quot;):&quot;]<br># That&#39;s 7 tokens for one function signature</pre><p>Underscores, camelCase, and symbols all split differently. This matters when you’re working on code-heavy prompts.</p><h3>4. Non-English text tokenizes less efficiently</h3><p>English dominates most training corpora, so English tokens are “fuller” — they pack more meaning per token. Hindi, Tamil, or Arabic text often requires more tokens to represent the same meaning. This has real billing implications for multilingual apps.</p><h3>How to Actually Count Tokens</h3><p>Don’t guess. Use the tools:</p><p><strong>For OpenAI:</strong></p><pre>pip install tiktoken</pre><pre>import tiktoken</pre><pre>enc = tiktoken.encoding_for_model(&quot;gpt-4&quot;)<br>tokens = enc.encode(&quot;My name is Arjun&quot;)<br>print(len(tokens))  # → 4</pre><p><strong>For Anthropic (Claude):</strong></p><pre>import anthropic</pre><pre>client = anthropic.Anthropic()<br>response = client.messages.count_tokens(<br>    model=&quot;claude-opus-4-5&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;My name is Arjun&quot;}]<br>)<br>print(response.input_tokens)</pre><p><strong>Quick online tools:</strong></p><ul><li><a href="https://platform.openai.com/tokenizer">platform.openai.com/tokenizer</a> — paste any text and see exactly how it splits</li></ul><h3>Why Does Any of This Matter?</h3><p>Because tokens are the <strong>unit of measurement for everything</strong> in the LLM world:</p><ul><li><strong>Billing</strong> is per token (we’ll dig into this in Part 2)</li><li><strong>Context window</strong> is measured in tokens (Part 4)</li><li><strong>Response speed</strong> depends on token count — more tokens = slower</li><li><strong>Thinking</strong> happens in tokens (Part 5)</li><li><strong>Hallucinations</strong> emerge from the token prediction process (Part 6)</li></ul><p>Every single concept in this series is downstream of this one. Tokens aren’t just a billing quirk — they are the atom that everything else is built from.</p><h3>The Mental Model to Keep</h3><p>After everything above, here’s the one thing to carry forward:</p><blockquote><strong><em>An LLM doesn’t read text. It reads a sequence of integer IDs, where each ID represents a learned chunk of language. The quality, cost, and behavior of every LLM interaction is shaped by how your text splits into those chunks.</em></strong></blockquote><p>The next time you type a prompt, you’re not sending words to the model. You’re sending a list of numbers — and the model’s entire world is built on top of them.</p><h3>What’s Next</h3><p>Now that you understand what tokens are, the natural next question is: <em>who’s counting them, and what does it cost?</em></p><p>In <strong>Part 2 — LLM Billing: How Input Tokens, Output Tokens and History Add Up</strong>, we’ll break down why your API bill is probably higher than you think — and exactly where those extra tokens are hiding.</p><p><em>Series: An Engineer’s Explorations in LLM Concepts ( Part 1 of 6 )</em></p><p><strong>Tags:</strong> #LLM #MachineLearning #NLP #SoftwareEngineering #AIEngineering #Tokenization #GPT #Claude #DeepLearning</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=212b459b0e96" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/tokens-the-atom-of-everything-in-large-language-models-212b459b0e96">Tokens: The Atom of Everything in Large Language Models</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>