<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Murali Krishnan Sankaran on Medium]]></title>
        <description><![CDATA[Stories by Murali Krishnan Sankaran on Medium]]></description>
        <link>https://medium.com/@smurali1717?source=rss-541d2777c31c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lT-qp3m7i0V3kAu-3UUTIw.jpeg</url>
            <title>Stories by Murali Krishnan Sankaran on Medium</title>
            <link>https://medium.com/@smurali1717?source=rss-541d2777c31c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:56:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@smurali1717/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Token Math in Production: Why Your Budget Is Lying to You]]></title>
            <link>https://medium.com/@smurali1717/token-math-in-production-why-your-budget-is-lying-to-you-e9961a0f9e23?source=rss-541d2777c31c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9961a0f9e23</guid>
            <dc:creator><![CDATA[Murali Krishnan Sankaran]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 15:08:40 GMT</pubDate>
            <atom:updated>2025-11-08T15:11:19.808Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Penny-wise, pound-foolish: Why Token Optimisation Backfires</blockquote><p>One of the most overlooked realities when using a foundation model — either directly or through fine-tuning — is the token economy. During prototyping, everything seems fine: responses are clean, token usage is low, and costs aren’t a concern. But once the system scales to 10,000+ requests per day, the numbers become daunting. Token limits vanish, and what looked “efficient” in testing quickly turns into a production budget drain. In response, many AI engineers chase false optimisation — trimming prompts, dropping context, or removing structure — in hopes of saving a few tokens per call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n4d_wZXVwpdq1k89" /></figure><p>Optimising the input prompt may seem like saving the token budget, but when the input deviates from the core structure, the model starts to stumble, and you will get junk responses. At last, you end up burning your additional tokens unknowingly. Real optimisation isn’t about shortening text; it’s about keeping it clear and consistent so the model gets it right the first time. The error rate percentage is the key ingredient in this equation, and that’s where actual token efficiency actually begins.</p><h4><strong>The True Equation — Why Your Token Math Is Lying to You</strong></h4><p>Most teams estimate LLM cost with a simple, tidy formula:</p><pre>Total cost ≈ (Input tokens × Input price) + (Output tokens × Output price)</pre><p>Looks logical, right? Except it’s completely wrong in the real world. That equation assumes every response is perfect on the first try — no retries, no malformed JSON, no hallucinations. But in production, that’s fantasy.</p><p>Now comes the part nobody puts in the spreadsheet — the error rate. Without structure, for example, roughly 10% of responses come back broken — missing fields, wrong totals, or invalid formats. Each one has to be regenerated, burning another complete request.</p><p>That isn’t optimisation; it’s self-sabotage. The tiny 1% token saving ends up inflating the total cost by 30–40%.</p><h3>Building a Smarter Buffer Strategy for Token Usage</h3><p>Every LLM or API project needs a safety cushion — a <strong>token buffer</strong> — that absorbs the chaos of retries, timeouts, and unpredictable output lengths. The trick is not guessing a number but <strong>designing a strategy that evolves with real data.</strong></p><p>So let’s break this down into <strong>two stages</strong>:</p><ol><li><strong>Unknown Stat</strong> — Build for Survival (When you have <strong>no data yet)</strong></li><li><strong>Known Stat</strong> — Build for Efficiency (When you have <strong>actual usage data)</strong></li></ol><h3>Stage 1: Unknown Stats — Build for Survival</h3><p>At this point, you have no logs and no visibility into errors. Your system is running purely on optimism and caffeine. It is more about guessing, protecting, and collecting the details.</p><p>So, plan separately for <strong>input</strong> and <strong>output</strong> tokens, because they’re billed differently:</p><pre>input_buffer  = input_tokens × (1 + buffer_percent)<br>output_buffer = output_tokens × (1 + buffer_percent)<br>cached_buffer  = cached_tokens × (1 + buffer_percent)</pre><blockquote><em>The “1” represents </em><strong><em>100% of your base token count</em></strong><em> — it keeps the full original estimate.<br>The </em><em>buffer_percent (for example, </em><em>0.2 = 20%) is the </em><strong><em>extra margin</em></strong><em> added on top.<br>Together, </em><em>(1 + buffer_percent) means “take the full estimate plus some safety room.”</em></blockquote><p>Then calculate your total estimated cost:</p><pre>total_cost = (input_buffer × input_rate) + (output_buffer × output_rate)  + (cached_buffer × cached_rate)</pre><blockquote><strong>Note:</strong><br>If you don’t have any cached data, remove the cached token and rate from the formula. Also, if your input and output prices are the same, use the same rate for both calculations.</blockquote><h3>Example</h3><pre>input_tokens  = 800<br>output_tokens = 200<br>buffer_percent = 1.2<br>input_rate  = $0.00000125 # OpenAI Rates<br>output_rate = $0.00001 # OpenAI Rates<br>per_day_transaction = 10,000<br>total_working_days = 22</pre><pre>input_buffer  = 800 × 1.2 = 960<br>output_buffer = 200 × 1.2 = 240</pre><pre>per_transaction_cost = (960 × 0.00000125) + (240 × 0.00001)<br>            = $0.0012 + $0.0024<br>            = $0.0036 per request</pre><pre>total_cost = per_day_transaction x per_transaction_cost x total_working_days</pre><pre>total_cost = $0.0036 x 10,000 x 22 =$792.00 </pre><p>That’s your <strong>early-stage cushion</strong>. When you know nothing, assume a <strong>20–25% buffer</strong> to stay safe.</p><p><strong>Before moving to Stage 2 — </strong>Don’t just collect logs — <strong>build the plumbing</strong> first. This stage is where you prepare your system for data-driven optimisation later.</p><p>You need to:</p><ul><li>Enable <strong>API call logging</strong> that captures token usage, response length, and error reason.</li><li>Record <strong>success vs. failure</strong> calls with timestamps.</li><li>Integrate <strong>governance and observability hooks</strong> (e.g., Watsonx Governance, Prometheus, OpenTelemetry) to track retries, latency, and token overflows.</li><li>Tag every request with metadata (model, prompt type, user flow) so you can segment usage data later.</li></ul><p>The goal isn’t to analyse yet— it’s to ensure your <strong>infrastructure is ready to generate clean, auditable metrics</strong> for Stage 2.</p><h3>Stage 2: Known Stats — Build for Efficiency</h3><p>Once your governance layer is in place and logs start streaming, it’s time to make decisions based on <strong>facts</strong>, not assumptions.</p><p>From your logs or governance dashboard, extract:</p><ul><li><strong>Average (mean)</strong> input and output tokens per request</li><li><strong>Standard deviation (σ)</strong> — how much the token counts fluctuate</li><li><strong>Error and retry percentages</strong></li></ul><p>This data reveals the consistency of your model. A small σ means stable token usage; a large σ means unpredictable behaviour.</p><p>Now, instead of a fixed 20% guess, switch to a <strong>data-driven adaptive buffer</strong>:</p><pre>input_buffer  = mean_input  + (k × std_input)<br>output_buffer = mean_output + (k × std_output)</pre><p>Where</p><ul><li><strong>k = 2</strong> covers ≈ 95% of normal cases (balanced safety and cost)</li><li><strong>k = 3</strong> covers ≈ 99.7% of cases (maximum safety for critical workloads)</li></ul><p>Then calculate the total cost:</p><pre>total_cost = (input_buffer × input_rate) + (output_buffer × output_rate)</pre><p>Next, tighten control:</p><ul><li><strong>Refine input prompts</strong> and <strong>constrain output formats</strong> to reduce token variance.</li><li>Track how these refinements lower <strong>error and retry rates</strong> over time.</li><li>Re-evaluate weekly until the required buffer stabilises — ideally below <strong>10–15%</strong>.</li></ul><p>At this stage, your buffer has evolved from a blind safety margin into a <strong>feedback-driven cost optimiser</strong> grounded in real production data.</p><h3><strong>When Verbose Structure Wins — and When Minimalism Makes More Sense</strong></h3><p>To decide whether to go <strong>verbose</strong> or <strong>minimal</strong>, you need more than intuition — it requires a bit of <strong>deep analysis</strong>. It’s not just about what you feed into the model or whether you get a decent-looking output. The <em>purpose</em> of the task decides the format.</p><p>If the task is <strong>subjective, creative, or loosely bounded</strong>, a verbose structure often adds no value. In fact, rigid structures can sometimes be confusing rather than clarifying. But in <strong>enterprise or precision-driven scenarios</strong>, structure isn’t a burden — it’s insurance. When you’re generating <strong>API-style responses, financial reconciliations, analytics summaries, or compliance reports</strong>, every token must mean something specific.</p><h3><strong>Key Takeaway</strong></h3><p>No matter the pricing model — <strong>dynamic, cached, or flat</strong> — the key is to:</p><ul><li>Combine <strong>accurate token math</strong> with a <strong>realistic buffer</strong>.</li><li>Track <strong>usage, success rates, and retries</strong> over time.</li><li>Continuously <strong>tune your multiplier</strong> as data improves.</li><li>That <strong>25% buffer</strong> isn’t a luxury — it’s protection against <strong>regeneration retries, API timeouts, random output-length variations, and caching inconsistencies across deployments.</strong></li><li>In practice, you’ll refine this number over time — <strong>start with a 20–25% buffer, collect logs on failed vs. successful requests, measure average token usage per request, and adjust the buffer percentage as real data accumulates.</strong></li></ul><p>Token pricing isn’t guesswork. It’s structured math trying to survive production chaos.</p><p>I’m planning a follow-up blog on <strong>GenAI non-functional requirements</strong> — what they really mean in practice and how to integrate them into your lifecycle.</p><p><strong>Keep watching this space for more….</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9961a0f9e23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>