<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tarush Singh on Medium]]></title>
        <description><![CDATA[Stories by Tarush Singh on Medium]]></description>
        <link>https://medium.com/@singh.tarus?source=rss-13f364d82365------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LEGXweuUQGgkPOSwxXCLkA.jpeg</url>
            <title>Stories by Tarush Singh on Medium</title>
            <link>https://medium.com/@singh.tarus?source=rss-13f364d82365------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:20:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@singh.tarus/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Letting LLMs Struggle with Tables — Make Them Think First]]></title>
            <link>https://medium.com/@singh.tarus/stop-letting-llms-struggle-with-tables-make-them-think-first-758ebe88342b?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/758ebe88342b</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[hallucinations-in-llms]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Tue, 05 May 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-05-05T12:31:01.619Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TiBEsJdSmvJNd1cOiKtPMA.png" /></figure><p>Most people try to use LLMs with tabular data in the most direct way possible:</p><blockquote><em>“Here is my Tabular data. Now answer my question.”</em></blockquote><p>At first, this feels reasonable. LLMs are good at understanding text, CSVs contain text, and the model should be able to reason over it.</p><p>But this breaks down very quickly.</p><p>The issue is not that the model is bad. The issue is that we are asking it to do too many things at once.</p><p>When an LLM receives a raw table, it has to:</p><ul><li>understand what the dataset is about</li><li>identify which columns matter</li><li>scan through noisy rows</li><li>perform filtering or grouping</li><li>count values correctly</li><li>finally generate an answer</li></ul><p>That is a lot of responsibility to push into a single prompt.</p><p>And worse, some of those steps are not really reasoning tasks. They are computation tasks.</p><p>For example, say we ask:</p><blockquote><strong><em>Which of the folks had penicillin-related allergies?</em></strong></blockquote><p>This sounds simple. But if the data lives inside a CSV, the model still needs to figure out which column contains allergy information, inspect the values in that column, notice spelling variations like Penicilin, and count the matching records.</p><p>That is exactly the kind of task where LLMs can become unreliable.</p><p>Not because the question is hard.</p><p>Because the context is messy.</p><p>So instead of forcing the model to read the whole table and answer directly, I tried a different approach:</p><blockquote><em>Let the LLM decide </em><strong><em>what needs to be computed</em></strong><em>, then let code perform the computation.</em></blockquote><p>In other words:</p><blockquote><strong><em>LLM as planner. Code as executor.</em></strong></blockquote><p>That small shift made the system much more reliable.</p><p>Instead of feeding the model an entire CSV, the pipeline first converts the user’s question into a structured directive. That directive tells the system which columns matter, whether a grouping operation is needed, what filters apply, and what smaller context should be sent back to the model.</p><p>The LLM does not need to count rows anymore.</p><p>It only needs to reason over a reduced, cleaner view of the data.</p><p>That is the core idea of this article.</p><p>We are not building a full SQL agent.<br> We are not dumping raw CSVs into a prompt.<br> We are not pretending embeddings alone can preserve table structure.</p><p>We are building a small bridge between natural language, structured data, and deterministic computation.</p><p>Next, let’s walk through how the dataset is first converted into something useful for retrieval.</p><h3>Turning a CSV Into Something an LLM Can Actually Understand</h3><p>Before we even get to querying the data, there’s a more fundamental problem:</p><blockquote><em>How do you </em>store<em> a CSV so that an LLM can retrieve it meaningfully?</em></blockquote><p>Most approaches go one of two ways:</p><ul><li>embed every row (breaks structure, noisy)</li><li>embed the entire CSV (too large, low signal)</li></ul><p>Neither works well.</p><p>Instead, I took a different approach:</p><blockquote><em>Represent the dataset as a </em><strong><em>semantic description of its structure</em></strong><em>, not its raw contents.</em></blockquote><h3>Step 1: Extract Structure, Not Just Data</h3><p>When loading a CSV, we don’t pass everything forward. We extract:</p><ul><li>column names</li><li>a small sample of values</li><li>enough signal to understand what each column represents</li></ul><p>Implementation:</p><pre>def load_csv(self):<br>    rows = load_csv_file(self.path)<br>    if not rows:<br>        return []<br>    header = rows[0]<br>    sample_data = rows[1:6]<br>    structured_columns = [<br>        {<br>            &quot;column_name&quot;: col,<br>            &quot;sample_values&quot;: [row[i] for row in sample_data if i &lt; len(row)]<br>        }<br>        for i, col in enumerate(header)<br>    ]</pre><p>This does something subtle but important:</p><blockquote><em>It compresses the dataset into </em><strong><em>just enough context</em></strong><em> for the LLM to reason about its schema.</em></blockquote><h3>Step 2: Let the LLM Describe the Dataset</h3><p>Now instead of embedding raw data, we prompt the LLM to generate a structured description:</p><pre>prompt = PromptTemplate.from_template(<br>    &quot;&quot;&quot;<br>    You are analyzing a dataset.<br>    Columns with sample values:<br>    {columns}<br>    Write a clear textual description...<br>    &quot;&quot;&quot;<br>)<br></pre><p>The output looks like:</p><ul><li>a short dataset summary</li><li>a list of column descriptions</li></ul><p>This becomes the <strong>representation stored in the vector database</strong>.</p><h3>Why This Matters</h3><p>At this point, we’ve made a key shift:</p><blockquote><em>We are no longer storing </em>data<em>.<br> We are storing </em><strong><em>understanding of the data</em></strong><em>.</em></blockquote><p>That means when a query comes in like:</p><blockquote><em>“penicillin-related allergies”</em></blockquote><p>The retrieval step doesn’t rely on matching raw rows.</p><p>It relies on:</p><ul><li>column semantics</li><li>dataset meaning</li><li>contextual similarity</li></ul><p>This is far more reliable.</p><h3>Step 3: Store It as a Document</h3><p>Instead of chunking rows, we store <strong>one document per dataset</strong>:</p><pre>Document(<br>    page_content=response.content,<br>    metadata={<br>        &quot;columns&quot;: header,<br>    },<br>)</pre><p>This is intentional.</p><ul><li>No fragmentation</li><li>No row-level noise</li><li>No broken relationships</li></ul><p>Just one clean representation.</p><h3>The Result</h3><p>After this step, the system can:</p><ul><li>retrieve the right dataset based on meaning</li><li>know what columns exist</li><li>understand what kind of operations are possible</li></ul><p>But it still hasn’t touched the actual data.</p><p>And that’s exactly what we want.</p><blockquote><em>At this stage, the system knows </em>what the data looks like<em>, but not yet </em>how to use it<em>.</em></blockquote><p>That decision comes next.</p><h3>Converting the Question Into a Query Plan</h3><p>At this point, I have:</p><ul><li>the dataset schema (from earlier)</li><li>the actual data loaded as a DataFrame</li><li>the user’s question</li></ul><p>Now, instead of asking the LLM to answer directly, I do something different:</p><blockquote><em>I ask it to tell me </em><strong><em>how to retrieve the answer</em></strong><em>.</em></blockquote><h3>The Key Shift</h3><p>Instead of:</p><blockquote><em>“Answer this question using the table”</em></blockquote><p>I prompt:</p><blockquote><em>“Given this table and question, generate a structured way to query the data”</em></blockquote><p>That structure looks like this:</p><pre>{<br>  &quot;identifier_columns&quot;: [&quot;ID&quot;],<br>  &quot;groupby_columns&quot;: [&quot;allergies&quot;],<br>  &quot;row_range&quot;: [],<br>  &quot;task&quot;: &quot;statistical summary&quot;,<br>  &quot;filters&quot;: {&quot;allergies&quot;: &quot;penicillin&quot;}<br>}</pre><p>This is not just a random format.</p><p>It encodes everything needed to reduce the dataset:</p><ul><li><strong>identifier_columns</strong> → how to uniquely identify rows if needed</li><li><strong>groupby_columns</strong> → what dimension to aggregate on</li><li><strong>task</strong> → what operation to perform (in this case, a summary)</li><li><strong>filters</strong> → what values we care about</li><li><strong>row_range</strong> → optional slicing if needed</li></ul><h3>How I Generate This</h3><p>Inside my response pipeline, I prompt the LLM like this:</p><pre>prompt = PromptTemplate(<br>                    input_variables=[&quot;content&quot;, &quot;question&quot;, &quot;total&quot;],<br>                    template=&quot;&quot;&quot;<br>                    You are a helpful assistant for answering questions about tabular data in a spreadsheet.<br>                    Your job is to analyze the question and return a JSON object indicating how to retrieve the relevant data.<br><br>                    Content (Column descriptions):<br>                    {content}<br><br>                    Question: {question}<br>                    Total rows: {total}<br><br>                    Rules:<br>                    - &quot;identifier_columns&quot;: Columns that uniquely identify a row (e.g. ID, name, code). Always include these when task is null so rows can be distinguished in the result. Never use these for grouping.<br>                    - &quot;groupby_columns&quot;: Select a list of columns that can be used for grouping. Can be empty if no grouping is needed.<br>                    - &quot;row_range&quot;: [start_index, end_index] (inclusive) of relevant rows. Leave as [] if not applicable.<br>                    - &quot;task&quot;: Choose from the options given below.<br>                        Options for &quot;task&quot;:<br>                        - &quot;statistical summary&quot;: Return a statistical summary (count, unique values)<br>                        - null: Return specific rows with identifier columns<br>                    - &quot;filters&quot;: For a provided groupby column, based on question derive an applicable filter value.<br>                    Output format:<br>                    {{<br>                        &quot;identifier_columns&quot;: [&lt;list of unique/ID column names&gt;],<br>                        &quot;groupby_columns&quot;: [&lt;list of categorical column names for grouping&gt;],<br>                        &quot;row_range&quot;: [start_row_index, end_row_index],<br>                        &quot;task&quot;: &quot;statistical summary&quot; | null,<br>                        &quot;filters&quot;: {{&lt;Key of each groupby_columns value : filter value based on question else null&gt;}}<br>                    }}<br><br>                    Return only valid JSON. No commentary, no markdown.<br>                    &quot;&quot;&quot;.strip()<br>                )</pre><blockquote><em>I’m forcing the model into a </em><strong><em>structured output that maps directly to code execution</em></strong><em>.</em></blockquote><h3>Why This Works So Well</h3><p>This step isolates what LLMs are actually good at:</p><ul><li>understanding intent</li><li>mapping natural language to structure</li><li>selecting relevant dimensions</li></ul><p>It removes what they’re bad at:</p><ul><li>counting</li><li>grouping</li><li>handling large noisy data</li></ul><h3>What This Unlocks</h3><p>At this stage, I’ve essentially converted:</p><blockquote><em>natural language → executable query plan</em></blockquote><p>And now, the system is no longer ambiguous.</p><p>It knows exactly:</p><ul><li>what to compute</li><li>how to compute it</li><li>what subset of data matters</li></ul><blockquote><em>This is the moment where the pipeline stops being “LLM-driven guessing”<br> and becomes </em><strong><em>LLM-guided computation</em></strong><em>.</em></blockquote><p>Next, I take this directive and actually execute it on the dataset.</p><p>That’s where things become deterministic.</p><h3>Executing the Plan (Where Hallucinations Disappear)</h3><p>Once I have the JSON directive, the LLM’s job is done.</p><p>From here on, I don’t rely on the model anymore.</p><blockquote><em>I switch to deterministic code.</em></blockquote><h3>Turning the Directive Into Computation</h3><p>The directive gets passed into a processing function:</p><pre>def process_tabular_directive(df: pd.DataFrame, directive: dict) -&gt; pd.DataFrame:<br>    identifier_columns = [c for c in directive.get(&quot;identifier_columns&quot;, []) or [] if c in df.columns]<br>    groupby_columns = [c for c in directive.get(&quot;groupby_columns&quot;, []) or []    if c in df.columns]<br>    row_range = directive.get(&quot;row_range&quot;, []) or []<br>    task = directive.get(&quot;task&quot;)<br>    task_str = (task or &quot;&quot;).strip().lower() if isinstance(task, str) else &quot;&quot;<br>    if task_str == &quot;statistical summary&quot;:<br>        if groupby_columns:<br>            return (<br>                df.groupby(groupby_columns, dropna=False)<br>                  .size()<br>                  .reset_index(name=&quot;count&quot;)<br>            )   </pre><p>There’s nothing “AI” happening here.</p><p>It’s just:</p><ul><li>Pandas</li><li>grouping</li><li>counting</li></ul><p>And that’s exactly the point.</p><h3>Why This Step Is Critical</h3><p>If I had asked the LLM to do this:</p><ul><li>it might miss rows</li><li>it might miscount</li><li>it might ignore edge cases</li><li>it might hallucinate</li></ul><p>But here:</p><blockquote><em>The output is guaranteed to be correct.</em></blockquote><h3>What the System Produces</h3><p>For the allergy example, the output becomes:</p><p>allergies count</p><p>Penicilin 2</p><p>bees 1</p><p>dust 2</p><p>grass/animal fur 1</p><p>mite 1</p><p>pollen15</p><p>pollen/hair dye 1</p><p>This is a massive reduction from the original dataset.</p><p>But more importantly:</p><blockquote><em>It is now </em><strong><em>perfectly structured for reasoning</em></strong><em>.</em></blockquote><h3>What Just Happened (And Why It Matters)</h3><p>Let’s zoom out for a second.</p><p>Before:</p><ul><li>LLM had to scan the full dataset</li><li>LLM had to infer structure</li><li>LLM had to compute counts</li></ul><p>Now:</p><ul><li>Structure is explicit</li><li>computation is done</li><li>only relevant information remains</li></ul><h3>The Hidden Win: Context Compression</h3><p>This step does something extremely valuable:</p><blockquote><em>It compresses the dataset into a </em><strong><em>high-signal representation</em></strong><em>.</em></blockquote><p>That means:</p><ul><li>fewer tokens sent to the LLM</li><li>faster responses</li><li>more accurate reasoning</li></ul><h3>The Key Principle</h3><blockquote><strong><em>Never ask an LLM to compute what code can compute deterministically.</em></strong></blockquote><p>Instead:</p><ul><li>let the LLM define the computation</li><li>let code execute it</li><li>feed the result back for reasoning</li></ul><h3>Why This Approach Actually Works</h3><p>This entire system boils down to one simple idea:</p><blockquote><strong><em>Separate reasoning from computation.</em></strong></blockquote><ul><li>The LLM decides <em>what needs to be done</em></li><li>The system executes it deterministically</li><li>The LLM explains the result</li></ul><p>Most pipelines try to do all three at once.</p><p>That’s where things break.</p><h3>Where This Goes Beyond CSVs</h3><p>While I built this around tabular data, the pattern is much bigger.</p><p>This same idea can be extended to:</p><ul><li>analytics systems</li><li>log analysis</li><li>structured RAG pipelines</li><li>dashboards powered by LLMs</li><li>even agent workflows</li></ul><p>Anywhere you have:</p><ul><li>structured data</li><li>repeatable operations</li><li>natural language queries</li></ul><p>this pattern applies.</p><h3>What I Learned Building This</h3><p>If there’s one thing this system made clear, it’s this:</p><blockquote><em>LLMs don’t fail because they are weak.<br> They fail because we overload them.</em></blockquote><p>We ask them to:</p><ul><li>understand</li><li>compute</li><li>and explain</li></ul><p>all in one shot.</p><p>That’s not what they’re best at.</p><h3>Closing Thought</h3><p>If you’re working with LLMs and structured data, try this shift:</p><ul><li>Don’t feed raw tables</li><li>Don’t rely on embeddings alone</li><li>Don’t force the model to compute</li></ul><p>Instead:</p><blockquote><strong><em>Let the model plan. Let code execute. Let the model explain.</em></strong></blockquote><p>That small change turns a fragile pipeline into a reliable system.</p><p>And more importantly:</p><blockquote><em>It lets the LLM focus on what it actually does best — reasoning.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=758ebe88342b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How LLMs Read Text: A Practical Guide to Tokenization Algorithms]]></title>
            <link>https://medium.com/@singh.tarus/how-llms-read-text-a-practical-guide-to-tokenization-algorithms-6ec61890abfb?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ec61890abfb</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[text-preprocessing]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 16:10:13 GMT</pubDate>
            <atom:updated>2026-03-24T16:10:13.001Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jh5WW892Z4rvDIAUb4Qs_A.png" /></figure><h3>🧠 Introduction</h3><p>Large Language Models (LLMs) don’t actually “read” text the way we do. They don’t see words, sentences, or meaning — at least not directly. Instead, everything you type gets broken down into smaller units called <strong>tokens</strong>, which are then converted into numbers that the model can process.</p><p>This step — <strong>tokenization</strong> — is the very first and one of the most critical stages in any NLP or LLM pipeline. It affects how efficiently a model uses its context window, how well it generalizes to new words, and even how much it costs to run queries in production systems.</p><p>But tokenization isn’t just about splitting text. Over time, multiple algorithms have been developed to balance vocabulary size, flexibility, and performance. From simple word splitting to advanced subword techniques used in modern models like GPT and LLaMA, each method reflects a different trade-off in how machines understand language.</p><p>In this guide, we’ll break down the most important tokenization algorithms used in practice and implement them step-by-step in Python — keeping everything simple, intuitive, and easy to follow.</p><h3>📌 Algorithms Covered in This Article</h3><ul><li>Word</li><li>Character</li><li>BPE</li><li>WordPiece</li><li>SentencePiece</li><li>ByteBPE</li></ul><h3>🔹 Word Tokenization</h3><h3>1. What is it? (Simple Explanation)</h3><p>Word tokenization means <strong>splitting a sentence into words using spaces</strong>.</p><p>👉 Example:</p><pre>&quot;I love machine learning&quot;</pre><p>becomes:</p><pre>[&quot;I&quot;, &quot;love&quot;, &quot;machine&quot;, &quot;learning&quot;]</pre><p>That’s it. No fancy logic just break the words wherever there is a space.</p><h3>2. Simple Python Code</h3><pre>def word_tokenizer(text):<br>    # Split sentence by spaces<br>    tokens = text.split()<br>    return tokens<br><br>text = &quot;I love machine learning&quot;<br>tokens = word_tokenizer(text)<br>print(tokens)</pre><pre>[&#39;I&#39;, &#39;love&#39;, &#39;machine&#39;, &#39;learning&#39;]</pre><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>Very simple and fast</li><li>Easy to understand</li><li>Works okay for clean text</li></ul><h3>❌ Weaknesses:</h3><ul><li>Fails on punctuation</li><li>&quot;hello!&quot; ≠ &quot;hello&quot;</li><li>Cannot handle new words</li><li>&quot;unhappiness&quot; is treated as one unknown word</li><li>Vocabulary becomes very large</li></ul><h3>🔁 Why We Need the Next Algorithm</h3><p>Word tokenization treats every word as a whole unit.</p><p>👉 Problem:</p><ul><li>If the model has never seen a word before → it gets confused</li><li>No way to break words into smaller parts</li></ul><p>👉 Solution:<br> We go smaller…</p><p>➡️ Next: <strong>Character Tokenization</strong></p><h3>🔹 Character Tokenization</h3><h3>1. What is it? (Simple Explanation)</h3><p>Character tokenization means <strong>breaking text into individual letters (characters)</strong>.</p><p>👉 Example:</p><pre>&quot;hello&quot;</pre><p>Becomes:</p><pre>[&quot;h&quot;, &quot;e&quot;, &quot;l&quot;, &quot;l&quot;, &quot;o&quot;]</pre><h3>2. Simple Python Code</h3><pre>def char_tokenizer(text):<br>    # Convert string into list of characters<br>    tokens = list(text)<br>    return tokens<br><br>text = &quot;hello&quot;<br>tokens = char_tokenizer(text)<br>print(tokens)</pre><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>No unknown words (every word can be built from characters)</li><li>Very small vocabulary (just letters, numbers, symbols)</li><li>Works for any language</li></ul><h3>❌ Weaknesses:</h3><ul><li>Sequences become very long</li><li>&quot;learning&quot; → 8 tokens instead of 1</li><li>Hard for models to learn meaning from characters</li><li>Slower processing due to long sequences</li></ul><h3>🔁 Why We Need the Next Algorithm</h3><p>Character tokenization solves the “unknown word” problem…</p><p>👉 But creates a new problem:</p><ul><li>Too many tokens</li><li>Hard to learn meaningful patterns</li></ul><p>👉 We need something in between:</p><ul><li>Not full words</li><li>Not single characters</li></ul><p>➡️ Next: <strong>BPE (Byte Pair Encoding)</strong></p><h3>🔹 BPE (Byte Pair Encoding)</h3><h3>1. What is it? (Simple Explanation)</h3><p>BPE is a method that <strong>builds tokens by merging common character pairs</strong>.</p><p>👉 Instead of:</p><ul><li>whole words (too big)</li><li>single characters (too small)</li></ul><p>It learns <strong>frequent chunks of words</strong>.</p><p>👉 Example:</p><p>Start with characters:</p><pre>&quot;low&quot;<br>→ [&quot;l&quot;, &quot;o&quot;, &quot;w&quot;]</pre><p>If &quot;lo&quot; appears often, BPE merges it:</p><pre>[&quot;lo&quot;, &quot;w&quot;]</pre><p>If &quot;low&quot; appears often, it merges again:</p><pre>[&quot;low&quot;]</pre><h3>2. Simple Python Code</h3><pre>from collections import Counter<br><br>def get_vocab(text):<br>    words = text.split()<br>    return Counter(words)<br><br>def split_words(vocab):<br>    return {&quot; &quot;.join(word): freq for word, freq in vocab.items()}<br><br>def get_pairs(vocab):<br>    pairs = Counter()<br>    <br>    for word, freq in vocab.items():<br>        symbols = word.split()<br>        <br>        for i in range(len(symbols) - 1):<br>            pairs[(symbols[i], symbols[i+1])] += freq<br>            <br>    return pairs<br><br>def merge_pair(pair, vocab):<br>    new_vocab = {}<br>    bigram = &quot; &quot;.join(pair)<br>    replacement = &quot;&quot;.join(pair)<br>    <br>    for word in vocab:<br>        new_word = word.replace(bigram, replacement)<br>        new_vocab[new_word] = vocab[word]<br>        <br>    return new_vocab<br><br><br>text = &quot;low lower lowest&quot;<br><br>vocab = split_words(get_vocab(text))<br><br>for i in range(3):<br>    pairs = get_pairs(vocab)<br>    best = max(pairs, key=pairs.get)<br>    vocab = merge_pair(best, vocab)<br>    print(f&quot;Step {i+1}: merged {best}&quot;)<br><br>print(vocab)</pre><pre>{&#39;low&#39;: 1, &#39;lowe r&#39;: 1, &#39;lowe s t&#39;: 1}</pre><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>Handles unknown words by breaking them into parts</li><li>Vocabulary stays smaller than word-based</li><li>Learns useful patterns like:</li><li>&quot;low&quot;</li><li>&quot;est&quot;</li><li>&quot;ing&quot;</li></ul><h3>❌ Weaknesses:</h3><ul><li>Merging is based only on frequency (not meaning)</li><li>Can create weird or suboptimal splits</li><li>Depends heavily on training data</li></ul><h3>🔁 Why We Need the Next Algorithm</h3><p>BPE improves things a lot…</p><p>👉 But it has a limitation:</p><ul><li>It only looks at <strong>frequency</strong>, not how meaningful a split is</li></ul><p>👉 Example:</p><ul><li>It may merge something common but not useful</li></ul><p>👉 We need a smarter way to pick tokens<br> ➡️ One that considers <strong>likelihood, not just frequency</strong></p><p>➡️ Next: <strong>WordPiece</strong></p><h3>🔹 WordPiece</h3><h3>1. What is it? (Simple Explanation)</h3><p>WordPiece is similar to BPE, but instead of just merging frequent pairs, it <strong>chooses tokens based on what gives the best overall representation of words</strong>.</p><p>👉 In simple terms:</p><ul><li>BPE = “merge what appears most”</li><li>WordPiece = “choose what makes words easier to understand”</li></ul><p>👉 Example:</p><p>Word:</p><pre>&quot;unhappiness&quot;</pre><p>WordPiece might break it into:</p><pre>[&quot;un&quot;, &quot;##happi&quot;, &quot;##ness&quot;]</pre><p>👉 Key idea:</p><ul><li>&quot;##&quot; means <strong>this token continues a word</strong></li></ul><p>👉 Another example:</p><pre>&quot;playing&quot;</pre><p>becomes:</p><pre>[&quot;play&quot;, &quot;##ing&quot;]</pre><h3>2. Simple Python Code</h3><pre>def wordpiece_tokenize(word, vocab):<br>    tokens = []<br>    i = 0<br>    <br>    while i &lt; len(word):<br>        j = len(word)<br>        found = None<br>        <br>        # Try longest match first<br>        while i &lt; j:<br>            sub = word[i:j]<br>            <br>            if i &gt; 0:<br>                sub = &quot;##&quot; + sub<br>                <br>            if sub in vocab:<br>                found = sub<br>                break<br>                <br>            j -= 1<br>        <br>        # If nothing found → unknown<br>        if found is None:<br>            return [&quot;[UNK]&quot;]<br>        <br>        tokens.append(found)<br>        i = j<br>        <br>    return tokens<br><br><br>vocab = {&quot;play&quot;, &quot;##ing&quot;, &quot;un&quot;, &quot;##happi&quot;, &quot;##ness&quot;}<br><br>print(wordpiece_tokenize(&quot;playing&quot;, vocab))<br>print(wordpiece_tokenize(&quot;unhappiness&quot;, vocab))</pre><pre>[&#39;play&#39;, &#39;##ing&#39;]<br>[&#39;un&#39;, &#39;##happi&#39;, &#39;##ness&#39;]</pre><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>Smarter than BPE (not just frequency-based)</li><li>Produces more meaningful subwords</li><li>Handles unknown words well</li><li>Used in real models like BERT</li></ul><h3>❌ Weaknesses:</h3><ul><li>Still depends on a fixed vocabulary</li><li>Can fail if a word cannot be split using known tokens</li><li>Requires more computation than BPE</li></ul><h3>🔁 Why We Need the Next Algorithm</h3><p>WordPiece is good…</p><p>👉 But it still has limits:</p><ul><li>It depends on spaces between words</li><li>Struggles with languages without clear spacing</li><li>Vocabulary is still fixed</li></ul><p>👉 We need something more flexible:</p><ul><li>Works directly on raw text</li><li>Handles any language easily</li></ul><p>➡️ Next: <strong>SentencePiece</strong></p><h3>🔹 SentencePiece</h3><h3>1. What is it? (Simple Explanation)</h3><p>SentencePiece is a tokenizer that <strong>does not rely on spaces</strong>.<br> It treats the text as a <strong>raw stream of characters</strong> and learns how to split it.</p><p>👉 Key idea:</p><ul><li>Spaces are just another character</li><li>The model learns tokens directly from text</li></ul><p>👉 Example:</p><pre>&quot;machine learning&quot;</pre><p>becomes:</p><pre>[&quot;▁machine&quot;, &quot;▁learning&quot;]</pre><p>👉 The symbol:</p><ul><li>&quot;▁&quot; = represents a space before the word</li></ul><p>👉 Another example:</p><pre>&quot;playing&quot;</pre><p>might become:</p><pre>[&quot;▁play&quot;, &quot;ing&quot;]</pre><h3>2. Simple Python Code</h3><pre>import sentencepiece as spm<br><br># Step 1: Create a small training file<br>with open(&quot;data.txt&quot;, &quot;w&quot;) as f:<br>    f.write(&quot;machine learning is fun\n&quot;)<br>    f.write(&quot;I love learning\n&quot;)<br><br># Step 2: Train tokenizer<br>spm.SentencePieceTrainer.train(<br>    input=&quot;data.txt&quot;,<br>    model_prefix=&quot;m&quot;,<br>    vocab_size=50 ###  Total number of tokens in the vocabulary<br>)<br><br># Step 3: Load model<br>sp = spm.SentencePieceProcessor(model_file=&quot;m.model&quot;)<br><br># Step 4: Tokenize<br>print(sp.encode(&quot;machine learning&quot;, out_type=str))<br>print(sp.encode(&quot;playing&quot;, out_type=str))</pre><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>Does not depend on spaces</li><li>Works for all languages</li><li>Very flexible and robust</li><li>Used in modern LLMs like LLaMA and T5</li></ul><h3>❌ Weaknesses:</h3><ul><li>Harder to understand than BPE/WordPiece</li><li>Requires training a model first</li><li>Output tokens can look less intuitive</li></ul><h3>🔁 Why We Need the Next Algorithm</h3><p>SentencePiece is very powerful…</p><p>👉 But there’s still a problem:</p><ul><li>It works on characters, not raw bytes</li><li>Some inputs (like emojis, special symbols, code) can still be tricky</li></ul><p>👉 We need something that:</p><ul><li>Works on <strong>any input at all</strong></li><li>Even raw binary data</li></ul><p>➡️ Next: <strong>ByteBPE (Byte-Level BPE)</strong></p><h3>🔹 ByteBPE (Byte-Level BPE)</h3><h3>1. What is it? (Simple Explanation)</h3><p>ByteBPE is like BPE, but instead of working on characters or words, it works on <strong>raw bytes</strong>.</p><p>👉 Simple idea:</p><ul><li>Everything (text, emoji, symbols) → converted into bytes</li><li>Then BPE merges common byte patterns</li></ul><p>👉 Example:</p><pre>&quot;hello 👋&quot;</pre><p>Internally becomes something like:</p><pre>[b&#39;h&#39;, b&#39;e&#39;, b&#39;l&#39;, b&#39;l&#39;, b&#39;o&#39;, b&#39; &#39;, b&#39;\xf0&#39;, b&#39;\x9f&#39;, b&#39;\x91&#39;, b&#39;\x8b&#39;]</pre><p>Then BPE merges frequent patterns into tokens.</p><p>👉 Output (simplified view):</p><pre>[&quot;hello&quot;, &quot;Ġ&quot;, &quot;👋&quot;]</pre><p>👉 &quot;Ġ&quot; represents a space</p><h3>2. Simple Python Code</h3><pre>from transformers import AutoTokenizer<br><br># Load GPT-style tokenizer (uses Byte-Level BPE)<br>tokenizer = AutoTokenizer.from_pretrained(&quot;gpt2&quot;)<br><br>text = &quot;hello 👋 world&quot;<br><br>tokens = tokenizer.tokenize(text)<br><br>print(tokens)</pre><pre>[&quot;hello&quot;, &quot;Ġ&quot;, &quot;👋&quot;]</pre><p>(or slightly different depending on tokenizer)</p><h3>3. Strengths and Weaknesses</h3><h3>✅ Strengths:</h3><ul><li>Works on any input (text, emoji, code, symbols)</li><li>No unknown tokens</li><li>Very robust and flexible</li><li>Used in GPT models</li></ul><h3>❌ Weaknesses:</h3><ul><li>Tokens can look strange (not human-friendly)</li><li>Harder to interpret manually</li><li>Still based on frequency, not meaning</li></ul><h3>🔁 Final Insight</h3><p>ByteBPE solves a key problem:</p><p>👉 Earlier methods:</p><ul><li>Struggled with special characters</li><li>Needed clean text</li></ul><p>👉 ByteBPE:</p><ul><li>Works on <strong>any raw input</strong></li><li>Makes LLMs more reliable in real-world use</li></ul><h3>✅ Wrap-Up Transition (You can use this next)</h3><p>At this point, you’ve covered:</p><ul><li>Word → simple but limited</li><li>Character → flexible but inefficient</li><li>BPE → balanced approach</li><li>WordPiece → smarter subwords</li><li>SentencePiece → space-independent</li><li>ByteBPE → fully robust</li></ul><p>👉 This is exactly how tokenization evolved to support modern LLMs.</p><h3>🧠 Conclusion</h3><p>Tokenization might look like a simple preprocessing step, but it plays a huge role in how LLMs understand and process text. From splitting by spaces to learning byte-level patterns, each algorithm we explored reflects a step toward better efficiency and flexibility.</p><p>We started with <strong>Word</strong> tokenization — simple but limited. Then moved to <strong>Character</strong> tokenization, which removed unknown words but made sequences too long. <strong>BPE</strong> introduced a balance by learning common subword patterns, while <strong>WordPiece</strong> improved this by choosing more meaningful splits. <strong>SentencePiece</strong> removed the need for spaces entirely, making it more flexible across languages. Finally, <strong>ByteBPE</strong> made tokenization fully robust by working directly with raw bytes, enabling modern LLMs to handle any kind of input.</p><p>The key takeaway is this:</p><p>👉 Tokenization is not just about splitting text — it’s about <strong>how efficiently a model can understand and represent language</strong>.</p><p>In real-world systems, this directly affects:</p><ul><li>how many tokens your input consumes</li><li>how fast your model responds</li><li>how well it handles edge cases</li></ul><p>Understanding these algorithms gives you a deeper view into how LLMs actually work under the hood — and helps you make better decisions when building or optimizing AI systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ec61890abfb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Sending Entire Chunks to Your LLM]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@singh.tarus/stop-sending-entire-chunks-to-your-llm-394ae31712b5?source=rss-13f364d82365------2"><img src="https://cdn-images-1.medium.com/max/960/1*aVjdxssdIholbrVH1exGoA.jpeg" width="960"></a></p><p class="medium-feed-snippet">Using Context Extraction to Reduce Tokens and Speed Up RAG Pipelines
Retrieval-Augmented Generation (RAG) has become the default&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@singh.tarus/stop-sending-entire-chunks-to-your-llm-394ae31712b5?source=rss-13f364d82365------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@singh.tarus/stop-sending-entire-chunks-to-your-llm-394ae31712b5?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/394ae31712b5</guid>
            <category><![CDATA[latency-optimization]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[langextract]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 15:27:47 GMT</pubDate>
            <atom:updated>2026-03-19T15:27:47.353Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Designing AI Systems That Get Cheaper Over Time]]></title>
            <link>https://medium.com/@singh.tarus/designing-ai-systems-that-get-cheaper-over-time-d6a304db9c50?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6a304db9c50</guid>
            <category><![CDATA[knowledge-distillation]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Tue, 23 Dec 2025 05:33:13 GMT</pubDate>
            <atom:updated>2025-12-23T05:33:13.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*470xy-3y4JHi1Y3ZznUafw.png" /></figure><p>Most production GenAI systems scale cost linearly: more users → more LLM calls → higher bills.</p><p>The common response is to “switch to a smaller model” or “optimize prompts.”<br> In practice, this often just shifts the problem.</p><p>In this article, I describe a different approach: <strong>treating large language models as teachers</strong>, not execution engines.<br> The idea is simple — use an LLM to label and structure behavior, then <strong>train smaller specialist models to handle the majority of requests</strong>.</p><p>As the system runs, the smaller model improves, LLM calls decrease, and <strong>cost per query drops over time</strong>.</p><p>To make this concrete, I’ll walk through a minimal example: a <strong>query routing system</strong> where an LLM teaches a lightweight classifier how to decide <em>when</em> complex reasoning or tools are actually needed.</p><h3>The Core Idea</h3><ul><li><strong>LLM</strong>: high-capacity, expensive, used sparingly</li><li><strong>Student model</strong>: cheap, fast, specialized</li><li><strong>Routing logic</strong>: student first, LLM as fallback</li><li><strong>Feedback loop</strong>: only high-confidence, successful decisions become training data</li></ul><p>This turns the LLM from a permanent dependency into a <strong>bootstrapping mechanism</strong>.</p><h3>From Idea to Implementation: Routing as the First Optimization</h3><p>At a glance, this system looks like “LLM distillation.”<br> That framing misses the point.</p><p><strong>What’s actually happening is simpler and more effective:</strong></p><blockquote><strong><em>We extract intelligence from an LLM and encode it into the cheapest possible execution model.</em></strong></blockquote><h3>The Problem</h3><p>Most GenAI systems work like this:</p><ol><li>Every query goes to a large LLM</li><li>The LLM decides what to do</li><li>The LLM executes the decision</li></ol><p>Even when the answer is trivial, the most expensive component is already involved.</p><h3>The Shift</h3><p>This implementation separates <strong>decision-making</strong> from <strong>execution</strong>.</p><p>Instead of asking an LLM to both decide <em>and</em> act, we introduce a lightweight <strong>routing step</strong> whose only job is to answer:</p><blockquote><strong><em>What is the cheapest way to correctly handle this query?</em></strong></blockquote><p>This is <strong>not</strong> a generation problem.<br> It’s a <strong>classification problem</strong>.</p><h3>Why Routing Works</h3><ul><li>Routing happens <em>before</em> reasoning</li><li>Routing decisions are discrete and repeatable</li><li>Routing can be learned by much smaller models</li></ul><p>Early in the system’s lifecycle:</p><ul><li>A large LLM performs routing</li><li>High-confidence decisions are logged</li></ul><p>Over time:</p><ul><li>A small specialist model learns the same routing behavior</li><li>The LLM becomes a fallback, not the default</li></ul><h3>Where the Cost Savings Come From</h3><blockquote><strong><em>The biggest cost reduction doesn’t come from shrinking models — it comes from changing the model class.</em></strong></blockquote><p>Replacing a generative reasoning step with a cheap execution primitive is where the real gains are. And this can act as the tip of the iceberg. We can eventually use mini-LLMs which are extremely cost efficient, to replace the critical thinking done by giant LLM like Gemini, ChatGPT, Claude, etc.</p><p>Let’s continue —</p><h3>Now What Follows</h3><p>With routing treated as a first-class decision, the rest of the system becomes straightforward:</p><ul><li>student model first</li><li>LLM only on low confidence</li><li>continuous improvement through logged examples</li></ul><p>The pseudocode below shows how this works end to end.</p><h3>Pseudocode: LLM-Trained Query Router</h3><h3>1️⃣ Teacher Labeling (LLM)</h3><pre>def teacher_route(query):<br>    &quot;&quot;&quot;<br>    Uses a large LLM to decide how the query should be handled.<br>    &quot;&quot;&quot;<br>    response = llm.generate(<br>        prompt=f&quot;&quot;&quot;<br>        Decide the correct handling strategy for the query.<br>        Choose one:<br>        - ROUTE_SIMPLE<br>        - ROUTE_RETRIEVAL<br>        - ROUTE_PLANNING<br>        - ROUTE_TOOL_CALL<br><br>        Query: {query}<br><br>        Return JSON with:<br>        route, confidence (0–1)<br>        &quot;&quot;&quot;<br>    )<br><br>    return {<br>        &quot;query&quot;: query,<br>        &quot;route&quot;: response[&quot;route&quot;],<br>        &quot;confidence&quot;: response[&quot;confidence&quot;]<br>    }</pre><h4>2️⃣ Logging High-Quality Training Data</h4><pre>def log_if_high_quality(result, execution_success):<br>    &quot;&quot;&quot;<br>    Only store examples that are safe to learn from.<br>    &quot;&quot;&quot;<br>    if result[&quot;confidence&quot;] &gt;= 0.85 and execution_success:<br>        dataset.append({<br>            &quot;text&quot;: result[&quot;query&quot;],<br>            &quot;label&quot;: result[&quot;route&quot;]<br>        })</pre><p><strong>Why this matters</strong><br> Most distillation pipelines fail because they blindly log everything.<br> This step prevents error amplification.</p><h3>3️⃣ Training the Student Model</h3><pre>def train_student(dataset):<br>    &quot;&quot;&quot;<br>    Train a cheap classifier on LLM-labeled routes.<br>    &quot;&quot;&quot;<br>    X = embed_texts([d[&quot;text&quot;] for d in dataset])<br>    y = [d[&quot;label&quot;] for d in dataset]<br><br>    model = LogisticRegression()<br>    model.fit(X, y)<br><br>    return model</pre><h4>4️⃣ Inference-Time Routing</h4><pre>def route_query(query, student_model, threshold=0.75):<br>    embedding = embed_text(query)<br>    probs = student_model.predict_proba([embedding])[0]<br><br>    best_route = argmax_label(probs)<br>    confidence = max(probs)<br><br>    if confidence &gt;= threshold:<br>        return best_route, &quot;student&quot;<br>    else:<br>        return teacher_route(query)[&quot;route&quot;], &quot;llm&quot;</pre><p>This is the <strong>cost-decay mechanism</strong>:</p><ul><li>early on → frequent LLM calls</li><li>later → student handles most traffic</li></ul><h4>5️⃣ Continuous Improvement Loop</h4><pre>for query in incoming_queries:<br>    route, source = route_query(query, student_model)<br><br>    success = execute(route, query)<br><br>    if source == &quot;llm&quot;:<br>        result = teacher_route(query)<br>        log_if_high_quality(result, success)<br><br>    if should_retrain(dataset):<br>        student_model = train_student(dataset)</pre><h3>What This Demonstrates</h3><p>This small routing example highlights three broader system-level ideas that matter more than the specific models used.</p><h3>1. LLMs as Teachers, Not Endpoints</h3><p>Large language models are most valuable when they <strong>create structure and supervision</strong>, not when they sit on the critical path of every request.</p><p>In this system, the LLM’s role is to:</p><ul><li>make high-quality decisions early on</li><li>label behavior</li><li>bootstrap cheaper execution models</li></ul><p>Once that knowledge is captured, repeated LLM calls become unnecessary.</p><h3>2. Smaller Models Should Be Specialists</h3><p>The student model here is not a “mini LLM.”</p><p>It is the <strong>simplest model capable of executing a specific behavior reliably</strong>.</p><p>For routing, this means:</p><ul><li>classification instead of generation</li><li>deterministic outputs</li><li>fast inference</li><li>easy evaluation</li></ul><p>This is why changing the <em>model class</em> matters more than shrinking parameter counts.</p><h3>3. Cost Reduction Is an Architectural Choice</h3><p>Most teams try to reduce cost by:</p><ul><li>tweaking prompts</li><li>switching providers</li><li>using smaller LLMs</li></ul><p>This system reduces cost by <strong>changing where intelligence lives</strong>.</p><p>Once routing is handled by a cheap model:</p><ul><li>latency drops</li><li>throughput increases</li><li>cost per query decays over time</li></ul><p>No prompt optimization can compete with that.</p><h3>Why This Pattern Scales</h3><p>The routing example is intentionally small, but the pattern generalizes.</p><p>The same LLM-to-specialist pipeline can be applied to:</p><ul><li>tool selection</li><li>SQL intent classification</li><li>safety gating</li><li>anomaly detection</li><li>fallback policy learning</li><li>or replacing the decision-making components of your agentic pipeline</li></ul><p>Each time, the goal is the same:</p><blockquote><em>move repeatable decisions out of generative models and into cheaper execution layers</em></blockquote><h3>Closing Thought</h3><p>As LLMs become more capable, the real challenge is no longer <em>what they can do</em> — it’s <strong>how often we need to call them</strong>.</p><p>Systems that improve over time by transferring intelligence into smaller, task-specific models will always outperform systems that rely on prompt engineering alone.</p><p>In practice, the most scalable GenAI systems won’t be built by shrinking models.<br> They’ll be built by <strong>changing the architecture around them</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6a304db9c50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracking LLM Hallucinations Using ReAct-Style Reasoning (Without Telling the Model the Rules)]]></title>
            <link>https://medium.com/@singh.tarus/tracking-llm-hallucinations-using-react-style-reasoning-without-telling-the-model-the-rules-855ca6147c29?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/855ca6147c29</guid>
            <category><![CDATA[llm-hallucinations]]></category>
            <category><![CDATA[react-prompting]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[chain-of-thought]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 19:51:29 GMT</pubDate>
            <atom:updated>2025-12-16T19:51:29.810Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6SHBqj-jHJMlJ1MF9xxSNg.png" /></figure><p>Most evaluations of Large Language Models (LLMs) focus on a single question:</p><blockquote>Did the model produce the correct final answer?</blockquote><p>This approach is simple, but it hides a critical failure mode:<br> <strong>LLMs can give correct answers for the wrong reasons.</strong></p><p>In real systems — analytics tools, financial workflows, internal decision engines, or autonomous agents — this is dangerous. A model that “gets lucky” today may fail catastrophically tomorrow.</p><p>This article presents a <strong>simple and interpretable method to track hallucinations</strong> by evaluating <em>how</em> a model reasons, not just <em>what</em> it outputs.</p><h3>The Core Idea (In Plain English)</h3><ol><li>We ask the LLM to solve a problem using <strong>ReAct-style step-by-step reasoning</strong>.</li><li>We <strong>do not tell the model which rule or framework to use</strong>.</li><li>We capture the model’s intermediate reasoning steps.</li><li>We score those steps on:</li></ol><ul><li><strong>Usefulness</strong></li><li><strong>Correctness</strong></li><li><strong>Risk</strong></li></ul><p>We store and analyse these steps to detect hallucination patterns.</p><p>This lets us answer a better question:</p><blockquote>“Did the model actually understand the problem — or did it just get lucky?”</blockquote><h3>A Simple Example (Math, Not Theory)</h3><p>Why We Intentionally Do Not Give the Rules</p><p>Consider a simple math expression:</p><p><strong>Question:</strong></p><pre>8 + 2 × (3 + 4)</pre><p>If we tell the model:</p><blockquote><em>“Solve this using PEMDAS”</em></blockquote><p>We are testing whether it can <strong>follow instructions</strong>.</p><p>If we <em>don’t</em> tell it that, we test something more important:</p><blockquote><strong><em>Can the model identify which reasoning framework applies?</em></strong></blockquote><p>This distinction matters in real-world tasks where:</p><ul><li>Rules are implicit</li><li>Context is ambiguous</li><li>The model must choose <em>how</em> to reason before executing</li></ul><p>Hallucinations often occur at this <strong>rule-selection step</strong>, not at calculation.</p><h3>Why Start with Math?</h3><p>Math provides:</p><ul><li>Deterministic ground truth</li><li>Clear step ordering</li><li>Easy verification</li></ul><p>This isolates <strong>reasoning quality</strong> from domain ambiguity.<br> The same method generalises to SQL generation, analytics, finance, and agent planning.</p><h3>Using ReAct to Capture Reasoning</h3><p>We use a ReAct-style format that separates reasoning into explicit steps:</p><ul><li><strong>Thought</strong> — what the model believes or plans</li><li><strong>Action</strong> — the operation it performs</li><li><strong>Observation</strong> — the result of that action</li><li><strong>Final Answer</strong> — the output</li></ul><p>Example (simplified):</p><pre>Thought: Parentheses indicate grouped operations.<br>Action: Compute (3 + 4).<br>Observation: 7<br>Thought: Multiplication should be resolved next.<br>Action: Compute 2 × 7.<br>Observation: 14<br>Thought: Add remaining values.<br>Action: Compute 8 + 14.<br>Observation: 22<br>Final Answer: 22</pre><p>Each <em>Thought</em> is treated as <strong>data</strong>, not disposable text.</p><h3>Why We Store Reasoning Separately</h3><p>Instead of discarding the chain of thought, we store it so we can:</p><ul><li>Compare reasoning across many runs</li><li>Identify recurring hallucination patterns</li><li>Detect risky reasoning early</li></ul><p>The final answer becomes only one signal — <br> The reasoning path becomes the primary one.</p><h3>Turning Reasoning into Measurable Metrics</h3><p>To make hallucination tracking concrete, each reasoning step is evaluated using <strong>three simple metrics</strong>. These are designed to be intuitive and interpretable, not academic.</p><h3>1. Usefulness</h3><p><strong>Question:</strong><br> Does this step meaningfully help solve the problem?</p><p><strong>Why it matters:</strong><br> LLMs often generate reasoning that sounds intelligent but does not move the solution forward.</p><p><strong>Examples:</strong></p><p>Reasoning Step Usefulness:</p><p>“Parentheses group operations, so I’ll evaluate them first.” -&gt; High</p><p>“There are several numbers in this expression.” -&gt; Low</p><p>Usefulness filters out:</p><ul><li>Filler reasoning</li><li>Circular explanations</li><li>Token-heavy but empty steps</li></ul><h3>2. Correctness</h3><p><strong>Question:</strong><br> Is the reasoning logically or factually correct?</p><p><strong>Why it matters:</strong><br> A model can reach the correct final answer while relying on <strong>incorrect internal reasoning</strong>.</p><p><strong>Examples:</strong></p><p>Reasoning Step Correctness</p><p>“Parentheses are evaluated before multiplication.” -&gt; Correct</p><p>“I’ll add numbers from left to right.” -&gt; Incorrect</p><p>This metric exposes <strong>hidden hallucinations</strong> that accuracy metrics miss.</p><h3>3. Risk</h3><p><strong>Question:</strong><br> How likely is this reasoning to fail on slightly different inputs?</p><p><strong>Why it matters:</strong><br> Some reasoning works only in simple cases and collapses in real-world scenarios.</p><p><strong>High-risk signals include:</strong></p><ul><li>Guessing</li><li>Overconfidence without explanation</li><li>“This usually works” logic</li><li>Rules applied without justification</li></ul><p><strong>Examples:</strong></p><p>Reasoning Step Risk:</p><p>“Parentheses explicitly group operations.” -&gt; Low</p><p>“This should be fine.” -&gt; High</p><p>Risk identifies <strong>answers that are accidentally correct</strong>.</p><h3>How These Metrics Work Together</h3><p>Each metric captures a different failure mode:</p><ul><li><strong>Usefulness</strong> → Is the reasoning doing real work?</li><li><strong>Correctness</strong> → Is it actually right?</li><li><strong>Risk</strong> → Will it generalise?</li></ul><p>Hallucinations often appear as:</p><ul><li>Correct final answer</li><li>Incorrect or risky reasoning</li><li>Confident language</li></ul><p>These metrics make that mismatch visible.</p><h3>A Simple Prompt to Enable This Method</h3><p>Below is a minimal prompt that works with any LLM.<br> Notice what’s missing: <strong>no mention of PEMDAS or operator precedence</strong>.</p><pre>You are solving a problem step by step.<br><br>Follow this format strictly:<br><br>Thought: Describe what you are thinking.<br>Action: Perform a single operation.<br>Observation: Result of that operation.<br><br>Repeat until you reach a conclusion.<br><br>Do not skip steps.<br>Do not jump to the final answer.<br><br>Question:<br>{QUESTION}</pre><p>This prompt exposes the model’s <strong>natural reasoning behaviour</strong>.</p><h3>Minimal Python Example</h3><p>This example:</p><ul><li>Sends a math question</li><li>Captures the reasoning trace</li><li>Produces raw data for evaluation</li></ul><pre>from openai import OpenAI<br><br>client = OpenAI()<br><br>PROMPT_TEMPLATE = &quot;&quot;&quot;<br>You are solving a problem step by step.<br><br>Follow this format strictly:<br><br>Thought: Describe what you are thinking.<br>Action: Perform a single operation.<br>Observation: Result of that operation.<br><br>Repeat until you reach a conclusion.<br><br>Do not skip steps.<br>Do not jump to the final answer.<br><br>Question:<br>{question}<br>&quot;&quot;&quot;<br><br>def run_reasoning(question):<br>    prompt = PROMPT_TEMPLATE.format(question=question)<br><br>    response = client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}<br>        ],<br>        temperature=0<br>    )<br><br>    return response.choices[0].message.content<br><br><br>if __name__ == &quot;__main__&quot;:<br>    question = &quot;8 + 2 * (3 + 4)&quot;<br>    reasoning_trace = run_reasoning(question)<br>    print(reasoning_trace)</pre><p>This output becomes your <strong>reasoning dataset</strong>.<br> Each Thought can now be scored on usefulness, correctness, and risk.</p><h3>Introducing a Scoring Agent</h3><p>To avoid manual evaluation, we introduce a <strong>Scoring Agent</strong> — a lightweight LLM prompt whose only job is to <strong>judge reasoning</strong>, not generate it.</p><p>The scoring agent receives:</p><ul><li>The original question</li><li>A single reasoning step</li><li>The step’s position in the reasoning chain</li></ul><p>It outputs <strong>structured scores</strong>, not prose.</p><h3>Scoring Agent Prompt (Conceptual)</h3><pre>You are evaluating a reasoning step produced by another AI.<br><br>Question:<br>{question}<br><br>Reasoning Step:<br>{thought}<br><br>Evaluate this step on a scale of 0 to 1 for:<br><br>- Usefulness: Does this step help solve the problem?<br>- Correctness: Is the reasoning logically correct?<br>- Risk: How likely is this step to cause errors later?<br><br>Return ONLY valid JSON:<br>{<br>  &quot;usefulness&quot;: number,<br>  &quot;correctness&quot;: number,<br>  &quot;risk&quot;: number<br>}</pre><p>This agent never sees the final answer.<br> It judges <strong>reasoning quality in isolation</strong>, which is critical for hallucination detection.</p><h3>Simple Scoring Agent Code Example</h3><pre>def score_reasoning_step(question, thought, client):<br>    prompt = f&quot;&quot;&quot;<br>You are evaluating a reasoning step produced by another AI.Question:<br>{question}<br>Reasoning Step:<br>{thought}<br>Evaluate this step on a scale of 0 to 1 for:<br>- usefulness<br>- correctness<br>- risk<br>Return ONLY valid JSON.<br>&quot;&quot;&quot;<br>    response = client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=0<br>    )<br>    return response.choices[0].message.content</pre><p>Each reasoning trace now becomes a <strong>list of scored steps</strong>.</p><h3>Metric Aggregation: Turning Scores into Hallucination Signals</h3><p>Once steps are scored, we aggregate them to evaluate the <strong>entire reasoning chain</strong>.</p><h3>1. Average Scores (Baseline)</h3><pre>avg_usefulness<br>avg_correctness<br>avg_risk</pre><p>This gives a high-level quality snapshot.</p><h3>2. Early-Error Detection</h3><p>Hallucinations often start <strong>early</strong>.</p><p>We track:</p><ul><li>First incorrect step index</li><li>First high-risk step index</li></ul><p>Early failures are far more dangerous than late ones.</p><h3>3. Risk-Weighted Correctness</h3><p>A useful composite score:</p><pre>trust_score = avg_correctness × (1 − avg_risk)</pre><p>This penalises answers that are:</p><ul><li>Correct but supported by risky reasoning</li></ul><h3>4. Hallucination Flags</h3><p>We flag a reasoning chain if:</p><ul><li>Final answer is correct <strong>but</strong></li><li>One or more steps have low correctness <strong>or</strong></li><li>Average risk exceeds a threshold</li></ul><p>This catches <em>accidental correctness</em>.</p><h3>Why Aggregation Matters</h3><p>Without aggregation, reasoning analysis is anecdotal.</p><p>With aggregation, we can:</p><ul><li>Compare models</li><li>Track improvement over time</li><li>Identify unstable reasoning styles</li><li>Decide when human review is required</li></ul><p>This turns reasoning into <strong>operational telemetry</strong>.</p><h3>Final Takeaway</h3><p>Hallucinations do not appear out of nowhere.<br> They show up first in the reasoning.</p><p>By:</p><ul><li>Capturing chain-of-thought,</li><li>Withholding explicit rules,</li><li>And scoring reasoning quality with clear metrics</li></ul><p>We can <strong>track hallucinations instead of guessing where they happened</strong>.</p><p>This shifts LLM evaluation from:</p><blockquote>“Did it work?”</blockquote><p>to:</p><blockquote>“Can we trust how it worked?”</blockquote><p>And that’s the difference between a clever demo and a system you can actually rely on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=855ca6147c29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Building a Self-Improving SQL Agent with Milvus and Reinforcement-Inspired Memory]]></title>
            <link>https://medium.com/@singh.tarus/building-a-self-improving-sql-agent-with-milvus-and-reinforcement-inspired-memory-d98942539aba?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/d98942539aba</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 02:04:05 GMT</pubDate>
            <atom:updated>2025-10-20T02:04:05.209Z</atom:updated>
            <content:encoded><![CDATA[<h4>Turning a static prompt into an evolving reasoning system</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vRwKfgTG2COwq0lWx2pew.png" /></figure><h3>💭 The Problem</h3><p>Most SQL-generating Agentic AI assistants behave in an amnesiacs and static manner.</p><p>You give them a question —</p><blockquote><em>“Show me EC2 cost for last month by region.”<br>They spit out a query, you run it, maybe it works… but the next time you ask something similar, they’ve already forgotten.</em></blockquote><p>That was the issue I hit during my <strong>FinOpsly</strong> project.<br> The system’s SQL prompt template was static — it never <em>learned</em> what worked. Every fix had to be manually baked into the prompt.</p><p>So the goal became clear:</p><blockquote><strong><em>Can we make the agent remember successful SQLs and reuse them intelligently?</em></strong></blockquote><h3>🚀 Enter the Self-Improving Agent</h3><p>Instead of retraining a model each time, we can build an <strong>adaptive prompt system</strong> powered by a <strong>Vector Database (Milvus)</strong> that remembers and prioritizes high-reward examples.</p><p>The loop looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ei-v5XOaDuTJoOa-aes2w.png" /></figure><h3>⚙️ Step-by-Step Breakdown</h3><p><strong>1. User Query</strong><br> A natural-language question from the user.</p><p><strong>2. Retrieve Prompt Template</strong><br> Load the appropriate base prompt for this task (for example, the FinOps SQL template).</p><p><strong>2a. Retrieve K Samples from Milvus</strong><br> Perform vector similarity search to fetch the top-K most semantically similar past queries.</p><p><strong>2b. Filter and Re-rank</strong><br> Keep only examples that executed successfully and have high rewards (for instance, reward &gt; 0.6).</p><p><strong>3. Extract Do’s &amp; Don’ts</strong><br> From these examples, infer what patterns worked and which led to errors. These become lightweight behavioral rules.</p><p><strong>3b. Apply to Template</strong><br> Merge the examples and rules into the base prompt to form a <em>dynamic</em> prompt.</p><p><strong>4. LLM Generation</strong><br> Send this prompt to the model to generate the new SQL query.</p><p><strong>5. Feedback Loop</strong><br> Execute the SQL, compute its reward, and push the new (query, SQL, reward) record back into Milvus.</p><h3>🧩 Designing the Reward</h3><p>Each generated SQL receives a combined score based on:</p><ul><li><strong>Execution success:</strong> +1 if the query runs cleanly</li><li><strong>Semantic correctness:</strong> +0.8 if it matches the user intent</li><li><strong>Efficiency:</strong> −0.1 for overly complex or slow queries</li><li><strong>User feedback:</strong> +0.2 if reused or marked correct</li></ul><p>This numeric reward (ranging from 0–1) determines what gets remembered. The values can be changed and updated to better suit your requirements. Like say you give higher weightage to efficiency and execution so you can give a different set of values to it.</p><h3>🗄️ Storing Experience in Milvus</h3><p>Each learning event becomes a Milvus record:</p><pre>{<br>  &quot;query&quot;: &quot;Show EC2 cost by region last month&quot;,<br>  &quot;sql&quot;: &quot;SELECT region, SUM(cost)...&quot;,<br>  &quot;reward_total&quot;: 0.85,<br>  &quot;reward_execution&quot;: 1.0,<br>  &quot;reward_correctness&quot;: 0.8,<br>  &quot;reward_efficiency&quot;: -0.1,<br>  &quot;reward_user_feedback&quot;: 0.2,<br>  &quot;executed&quot;: True,<br>  &quot;embedding&quot;: embed(&quot;Show EC2 cost by region last month&quot;)<br>}</pre><p><strong>Schema overview</strong></p><ul><li>id — primary key (auto-increment)</li><li>query — the user’s question text</li><li>sql — generated SQL statement</li><li>reward_total — overall weighted score used for ranking</li><li>reward_execution — success/failure of SQL execution</li><li>reward_correctness — how semantically correct the SQL was</li><li>reward_efficiency — penalty for runtime or unnecessary complexity</li><li>reward_user_feedback — optional human or behavioral score</li><li>executed — boolean flag for execution success</li><li>embedding — vector representation of the query</li></ul><p>Mlivus Schema:</p><pre>collection_schema = {<br>  &quot;fields&quot;: [<br>    {&quot;name&quot;: &quot;id&quot;, &quot;type&quot;: &quot;INT64&quot;, &quot;is_primary&quot;: True, &quot;auto_id&quot;: True},<br>    {&quot;name&quot;: &quot;query&quot;, &quot;type&quot;: &quot;VARCHAR&quot;, &quot;max_length&quot;: 512},<br>    {&quot;name&quot;: &quot;sql&quot;, &quot;type&quot;: &quot;VARCHAR&quot;, &quot;max_length&quot;: 2048},<br>    {&quot;name&quot;: &quot;reward_total&quot;, &quot;type&quot;: &quot;FLOAT&quot;},<br>    {&quot;name&quot;: &quot;reward_execution&quot;, &quot;type&quot;: &quot;FLOAT&quot;},<br>    {&quot;name&quot;: &quot;reward_correctness&quot;, &quot;type&quot;: &quot;FLOAT&quot;},<br>    {&quot;name&quot;: &quot;reward_efficiency&quot;, &quot;type&quot;: &quot;FLOAT&quot;},<br>    {&quot;name&quot;: &quot;reward_user_feedback&quot;, &quot;type&quot;: &quot;FLOAT&quot;},<br>    {&quot;name&quot;: &quot;executed&quot;, &quot;type&quot;: &quot;BOOL&quot;},<br>    {&quot;name&quot;: &quot;embedding&quot;, &quot;type&quot;: &quot;FLOAT_VECTOR&quot;, &quot;dim&quot;: 384}<br>  ]<br>}</pre><p>This richer structure lets the system:</p><ul><li>Re-weight rewards dynamically (for instance, prioritize correctness later).</li><li>Perform advanced filtering, such as<br> expr=&quot;reward_correctness &gt; 0.7 and reward_efficiency &gt; -0.2&quot;.</li><li>Diagnose weaknesses (e.g., frequent low efficiency scores).</li></ul><h3>🔍 Retrieving the Best Memories</h3><pre>results = collection.search(<br>    data=[embed(query)],<br>    anns_field=&quot;embedding&quot;,<br>    limit=10,<br>    param={&quot;metric_type&quot;: &quot;IP&quot;, &quot;params&quot;: {&quot;nprobe&quot;: 10}},<br>    expr=&quot;reward_total &gt; 0.6 and executed == true&quot;,<br>    output_fields=[&quot;query&quot;, &quot;sql&quot;, &quot;reward_total&quot;]<br>)<br><br>for r in results[0]:<br>    sim = r.distance<br>    rew = r.entity[&quot;reward_total&quot;]<br>    r.combined = 0.6*sim + 0.4*rew<br><br>top_examples = sorted(results[0], key=lambda x: x.combined, reverse=True)[:3]</pre><p>Hybrid retrieval balances <strong>semantic similarity</strong> with <strong>historical success</strong> so that the few-shot examples injected into the next prompt are both relevant <em>and</em> reliable.</p><h3>🧠 Building the Dynamic Prompt</h3><pre>def build_prompt(query, examples):<br>    ctx = &quot;\n\n&quot;.join([<br>        f&quot;User: {e.entity[&#39;query&#39;]}\nSQL: {e.entity[&#39;sql&#39;]}&quot;<br>        for e in examples<br>    ])<br>    return f&quot;&quot;&quot;<br>You are a FinOps SQL generation assistant.<br>Learn from these successful examples:<br><br>{ctx}<br><br>Now generate SQL for:<br>User: {query}<br>SQL:<br>&quot;&quot;&quot;</pre><p>Each prompt is freshly constructed using the highest-rewarded, most relevant examples retrieved from Milvus.</p><h3>🔁 Continuous Self-Improvement</h3><p>After execution:</p><pre>collection.insert([{<br>  &quot;query&quot;: user_query,<br>  &quot;sql&quot;: generated_sql,<br>  &quot;reward_total&quot;: reward_total,<br>  &quot;reward_execution&quot;: reward_exec,<br>  &quot;reward_correctness&quot;: reward_corr,<br>  &quot;reward_efficiency&quot;: reward_eff,<br>  &quot;reward_user_feedback&quot;: reward_user,<br>  &quot;executed&quot;: success_flag,<br>  &quot;embedding&quot;: embed(user_query)<br>}])<br>collection.flush()</pre><p>Optionally you could also prune low performers:</p><pre>collection.delete(expr=&quot;reward_total &lt; 0.3 or executed == false&quot;)</pre><p>Each round of generation updates the memory, allowing the agent to evolve without ever retraining its LLM backbone.</p><h3>🧩 Why This Architecture Works</h3><p><strong>Before — Static Agent</strong></p><ul><li>One fixed prompt</li><li>No memory or feedback</li><li>Manual updates only</li></ul><p><strong>Now — Adaptive Memory Agent</strong></p><ul><li>Builds query-specific prompts</li><li>Learns from high-reward history</li><li>Continuously refines its reasoning</li></ul><p>Milvus becomes the agent’s <strong>long-term policy memory</strong> — a structured repository of what worked and why.</p><h3>🔮 Future Enhancements</h3><ul><li>Add an <strong>LLM critic</strong> to automatically score correctness.</li><li>Train a lightweight <strong>reward model</strong> using the detailed breakdown fields.</li><li>Experiment with <strong>PPO or DPO fine-tuning</strong> once enough examples exist.</li><li>Extend memory sharing across multiple agents</li></ul><h3>✨ Closing Thought</h3><blockquote><em>You don’t always need to retrain your model to make it smarter.<br> Sometimes, all you need is memory — and a good reason to remember.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d98942539aba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Goldfish to Elephant: How Vector Database Memory Makes AI Agents Actually Smart]]></title>
            <link>https://medium.com/@singh.tarus/from-goldfish-to-elephant-how-vector-database-memory-makes-ai-agents-actually-smart-edf572061c30?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf572061c30</guid>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[langchain-agents]]></category>
            <category><![CDATA[ai-agent-memory]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 01:04:06 GMT</pubDate>
            <atom:updated>2025-07-28T01:04:06.326Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0OpmC8KTf0vdS3ykPyFXzg.png" /><figcaption>Buffer memory VS Vectorized memory</figcaption></figure><h3>Why Does My AI Agent Forget Everything?</h3><p>Ever had an AI chatbot or data assistant that seemed sharp — until you asked a follow-up question referencing something you’d said before? Suddenly, it’s lost, confused, or asking you to repeat yourself.</p><p>Most AI agents, even those built with state-of-the-art frameworks like LangChain, use what’s called <strong>buffered memory</strong> — they only “remember” the last N turns. Everything else is, well, goldfish memory.</p><p>But what if your agent could remember <em>everything</em>, and search its memory by meaning — not just by recency? That’s what <strong>vector database memory</strong> (with tools like FAISS) unlocks.</p><h3>Pro Tip: Use Vector DBs for AI Agent Question Management</h3><p>If you’re building an AI agent that needs to handle lots of user questions, <strong>don’t rely on windowed memory</strong> (which will make your bot forgetful and frustrating).</p><p><strong>Instead:</strong></p><blockquote><strong><em>Use a vector database to store every question — plus its context, filters, and metadata.</em></strong></blockquote><ul><li>Instantly recall any past question, no matter how long ago.</li><li>Find the <em>most similar</em> question by meaning, not just by keyword or recency.</li><li>Merge old context with new questions for seamless, “I-remember-you” follow-ups.</li></ul><p><strong>Bottom line:</strong></p><blockquote><strong><em>Vector DBs turn your AI agent from a goldfish into an elephant.</em></strong><em><br> They make your agent truly </em>smart<em> about the questions it’s seen and the conversations it’s had.</em></blockquote><h3>How Does Standard LangChain Memory Work?</h3><p>LangChain’s default memory modules include:</p><ul><li><strong>ConversationBufferMemory:</strong> remembers the last N messages.</li><li><strong>ConversationSummaryMemory:</strong> keeps a summary (“gist”) of old chats.</li></ul><p>They’re easy to use and fine for short, linear conversations. But once you need to reference something from page 1 while you’re on page 10, buffer memory falls flat.</p><h3>What Makes Vector Database (VDB) Memory Different?</h3><p>A <strong>vector database</strong> stores every message (or question, answer, and more) as an embedding — a numeric representation that captures meaning. It also stores <em>metadata</em> (like filters, user, query intent, etc).</p><p>When you ask something new, the agent:</p><ul><li>Embeds your query,</li><li>Searches the VDB for the most <em>similar</em> prior question (even if it’s 500 messages old!),</li><li>Recovers the <strong>full context</strong> and uses it to answer — often <em>better than a human</em>.</li></ul><h3>Hands-On Example: Build a Smart Agent Memory in Python</h3><p>Here’s a <strong>full, minimal working demo</strong> (using FAISS, LangChain, and OpenAI embeddings):</p><pre>import faiss<br>from langchain_community.docstore.in_memory import InMemoryDocstore<br>from langchain_community.vectorstores import FAISS<br>from langchain_openai import OpenAIEmbeddings<br>from uuid import uuid4<br>from langchain_core.documents import Document<br><br># 1. Set up the embedding model and FAISS index<br>embeddings = OpenAIEmbeddings(model=&quot;text-embedding-3-large&quot;)<br>embeddings_dim = len(embeddings.embed_query(&quot;hello world&quot;))<br>index = faiss.IndexFlatL2(embeddings_dim)<br>vector_store = FAISS(<br>    embedding_function=embeddings,<br>    index=index,<br>    docstore=InMemoryDocstore(),<br>    index_to_docstore_id={},<br>)</pre><p>With `FAISS` setup now we will test out our system by passing a set of test examples. In this example, we are going to pass and store the <strong>Question </strong>and the <strong>Metadata </strong>associated with the response generation from one of my agents.</p><pre># 2. Example questions, each with metadata<br>questions = [<br>    {<br>        &quot;original_query&quot;: &quot;What was my AWS EC2 cost last month?&quot;,<br>        &quot;applied_filters&quot;: {<br>            &quot;get_filtered_aws_cur_data&quot;: {<br>                &quot;metrics&quot;: [&quot;lineItem/NetUnblendedCost&quot;],<br>                &quot;dimensions&quot;: [&quot;bill/BillingEntity&quot;, &quot;BillingPeriodStartDate_Month&quot;],<br>                &quot;filters&quot;: [<br>                    {&quot;dimension&quot;: &quot;BillingPeriodStartDate_Month&quot;, &quot;operator&quot;: &quot;=&quot;, &quot;value&quot;: &quot;202505&quot;}<br>                ]<br>            }<br>        }<br>    },<br>    {<br>        &quot;original_query&quot;: &quot;what was my AWS usage by services for last week?&quot;,<br>        &quot;applied_filters&quot;: {<br>            &quot;get_filtered_aws_cur_data&quot;: {<br>                &quot;metrics&quot;: [&quot;lineItem/UsageAmount&quot;],<br>                &quot;dimensions&quot;: [<br>                    &quot;lineItem/UsageStartDate&quot;,<br>                    &quot;lineItem/ProductCode&quot;,<br>                    &quot;product/servicecode&quot;<br>                ],<br>                &quot;filters&quot;: [<br>                    {<br>                        &quot;dimension&quot;: &quot;lineItem/UsageStartDate&quot;,<br>                        &quot;operator&quot;: &quot;BETWEEN&quot;,<br>                        &quot;value&quot;: &quot;&#39;2025-07-07&#39; AND &#39;2025-07-14&#39;&quot;<br>                    }<br>                ]<br>            }<br>        }<br>    },<br>    {<br>        &quot;original_query&quot;: &quot;what was my EC2 usage by services for May?&quot;,<br>        &quot;applied_filters&quot;: {<br>            &quot;get_filtered_aws_cur_data&quot;: {<br>                &quot;metrics&quot;: [&quot;lineItem/UsageAmount&quot;, &quot;effectiveCostTotal&quot;],<br>                &quot;dimensions&quot;: [<br>                    &quot;lineItem/ProductCode&quot;,<br>                    &quot;product/servicecode&quot;,<br>                    &quot;BillingPeriodStartDate_Month&quot;,<br>                    &quot;pricing/unit&quot;<br>                ],<br>                &quot;filters&quot;: [<br>                    {<br>                        &quot;dimension&quot;: &quot;lineItem/ProductCode&quot;,<br>                        &quot;operator&quot;: &quot;ILIKE&quot;,<br>                        &quot;value&quot;: &quot;%AmazonEC2%&quot;<br>                    },<br>                    {<br>                        &quot;dimension&quot;: &quot;BillingPeriodStartDate_Month&quot;,<br>                        &quot;operator&quot;: &quot;=&quot;,<br>                        &quot;value&quot;: &quot;202505&quot;<br>                    }<br>                ]<br>            }<br>        }<br>    }<br>]</pre><p>Now we will be adding a way to keep track of the questions correctly</p><p>Every document (question, answer, etc.) added to the vector database gets a <strong>Universally Unique Identifier (UUID)</strong>.</p><pre>documents = [<br>    Document(<br>        page_content=q[&quot;original_query&quot;],<br>        metadata={&quot;applied_filters&quot;: q[&quot;applied_filters&quot;], &quot;question_idx&quot;: idx + 1}<br>    )<br>    for idx, q in enumerate(questions)<br>]<br><br># 4. Create unique IDs for each document<br>uuids = [str(uuid4()) for _ in documents]</pre><p>This ensures each entry is uniquely tracked — so you can reliably retrieve, update, or reference any item, even if other data is added or removed later.</p><p>But WHY?</p><ul><li>Without UUIDs, if you add or remove documents, your indexes might get misaligned and you could retrieve the wrong document.<br> With UUIDs, every item is always referenced correctly — no collisions, no confusion.</li></ul><p>And finally, we take these information and store them in the VDB and ask question to see if it retrieves the right and relevant question</p><pre># 5. Add documents to the FAISS vector store<br>vector_store.add_documents(documents=documents, ids=uuids)<br><br># 6. Query: Find the most similar prior question to a new query<br>results = vector_store.similarity_search(<br>    query = &quot;Now usage costs for this week &quot;,<br>    k=1,<br>    filter=None<br>)<br><br>if results:<br>    top = results[0]<br>    print(&quot;Retrieved question:&quot;, top.page_content)<br>    print(&quot;Metadata:&quot;, top.metadata)<br>else:<br>    print(&quot;No match found!&quot;)</pre><p>So does it work?</p><pre>Retrieved question: what was my AWS usage by services for last week?<br>Metadata: {&#39;applied_filters&#39;: <br>  {&#39;get_filtered_aws_cur_data&#39;: <br>    {&#39;metrics&#39;: <br>      [&#39;lineItem/UsageAmount&#39;], <br>    &#39;dimensions&#39;: [<br>      &#39;lineItem/UsageStartDate&#39;, &#39;lineItem/ProductCode&#39;, &#39;product/servicecode&#39;<br>     ], <br>    &#39;filters&#39;: [<br>        {&#39;dimension&#39;: &#39;lineItem/UsageStartDate&#39;, &#39;operator&#39;: &#39;BETWEEN&#39;, &#39;value&#39;: &quot;&#39;2025-07-07&#39; AND &#39;2025-07-14&#39;&quot;}<br>          ]}}, <br>    &#39;question_idx&#39;: 2}</pre><p><strong>Result:</strong><br> If the new query is similar to a prior question, you’ll retrieve that context instantly — <strong>no matter how old it is.</strong></p><h3>Merits of Vector Database Memory</h3><ul><li><strong>Infinite recall:</strong> Never lose context to a chat window limit.</li><li><strong>Semantic understanding:</strong> Finds what you <em>mean</em>, not just what you typed last.</li><li><strong>Structured intelligence:</strong> Merge filters, carry over slots, reconstruct SQL or API calls on the fly.</li><li><strong>Persistent &amp; scalable:</strong> Survives agent/server restarts; grows with your knowledge base.</li><li><strong>Flexible querying:</strong> Retrieve by meaning, metadata, or filter — auditable and powerful.</li></ul><h3>Demerits and Caveats</h3><ul><li><strong>Slightly more setup:</strong> You’ll need to set up a vector DB (but it’s just a few lines with FAISS or Chroma).</li><li><strong>Embedding quality matters:</strong> Poor embeddings = poor retrieval.</li><li><strong>Resource use:</strong> For truly huge data, you may need distributed/managed vector stores.</li><li><strong>Can retrieve “wrong” memory:</strong> Needs thoughtful prompting or slot-merging logic for follow-ups.</li><li><strong>Not a drop-in for all chatbots:</strong> Some use-cases (ultra-short chats, no history) may not benefit.</li></ul><h3>Why Not Just Use LangChain Memory?</h3><h3>LangChain Buffer Memory (the default):</h3><ul><li><strong>Remembers all turns?</strong><br> ❌ Only remembers the last <em>N</em> messages (windowed memory).</li><li><strong>Recall by meaning?</strong><br> ❌ Only retrieves based on recency, not what you <em>meant</em>.</li><li><strong>Structured context?</strong><br> ❌ Stores just plain text — no metadata, filters, or context.</li><li><strong>Audit/filter/search?</strong><br> ❌ No way to filter, search, or audit by context.</li><li><strong>Survives restarts?</strong><br> ❌ Memory is lost if the app/server restarts.</li><li><strong>Flexible carryover?</strong><br> ❌ Hard to carry over slots, filters, or context across questions.</li></ul><h3>Vector DB Memory (FAISS, etc.):</h3><ul><li><strong>Remembers all turns?</strong><br> ✅ Stores everything, with infinite, persistent memory.</li><li><strong>Recall by meaning?</strong><br> ✅ Finds and retrieves by semantic similarity, not just recency.</li><li><strong>Structured context?</strong><br> ✅ Stores both text <em>and</em> rich metadata (filters, slots, etc.).</li><li><strong>Audit/filter/search?</strong><br> ✅ You can audit, search, and filter any part of memory.</li><li><strong>Survives restarts?</strong><br> ✅ Memory persists even after restarting your agent.</li><li><strong>Flexible carryover?</strong><br> ✅ Seamlessly merges old and new context for smarter follow-ups.</li></ul><p><strong>LangChain buffer memory = scrolling chat window.<br>Vector DB memory = a searchable, intelligent brain.</strong></p><h3>Conclusion</h3><p>As LLM-powered AI agents move from “chatbot” to “analyst, assistant, or teammate,” the <strong>old buffer-memory approach is not enough</strong>.<br> <strong>Vector database memory</strong> is not just a technical upgrade — it’s a total rethink of what agent memory can be:</p><ul><li><strong>Smart.</strong></li><li><strong>Scalable.</strong></li><li><strong>Truly helpful.</strong></li></ul><p>If you want your agent to finally “remember” — and <em>use</em> — the entire history of its interactions, give vector database memory a try.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf572061c30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP from Scratch: Building a Python Agent that Understands and Executes Your Code]]></title>
            <link>https://medium.com/@singh.tarus/mcp-from-scratch-building-a-python-agent-that-understands-and-executes-your-code-b51a4687a732?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/b51a4687a732</guid>
            <category><![CDATA[openai-api]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[ai-agentic-workflow]]></category>
            <category><![CDATA[langchain-agents]]></category>
            <category><![CDATA[llm-agent]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Thu, 26 Jun 2025 23:17:37 GMT</pubDate>
            <atom:updated>2025-06-26T23:17:37.134Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A hands-on guide to building an AI agent powered by docstrings, LangChain, and the Model Context Protocol (MCP)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i6JqCkIa_xLg3D7qCpHEEQ.png" /></figure><h3>Introduction</h3><p>In this tutorial, we’ll build a Model Context Protocol (MCP)-inspired system from scratch — a lightweight agent that can parse class and method docstrings, reason about user queries, select methods dynamically, and execute them in sequence.</p><p>If you want to check out the code, you can <strong>skip straight to the complete implementation on GitHub</strong>:</p><blockquote><strong>👉 </strong><a href="https://github.com/your-username/AutoCLASS"><strong>AutoClass GitHub Repository</strong></a></blockquote><p>Otherwise, let’s walk through how to build the system step by step — from class registration to intelligent execution with dependency resolution.</p><p>Imagine being able to hand an LLM a plain Python class — no wrappers, no APIs, no extra code — and have it <em>understand</em>, <em>select</em>, and <em>execute</em> its methods like a smart teammate.</p><p>That’s the promise of <strong>AutoClass</strong>, a lightweight execution framework built around the <strong>Model Context Protocol (MCP)</strong> — a design pattern where models can reason over code by leveraging structured context. With MCP, your classes become <strong>self-describing agents</strong>, exposing method metadata through readable docstrings that an LLM can parse and act on.</p><p>In this tutorial, we’ll walk you through building AutoClass from scratch:</p><ul><li>Parsing Python classes and extracting rich context from their docstrings</li><li>Creating an LLM-powered engine that understands <em>what method to run</em> and <em>with what inputs</em> — based only on user intent</li><li>Designing a multi-stage pipeline that resolves dependencies between methods automatically</li><li>Enabling modular class registration, execution reuse, and optional streamlit-based visualization</li></ul><p>Just Python, LangChain, and structured reasoning.</p><h3>How It Works (In Plain English)</h3><p>The core idea is simple:</p><p><strong>If your code can describe itself clearly, an LLM can learn to use it.</strong></p><p>Instead of relying on APIs, wrappers, or hardcoded schemas, we let regular Python classes act as <em>self-describing modules</em>. Each method comes with a structured docstring — a short blueprint of what it does, what it needs, and what it returns.</p><h3>🔁 Execution Flow</h3><p><strong>1. Write normal Python classes and methods</strong><br> But make sure each method has a docstring structured like this:</p><pre>- Description: This method adds two numbers.<br>- List of parameters:<br>    - param a: First number :type: int or float<br>    - param b: Second number :type: int or float<br>:return: Sum of a and b :rtype: int or float</pre><p><strong>2. Parse classes and build context</strong><br> Using Python’s inspect module; we extract each method:</p><ul><li>Description</li><li>Parameters and types</li><li>Return value (if any)</li></ul><p>This metadata becomes part of a <strong>shared context</strong>, like a local knowledge base for the agent.</p><p><strong>3. User’s asks a question or gives a task</strong><br> For example:<br> “Take 5 and 2, multiply them, then add that to 6 divided by 4.”</p><p><strong>4. LLM selects which methods to run</strong><br> Based on the method descriptions and names, the model identifies:</p><ul><li>Which class(es) to use</li><li>Which method(s) to invoke</li></ul><p><strong>5. LLM extracts values from the query</strong><br> The agent also maps values like 5, 2, 6, 4 to the right parameters (a, b) for each method.</p><p><strong>6. The agent runs the selected methods</strong><br> It executes each method with the correct parameters.<br> If a method input depends on the output of another, the agent waits — resolving dependencies automatically.</p><p><strong>7. Outputs are saved and reused</strong><br> Each result is stored in a shared dictionary, so future steps (or even future queries) can reference them without recomputing.</p><p>This pattern — where code becomes <em>self-explaining</em> and <em>model-controllable</em> — is what we call the <strong>Model Context Protocol</strong> (MCP). It lets AI agents reason about real code using nothing more than the structure you already maintain.</p><h3>Now with basics done, we develop the code</h3><h3>📦 Step 1: Initializing the Agent</h3><p>The Agent class sets up the LLM and initializes internal storage:</p><pre>class Agent:<br>    def __init__(self, model_name: str = &quot;gpt-4o-mini&quot;, temperature: float = 0.1):<br>        self.llm = ChatOpenAI(model_name=model_name, temperature=temperature)<br>        self.context = {}<br>        self.registered_class = {}<br>        self.method_docs = {}<br>        self.pipeline = None</pre><p>For this task, we will use GPT-4O-Mini but we make sure that users have the ability to pass any <strong>OpenAI model.</strong></p><h3>🧠 Step 2: Registering Classes with Structured Docstrings</h3><p>When a class instance is passed to register_class(), its docstring and method-level docstrings are parsed and stored.py</p><pre>  def register_class(self, instance, alias=None):<br>      class_name = alias or instance.__class__.__name__<br>      self.registered_class[class_name] = instance<br>  <br>      class_doc = inspect.getdoc(instance.__class__)<br>      class_meta = {<br>          &quot;class_name&quot;: class_name,<br>          &quot;class_description&quot;: class_doc or &quot;&quot;,<br>          &quot;methods&quot;: []<br>      }<br>  <br>      for name, func in inspect.getmembers(instance, predicate=inspect.ismethod):<br>          if name.startswith(&quot;_&quot;):<br>              continue<br>          doc = inspect.getdoc(func)<br>          if doc:<br>              parsed = self.parse_docstring(doc)<br>              method_data = {<br>                  &quot;method&quot;: name,<br>                  &quot;method_description&quot;: parsed[&quot;description&quot;],<br>                  &quot;inputs&quot;: parsed[&quot;inputs&quot;],<br>                  &quot;output&quot;: parsed[&quot;output&quot;],<br>                  &quot;raw_doc&quot;: doc<br>              }<br>              class_meta[&quot;methods&quot;].append(method_data)<br>  <br>      self.context.setdefault(&quot;classes&quot;, []).append(class_meta)</pre><p>This register class will allow users to add new classes and make the system scalable.</p><h3>📃 Step 3: Parsing Docstrings into Usable Structures</h3><p>Docstrings are parsed using a simple format (with - param, :return:, :rtype: tags).</p><pre>  def parse_docstring(self, docstring):<br>      parsed = {&quot;description&quot;: &quot;&quot;, &quot;inputs&quot;: {}, &quot;output&quot;: &quot;&quot;}<br>      lines = docstring.strip().splitlines()<br>  <br>      for line in lines:<br>          line = line.strip()<br>          if line.lower().startswith(&quot;- description:&quot;):<br>              parsed[&quot;description&quot;] = line.split(&quot;:&quot;, 1)[1].strip()<br>          elif line.startswith(&quot;- param&quot;):<br>              match = re.match(r&quot;- param (\w+): .*?:type:\s*(.*)&quot;, line)<br>              if match:<br>                  param, param_type = match.groups()<br>                  parsed[&quot;inputs&quot;][param] = param_type<br>          elif line.startswith(&quot;:rtype:&quot;):<br>              parsed[&quot;output&quot;] = line.split(&quot;:&quot;, 1)[1].strip()<br>          elif line.startswith(&quot;:return:&quot;) and not parsed[&quot;output&quot;]:<br>              parsed[&quot;output&quot;] = line.split(&quot;:&quot;, 1)[1].strip()<br>  <br>      return parsed</pre><h3>Step 4: Selecting Relevant Classes and Methods with LLM</h3><p>The method llm_choose_class_method() analyzes the user query and returns which class/methods are relevant.</p><pre>    def llm_choose_class_method(self, query):<br>        prompt = PromptTemplate(<br>            input_variables=[&quot;classes&quot;,&quot;user_query&quot;],<br>            template=&quot;&quot;&quot;<br>                You are an AI assistant that can choose a class to handle a user query.<br><br>                List of classes:<br>                {classes}<br><br>                User query: {user_query}<br><br>                Instruction:<br>                You are given a user query and a list of classes with their associated methods and method descriptions.<br><br>                Your task is to:<br>                1. Analyze the user query and determine which class or classes contain methods that can help answer the query.<br>                2. For each selected class, return a list of method names (only the names) that are relevant to the query.<br>                3. The output must be a dictionary in the following format:<br><br>                {{<br>                    &#39;class_name1&#39;: [&#39;method1&#39;, &#39;method2&#39;],<br>                    &#39;class_name2&#39;: [&#39;method3&#39;]<br>                }}<br>                <br>                4. If a output to a query depends on execution of a set of methods first, include them in the format of &lt;class_name&gt;.&lt;method_name&gt;<br><br>                Rules:<br>                - Only include classes that contain at least one relevant method.<br>                - Only include methods that directly relate to the query.<br>                - If no methods match, return an empty dictionary: {{}}<br>                - Do not return any explanation or extra text—**only the dictionary output**.<br>                - You should not change the class name and use the exact class name as it is. <br>                &quot;&quot;&quot;)<br>        classes_desc = &quot;\n&quot;.join(<br>            [<br>                f&quot;Class: {cls.get(&#39;class_name&#39;, &#39;Unknown&#39;)} — Method: {met.get(&#39;method&#39;, &#39;Unnamed&#39;)} — Description: {met.get(&#39;method_description&#39;, &#39;&#39;)}&quot;<br>                for cls in self.context.get(&quot;classes&quot;, [])<br>                for met in cls.get(&quot;methods&quot;, [])<br>            ]<br>        )<br>        #print(classes_desc)<br>        message = HumanMessage(<br>                content = prompt.format(classes = classes_desc, user_query = query)<br>            )<br>        <br>        resp = self.llm([message]).content.strip()<br>        <br>        try:<br>            resp = ast.literal_eval(resp)<br>            #print(resp)<br>        except:<br>            print(&quot;Error converting to dictionary. LLM dind&#39;t provide dictionary formatted output&quot;)<br>        self.pipeline = self.get_method_context_subset(resp)<br>        return resp</pre><p>In this, we give it a structured template and clear instructions on the output format. The `Rules` make sure that the model doesn’t select or create some new variables. By relying on the classes and methods doc strings, the LLM will be able to pick the right class and method.</p><h3>📌 Step 5: Filtering Out Only the Relevant Pipeline</h3><p>Once we have the method selection, we prune the original context to gain the descriptions and other content for methods inside of our registered classes:</p><pre>def get_method_context_subset(self, selected_dict=None):<br>        &quot;&quot;&quot;<br>        Prunes self.context[&#39;classes&#39;] based on selected class/methods dictionary.<br>        Returns a filtered structure with only relevant class/method pairs.<br>        &quot;&quot;&quot;<br>        pruned_context = {&quot;classes&quot;: []}<br>        selected_dict = selected_dict or self.pipeline<br>        for cls in self.context.get(&quot;classes&quot;, []):<br>            class_name = cls[&quot;class_name&quot;]<br>            if class_name not in selected_dict:<br>                continue<br><br>            # Filter methods<br>            selected_methods = []<br>            for method in cls[&quot;methods&quot;]:<br>                if method[&quot;method&quot;] in selected_dict[class_name]:<br>                    method.pop(&#39;method_description&#39;)<br>                    method.pop(&#39;raw_doc&#39;)<br>                    selected_methods.append(method)<br><br>            if selected_methods:<br>                pruned_context[&quot;classes&quot;].append({<br>                    &quot;class_name&quot;: class_name,<br>                    # &quot;class_description&quot;: cls[&quot;class_description&quot;],<br>                    &quot;methods&quot;: selected_methods<br>                })<br>        return pruned_context</pre><h3>🤖 Step 6: Filling in the Inputs via LLM</h3><p>After selection, we let the LLM extract input values based on user’s query if the original query has input variables:</p><pre>    def llm_determine_input_parameters(self, query):<br>        context = &quot;\n&quot;.join(<br>            [<br>                f&quot;&quot;&quot;Class: {cls[&#39;class_name&#39;]}<br>        Method: {met[&#39;method&#39;]}<br>        Inputs: {&#39;, &#39;.join([f&quot;{k} ({v})&quot; for k, v in met.get(&#39;inputs&#39;, {}).items()])}<br>        Output: {met.get(&#39;output&#39;, &#39;N/A&#39;)}&quot;&quot;&quot;<br>                for cls in self.pipeline.get(&quot;classes&quot;, [])<br>                for met in cls.get(&quot;methods&quot;, [])<br>            ]<br>        )<br>        prompt = PromptTemplate(<br>            input_variables=[&quot;conexts&quot;, &quot;query&quot;],<br>            template=&quot;&quot;&quot;<br>                You are an AI assistant that analyzes a user query and fills in input values for methods defined in the provided context.<br><br>                Context:<br>                {contexts}<br><br>                Query:<br>                {query}<br><br>                Instructions:<br><br>                - For each method, check if its required inputs are directly provided in the query.<br>                - If an input value is not found in the query, check if any previous method&#39;s output can be used instead.<br>                - If a method depends on a previous result, represent that input as &quot;ClassName.methodName&quot;.<br>                - Only use such references if they logically make sense — do NOT force linking all methods.<br>                - Preserve the structure: Do NOT rename keys, drop methods, or add extra fields.<br>                - Your output should include ONLY the updated JSON pipeline structure with filled `inputs`.<br><br><br>                Expected Output format:<br>                {{<br>                &quot;classes&quot;: [<br>                    {{<br>                    &quot;class_name&quot;: &quot;SomeClass&quot;,<br>                    &quot;methods&quot;: [<br>                        {{<br>                        &quot;method&quot;: &quot;some_method&quot;,<br>                        &quot;inputs&quot;: {{<br>                            &quot;param1&quot;: &quot;value1&quot;,<br>                            &quot;param2&quot;: 42<br>                        }},<br>                        &quot;output&quot;: &quot;...&quot;<br>                        }}<br>                    ]<br>                    }},<br>                    ...<br>                ]<br>                }}<br><br>                Only output this JSON structure. Do not explain anything.<br>                &quot;&quot;&quot;<br>                    )<br><br>        # Format and send prompt to LLM<br>        formatted_prompt = prompt.format(contexts=context, query=query)<br>        message = HumanMessage(content=formatted_prompt)<br>        response = self.llm([message]).content.strip()<br><br>        try:<br>            parsed = ast.literal_eval(response)<br>            self.pipeline = parsed<br>            print(&quot;\n✅ Filled pipeline with inputs:\n&quot;)<br>            return parsed<br>        except Exception as e:<br>            print(&quot;❌ Error parsing LLM response:&quot;, e)<br>            print(&quot;Raw response:\n&quot;, response)<br>            return {}</pre><p>For this, we again use a structured template and clear instructions about our expected format.</p><h3>🧩 Step 7: Executing the Pipeline with Dependency Resolution</h3><p>Finally, the pipeline is executed. As each are connected to the previous stage, the final JSON generated has a very structured way of executing methods and their order. If a method relies on the output of another, it waits until that output is available:</p><pre>def get_method_context_subset(self, selected_dict=None):<br>        &quot;&quot;&quot;<br>        Prunes self.context[&#39;classes&#39;] based on selected class/methods dictionary.<br>        Returns a filtered structure with only relevant class/method pairs.<br>        &quot;&quot;&quot;<br>        pruned_context = {&quot;classes&quot;: []}<br>        selected_dict = selected_dict or self.pipeline<br>        for cls in self.context.get(&quot;classes&quot;, []):<br>            class_name = cls[&quot;class_name&quot;]<br>            if class_name not in selected_dict:<br>                continue<br><br>            # Filter methods<br>            selected_methods = []<br>            for method in cls[&quot;methods&quot;]:<br>                if method[&quot;method&quot;] in selected_dict[class_name]:<br>                    method.pop(&#39;method_description&#39;)<br>                    method.pop(&#39;raw_doc&#39;)<br>                    selected_methods.append(method)<br><br>            if selected_methods:<br>                pruned_context[&quot;classes&quot;].append({<br>                    &quot;class_name&quot;: class_name,<br>                    # &quot;class_description&quot;: cls[&quot;class_description&quot;],<br>                    &quot;methods&quot;: selected_methods<br>                })<br>        return pruned_context<br>    <br>    def run_pipeline_with_dependencies(self, pipeline=None, max_passes=5):<br>        &quot;&quot;&quot;<br>        Executes a structured pipeline that may include method dependencies.<br>        Respects dependency ordering using &quot;Class.method&quot; notation in inputs.<br>        &quot;&quot;&quot;<br>        pipeline = pipeline or self.pipeline<br>        results = {}<br>        context = self.context<br>        unresolved_methods = []<br>        executed = set()<br><br>        # Flatten all method steps<br>        steps = []<br>        for cls in pipeline.get(&quot;classes&quot;, []):<br>            class_name = cls[&quot;class_name&quot;]<br>            instance = self.registered_class.get(class_name)<br>            if not instance:<br>                print(f&quot;❌ Class &#39;{class_name}&#39; is not registered.&quot;)<br>                continue<br>            for method in cls[&quot;methods&quot;]:<br>                steps.append({<br>                    &quot;class&quot;: class_name,<br>                    &quot;instance&quot;: instance,<br>                    &quot;method_name&quot;: method[&quot;method&quot;],<br>                    &quot;inputs&quot;: method[&quot;inputs&quot;]<br>                })<br><br>        # Resolve dependencies in multiple passes<br>        for _ in range(max_passes):<br>            progress = False<br>            remaining = []<br><br>            for step in steps:<br>                key = f&quot;{step[&#39;class&#39;]}.{step[&#39;method_name&#39;]}&quot;<br>                if key in executed:<br>                    continue<br><br>                resolved_inputs = {}<br>                inputs = step[&quot;inputs&quot;]<br>                can_execute = True<br><br>                for param, val in inputs.items():<br>                    if isinstance(val, (int, float, str)) and not isinstance(val, str) or &quot;.&quot; not in str(val):<br>                        resolved_inputs[param] = val<br>                    elif isinstance(val, str) and val in results:<br>                        resolved_inputs[param] = results[val]<br>                    elif isinstance(val, str) and val in context:<br>                        resolved_inputs[param] = context[val]<br>                    else:<br>                        can_execute = False<br>                        break<br><br>                if can_execute:<br>                    try:<br>                        method_fn = getattr(step[&quot;instance&quot;], step[&quot;method_name&quot;])<br>                        print(f&quot;🚀 Executing {key} with inputs: {resolved_inputs}&quot;)<br>                        output = method_fn(**resolved_inputs)<br><br>                        results[key] = output<br>                        context[key] = output<br>                        if isinstance(output, dict):<br>                            context.update(output)<br><br>                        executed.add(key)<br>                        progress = True<br>                    except Exception as e:<br>                        print(f&quot;❌ Error executing {key}: {e}&quot;)<br>                else:<br>                    remaining.append(step)<br><br>            if not progress:<br>                break<br>            steps = remaining<br><br>        # Final report<br>        if remaining:<br>            print(&quot;\n⚠️ Unresolved methods due to missing inputs or errors:&quot;)<br>            for r in remaining:<br>                print(f&quot; - {r[&#39;class&#39;]}.{r[&#39;method_name&#39;]} (inputs: {r[&#39;inputs&#39;]})&quot;)<br><br>        return results   </pre><h3>🔁 Final Usage Pattern (Sample Flow)</h3><p>We first set up our 2 classes. For this example, we will have a simple class that can perform arithmetic operations and string manipulation for now.</p><p>Keeping in line with how we have structured our thought process:</p><blockquote><strong>CLASS 1:</strong></blockquote><pre><br><br>class StringUtils:<br>    &quot;&quot;&quot;<br>    This class provides a set of methods for common string manipulation operations such as formatting, case conversion, and analysis.<br>    &quot;&quot;&quot;<br><br>    def __init__(self):<br>        self.name = &quot;string_utilities&quot;<br><br>    def to_upper(self, text):<br>        &#39;&#39;&#39;<br>        - Description: Converts the input string to uppercase.<br>        - List of parameters:<br>            - param text: Input string :type: str<br>        :return: Uppercase version of the input string :rtype: str<br>        &#39;&#39;&#39;<br>        return text.upper()<br><br>    def to_lower(self, text):<br>        &#39;&#39;&#39;<br>        - Description: Converts the input string to lowercase.<br>        - List of parameters:<br>            - param text: Input string :type: str<br>        :return: Lowercase version of the input string :rtype: str<br>        &#39;&#39;&#39;<br>        return text.lower()<br><br>    def count_words(self, text):<br>        &#39;&#39;&#39;<br>        - Description: Counts the number of words in the input string.<br>        - List of parameters:<br>            - param text: Input string :type: str<br>        :return: Total number of words in the input string :rtype: int<br>        &#39;&#39;&#39;<br>        return len(text.split())<br><br>    def reverse_string(self, text):<br>        &#39;&#39;&#39;<br>        - Description: Reverses the characters in the input string.<br>        - List of parameters:<br>            - param text: Input string :type: str<br>        :return: Reversed string :rtype: str<br>        &#39;&#39;&#39;<br>        return text[::-1]<br><br>    def contains_substring(self, text, substring):<br>        &#39;&#39;&#39;<br>        - Description: Checks whether the input string contains the specified substring.<br>        - List of parameters:<br>            - param text: Input string :type: str<br>            - param substring: Substring to check :type: str<br>        :return: True if the substring is found in the text, else False :rtype: bool<br>        &#39;&#39;&#39;<br>        return substring in text</pre><blockquote><strong>CLASS 2:</strong></blockquote><pre>class ArithmeticOperations:<br>    &quot;&quot;&quot;<br>    This example cass contains a set of methods designed to perform basic arithmetic operations.<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        self.name = &quot;example&quot;<br><br>    def add(self, a, b):<br>        &#39;&#39;&#39;<br>        - Description: This method adds two numbers.<br>        - List of parameters:<br>            - param a: First number :type: int or float<br>            - param b: Second number :type: int or float<br>        :return: Sum of a and b :rtype: int or float<br>        &#39;&#39;&#39;<br>        return a + b<br><br>    def subtract(self, a, b):<br>        &#39;&#39;&#39;<br>        - Description: This method subtracts two numbers.<br>        - List of parameters:<br>            - param a: First number :type: int or float<br>            - param b: Second number :type: int or float<br>        :return: Difference of a and b :rtype: int or float<br>        &#39;&#39;&#39;<br>        return a - b<br><br>    def multiply(self, a, b):<br>        &#39;&#39;&#39;<br>        - Description: This method multiplies two numbers.<br>        - List of parameters:<br>            - param a: First number :type: int or float<br>            - param b: Second number :type: int or float<br>        :return: Product of a and b :rtype: int or float<br>        &#39;&#39;&#39;<br>        return a * b<br><br>    def divide(self, a, b):<br>        &#39;&#39;&#39;<br>        - Description: This method divides two numbers.<br>        - List of parameters:<br>            - param a: First number :type: int or float<br>            - param b: Second number :type: int or float<br>        :return: Quotient of a and b :rtype: int or float<br>        &#39;&#39;&#39;<br>        if b == 0:<br>            raise ValueError(&quot;Cannot divide by zero&quot;)<br>        return a / b</pre><p>Now with our two classes, how do we use our AI agent accordingly?</p><pre>   query = input(&quot;Query: &quot;)<br>   ex = ArithmeticOperations()<br>   st = StringUtils()<br>   ag = Agent()<br>   ag.register_class(ex, alias=&quot;ArithmeticOperations&quot;)<br>   ag.register_class(st, alias=&#39;StringUtils&#39;)<br>   methods = ag.list_methods()<br>   classes = ag.list_classes()<br>   ag.llm_choose_class_method(query=query)<br>   ag.llm_determine_input_parameters(query = query)<br>   print(json.dumps(ag.get_current_pipeline(), indent = 4))<br>   pipeline = ag.get_current_pipeline()<br>   results = ag.run_pipeline_with_dependencies(pipeline=pipeline)<br>   print(results)</pre><h3>Let’s see it in action…</h3><p>Our query will be</p><blockquote>Perform the task (5+3) * 2</blockquote><p>Output:</p><pre>✅ Filled pipeline with inputs:<br><br>{<br>    &quot;classes&quot;: [<br>        {<br>            &quot;class_name&quot;: &quot;ArithmeticOperations&quot;,<br>            &quot;methods&quot;: [<br>                {<br>                    &quot;method&quot;: &quot;add&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: 5,<br>                        &quot;b&quot;: 3<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                },<br>                {<br>                    &quot;method&quot;: &quot;multiply&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: &quot;ArithmeticOperations.add&quot;,<br>                        &quot;b&quot;: 2<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                }<br>            ]<br>        }<br>    ]<br>}<br>🚀 Executing ArithmeticOperations.add with inputs: {&#39;a&#39;: 5, &#39;b&#39;: 3}<br>🚀 Executing ArithmeticOperations.multiply with inputs: {&#39;a&#39;: 8, &#39;b&#39;: 2}<br>{&#39;ArithmeticOperations.add&#39;: 8, &#39;ArithmeticOperations.multiply&#39;: 16}</pre><p>As we can see it broke the question down on what things it needed to do first and performed the necessary actions to achieve it. Now what if we want to see if the methods called are dependent on the execution of other methods?</p><blockquote>5 * 2, 6 / 3 and add their results.</blockquote><pre>✅ Filled pipeline with inputs:<br><br>{<br>    &quot;classes&quot;: [<br>        {<br>            &quot;class_name&quot;: &quot;ArithmeticOperations&quot;,<br>            &quot;methods&quot;: [<br>                {<br>                    &quot;method&quot;: &quot;multiply&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: 5,<br>                        &quot;b&quot;: 2<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                },<br>                {<br>                    &quot;method&quot;: &quot;divide&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: 6,<br>                        &quot;b&quot;: 3<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                },<br>                {<br>                    &quot;method&quot;: &quot;add&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: &quot;ArithmeticOperations.multiply&quot;,<br>                        &quot;b&quot;: &quot;ArithmeticOperations.divide&quot;<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                }<br>            ]<br>        }<br>    ]<br>}<br>🚀 Executing ArithmeticOperations.multiply with inputs: {&#39;a&#39;: 5, &#39;b&#39;: 2}<br>🚀 Executing ArithmeticOperations.divide with inputs: {&#39;a&#39;: 6, &#39;b&#39;: 3}<br>🚀 Executing ArithmeticOperations.add with inputs: {&#39;a&#39;: 10, &#39;b&#39;: 2.0}<br>{&#39;ArithmeticOperations.multiply&#39;: 10, &#39;ArithmeticOperations.divide&#39;: 2.0, &#39;ArithmeticOperations.add&#39;: 12.0}</pre><p>What if different classes and methods are called?</p><blockquote>Perform the task of 5 *2 and reverse the string MCP</blockquote><pre>✅ Filled pipeline with inputs:<br><br>{<br>    &quot;classes&quot;: [<br>        {<br>            &quot;class_name&quot;: &quot;ArithmeticOperations&quot;,<br>            &quot;methods&quot;: [<br>                {<br>                    &quot;method&quot;: &quot;multiply&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;a&quot;: 5,<br>                        &quot;b&quot;: 2<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                }<br>            ]<br>        },<br>        {<br>            &quot;class_name&quot;: &quot;StringUtils&quot;,<br>            &quot;methods&quot;: [<br>                {<br>                    &quot;method&quot;: &quot;reverse_string&quot;,<br>                    &quot;inputs&quot;: {<br>                        &quot;text&quot;: &quot;MCP&quot;<br>                    },<br>                    &quot;output&quot;: &quot;...&quot;<br>                }<br>            ]<br>        }<br>    ]<br>}<br>🚀 Executing ArithmeticOperations.multiply with inputs: {&#39;a&#39;: 5, &#39;b&#39;: 2}<br>🚀 Executing StringUtils.reverse_string with inputs: {&#39;text&#39;: &#39;MCP&#39;}<br>{&#39;ArithmeticOperations.multiply&#39;: 10, &#39;StringUtils.reverse_string&#39;: &#39;PCM&#39;}</pre><p>As we can see it performed the math operations and reversed the string as well.</p><h3>🚀 Wrapping Up</h3><p>In this tutorial, we built a flexible, agentic system that dynamically interprets class structures and method logic using structured docstrings and the <strong>Model Context Protocol (MCP)</strong> approach. By enabling LLMs to reason over class definitions and method dependencies, we moved beyond traditional scripting into a world where AI can autonomously orchestrate execution pipelines.</p><p>This framework is just the beginning — with persistence, memory, user-defined variables, and external toolchains (like file I/O or APIs), you can evolve this into a powerful autonomous agent system.</p><p>If you’re curious to:</p><ul><li>Integrate persistence and memory across sessions</li><li>Extend to UI-based execution (e.g., with Streamlit)</li><li>Handle long chains of reasoning or real-world automation</li></ul><p>Then stay tuned — more advanced guides are coming soon.</p><h3>🧠 Got Feedback or Ideas?</h3><p>If you’ve got ideas to improve or adapt this, open a discussion or drop a PR on <a href="https://github.com/TarushS-1996/AutoCLASS">GitHub</a>. I’d love to hear how you’re using this framework — whether for automation, education, or building AI-native tools.</p><h3>🙏 Thanks for Reading!</h3><p>If this helped you understand MCP systems or agentic architectures better, give it a share or follow. Until next time — happy building!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b51a4687a732" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Context enhancement]]></title>
            <link>https://medium.com/@singh.tarus/llm-context-enhancement-ed0b005c3673?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed0b005c3673</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[content-enrichment]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Thu, 29 May 2025 04:25:36 GMT</pubDate>
            <atom:updated>2025-05-29T04:25:36.122Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsgtJm2Kz4MXUABVi6nj1Q.png" /></figure><p>Large Language Models (LLMs) like GPT-4 and Claude are impressive. They can generate poems, write code, summarize articles, and answer questions on just about anything. But they still have one big weakness: <strong>limited context</strong>.</p><p>LLMs are like brilliant students with short-term memory. You give them some information in a prompt; they make sense of it and respond. But if they don’t see it in the prompt, it’s like it doesn’t exist. This is where most hallucinations, inaccuracies, or vague answers come from.</p><p>So — what if we could <strong>feed them smarter, richer, more structured context</strong>? What if we could use <strong>machine learning (ML)</strong> techniques to <em>curate</em>, <em>filter</em>, and <em>enrich</em> the data that surrounds a query — before it ever reaches the LLM?</p><p>That’s the idea behind <strong>Contextual Enrichment</strong> — a layer of intelligence that preprocesses context to give LLMs a clearer, more relevant understanding of the world they’re being asked to respond to.</p><h3>The Problem: Context Windows Aren’t Enough</h3><p>LLMs operate within a fixed-size “context window” — a few thousand tokens of text, which includes the user’s question, some background, and maybe a few examples.</p><p>But here’s the catch:</p><ul><li>Most real-world knowledge doesn’t fit in that window.</li><li>Not all parts of the context are equally useful.</li><li>And some of the most relevant data might be stored in a format the model can’t directly interpret (tables, logs, timelines, knowledge graphs, etc.).</li></ul><p>That’s where machine learning comes in — not to replace the LLM, but to <strong>act as a smart prep layer</strong>.</p><h3>The Fix: ML as a Context Enricher</h3><p>Here’s how you can make an LLM <em>smarter</em> without touching the model itself:</p><h3>🔍 1. Retrieval-Augmented Generation (RAG)</h3><p>Pull in relevant documents from a database based on the user’s question and then inject those into the prompt. Great for:</p><ul><li>Customer support</li><li>Legal research</li><li>Internal knowledge bases</li></ul><h3>🎯 2. Clustering + Metadata Analysis</h3><p>Use unsupervised learning (like KMeans, ARIMA, etc.) to group related content, and tag new queries with their most likely topic cluster. Helps in:</p><ul><li>Trend detection</li><li>Ticket triaging</li><li>Dynamic response generation</li></ul><h3>🧠 3. Contextual Embeddings</h3><p>Replace keyword matching with embedding similarity — so semantically related content gets included, even if the wording is different.</p><h3>🧾 4. Domain Fine-Tuning</h3><p>Take a base LLM and fine-tune it on domain-specific examples. This doesn’t enrich the prompt context directly but <strong>teaches the model</strong> to interpret the context better.</p><h3>⏳ 5. Sequence Modeling for Long-Term Memory</h3><p>Use state-space models or memory layers to help LLMs retain context across multiple interactions or long documents.</p><h3>Building the Bridge: A Modular Pipeline</h3><p>To tie this all together, imagine a pipeline like this:</p><ol><li><strong>User submits a query</strong></li><li><strong>Contextual data is extracted from databases and logs</strong></li><li><strong>ML methods analyze and enrich this data</strong> to extract deeper insights and highlight key information</li><li><strong>The enriched context is combined with the query</strong> in a structured prompt template — with clear instructions on what the LLM is being shown</li><li><strong>The LLM responds</strong>, now with far greater understanding of the situation</li></ol><p>You’re not just throwing more data at the model — you’re <strong>curating and shaping</strong> it, so the model sees only what’s most useful and meaningful. This layered approach takes advantage of the LLM’s pattern recognition abilities, leading to far <strong>richer, more accurate, and grounded responses</strong>.</p><h3>Final Thought</h3><p>Language models are already powerful. But with the right kind of context — <strong>smart, structured, and enriched using ML</strong> — they become <em>truly intelligent</em> assistants. Whether you’re building a customer support tool, a knowledge engine, or a research assistant, <strong>contextual enrichment</strong> may be the missing link between “just okay” and “incredible.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed0b005c3673" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Supercharging PyTorch Training: 10 GPU Optimizations with Functional Code]]></title>
            <link>https://medium.com/@singh.tarus/supercharging-pytorch-training-10-gpu-optimizations-with-functional-code-f50b8f719bad?source=rss-13f364d82365------2</link>
            <guid isPermaLink="false">https://medium.com/p/f50b8f719bad</guid>
            <category><![CDATA[deeplearningoptimization]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machinelearningalgorithms]]></category>
            <category><![CDATA[model-training-tips]]></category>
            <dc:creator><![CDATA[Tarush Singh]]></dc:creator>
            <pubDate>Sat, 19 Apr 2025 03:46:55 GMT</pubDate>
            <atom:updated>2025-04-19T03:54:53.123Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mIrOJUym5w6Z9940-04jQ.png" /></figure><p>We have all been there</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*wV4dDKEejm8N4s_grRNwEQ.jpeg" /></figure><h3>Fear not we have CUDA. But how, when and what to use?</h3><p>CUDA and DL frameworks like PyTorch offer a slew of tools and methods to help AI engineers leverage the pipeline. And the good thing is training deep learning models efficiently is not just about bigger GPUs. It’s about <strong>making the most of the hardware you already have</strong>. In this guide, we explore 10 proven PyTorch optimizations that can dramatically boost your model’s training speed, reduce memory usage, and eliminate bottlenecks.</p><p>Whether you’re working with a <strong>mid-tier GPU like the RTX 3060</strong> or a high-end A100, these techniques will help you train smarter.</p><h3>📋 Quick List of Optimizations Used:</h3><ul><li>.to(device) for device-aware tensor operations</li><li>pin_memory=True to speed up data transfer from CPU to GPU</li><li>cudnn.benchmark = True to optimize CNN kernel selection</li><li>torch.cuda.amp.autocast() for mixed precision training</li><li>torch.compile() to enable graph-level optimization (PyTorch 2.0+)</li><li>torch.jit.script() to reduce Python overhead during inference</li><li>torch.utils.checkpoint() for saving VRAM on large models</li><li>torch.nn.DataParallel / DistributedDataParallel for multi-GPU training</li><li>num_workers=4 in DataLoader for parallelized data loading</li><li>torch.profiler.profile() to identify and eliminate performance bottlenecks</li></ul><h3>🔧 1. .to(device) – Efficient Tensor Transfers</h3><p>Always move your tensors and models to the GPU explicitly.</p><pre>import torch<br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>model = model.to(device)<br>inputs = inputs.to(device)</pre><p>✅ Prevents accidental CPU processing and speeds up tensor operations.</p><p><strong>When to use:</strong> Every time you initialize a model or load inputs/targets. If you forget this, your model might run on the CPU, causing huge slowdowns even on GPU-equipped machines.</p><h3>⚡ 2. pin_memory=True – Faster Data Loading</h3><p>Enable pinned memory for asynchronous CPU-to-GPU data transfers.</p><pre>from torch.utils.data import DataLoader<br>train_loader = DataLoader(dataset, batch_size=64, <br>                   pin_memory=True, num_workers=4)</pre><p>✅ Boosts data loading performance, especially when GPU is faster than CPU.</p><p><strong>When to use:</strong> Always enable when training on GPU. Pinned memory ensures the data loader prepares batches in memory that’s easier and faster to transfer to the GPU.</p><h3>⚙️ 3. cudnn.benchmark = True – Faster CNN Training</h3><p>Let cuDNN find the fastest convolution algorithm for your model.</p><pre>import torch.backends.cudnn as cudnn<br>cudnn.benchmark = True</pre><p>✅ Great for models with fixed input sizes like image classifiers.</p><p><strong>When to use:</strong> If your input dimensions don’t change between iterations (e.g., 224x224 images). cuDNN will benchmark once and reuse the best-performing convolution algorithm.</p><h3>🧠 4. torch.cuda.amp.autocast() – Mixed Precision Training</h3><p>Leverage FP16 where safe for faster training and lower memory usage.</p><pre>scaler = torch.cuda.amp.GradScaler()<br>for inputs, targets in train_loader:<br>    inputs, targets = inputs.to(device), targets.to(device)<br>    with torch.cuda.amp.autocast():<br>        outputs = model(inputs)<br>        loss = criterion(outputs, targets)<br>    scaler.scale(loss).backward()<br>    scaler.step(optimizer)<br>    scaler.update()</pre><p>✅ Up to 2x speedup and lower VRAM consumption.</p><p><strong>When to use:</strong> Any modern GPU (e.g., RTX series) supports mixed precision. This is especially helpful when training large models or using high-resolution inputs.</p><h3>🏎️ 5. torch.compile() – PyTorch 2.0 JIT Compilation</h3><p>Speed up your model by compiling it into an optimized computation graph.</p><pre>model = torch.compile(model)</pre><p>✅ 20–30% speedup by fusing ops and reducing kernel launches.</p><p><strong>When to use:</strong> Right before training begins. Works best for transformer models or anything with repetitive layers. Requires PyTorch 2.0+.</p><h3>✂️ 6. torch.jit.script() – JIT for Inference</h3><p>Optimize models for inference by converting them to TorchScript.</p><pre>scripted_model = torch.jit.script(model)<br>torch.jit.save(scripted_model, &quot;optimized_model.pt&quot;)</pre><p>✅ Reduces Python overhead during inference, making deployment faster.</p><p><strong>When to use:</strong> Before deploying models to production, especially in C++ inference environments or mobile devices. It helps remove Python dependency.</p><h3>🧩 7. torch.utils.checkpoint() – Save VRAM for Deep Models</h3><p>Recompute intermediate activations during the backward pass to save memory.</p><pre>from torch.utils.checkpoint import checkpoint<br>def custom_forward(x):<br>    return model(x)<br>outputs = checkpoint(custom_forward, inputs)</pre><p>✅ Useful when training deep networks like transformers on limited VRAM.</p><p><strong>When to use:</strong> When you get out-of-memory errors or when using large batch sizes or models like ResNet152, ViTs, or GPT-style architectures.</p><h3>🧪 8. torch.nn.DataParallel or DistributedDataParallel – Multi-GPU Scaling</h3><p>Use multiple GPUs to speed up training (DDP is preferred).</p><pre>from torch.nn.parallel import DistributedDataParallel as DDP<br>model = DDP(model)</pre><p>✅ Best for scaling to multiple GPUs without manual splitting.</p><p><strong>When to use:</strong> When training on 2+ GPUs. Use DDP for the best performance and lowest communication overhead. DataParallel is easier for quick setups.</p><h3>📦 9. num_workers=4 in DataLoader – Faster Dataset Loading</h3><p>Utilize CPU cores to load data in parallel.</p><pre>train_loader = DataLoader(dataset, batch_size=64, shuffle=True, num_workers=4)</pre><p>✅ Prevents data loading from becoming the training bottleneck.</p><p><strong>When to use:</strong> Always, unless you’re I/O bound or working on low-core systems. Adjust num_workers based on your CPU core count for optimal performance.</p><h3>📊 10. torch.profiler.profile() – Find Bottlenecks</h3><p>Use PyTorch’s built-in profiler to find CPU/GPU performance issues.</p><pre>import torch.profiler<br>with torch.profiler.profile(<br>    activities=[<br>        torch.profiler.ProfilerActivity.CPU,<br>        torch.profiler.ProfilerActivity.CUDA<br>    ],<br>    schedule=torch.profiler.schedule(wait=1, warmup=1, active=2, repeat=1), on_trace_ready=torch.profiler.tensorboard_trace_handler(&quot;./log&quot;)) as prof:<br>    for inputs, labels in train_loader:<br>        inputs, labels = inputs.to(device), labels.to(device)<br>        outputs = model(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br>        prof.step()</pre><p>✅ Export logs to TensorBoard and spot slow ops or memory spikes.</p><p><strong>When to use:</strong> During model development or optimization. Especially useful when training is slow or GPU is underutilized.</p><h3>🚀 Wrapping Up</h3><p>By applying even a few of these optimizations, you’ll likely see <strong>faster epochs, better memory utilization, and more stable training</strong>. Combine them all, and your training pipeline becomes a finely tuned performance machine.</p><p>🧠 Want a real-world benchmark on your RTX 3060 or AMD GPU? Let me know!</p><h3><strong>👤 About the Author</strong></h3><p>I’m a machine learning engineer passionate about performance optimization and system-level AI engineering. I work extensively with PyTorch, CUDA, distributed training, and real-time inference. My goal is to make deep learning both accessible and efficient — on everything from consumer GPUs to scalable multi-node clusters. If you’re into model compression, knowledge distillation, or building smarter AI pipelines, feel free to connect!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f50b8f719bad" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>