<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ishveen Kaur on Medium]]></title>
        <description><![CDATA[Stories by Ishveen Kaur on Medium]]></description>
        <link>https://medium.com/@ishveen?source=rss-75aefbf1f3d5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-Zol8QgJuJ5nHOoxAOGGFw.jpeg</url>
            <title>Stories by Ishveen Kaur on Medium</title>
            <link>https://medium.com/@ishveen?source=rss-75aefbf1f3d5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:41:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ishveen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rebuilding Classroombot.ai]]></title>
            <link>https://medium.com/@ishveen/rebuilding-classroombot-ai-09c6afac63a4?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/09c6afac63a4</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[socratic-method]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 02:37:55 GMT</pubDate>
            <atom:updated>2026-11-11T02:37:55.999Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3nUgbO-GU4g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3nUgbO-GU4g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3nUgbO-GU4g%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0babfacbf51aa0ac01abcd9bb74a5cbc/href">https://medium.com/media/0babfacbf51aa0ac01abcd9bb74a5cbc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5ItAcgmVrvJQOQdo1_Oow.png" /></figure><p>In a world with chatbots and AI tools at the hands of any student, how can professors trust that their students are actually doing their work and learning the material? Moreover, with AI chatbots hallucinating answers so often, how can we trust AI to provide students with the right answers? With Classroombot.ai, students can use AI responsibly to learn rather than quickly get answers. ClassroomBot.ai uses a Socratic engine that asks questions to guide the students to the answer rather than handing it to them. Also, teachers can see the questions students ask and how students are using AI to learn. Rather than answering the same simple questions in office hours, professors can see what students struggle with, what they want to learn, and what could possibly be added to their curriculum, and this would all be presented in a neat, organized, and informative dashboard. That’s the goal of Classroombot.ai: to create trust between students and teachers, and to build trust in AI to teach students responsibly. It gives an opportunity for students to learn with AI responsibly, and it allows teachers to learn about their students and how they can improve their courses. Furthermore, Classroombot.ai’s architecture gives the model access only to the course material and it must directly answer and cite from this material, so students can trust what it tells them.</p><p>But the question is, how can an app balance the student experience, teacher dashboard, accuracy, and trust between teachers and students?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2m2Ft0I0jLrUorjXQ7wDdQ.png" /><figcaption>How this app will balance</figcaption></figure><h3>How the App Works Right Now</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DqoWW8Yb9nwnlc5SMFjuTg.png" /><figcaption>Simple architecture</figcaption></figure><p>Classroombot.ai is a chatbot that is designed to answer only from a course’s own materials and tries to teach through questions rather than answers. It has two parts: a chat window for students and an analytics dashboard for the professor.</p><p>Classroombot.ai uses RAG: before answering, it searches the course’s own materials for the passages most related to the question, then hands those passages to the AI model along with the question.</p><p>When a student asks something, the AI model (Google’s Gemini) first sorts the question into one of five types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XY0HYPz9N7tV7mNx_u-Tew.png" /></figure><p>It also tries to read the student’s intent: are they learning something new, asking for clarification, applying an idea, debugging, checking their understanding, or exploring?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLomZtiVNm2omqnjCUTLjg.png" /></figure><p>For conceptual and code questions, the bot is meant to walk the student through a sequence instead of answering outright:</p><ol><li><strong>Diagnose:</strong> ask a question to find out what the student already knows.</li><li><strong>Probe:</strong> ask a “why” or “how” question that pushes the student’s reasoning, and continue to ask targeted questions until the student is stuck or wrong.</li><li><strong>Teach:</strong> give a clear explanation, citing the course material it came from.</li><li><strong>Check:</strong> ask a short question to confirm the student understood.</li></ol><p>The system runs as seven separate services in the code: one gathers course files, one prepares and searches the passages, one is the tutor itself, one handles the professor’s analytics, and the rest handle login, the database, and the website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTbRx8x8sdrgaqSq_2yB7w.png" /><figcaption>Architecture</figcaption></figure><h3>Improvements Needed</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5P5VtuBxoSqbQrZWwFRKSQ.png" /></figure><ul><li><strong><em>Problem</em></strong>: Every phase definition, limit, and the phrase “don’t give away the answer” is in a single long prompt. Gemini is asked to follow it, but nothing forces it to, and a model under pressure from a persistent student will often simply comply with the student because of its user-friendly training. → <strong><em>Fix</em></strong>: My goal is a tutor whose Socratic behavior is built in, not just requested. In order to achieve this, the code must change greatly. Instead of leaving the decisions to the model, our own system makes the decisions (is this question in scope, which passages are relevant, which teaching phase are we in, etc.) before the model writes a word or receives the user’s question, and checks the result afterward. The code will also block any responses that give away any answers when it is not appropriate to.</li><li><strong><em>Problem</em></strong><em>:</em> The model writes its response first, and only afterward does the code try to work out which phase that response was in. The method is flawed because it just checks whether the tutor’s response contains a question mark and how long it is. It should be looking at the student’s message to decide the phase, not the tutor’s. So the phase never actually shapes what gets written, and it’s identified incorrectly most of the time. Also, the code treats “I don’t know” as frustration, which jumps straight to the explanation. That is the most common thing a confused student types, and it bypasses all the questioning. → <strong><em>Fix</em></strong><em>:</em> The phase will be identified before the model builds a response so that the model can answer with the phase in mind. Instead of checking the tutor’s response length and question marks, a separate model will read the student’s message and decide which phase the conversation is in. That also fixes the “I don’t know” problem since a student saying they don’t know becomes a signal to back up and ask a simpler question rather than being a reason to skip to an answer.</li><li><strong><em>Problem</em></strong><em>:</em> When a response lacks a source, the code appends a generic source line, meaning it is citing nothing. Every question is compared against every passage one at a time. There is no keyword search alongside meaning-based search, no second pass to re-rank the results, and the system always takes the top five matches without knowing how good they are. So it can’t tell which passages are the most relevant accurately, which is exactly the judgment out-of-scope redirection needs. → <strong><em>Fix</em></strong><em>:</em> The code will make sure that all citations are accurate before returning the model’s response to the user. Currently, retrieval is very simple, with no index, keyword search, or reranking, and I plan to add all three.</li><li><strong><em>Problem</em></strong><em>:</em> Several lab notebooks given to students already contain the completed solution code, pasted in. Those notebooks are searchable, so a tutor citing its sources could hand a student the answer key. The lecture chunking and parsing is very basic and some slides have text parsed that are not relevant to what the professor may be trying to teach. → <strong><em>Fix</em></strong><em>:</em> Solution code will be removed from lab notebooks before they’re added into the RAG documents, so the tutor won’t accidentally give it to the student. Also, I plan to parse and chunk the lectures and code appropriately by using methods such as a VLM to describe lecture slides or chunk code by sections in the labs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SwjZ8zRGzytbhyd5M04Cw.png" /><figcaption>Doc Parsing with a VLM</figcaption></figure><ul><li><strong><em>Problem</em></strong>: The tutor only remembers the current conversation, but it keeps no record of what a student understands or struggles with, so every session starts from scratch. → <strong><em>Fix</em></strong>: The tutor will keep a short profile of each student, per course: what they’ve shown they understand, what they’ve struggled with, and roughly what level they’re at. For a new student, the tutor’s first job is to gauge their level. Entries that keep proving useful are reinforced, and ones that go unused fade.</li><li><strong><em>Problem</em></strong><em>:</em> As of now, the dashboard is very basic and doesn’t collect data of what questions students typically ask and what topics they are most interested in. → <strong><em>Fix</em></strong><em>:</em> Professors will be able to read individual conversations with student names removed, along with trends across the whole class: which topics come up most, where students get stuck, and which questions the course material doesn’t cover. Students will know this before they start using the tutor. The tutor does keep a profile of each student so it can adjust to their level, but that profile stays with the tutor and isn’t shown to the professor.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhprqUUTQrjlKDtWXBA2kQ.png" /></figure><ul><li><strong><em>Problem</em></strong><em>:</em> very course is taught by the same general model, Gemini, that is told what to do only by a prompt. → <strong><em>Fix</em></strong><em>:</em> A giant general-purpose model is trained to be helpful, and to a student “helpful” usually means they need an answer right away. A prompt asking it to hold back from doing this is fighting its training. A smaller model can be trained on thousands of examples of good Socratic teaching until that behavior is its own behavior. These examples will come from Gemini playing simulated students with different personalities, and it will also simulate the tutor giving responses, and the conversations that pass the rubric will be used as training data. It also runs on infrastructure we control and costs a lot less per conversation. The trade-off is that a small model is worse at complex reasoning and multitasking. That is exactly why our code handles scope, retrieval, and phase, so the model’s only job is to write one good response. The rubric will help define what good teaching and bad teaching looks like for each phase. This will help evaluate responses. To improve the performance, SFT and RL will be performed so that the model can learn from ideal responses. The rubric will be used as the reward for RL. Gemini grades responses against the rubric, with people checking its grades. However, before performing RL, we must perform SFT. SFT will help the model learn how its tone should be using ideal examples, while RL will be used to improve on meeting the criteria of the rubric. It’s important that the rubric captures all the aspects that a good response should have so that it can use this rubric as a reward model to improve.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7UVnBeuo5cktaf_fppKtw.png" /><figcaption>The Rubric</figcaption></figure><h3>How it Benefits Students</h3><p>Students can benefit from this app in many ways because these chatbots are made specifically for each course.</p><ul><li>The students can learn from chatbots rather than just getting answers from them, which will help them better understand the course material.</li><li>The chatbot will be trained using SFT and RL, and each course will get its own chatbot with specific training based on what course it is, which means that it will teach better than a generic chatbot that is not trained on the course.</li><li>Students can get help at any time rather than having to wait to ask the professor</li><li>Not only does the tutor chatbot answer questions or teach lessons, but it also guides the student by using the four phases mentioned before. This creates an environment where the student learns to answer questions and think through concepts rather than studying from the answers a chatbot gave it.</li></ul><p>Although this app benefits students, what stops a student from making their own custom GPT, uploading the course data, and making it follow this Socratic method so that they can learn on their own? It may not be as accurate and useful as our app, but it will get the job done for most students. The true benefit is the connection between the teachers and students, because all teachers know that their students use AI, but this app will show teachers how the students use it.</p><h3>How it Benefits Teachers</h3><p>The teachers will have a dashboard filled with data about the questions that students ask and what topics they might be stuck on.</p><p>They can also see <em>how</em> students use AI, which gives them a better sense of how students learn and how they reason through things. Moreover, by seeing how students interact with AI, they can learn which of the tutor’s teaching methods work and which don’t</p><p>By seeing when students ask questions related to the course but not specifically in the course, the professor can update their course to include information that they might have missed. If students are confused on one concept frequently, the professor can work on improving that portion of their lecture.</p><p>By building this connection of AI usage between students and teachers, students can learn by using AI responsibly while professors can continue to improve their courses with better material</p><h3>How we Ensure Accuracy</h3><p>To ensure that the students are receiving accurate responses and teaching the professor’s material well, the code will ensure that answers to a user’s questions will only come from course materials. The response should also cite this information so that the user can check it against the course’s data. If a user’s question is on topic but doesn’t have any course material on it, it will be marked as a gap that a teacher can fill.</p><p>Each course gets its own model, trained using SFT and RL on simulated tutoring conversations built from that course’s materials. That training doesn’t teach the model new facts. Instead, it teaches the model how to teach this particular course: its vocabulary, the mistakes students commonly make, and eventually the professor’s own way of explaining things. Retrieval is what keeps the tutor accurate, but the course-specific training is what makes it feel like the student is in the classroom.</p><h3>Summary</h3><p>To make this app the best it can be, we must make the following changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IU7x6yx3iYhI5JlTn5O2Og.png" /><figcaption>The plan</figcaption></figure><p>When these changes are made, students will get a tutor built for their specific course that teaches them rather than answers for them. Teachers will get a dashboard that shows where students are stuck, what they ask the most, and what the course is missing. Since every answer comes from the course’s own material, both can trust what the tutor says.</p><p>So how does an app balance the student experience, the teacher dashboard, accuracy, and trust? A student who is guided to an answer creates a conversation that the professor can learn from. A dashboard built from those conversations shows the professor how their class actually learns and how they can improve. Accuracy of responses creates trust in the chatbot for both the student and the professor. Anyone can build a Socratic chatbot, but Classroombot.ai creates trust in AI usage between teachers and students so that they can learn from each other.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09c6afac63a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I made a stock-analysis LLM ~100× cheaper to run]]></title>
            <link>https://medium.com/@ishveen/how-i-made-a-stock-analysis-llm-100-cheaper-to-run-07e8de3240d0?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/07e8de3240d0</guid>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[stocks]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 14:25:51 GMT</pubDate>
            <atom:updated>2026-06-19T14:25:51.228Z</atom:updated>
            <content:encoded><![CDATA[<p>This summer, I decided to intern on a real project and spent a chunk of time turning a terrible model server into a faster one, measuring every single step so I could say “implementing X made this faster” instead of “trust me, it’s faster now.” I’m going to build it from the ground up: what the app even is, why inference speed is so important, the theory behind every parameter, the code that implements these parameters, the code I had to write, and then the results ranked.</p><p>If you’ve never touched an inference engine before, don’t worry; I explain the vocabulary as it comes up.</p><h3>Part 1 — What StockTickers.ai actually does</h3><p>StockTickers.ai is a research platform for stocks. For every ticker in our database (AAPL, TSLA, NVDA, …) we pull a pile of market data from a provider called <strong>FMP</strong> (Financial Modeling Prep): company profile, income statements, growth rates, revenue segments, earnings history, insider trades, analyst ratings, and more. Fourteen tables’ worth, per ticker.</p><p>Raw tables are not insight. A pile of numbers doesn’t tell a <em>newcomer</em> whether a stock is scary, and it doesn’t tell a <em>day trader</em> where the momentum is. So we do a step called <strong>LLM enrichment</strong>: we feed that data to a language model (we run <strong>Gemma-4-E4B</strong>) and ask it to write a structured, readable analysis on top of the numbers.</p><blockquote><strong><em>What “Gemma-4-E4B” means.</em></strong><em> Gemma is Google’s family of open-weight models. The </em><em>E4B suffix is the unusual part. The “E” stands for effective parameters. While it reasons with the depth of a larger model, it actively uses an effective 4 billion parameters (about 8B total with embeddings) during inference, preserving battery life and RAM. For our purposes just read it as &quot;a small, fast ~4B open model.&quot; Its size is exactly why it fits on one cheap GPU, and why some of the optimizations below (like pre-quantized checkpoints) turned out not to matter.</em></blockquote><h3>The four personas</h3><p>Here’s the part that multiplies our workload. We generate <strong>four </strong>analyses per ticker, one for each reader persona:</p><ul><li>Newcomer: for someone new to investing; max output tokens: 800</li><li>Retail: A regular retail investor; max output tokens: 1200</li><li>Trader: Active/day trader; max output tokens: 2200</li><li>Analyst: Stock market pro; max output tokens: 2400</li></ul><p>Each persona gets the <em>same underlying data</em> but a different “style guide” telling the model who it’s writing for and what to emphasize. The model returns structured JSON, which we store and serve in the app. The prompt shape: an intro, a persona style guide, the data block (FMP data tables), and “return only valid JSON.”</p><h3>Part 2 — Why optimize enrichment specifically</h3><p>Let’s do the math. Our database has about all US market tickers in the active watchlist. Right now, I tested on 100 tickers. That’s already <strong>400 model calls</strong> per full refresh. We want to refresh nightly. Each call generates up to about 2,400 tokens. We’re talking hundreds of thousands of generated tokens, every night, forever.</p><p>The model runs on <strong>Modal</strong>, a cloud platform that rents us a <strong>GPU</strong> by the second and only charges while the GPU is actually running. We use an <strong>L4</strong> — a 24 GB GPU, the cheap but effective option. So our cost and our nightly run-time are both a direct function of one thing:</p><blockquote><strong><em>How many tokens can we generate per second per GPU, and how few GPU-seconds does the whole job take?</em></strong></blockquote><p>This is a <strong>throughput</strong> problem, not a <strong>latency</strong> problem:</p><ul><li><strong>Latency</strong> = how long <em>one</em> request takes. Matters when a human is waiting on a chatbot.</li><li><strong>Throughput</strong> = how <em>many</em> requests finish per second across the whole batch. Matters when you’re grinding through a known list and no single item is urgent.</li></ul><p>Nobody is sitting there refreshing the page waiting for AAPL’s trader view to render at 2am. We just want the whole pile done fast and cheap. So we tune for <strong>total tokens/sec across many concurrent requests</strong>, even if any single request gets a hair slower.</p><h3>Part 3 — Why we run our own model on Modal (and not just call an API)</h3><p>Prior to working on this project, I assumed that if you wanted an LLM in your app, you just called an API. You sign up with OpenAI or Anthropic or Google, you get a key, you send a prompt, you get text back, and you pay per token. Simple. So when I learned that we were running our own copy of Gemma on rented GPUs instead, my first question was the obvious one: why make life harder? It turns out there are good reasons, and understanding them is the whole reason this article exists.</p><h4>Quick definition: what “inference” actually is</h4><p>When people say “inference,” they just mean running a trained model to get an output. Training is the expensive, one-time process of teaching the model. Inference is every time after that, when you actually use it. Each persona analysis we generate is one inference call. So when I talk about “optimizing inference,” I mean making each of those calls, and all of them together, faster and cheaper to run.</p><p>There are two ways to do inference. You can rent it as a service (an API), where someone else runs the model and you pay per token. Or you can host the model yourself, where you rent a GPU, load the model onto it, and run the calls directly. Both give you the same text back. They have very different economics, and that difference is the whole story here.</p><h4>What Modal is, and why we use it</h4><p>Modal is a serverless GPU platform. “Serverless” sounds like there are no servers, but what it really means is that you don’t have to manage them. You write your code, you tell Modal which GPU you want (we use an L4), and Modal handles spinning up the machine, loading your model, and tearing it all down when you are done. The part that matters most for us: Modal bills by the GPU-second, and it can scale to zero. When no enrichment job is running, we have zero GPUs on and we pay nothing. When the nightly job kicks off, GPUs spin up, do the work, and shut back down.</p><p>Compare that to the old way of self-hosting, where you would rent a GPU server by the month and pay for it whether it was working or sitting idle at 3am. Modal gives us the cost profile of an API (pay only for what you use) with the control of self-hosting (it is our model, our settings, our optimizations). That combination is exactly what we need, because it gives us customizability without a huge burden of handling the server.</p><h4>So why not just use an API?</h4><p>For a lot of projects, an API is the right call, and I want to be honest about that. APIs are effortless. There is no GPU to configure, no model to load, no out-of-memory error to debug at midnight. If our app made a few LLM calls a day, I would say use an API and never think about it again.</p><p>But look back at our numbers. One ticker is four persona analyses, our universe is meant to grow into the thousands of tickers, and we want to refresh nightly. That is a job that generates hundreds of thousands of tokens every single night, forever. At that volume, per-token API pricing skyrockets into something unaffordable. Self-hosting a small open model like Gemma on a cheap GPU can be many times cheaper per token at that scale. We also get full control: we pick the exact model, we can quantize it, we can run it in batch mode, and nobody deprecates our model out from under us.</p><h4>If self-hosting is cheaper, why doesn’t everyone do it?</h4><p>This is the question that took me the longest to really understand, and it is the heart of this whole article.</p><p>Self-hosted inference is only cheaper if you optimize it. That cheapness is not something you get for free just by renting a GPU. Out of the box, a naively configured self-hosted model can actually be slower and more expensive than just paying an API, because you are renting a whole GPU by the second and only using a fraction of it. If your GPU sits there processing one request at a time while the rest of its capacity goes to waste, you are burning money.</p><p>The reason an API feels cheap per call is that the provider has already done a massive amount of engineering to pack many users’ requests onto the same hardware efficiently. They optimized it for you, and the per-token price you pay quietly includes all of that work plus their margin.</p><p>When you self-host, you are taking that engineering on yourself. The potential savings are real and large, but you only unlock them by doing the work: batching many requests together, fitting more of them into GPU memory at once, compressing the model, scaling across GPUs only when you actually need to, and measuring all of it so you know it genuinely helped. That work is exactly what the rest of this article is about. We went from a setup that managed about 19 tokens per second to one that does over 2,600, and the gap between those two numbers is the gap between “self-hosting is a waste of money” and “self-hosting is the cheapest option we have.”</p><p>So the trade is simple to state and hard to earn. An API is expensive but effortless. Self-hosting is cheap but only if you put in the optimization work. The rest of this article is how we put it in.</p><h3>Part 4 — The theory: every knob, and why it moves the needle</h3><p>Before any code, the vocabulary. These concepts explain a lot of what we tuned.</p><p><strong>Token.</strong> ≈¾ of a word. Models read and write in tokens. “Analyze TSLA” is ~3 tokens.</p><p><strong>vLLM.</strong> The open-source <strong>inference engine</strong> we run the model inside. “Inference engine” just means the program that loads the model weights and actually generates text from them. It handles the batching, the memory management, the GPU kernels. Every knob in this article is either a vLLM flag or a Modal (the hosting platform) setting. When I say “the engine,” I mean vLLM.</p><p><strong>GPU / VRAM.</strong> The GPU runs the model; <strong>VRAM</strong> is its dedicated memory. Our GPU is an <strong>L4</strong> (NVIDIA’s cheap-but-capable data-center card) with <strong>24 GB</strong> of VRAM. The model <em>and</em> its working memory must both fit in those 24 GB. This single number — 24 GB — turns out to be the constraint behind almost every limit we hit. The L4 also has dedicated <strong>FP8 Tensor Cores</strong> : hardware that does 8-bit matrix math at roughly double the speed of 16-bit .</p><p><strong>Prefill vs. decode</strong> (a lot of explanations below hinge on this). Generating a response happens in two phases. <strong>Prefill</strong> = the model reads your entire input prompt in one big parallel pass and builds its initial notes. <strong>Decode</strong> = it then generates the output one token at a time, each new token depending on all the previous ones. Prefill is one fat parallel chunk of work up front; decode is a long sequence of small steps. Some optimizations only help one phase, but the most effective ones help both phases.</p><p><strong>KV cache.</strong> During decode, the model keeps running “notes” (the keys and values from attention) about everything it’s read and written so far, so it doesn’t recompute them every step. Those notes are the <strong>KV cache</strong>, they live in VRAM, and they are the main thing limiting <strong>how many requests can run at once</strong>. More free VRAM for KV cache = more simultaneous requests. This is the lever behind half the optimizations.</p><p><strong>Batching.</strong> Instead of handling requests one at a time, the engine processes many in the <em>same</em> GPU step. Plain (“static”) batching makes every order wait until a full tray is assembled, and the whole tray must finish before <em>any</em> new order starts. <strong>Continuous</strong> batching is the upgrade: vLLM slots new requests into an <em>in-flight</em> batch on every token boundary and evicts finished ones immediately, so the GPU doesn’t have to wait for all the requests to finish before putting new ones. When one slot is done, another slot can fill in. This is what makes continuous batching dramatically better than static batching, and it’s the single biggest structural win in this whole project.</p><p><strong>Concurrency.</strong> How many requests are in flight at the same instant.</p><p><strong>Quantization, FP8, and bf16.</strong> The model’s numbers (“weights”) have to be stored in <em>some</em> numeric format. <strong>bf16</strong> (bfloat16, 16-bit) is the default, full-precision format (“bf16” = “the normal, uncompressed model”). <strong>Quantization</strong> means storing those numbers less precisely to save memory and run the math faster. <strong>FP8</strong> is the 8-bit format we quantize to: half the bytes of bf16, and (on the L4’s FP8 cores) faster math. So “bf16 vs FP8” throughout = “uncompressed vs compressed.”</p><p>With those in hand, here are the actual knobs we changed.</p><h4>The levers that matter</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATKNhkg6K-DKwETPr1Lh-Q.png" /><figcaption>Parameters that we changed</figcaption></figure><p>If you don’t understand these parameters, you can understand it with this “kitchen analogy”. Each request is an order. max_containers = how many kitchens you open. max_inputs = orders one kitchen accepts at once. max_num_seqs = orders it cooks simultaneously on the stove. gpu_mem_util = how much counter space it uses. max_model_len = how much counter space it <em>reserves per order no matter how small the order is</em>. Quantization = a more compact recipe that takes half the space and cooks faster. Almost every problem we hit was &quot;ran out of counter space&quot; (24 GB).</p><h4>One workload-specific term: prefix caching</h4><p>This one deserves its own note because it’s a near-perfect fit on paper. Our four personas all share a long instruction preamble and differ only in the persona guide + the (identical, per ticker) data. vLLM splits the prompt into 16-token blocks and remembers the KV for each block; on a new request it reuses blocks <strong>from the start of the prompt up to the first token that differs</strong>. Matching is exact, prefix-only — one changed character breaks reuse from that point.</p><p>So the rule is: <strong>put the constant instructions first, the variable stuff last.</strong> We do exactly that. The theory said this should give 50–80% savings. There was a part where I switched the order where text came in the prompts to show the drastic difference in throughput and KV cache usage.</p><h3>Part 5 — The phases: the climb from worst to best</h3><p>The whole showcase is built as a <strong>staircase</strong>. We start from a deliberately crippled config (“what does <em>no</em> optimization look like?”) and turn on one group of levers at a time, measuring the gain at each step. The discipline here matters: if you flip five knobs at once and the number moves, you’ve learned nothing about <em>which</em> knob did it. So each phase changes one thing.</p><p>Here are the phases <strong>in the order we actually ran them.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATKNhkg6K-DKwETPr1Lh-Q.png" /><figcaption>Phases (not including the bonus phase at the end)</figcaption></figure><p>Each row keeps everything from the row above and changes the <strong>one</strong> bolded thing. That’s what makes it a fair test.</p><blockquote><strong><em>“Prefill-heavy knobs”</em></strong><em> (the </em><em>tuned phase) means two specific settings: </em><em>max-num-batched-tokens (how many prompt tokens vLLM is allowed to chew through in a single prefill step) and </em><em>max_num_seqs raised to 256 (more sequences in flight). Both only pay off when </em>prefill<em> is your bottleneck — i.e. when prompts are very long. The bet was that bigger prefill budgets would speed our ingestion; spoiler in Part 9, they did nothing, because our prompts are short enough to already fit the default budget.</em></blockquote><p>Three phases from the original plan got <strong>cut or deferred</strong>, and the reasons are themselves findings:</p><ul><li><strong>no-flashinfer — dropped.</strong> FlashInfer is a faster set of <em>attention</em> kernels (attention is the model&#39;s single most expensive calculation) we planned to A/B. But it&#39;s <strong>incompatible with Gemma-4.</strong> Attention is computed in parallel &quot;heads,&quot; and each head works over a vector of a fixed width — the <strong>head size</strong>. FlashInfer only ships optimized kernels for certain head sizes (64, 128, …); Gemma-4 uses <strong>head size 256</strong>, for which FlashInfer has no kernel, so the engine literally won&#39;t boot with it. There&#39;s nothing to isolate when the lever can&#39;t be turned on at all.</li><li><strong>offline — built separately.</strong> The offline batch interface (the real nightly-job workhorse) isn&#39;t a config flip — it&#39;s a different <em>way of running the model entirely</em>, with no HTTP server. It gets its own runner and its own metrics. More on this in Part 9, because it&#39;s a star.</li></ul><h3>Part 6 — The backend code: one switch, a book of presets</h3><p>How do you run “a dozen different model configurations” without hand-editing a dozen settings every time? You build <strong>one switch and a book of presets.</strong></p><p>The inference server has a single word near the top:</p><pre>PROFILE = &quot;quant&quot;   # ← the one word you change</pre><p>…and a list called CONFIGS — the book of presets, like the scene modes on a camera (Portrait, Night, Sports). Each entry is a named bundle with every setting filled in. The rest of the file reads its settings <em>from the matching bundle</em>. So flipping PROFILE quietly rewires the whole server, and because nothing else changes, any measured difference is attributable to that preset alone.</p><p>The routine for every phase was just:</p><ol><li><strong>Change one word</strong> — PROFILE = &quot;&lt;phase&gt;&quot;.</li><li><strong>Deploy</strong> — modal deploy gemma_modal.py (same app, same URL every time).</li><li><strong>Glance at the boot log</strong> to confirm the settings took and nothing OOM’d.</li><li><strong>Benchmark</strong> that phase, save the numbers.</li><li>Next phase, repeat.</li></ol><p>A couple of gotchas worth surfacing because they cost me time:</p><ul><li><strong>Turning prefix caching <em>off</em> needs an explicit “off.”</strong> vLLM leaves prefix caching <strong>on by default</strong>, so <em>not mentioning it</em> doesn’t disable it. I actually caught a bug where worst was secretly running <em>with</em> prefix caching (hit rates of 45–71% where they should&#39;ve been 0). Fixed by adding an explicit --no-enable-prefix-caching to the early phases. (See Part 8.)</li><li><strong>Counter space (24 GB) was the recurring wall.</strong> It’s why worst&#39;s huge context couldn&#39;t even boot, why 512-sequences OOM&#39;d, and why FP8 helped so much (it freed space). Almost every failure traced back to this one ceiling.</li></ul><p>The offline mode is a <strong>different animal</strong> — not a preset, not deployed. Instead of an always-on kitchen taking orders over the internet, you hand the kitchen the <em>entire night’s order list at once</em> and let it cook the whole pile with no internet in between:</p><pre>modal run gemma_modal.py::batch_enrich --prompts-file offline_prompts.jsonl</pre><p>It loads the model once and runs llm.chat() <strong>in-process</strong> — a direct Python call into vLLM, no HTTP, no network, no per-request web overhead. That&#39;s the cheapest way to grind a big list, and it&#39;s exactly what the nightly enrichment job is.</p><h3>Part 7 — The code I had to write</h3><p>A showcase is only worth anything if you <strong>measure</strong> the gains. This is the part I actually built.</p><p>The whole file is benchmark_enrich.py. It calls the <em>exact</em> production functions that are contained in my app— llm_persona._build_persona_prompt to build the prompt and llm_client.generate_json to hit the model — with one deliberate change: it feeds <strong>2 real FMP tables instead of all 14</strong>, to keep prompt length controlled across the sweep. Everything else (persona specs, per-persona token budgets, response_format={&quot;type&quot;:&quot;json_object&quot;}, the OpenAI client) is identical to what the app does in production. So the numbers show us how fast <em>our actual job</em> goes.</p><h4>The harness, in pieces</h4><p><strong>A closed-loop, time-boxed concurrency sweep.</strong> At each concurrency level C (1, 8, 32, 64, 128) it keeps C tickers being enriched at once for a fixed window (180s) and counts how many finish — literally “how many tickers in 3 minutes.” One worker enriches one ticker = 4 sequential persona calls, exactly mirroring llm_persona.run. So C workers = up to C concurrent calls to the server. The core loop (benchmark_enrich.py:190):</p><pre>def worker():<br>    while time.perf_counter() &lt; deadline:<br>        with lock:<br>            sym = pool[cursor[0] % len(pool)]<br>            cursor[0] += 1<br>        recs = _enrich_one_ticker(cache[sym])   # 4 persona calls, sequential<br>        with lock:<br>            calls.extend(recs)<br>            tickers_done[0] += 1</pre><p><strong>A background </strong><strong>/metrics sampler.</strong> While the sweep runs, a daemon thread polls vLLM&#39;s Prometheus /metrics endpoint every few seconds to capture <strong>KV-cache usage %</strong>, <strong>prefix-hit rate</strong>, and <strong>peak concurrent running requests</strong>. This is what let me see <em>inside</em> the engine rather than just timing requests from outside — and it&#39;s how I caught the biggest surprise of the whole project (next section).</p><p><strong>Per-level stats.</strong> For each level it records tokens/sec, tickers/min, calls/sec, success rate, and the <strong>p50/p95/p99 latency percentiles</strong> — because the headline question for a throughput job is “does latency stay bounded as load rises, or does it explode?”</p><p><strong>A one-time cold-start measurement.</strong> measure_startup hammers the endpoint until it gets a first successful response, timing how long the GPU took to boot from zero — because with min_containers=0 (scale to zero when idle, pay nothing), every run starts cold, and that cold-start cost is real.</p><p><strong>An offline export + report path.</strong> Since offline mode has no server to hit, I added export_offline_prompts — it writes every prompt the sweep <em>would</em> have sent into a JSONL file (DB-only, no GPU, no cost), using the identical prompts. Then offline_report parses the runner&#39;s output and computes the <strong>same</strong> metrics (tok/s, $/ticker, JSON-parse %) so offline is directly comparable to the HTTP phases. This is the bridge that made &quot;is the offline batch interface actually better?&quot; an answerable question.</p><p>Here is the full code + explanations for this benchmarking file that would run for every phase:</p><p>Here’s a top-to-bottom walkthrough of benchmark_enrich.py, snippet then explanation, in the order the file is written. You can paste this straight into Part 7. I kept the prose em-dash-free to match your Modal section.</p><h4>Imports</h4><pre>from __future__ import annotations<br>import csv<br>import json<br>import statistics<br>import threading<br>import time<br>from pathlib import Path<br>import requests<br>from sqlalchemy import text<br>from . import llm_client, llm_persona<br>from .config import MODAL_LLM_BASE_URL, logger<br>from .db import get_engine</pre><p>threading and time are how it drives concurrency and measures latency, requests is how it scrapes the engine&#39;s metrics, sqlalchemy.text is for the database queries, and the local imports (llm_client, llm_persona) are the actual production modules. By importing the real llm_persona instead of reimplementing it, the benchmark physically cannot drift from what the app does, and this allows us to simulate what would actually happen when we are running requests for our app.</p><h4>Personas and their token budgets</h4><pre>PERSONAS = (&quot;newcomer&quot;, &quot;retail&quot;, &quot;trader&quot;, &quot;analyst&quot;)<br># Identical to llm_persona._generate_view&#39;s hardcoded map.<br>MAX_OUT = {&quot;newcomer&quot;: 800, &quot;retail&quot;: 1200, &quot;trader&quot;: 2200, &quot;analyst&quot;: 2400}</pre><p>These are the four personas, and MAX_OUT is how many output tokens each one is allowed to generate (we mentioned in earlier sections).</p><h4>Step 1, gathering the trimmed input (2 tables instead of 14)</h4><pre>def gather_trimmed_inputs(symbol: str, conn) -&gt; dict:<br>    out: dict = {&quot;symbol&quot;: symbol}<br># Table 1 - assets (company profile)<br>    row = conn.execute(text(&quot;&quot;&quot;<br>        SELECT symbol, name, sector, industry, exchange<br>        FROM assets WHERE symbol = :s<br>    &quot;&quot;&quot;), {&quot;s&quot;: symbol}).mappings().first()<br>    out[&quot;profile&quot;] = dict(row) if row else {}<br>    # Table 2 - income_statements (latest quarter/FY), derived margins like prod<br>    row = conn.execute(text(&quot;&quot;&quot;<br>        SELECT period, report_date, revenue, gross_profit, operating_income, net_income, eps,<br>               ROUND((gross_profit::numeric / NULLIF(revenue, 0) * 100)::numeric, 2) AS gross_margin_pct,<br>               ROUND((operating_income::numeric / NULLIF(revenue, 0) * 100)::numeric, 2) AS operating_margin_pct<br>        FROM income_statements WHERE symbol = :s<br>        ORDER BY report_date DESC LIMIT 1<br>    &quot;&quot;&quot;), {&quot;s&quot;: symbol}).mappings().first()<br>    out[&quot;latest_income&quot;] = { ... } if row else {}<br>    return out</pre><p>This is the one deliberate difference from production. The real enrichment pulls all 14 FMP tables, this pulls only 2 (company profile + latest income statement). The point is to keep the prompt length controlled and identical across every phase, so that when throughput changes between phases I know it came from the server config and not from one ticker happening to have a longer data block. This also saved time when running 100+ requests at once for each phase.</p><h4>Choosing which tickers are eligible</h4><pre>def eligible_pool() -&gt; list[str]:<br>    &quot;&quot;&quot;Active watchlist tickers that actually have the 2 real tables loaded.&quot;&quot;&quot;<br>    engine = get_engine()<br>    with engine.begin() as conn:<br>        rows = conn.execute(text(&quot;&quot;&quot;<br>            SELECT w.symbol<br>            FROM watchlist w<br>            JOIN assets a ON a.symbol = w.symbol<br>            JOIN income_statements i ON i.symbol = w.symbol<br>            WHERE w.active<br>            GROUP BY w.symbol<br>            ORDER BY w.symbol<br>        &quot;&quot;&quot;)).fetchall()<br>    return [r[0] for r in rows]</pre><p>This builds the list of tickers the benchmark is allowed to use.A ticker only qualifies if it has a row in both assets and income_statements, because those are the two tables gather_trimmed_inputs reads. If a ticker is missing one, its prompt would be half-empty and would pollute the measurement.</p><h4>Pre-caching the inputs so we measure the model, not the database</h4><pre>def precache_inputs(pool: list[str]) -&gt; dict[str, dict]:<br>    &quot;&quot;&quot;Gather the trimmed real inputs once per ticker (the LLM call is the<br>    workload we measure; re-querying per request would just add DB noise).&quot;&quot;&quot;<br>    engine = get_engine()<br>    cache: dict[str, dict] = {}<br>    with engine.begin() as conn:<br>        for sym in pool:<br>            cache[sym] = gather_trimmed_inputs(sym, conn)<br>    return cache</pre><p>We query the database once up front and stash every ticker’s input in a dict. During the actual timed benchmark we never touch the database again, we just read from this cache. The thing we want to measure is the LLM call, so any database latency mixed into the timing would skew the results. This isolates the variable we care about.</p><h4>Step 2, one persona call</h4><pre>def _one_persona_call(persona: str, inputs: dict) -&gt; dict:<br>    prompt = llm_persona._build_persona_prompt(persona, inputs)   # real template<br>    t0 = time.perf_counter()<br>    try:<br>        res = llm_client.generate_json(prompt, tier=&quot;flash&quot;,<br>                                       max_output_tokens=MAX_OUT[persona])  # real call<br>        dt = time.perf_counter() - t0<br>        return {&quot;ok&quot;: True, &quot;latency&quot;: dt, &quot;persona&quot;: persona,<br>                &quot;in_tokens&quot;: res.input_tokens, &quot;out_tokens&quot;: res.output_tokens}<br>    except Exception as e:  # noqa: BLE001<br>        return {&quot;ok&quot;: False, &quot;latency&quot;: time.perf_counter() - t0,<br>                &quot;persona&quot;: persona, &quot;error&quot;: str(e)[:140]}</pre><p>A single persona analysis, wrapped in a timer. It calls the two real production functions (_build_persona_prompt and generate_json). time.perf_counter() brackets only the model call to get clean latency. The try/except is deliberate: under heavy load some calls will fail or time out, and instead of crashing the run, a failure is recorded as a structured result with ok: False. That is what lets the harness compute a success rate later instead of just dying. Note it also captures in_tokens and out_tokens returned by the client, which feed the tokens-per-second math.</p><h4>One ticker = four sequential calls</h4><pre>def _enrich_one_ticker(inputs: dict) -&gt; list[dict]:<br>    &quot;&quot;&quot;4 persona calls, sequential — exactly llm_persona.run&#39;s loop.&quot;&quot;&quot;<br>    return [_one_persona_call(p, inputs) for p in PERSONAS]</pre><p>Enriching one ticker is four persona calls run one after another, exactly as production does it. This matters for the concurrency model below. When the benchmark runs “8 concurrent workers,” each worker is processing a ticker as this 4-call sequence, so the concurrency is at the ticker level, which mirrors reality.</p><h4>Step 3, finding the metrics endpoint</h4><pre>def _metrics_root() -&gt; str:<br>    # MODAL_LLM_BASE_URL ends with /v1; /metrics is served at the app root.<br>    return MODAL_LLM_BASE_URL.rsplit(&quot;/v1&quot;, 1)[0].rstrip(&quot;/&quot;)</pre><p>A tiny helper. The model is called at a URL ending in /v1, but vLLM serves its Prometheus metrics at the app root under /metrics. This strips the /v1 suffix so we can build the metrics URL. This step was key so that we could measure the data.</p><h4>Scraping vLLM’s internal metrics</h4><pre>def fetch_metrics() -&gt; dict | None:<br>    try:<br>        r = requests.get(f&quot;{_metrics_root()}/metrics&quot;, timeout=10)<br>        if not r.ok:<br>            return None<br>        out: dict[str, float] = {}<br>        for line in r.text.splitlines():<br>            line = line.strip()<br>            if not line or line.startswith(&quot;#&quot;) or not line.startswith(&quot;vllm:&quot;):<br>                continue<br>            # name{labels} value   /   name value<br>            try:<br>                left, val = line.rsplit(&quot; &quot;, 1)<br>                name = left.split(&quot;{&quot;, 1)[0]<br>                out[name] = out.get(name, 0.0) + float(val)<br>            except ValueError:<br>                continue<br>        return out<br>    except Exception:<br>        return None</pre><p>This is how the harness sees inside the engine rather than only timing it from the outside. Prometheus metrics come back as plain text, one metric_name{labels} value per line. The loop skips comments and anything that is not a vllm: metric, splits each line into name and value, drops the {labels} part, and sums values that share a name. That summing is intentional and also has a known side effect: under scale-out with multiple replicas, gauges get added together across replicas, which is the source of the &quot;prefix-hit 343%&quot; artifact noted in the results. Everything is wrapped so that if the endpoint is down (for example during a cold start) it returns None instead of causing any errors.</p><h4>Percentile helper</h4><pre>def _pct(values: list[float], p: float) -&gt; float:<br>    if not values:<br>        return 0.0<br>    s = sorted(values)<br>    k = (len(s) - 1) * (p / 100.0)<br>    lo = int(k); hi = min(lo + 1, len(s) - 1)<br>    return round(s[lo] + (s[hi] - s[lo]) * (k - lo), 3)</pre><p>A standard linear-interpolation percentile. Given a list of latencies and a percentile like 95, it sorts them and interpolates between the two nearest samples. This is what produces the p50, p95, and p99 latency numbers. Percentiles matter more than the average for a throughput job, because the average can look fine while the p95 quietly explodes.</p><h4>Step 4, running one concurrency level</h4><p>This is the biggest function, so here it is in pieces.</p><p><strong>Setup:</strong></p><pre>def run_level(concurrency: int, window: float, pool: list[str],<br>              cache: dict[str, dict]) -&gt; dict:<br>    calls: list[dict] = []<br>    tickers_done = [0]<br>    lock = threading.Lock()<br>    deadline = time.perf_counter() + window<br>    kv_peak = [0.0]<br>    run_peak = [0.0]<br>    stop = threading.Event()<br>    m_start = fetch_metrics()<br>    cursor = [0]</pre><p>This sets up one level of the sweep, for example C=32. The design is closed-loop and time-boxed: rather than sending a fixed number of requests, it runs for a fixed window (180 seconds) and counts how much got done. deadline is when to stop. The single-element lists (tickers_done, kv_peak, cursor) are a Python trick to get a mutable value that the inner thread functions can update by reference. lock protects the shared counters from race conditions across threads, and m_start grabs a metrics snapshot before the work begins so we can diff prefix-cache counters at the end.</p><p><strong>The background sampler thread:</strong></p><pre>def sampler():<br>        while not stop.is_set():<br>            m = fetch_metrics()<br>            if m:<br>                # vLLM V1 renamed gpu_cache_usage_perc → kv_cache_usage_perc;<br>                # accept either so KV% is captured across engine versions.<br>                kv = m.get(&quot;vllm:kv_cache_usage_perc&quot;)<br>                if kv is None:<br>                    kv = m.get(&quot;vllm:gpu_cache_usage_perc&quot;)<br>                if kv is not None:<br>                    kv_peak[0] = max(kv_peak[0], kv * 100 if kv &lt;= 1.0 else kv)<br>                rn = m.get(&quot;vllm:num_requests_running&quot;)<br>                if rn is not None:<br>                    run_peak[0] = max(run_peak[0], rn)<br>            stop.wait(10)</pre><p>While the benchmark runs, this thread polls the engine every 10 seconds and records the high-water mark of two things: KV cache usage and how many requests the engine actually had running concurrently. The two-name lookup for KV is the fix for the metric-rename bug, it tries the new name first and falls back to the old one. The kv * 100 if kv &lt;= 1.0 handles whether the engine reports a fraction or a percent. That num_requests_running peak is the single most valuable number this thread captures, because it is how we discovered the real bottleneck in baseline (it capped at ~31, proving the wall was Modal&#39;s admission gate, not the vLLM batch).</p><p><strong>The worker threads:</strong></p><pre>def worker():<br>        while time.perf_counter() &lt; deadline:<br>            with lock:<br>                sym = pool[cursor[0] % len(pool)]<br>                cursor[0] += 1<br>            recs = _enrich_one_ticker(cache[sym])<br>            with lock:<br>                calls.extend(recs)<br>                tickers_done[0] += 1</pre><p>This is the load generator. Each worker loops until the deadline, grabbing the next ticker from the pool (the cursor wraps around with the modulo, so a small pool can drive a long run), enriching it as the 4-call sequence, then recording the results. The lock is held only briefly around the shared state, not during the slow model call, so the workers genuinely run in parallel. The number of these workers running at once is the concurrency level.</p><p><strong>Orchestration:</strong></p><pre>t0 = time.perf_counter()<br>    sampler_t = threading.Thread(target=sampler, daemon=True)<br>    sampler_t.start()<br>    workers = [threading.Thread(target=worker) for _ in range(concurrency)]<br>    for w in workers:<br>        w.start()<br>    for w in workers:<br>        w.join()<br>    elapsed = time.perf_counter() - t0<br>    stop.set()<br>    sampler_t.join(timeout=2)</pre><p>This launches everything and waits. One sampler thread starts, then concurrency worker threads start, then the main thread blocks on join() until every worker hits the deadline and exits. Once they are done it stamps elapsed, signals the sampler to stop, and waits briefly for it to wind down. This is the actual timed window.</p><p><strong>Computing the prefix-cache hit rate:</strong></p><pre>m_end = fetch_metrics()<br>    prefix_hit = None<br>    if m_start and m_end:<br>        q0, q1 = m_start.get(&quot;vllm:prefix_cache_queries_total&quot;), m_end.get(&quot;vllm:prefix_cache_queries_total&quot;)<br>        h0, h1 = m_start.get(&quot;vllm:prefix_cache_hits_total&quot;), m_end.get(&quot;vllm:prefix_cache_hits_total&quot;)<br>        if None not in (q0, q1, h0, h1) and (q1 - q0) &gt; 0:<br>            prefix_hit = round((h1 - h0) / (q1 - q0) * 100, 1)</pre><p>Prefix-cache hits and queries are cumulative counters, so a single reading is meaningless. The trick is to diff: take the snapshot from before the run (m_start) and after (m_end), and compute hits-during-this-window over queries-during-this-window. This gives the hit rate for just this level rather than since the server booted. The guard (q1 - q0) &gt; 0 avoids dividing by zero if no queries happened.</p><p><strong>Assembling the per-level result:</strong></p><pre>ok = [c for c in calls if c[&quot;ok&quot;]]<br>    fail = [c for c in calls if not c[&quot;ok&quot;]]<br>    lat = [c[&quot;latency&quot;] for c in ok]<br>    out_tok = sum(c.get(&quot;out_tokens&quot;, 0) for c in ok)<br>    res = {<br>            &quot;concurrency&quot;: concurrency,<br>            &quot;tickers_completed&quot;: tickers_done[0],<br>            &quot;tickers_per_min&quot;: round(tickers_done[0] / elapsed * 60, 1),<br>            &quot;persona_calls_failed&quot;: len(fail),<br>            &quot;success_rate_pct&quot;: round(len(ok) / max(1, len(calls)) * 100, 1),<br>            &quot;tokens_per_s&quot;: round(out_tok / elapsed, 1),<br>            &quot;latency_p95_s&quot;: _pct(lat, 95),<br>            &quot;kv_cache_used_pct_peak&quot;: round(kv_peak[0], 1) if kv_peak[0] else None,<br>            &quot;prefix_hit_pct&quot;: prefix_hit,<br>            &quot;max_running_observed&quot;: run_peak[0] or None,<br>            ...<br>        }<br>        return res</pre><p>(Trimmed for space, the real dict has p50/p99, calls/sec, and more.) This turns the raw call records into the headline numbers. It splits successes from failures, pulls latencies only from the successful calls, and totals the output tokens. The key derived metrics are tokens_per_s (the main throughput figure), tickers_per_min (the business-facing version of the same thing), success_rate_pct, latency_p95_s, and the two engine-internal peaks captured by the sampler. This dict is one row of a phase&#39;s results.</p><h4>Step 5, measuring cold start</h4><pre>def measure_startup(pool: list[str], cache: dict[str, dict],<br>                    max_wait: float, cold_threshold: float) -&gt; dict:<br>    t0 = time.perf_counter()<br>    sym = pool[0]<br>    while time.perf_counter() - t0 &lt; max_wait:<br>        r = _one_persona_call(&quot;newcomer&quot;, cache[sym])<br>        if r[&quot;ok&quot;]:<br>            startup = round(time.perf_counter() - t0, 2)<br>            cold = startup &gt;= cold_threshold<br>            return {&quot;startup_seconds&quot;: startup, &quot;was_cold&quot;: cold}<br>        last = r.get(&quot;error&quot;, &quot;&quot;)<br>        time.sleep(3)<br>    return {&quot;startup_seconds&quot;: None, &quot;was_cold&quot;: None, &quot;error&quot;: last}</pre><p>Run once per phase before the sweep. Because every phase scales to zero when idle, the first request has to wait for the GPU to boot and load the model, and that cold-start cost is real money. This function just retries a single call every 3 seconds until one succeeds, timing how long that took. The cold_threshold is how it decides whether what it measured was a genuine cold boot or just a warm replica answering fast, which is the basis for the &quot;(warm)&quot; caveats in the results table. This is important to isolate because this number can skew the results.</p><h4>Step 6, the phase driver</h4><pre>def run(phase: str, concurrency: str = &quot;1,8,32,64,128&quot;, window: float = 180.0,<br>        out_dir: str = &quot;./logs/bench&quot;, ...) -&gt; dict:<br>    levels = [int(c) for c in concurrency.split(&quot;,&quot;) if c.strip()]<br>    pool = eligible_pool()<br>    if not pool:<br>        raise SystemExit(&quot;benchmark-enrich: no tickers have both `assets` and &quot;<br>                         &quot;`income_statements` loaded. ...&quot;)<br>    cache = precache_inputs(pool)<br>    startup = measure_startup(pool, cache, warmup_timeout, cold_threshold)<br>        level_results = []<br>        for c in levels:<br>            level_results.append(run_level(c, window, pool, cache))<br>            time.sleep(settle)<br>        result = { &quot;phase&quot;: phase, &quot;model&quot;: llm_client.MODAL_LLM_MODEL,<br>                   &quot;tables_used&quot;: [&quot;assets&quot;, &quot;income_statements&quot;],<br>                   &quot;startup&quot;: startup, &quot;levels&quot;: level_results, ... }<br>        out = Path(out_dir)<br>        out.mkdir(parents=True, exist_ok=True)<br>        (out / f&quot;{phase}.json&quot;).write_text(json.dumps(result, indent=2))<br>        with (out / f&quot;{phase}.csv&quot;).open(&quot;w&quot;, newline=&quot;&quot;) as f:<br>            ...<br>        return result</pre><p>This is the top-level orchestrator for one phase. It parses the concurrency list (&quot;1,8,32,64,128&quot; becomes the sweep), builds the eligible pool, and bails early with a helpful message if no tickers qualify. Then it measures cold start once, runs each concurrency level in turn (with a short settle pause between them so one level&#39;s load does not bleed into the next), and writes everything to both a JSON file and a CSV named after the phase. Those per-phase files are what the comparison step reads later. This is the function behind &quot;run the sweep and save the numbers.&quot;</p><h4>Exporting prompts for the offline runner</h4><pre>def export_offline_prompts(out_dir: str = &quot;./logs/bench&quot;,<br>                           full: bool = False, limit: int | None = None,<br>                           symbols: list[str] | None = None) -&gt; dict:<br>    ...<br>    if full:<br>        cache = {sym: llm_persona._gather_inputs(sym) for sym in pool}  # all 14 tables<br>        out_name = &quot;offline_prompts.full.jsonl&quot;<br>    else:<br>        cache = precache_inputs(pool)  # 2-table trimmed<br>        out_name = &quot;offline_prompts.jsonl&quot;<br>    ...<br>    with out_path.open(&quot;w&quot;) as f:<br>        for sym in pool:<br>            inputs = cache[sym]<br>            for persona in PERSONAS:<br>                prompt = llm_persona._build_persona_prompt(persona, inputs)<br>                rec = {<br>                    &quot;id&quot;: f&quot;{sym}:{persona}&quot;,<br>                    &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>                    &quot;max_tokens&quot;: MAX_OUT[persona],<br>                    &quot;temperature&quot;: 0.2,<br>                }<br>                f.write(json.dumps(rec) + &quot;\n&quot;)</pre><p>Offline mode has no server to drive, so instead of sending requests, this writes every request it would have sent into a JSONL file that the Modal batch runner consumes. It is database-only, no GPU and no cost. The full flag is the switch between the two experiments: False uses the 2-table trimmed input (matching the throughput sweep), True uses the real 14-table production input (used for the long-prompt quality test in Part 10). Each line is one request with a TICKER:persona id so results can be joined back afterward, and crucially it builds the prompt with the same _build_persona_prompt the live path uses, so offline and HTTP are running identical prompts.</p><h4>Scoring JSON validity the way the app would</h4><pre>def _lenient_json_ok(text_out: str) -&gt; bool:<br>    &quot;&quot;&quot;Mirror llm_client&#39;s parsing: strip leading ``` fences, then raw_decode<br>    (accept the first complete JSON object).&quot;&quot;&quot;<br>    t = (text_out or &quot;&quot;).strip()<br>    if t.startswith(&quot;```&quot;):<br>        t = &quot;\n&quot;.join(l for l in t.split(&quot;\n&quot;) if not l.startswith(&quot;```&quot;))<br>    try:<br>        json.JSONDecoder().raw_decode(t)<br>        return True<br>    except Exception:<br>        return False</pre><p>The HTTP path forces valid JSON via response_format, but the offline runner does not, so we need to check whether the offline outputs would actually parse. The important detail is that this mirrors the app&#39;s own lenient parser exactly: it strips markdown code fences and uses raw_decode, which accepts the first complete JSON object even if there is trailing junk. Using the same forgiving rules the production app uses means the &quot;99.8% parse success&quot; number reflects what the app would really accept, not a stricter standard the app does not apply.</p><h4>Making offline comparable to the HTTP phases</h4><pre>def offline_report(out_dir=..., results=&quot;offline_results.jsonl&quot;,<br>                   wall_s=None, process_s=None, usd_per_hr=None,<br>                   offline_gpus: int = 1, http_gpus: int = 2) -&gt; None:<br>    ...<br>    n = ok = comp = prompt = 0<br>    with path.open() as f:<br>        for line in f:<br>            o = json.loads(line)<br>            n += 1<br>            comp += int(o.get(&quot;completion_tokens&quot;) or 0)<br>            if _lenient_json_ok(o.get(&quot;text&quot;, &quot;&quot;)):<br>                ok += 1<br>def block(label, secs):<br>        toks = comp / secs<br>        tpm = n_tickers / (secs / 60.0)<br>        if usd_per_s:<br>            cost = offline_gpus * secs * usd_per_s / n_tickers<br>            ...<br>    block(&quot;process&quot;, process_s)   # cold start excluded<br>    block(&quot;wall&quot;, wall_s)         # true single-run cost</pre><p>This translates the offline runner’s raw output into the same metrics as the HTTP phases so the comparison is fair. It totals the output tokens, checks parse success, then reports two cost bases: process (steady-state, cold start excluded, the right thing to compare against the HTTP phases) and wall (the true single-run cost including cold start). The offline_gpus versus http_gpus parameters are the GPU-count normalization, which is what makes the per-GPU comparison honest. The rest of the function (not shown) prints the HTTP phases side by side using their saved JSON files.</p><h4>The comparison table</h4><pre>def compare(out_dir: str = &quot;./logs/bench&quot;) -&gt; None:<br>    phases = [&quot;worst&quot;, &quot;baseline&quot;, &quot;best&quot;, &quot;prefix&quot;, &quot;quant&quot;, &quot;fp8-ckpt&quot;, &quot;tuned&quot;]<br>    loaded = {p: json.loads((out / f&quot;{p}.json&quot;).read_text())<br>              for p in phases if (out / f&quot;{p}.json&quot;).exists()}<br>    if not loaded:<br>        raise SystemExit(f&quot;no phase result files in {out_dir}&quot;)<br>def at(res, c, key):<br>        for lv in res[&quot;levels&quot;]:<br>            if lv[&quot;concurrency&quot;] == c:<br>                return lv.get(key, &quot;-&quot;)<br>        return &quot;-&quot;<br>    hdr = (&quot;phase&quot;, &quot;startup_s&quot;, &quot;tok/s@1&quot;, &quot;tok/s@32&quot;, &quot;tok/s@128&quot;,<br>           &quot;p95@128&quot;, &quot;prefix%@128&quot;, &quot;KV%@128&quot;, &quot;tickers/min@128&quot;)<br>    ...</pre><p>The final piece. After all phases have run and written their JSON files, this reads them back and prints the side-by-side table that the whole project exists to produce. It only loads phases whose files actually exist, so it works even when some phases were skipped. The little at() helper pulls a specific metric at a specific concurrency level out of a phase&#39;s results, which is how it lines up &quot;tok/s at C=128&quot; across every phase in one row. This is the function that turns nine separate runs into the one comparison you can actually read.</p><h3>Part 8 — How it actually went: the climb, step by step</h3><p>The tidy phase table in Part 5 makes this look like a clean staircase. It wasn’t. The real process was a diary of boot failures, a cloud outage, two separate out-of-memory crashes, a bug where a phase was secretly running <em>with</em> the optimization it was supposed to lack, and one confidently-wrong hypothesis. Here’s the story:</p><p>The loop for <em>every</em> phase was the same five moves: <strong>(1)</strong> set PROFILE = &quot;&lt;phase&gt;&quot;, <strong>(2)</strong> modal deploy, <strong>(3)</strong> read the boot log to confirm the settings took and nothing OOM&#39;d, <strong>(4)</strong> run the concurrency sweep and save the numbers, <strong>(5)</strong> record peak replica count and move on. Simple in theory. Here&#39;s where each step actually went.</p><h4>Step 0 — Build the switch and the harness first</h4><p>Before measuring anything, two pieces had to exist: the PROFILE switch + preset book in gemma_modal.py (Part 6), and the real-enrichment benchmark harness (Part 7). The harness drives the <em>real</em> production code path on <em>real</em> FMP data, not a synthetic load. This is what made every later number trustworthy. Everything below is that harness pointed at one deployed phase at a time.</p><h4>Step 1 — worst, and the fight to even boot the floor</h4><p>You’d think the <em>deliberately bad</em> config would be the easy one. It took <strong>five attempts.</strong></p><ul><li><strong>Attempt 1 — wouldn’t boot.</strong> I deployed worst with the planned 131,072-token context at gpu_memory_utilization=0.70. The engine never came up; the harness&#39;s startup probe timed out after 10 minutes. Root cause was <em>the very lesson the phase exists to teach</em>: a 131,072 context is too big to fit even a <strong>single</strong> sequence&#39;s KV cache in 24 GB at 0.70 util, so vLLM refuses to start. The irony was perfect, but a floor phase still has to <em>run</em> to produce a floor number.</li><li><strong>The fix.</strong> Bump gpu_memory_utilization 0.70 → <strong>0.80</strong> for worst only — just enough headroom to fit one 131,072-token sequence. It keeps worst&#39;s real identity (serialized, absurd context wasting KV) while actually booting. No context reduction needed.</li><li><strong>Attempt 3 — a red herring.</strong> The next cold-start run sat ~8.5 min with no response and <em>looked</em> exactly like the OOM pattern again. It turned out to be a <strong>temporary Modal-side outage</strong>, not our config. Lesson: not every failure is yours. I tore everything down and waited it out (this ate up three hours of my time) :(</li><li><strong>Attempt 4 — the phase was cheating.</strong> It finally ran… but the logs showed a prefix-cache <strong>hit rate of 45–71%</strong> on a phase that’s supposed to have <em>no</em> caching. vLLM enables prefix caching <strong>by default</strong>, and worst had never explicitly turned it off. So this wasn&#39;t the clean floor — it was &quot;worst-but-secretly-cached.&quot; I added an explicit --no-enable-prefix-caching.</li><li><strong>Attempt 5 — the real floor.</strong> With caching genuinely off (hit rate now 0%), worst posted <strong>~19 tok/s, flat</strong>, while p95 latency exploded <strong>47s → 377s</strong> from C=1 to C=8. I deliberately ran only C=1 and C=8 and <strong>killed it at the C=32 start</strong> — even C=8 took ~16 minutes to drain on a one-at-a-time server, so higher levels were pure cost for zero new insight.</li></ul><p><strong>What Step 1 taught me before I’d optimized a single thing:</strong> too-big a context can’t even boot, “off by default” is a lie you must check, the cloud has bad days, and your measurement tools have bugs too.</p><h4>Step 2 — baseline, where the real bottleneck hid</h4><p>This was the first <em>good</em> number, and the first genuine surprise. I turned on real batching (max_inputs=32, max_num_seqs=64, context down to 16k) and ran the full 1→128 sweep.</p><p>The payoff was immediate and exactly as theory predicted: at C=8, throughput jumped <strong>19 → 115 tok/s (~6×)</strong> and p95 dropped <strong>377s → 63s</strong>. Continuous batching is the single biggest structural win, full stop.</p><p>But throughput <strong>plateaued at ~472 tok/s at C=64 and then <em>regressed</em> at C=128</strong> (441 tok/s, p95 doubling to 246s). My live guess was the vLLM batch size (max_num_seqs=64). To check, I ran an <strong>external </strong><strong>/metrics sampler</strong> — a little host-side script polling the engine every few seconds during the sweep. It showed the truth: concurrent running requests <strong>capped at ~31</strong> and KV peaked at only <strong>~25%</strong>. So vLLM never even <em>saw</em> more than ~32 sequences, and memory was nowhere near full. The real wall was <strong>max_inputs=32 . Modal&#39;s per-container admission gate sitting in front of the engine, </strong>not the engine&#39;s batch at all. With one container, request #33 just queues at Modal. I corrected my own live commentary in the log. <strong>Without looking inside the engine, I&#39;d have tuned the wrong knob.</strong> This finding is exactly what motivated best&#39;s scale-out.</p><h4>Step 3 — best, and a fast engine that didn&#39;t fit</h4><p>The plan for best included <strong>FlashInfer</strong>, a faster attention backend, straight from Modal&#39;s own &quot;maximum throughput&quot; guide. First deploy, the engine refused to boot: Reason: [&#39;head_size not supported&#39;]. FlashInfer only ships kernels for certain attention head sizes; <strong>Gemma-4 uses head size 256, which it doesn&#39;t support.</strong> Not a misconfig — the fast engine simply has no part that fits this model. I removed it and fell back to the default backend. (This also retroactively killed the planned no-flashinfer isolation phase: you can&#39;t measure turning off something that was never on.)</p><p>With FlashInfer gone, best (CUDA graphs on, async scheduling, 2-replica scale-out) delivered <strong>1548 tok/s at C=128 — 3.5× baseline — with 3.4× lower p95</strong>, and crucially it <em>kept climbing</em> from C=64 to C=128 where baseline had flatlined. Scale-out had broken baseline&#39;s single-container admission wall, exactly as the Step 2 diagnosis predicted.</p><h4>Step 4 — prefix, and the clever idea that backfired</h4><p>Turning prefix caching <strong>on</strong> (the only change from best) gave a steady ~50–60% hit rate and a real <strong>+25% at C=8</strong>, fading to roughly a tie at full load — exactly the prefill-only behavior the theory predicts. Fine. Expected.</p><p>Then I got clever (well not really becasue I was wrong). The four personas of one ticker run back-to-back with <em>identical data</em>. So I hypothesized: move the big ~1500-token <strong>data block to the front</strong> of the prompt, ahead of the persona guide, and a ticker’s calls 2–4 could reuse that whole block instead of just the ~620-token persona prefix. On paper, ~1.9× more cached tokens per ticker. Obvious win. I built it as prefix-reordered and ran it.</p><p>The hit rate went <strong>down</strong> — 61% → 42% at C=8 — and throughput tied or got slightly worse. I killed it after C=8 once the trend was unambiguous and reverted. <strong>Why I was wrong:</strong> I forgot about <em>cross-worker sharing</em>. In the original order there are only <strong>4 distinct </strong><strong>[intro][persona] prefixes in the entire workload</strong>, so they stay permanently hot in cache and get hit by every worker on nearly every call. Reordering fragments the cache into <strong>per-ticker data blocks that only the one worker handling that ticker can reuse</strong>. A small, shared, always-hot prefix beats a big, fragmented, single-use one.</p><p>What went wrong: I was under the impression that the KV cache would wipe out the cache of the persona if the next request was not that same persona. Because the same data was being used back to back, I thought it’d be smarter to put the data first so that it would cache the data back to back. However, the personas actually stayed in the cache, so even if the same persona didn’t come back until four requests later, it was still saved in the cache.</p><h4>Step 5 — quant (FP8), the phase I expected to fail</h4><p>FP8 was the <strong>at-risk</strong> phase — there was a real chance Gemma-4 wouldn’t quantize cleanly on the fly, or that output would degrade. So I treated it as a verification exercise, not just a run:</p><ul><li><strong>Does it boot?</strong> Yes — cold start 427s (~1.6× the bf16 phases; the extra time is on-the-fly weight quantization at load).</li><li><strong>Is FP8 <em>actually</em> active,</strong> or did it silently fall back to bf16? I checked the warm /metrics cache_config_info and confirmed cache_dtype=&quot;fp8&quot;.</li><li><strong>Did output break?</strong> JSON-parse fails stayed at 0–3 across the sweep — no quality collapse.</li></ul><p>The result was the headline of the whole project: <strong>2640 tok/s at C=128–1.7× the bf16 configs — with the <em>lowest</em> p95 (44s) of any phase.</strong> FP8 won because it’s two wins at once (faster math on the L4’s FP8 cores <em>and</em> a half-size KV cache), and unlike prefix caching it helped at <em>every</em> load.</p><h4>Step 6 — fp8-ckpt, a clean negative result</h4><p>The idea: skip the 180s of boot-time quantization by loading a <strong>pre-built</strong> FP8 checkpoint. It <strong>didn’t help. </strong>Throughput matched quant within noise, and cold start was actually slightly <em>worse</em> (461s vs 427s). Why: the model is small enough that on-the-fly quantization is already cheap, and the pre-built checkpoint still has to be downloaded/loaded at boot, which offsets any saving (essentially this makes no difference, both quantization methods are acceptable)</p><h4>Step 7 — tuned, the same OOM lesson twice</h4><p>The prefill-heavy experiment: max-num-batched-tokens=8192 + max_num_seqs=512. <strong>Attempt 1 wouldn&#39;t boot</strong> :Available KV cache memory: -0.78 GiB. In bf16 on the 24 GB L4, the 14.4 GiB model plus the CUDA-graph capture for 512 sequences plus the inflated prefill buffer drove the KV budget <em>negative</em>. I dropped max_num_seqs to 256, which booted, but now it was identical to prefix, and sure enough it measured <strong>identical to </strong><strong>prefix</strong> at every level. The bigger prefill budget did nothing because our prompts are only ~2k tokens and already fit the default. A clean &quot;tested it, doesn&#39;t matter for us.&quot; (Note this OOM comes back in Step 8.)</p><h4>Step 8 — offline, the workhorse, run a different way</h4><p>Offline is different way of running the model entirely (no HTTP server; hand it the whole list at once). So it took a different pipeline:</p><ol><li><strong>Export the work with zero GPU cost.</strong> I added a collector mode that writes every prompt the sweep <em>would</em> have sent into a JSONL file (offline_prompts.jsonl) : 404 requests (101 tickers × 4 personas), identical prompts, pure DB read, no model calls.</li><li><strong>First run OOM’d — and it was the <em>same</em> mistake as Step 7.</strong> The offline runner’s engine config had inherited the tuned knobs (max_num_seqs=512), so it blew the KV budget on the 24 GB L4 exactly like tuned attempt 1. Dropped it to 256 (matching prefix), and it booted.</li><li><strong>Ran all 404.</strong> Process time 236s, <strong>979 tok/s on a single GPU</strong>, and a JSON-parse success rate of <strong>99.8%, </strong>which mattered because offline doesn’t <em>enforce</em> JSON the way the HTTP server does, and I’d promised to measure rather than assume that asymmetry was harmless.</li></ol><p>The comparison required care, and I built a report mode to enforce it: compare offline to the phase that ran the <strong>same engine</strong> (this run was bf16, so it compares to prefix, not quant), and normalize for GPU count. On that fair footing, offline batch was <strong>+33% per GPU and ~30% cheaper per ticker</strong> than the HTTP server — purely from deleting the HTTP/client overhead and keeping the batch full from t=0.</p><h4>Step 9 — The quality A/B, because “faster” isn’t “safe”</h4><p>Every number so far said FP8 is faster. None of them said it’s still <em>correct</em>. For a finance app that cites revenue and EPS, that’s the question that actually gates shipping. So the final step was a blind, rubric-scored A/B of FP8 vs bf16: run once at short prompts, then <strong>again at full 14-table production-length prompts</strong> to stress the theoretical long-context FP8 risk. (The data prep itself was a mini-project: only one ticker had all 14 tables loaded, so I refreshed and enriched seven more to match before the test was even possible.) Both lengths came back <strong>parity, zero numeric drift</strong> (look at Part 10 to learn more)</p><h4>The shape of the whole thing</h4><p>If you zoom out, almost every step had the same structure: <em>plan a clean experiment → hit a wall that the 24 GB GPU or a default setting or the cloud put in front of me → diagnose it with logs or the </em><em>/metrics sampler → fix the smallest thing → re-run → record the honest number, caveats and all.</em> The wins (batching, FP8) were where the theory said they&#39;d be. The <em>learning</em> was almost entirely in the failures: the secret-caching floor, the admission-gate wall, the backfired reorder, the two identical OOMs. That&#39;s why the run log is written as a diary and not a results-only summary: the mistakes are the part worth keeping.</p><h3>Part 9 — Results</h3><p>Here’s the master table (peak/representative numbers; tok/s and p95 at C=128 for the HTTP phases; offline is process-basis).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZugtyrwOib8Rjr-imWHGHw.png" /><figcaption>Comparison</figcaption></figure><h4>Why I rank per-GPU, not raw tok/s</h4><p>Here’s a subtlety that’s easy to get wrong, and it’s the one I most wanted to get right. <strong>Raw peak tok/s mixes GPU counts.</strong> The scale-out phases (best, prefix, quant, etc.) used <strong>2 GPUs</strong>; baseline, offline, and worst used <strong>1</strong>. If you rank by raw throughput, you&#39;re partly ranking &quot;who got to use more hardware,&quot; which tells you what the <em>deployed service delivers</em> but not which <em>technique</em> is actually more efficient.</p><p>The fair way to judge a technique is <strong>throughput per GPU</strong> (and ultimately <strong>$/ticker</strong>). So:</p><p><strong>Ranked by throughput PER GPU (the honest efficiency ranking):</strong></p><ol><li>quant — <strong>~1320 tok/s/GPU</strong> 🏆</li><li>fp8-ckpt — ~1269/GPU</li><li><strong>offline — 979/GPU</strong> ← this is the headline. It&#39;s <em>bf16 on a single GPU</em> and it still beats every 2-GPU bf16 HTTP config below it.</li><li>best — ~774/GPU</li><li>prefix — ~737/GPU</li><li>tuned — ~732/GPU</li><li>baseline — 441/GPU</li><li>worst — 19/GPU</li></ol><p><strong>Ranked by cost ($/ticker, steady-state — the production metric):</strong></p><ol><li>quant / fp8-ckpt — <strong>$0.0004</strong> (tie) 🏆</li><li>offline — $0.0005 (bf16; would be cheapest of all with FP8 added)</li><li>best / prefix / tuned — $0.0007</li><li>baseline — $0.0022</li><li>worst — couldn&#39;t even complete a full sweep</li></ol><blockquote><strong><em>How </em></strong><strong><em>$/ticker is computed</em></strong><em> (so you can sanity-check the table). There&#39;s no hidden accounting — it&#39;s GPU rent divided by work done:</em></blockquote><blockquote><em>$/ticker = (GPUs × seconds × $-per-GPU-second) ÷ tickers_processed</em></blockquote><blockquote><em>Modal bills by the GPU-second; we use its published </em><strong><em>L4 on-demand rate ≈ $0.80/hr</em></strong><em> (= $0.000222/sec). For the HTTP phases we derive </em><em>seconds ÷ ticker from each phase&#39;s measured </em>tickers/min<em>, multiply by the GPU count it ran on (scale-out = 2), and apply the rate. For offline it&#39;s literally </em><em>(1 GPU × process_seconds × rate) ÷ 101 tickers. Worked example for </em><em>quant: at its peak it cleared ~66 tickers/min on 2 GPUs → 2 GPUs × (60/66 s) × $0.000222 ≈ </em><strong><em>$0.0004/ticker</em></strong><em>. The code that does this is </em><em>offline_report() in benchmark_enrich.py:395. All figures are </em><strong><em>steady-state</em></strong><em> (cold start excluded) unless labelled &quot;wall.&quot;</em></blockquote><p>The story end-to-end, per GPU so it’s fair across replica counts: <strong>~19 tok/s serialized → ~115 with batching (~6×) → per-GPU tuning lifts the ceiling → FP8 multiplies everything ~1.5–1.7× → switching HTTP to offline adds another ~1.3× in efficiency.</strong> Stacked, FP8 + batching + offline is roughly <strong>two orders of magnitude</strong> over the serialized floor on the <em>same hardware</em>.</p><h4>Per-stage: what worked, what didn’t, and why</h4><p><strong>worst (the floor, 19 tok/s flat).</strong> With max_num_seqs=1, one sequence goes through the GPU at a time. A single sequence&#39;s matmuls can&#39;t fill the L4&#39;s compute units, so the GPU mostly idles and the run is pure latency — throughput is <em>independent of load</em> (everything queues) and p95 grows linearly (47s → 377s from C=1 → C=8). A subtle finding: worst at C=1 (19.1 tok/s) is actually a <em>hair faster</em> than baseline at C=1 (16.8), because with a single request the batching scheduler is pure overhead with nothing to batch. <strong>Batching only pays once there&#39;s contention.</strong></p><p><strong>baseline (batching, ~6× at moderate load).</strong> Raising the caps lets vLLM run many sequences in one forward pass; the GPU&#39;s compute units fill and throughput finally scales with concurrency (C=8: 19 → 115 tok/s). This is the <strong>single biggest structural jump</strong> and it&#39;s exactly where the theory said it&#39;d be.</p><p><strong>best (per-GPU tuning + scale-out, 3.5× baseline).</strong> Three independent wins stacked: CUDA graphs (why even C=1 improved 1.5× — launch overhead dominates at low batch), async scheduling, and scaling to a 2nd replica (which broke baseline&#39;s wall — throughput kept <em>climbing</em> C=64 → C=128, 1046 → 1548). Note the 1548 peak is <strong>budget-limited</strong>: I capped max_containers at 2 to control cost. With 5 replicas it&#39;d be ~2.5× higher. Scale-out raises throughput linearly with GPUs — the headroom is &quot;however many GPUs you&#39;ll pay for.&quot;</p><p><strong>prefix (prefix caching, big at light load, fades under load).</strong> It holds a steady ~50–60% hit rate and gives a nice <strong>+25% at C=8</strong>, shrinking to roughly a tie at C=128. Why it fades: prefix caching only saves <strong>prefill</strong> compute; as concurrency rises and the batch fills with <strong>decode</strong> work, prefill is a smaller slice of GPU time, so the relative payoff shrinks. But it&#39;s <em>strictly nice</em> otherwise — better p95, fewer fails, lower KV at every level — so we keep it on. (Why only ~50–60% and not 90%+? The shared prefix is ~620 tokens of a ~2k-token prompt; the per-ticker data is the rest, and it&#39;s unique. That&#39;s structural, near the ceiling for this prompt shape.)</p><h4>The other surprise: baseline’s real bottleneck</h4><p>baseline plateaued at 472 tok/s and then <em>regressed</em> at C=128 (441 tok/s, p95 doubling to 246s). My first guess was the vLLM batch size (max_num_seqs=64). The /metrics sampler proved me wrong: concurrent running capped at <strong>~31</strong> and KV peaked at only <strong>~25%</strong>. So vLLM never saw more than ~32 sequences — the real wall was <strong>max_inputs=32, Modal&#39;s per-container <em>admission</em> gate</strong>, sitting <em>in front of</em> the engine, not the engine&#39;s batch. With one container, request #33 just queues at Modal. And KV at 25% means memory wasn&#39;t remotely the limit — the throughput left on the table was pure admission-cap. That&#39;s <em>exactly</em> what best fixed by scaling out (2 replicas × 32 = more admission). Without the in-engine sampler I&#39;d have &quot;fixed&quot; the wrong knob.</p><h3>Part 10 — Did quantization hurt quality? (The question everyone asks about FP8)</h3><p>FP8 was the throughput winner, but compression always raises the eyebrow: <strong>did we trade speed for dumber answers?</strong> This is the classic FP8 worry. Lower-precision numbers, especially in a long KV cache, can in theory corrupt the model’s recall of figures. For a <em>finance</em> app where the model is citing revenue and EPS, getting a number wrong is a serious failure. So I tested it properly.</p><p><strong>Method:</strong> run the offline runner twice over the <strong>same 32 prompts</strong> (8 tickers × 4 personas) , once <strong>bf16</strong>, once <strong>FP8, t</strong>hen have a <strong>blind LLM judge</strong> (Claude) score each pair. Outputs shown as “Output A / Output B” in randomized order, engine identity hidden in a side key, un-blinded only after scoring. Graded on a 7-criterion weighted rubric: faithfulness (30%), hallucination control (20%), coherence (15%), fluency (10%), schema validity (10%), persona fit (10%), completeness (5%).</p><p><strong>Verdict: statistical parity. No degradation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1kwZ3DOC2Q60d-2W00tMA.png" /></figure><p>I ran it at <strong>two prompt lengths</strong> specifically to stress the theoretical risk. The 14-table test uses the full production data (~10× more numbers per prompt — price, 5-year growth, segment revenues, 8 quarters of earnings, Altman Z, transcript figures) and therefore a ~5× larger KV cache, which is exactly where FP8 KV degradation <em>should</em> show up if it’s going to.</p><p>It didn’t. <strong>FP8 reproduced every in-data figure correctly at both lengths.</strong> The theoretical long-context FP8 problem simply did not appear for this model and workload. If anything FP8 edged ahead, and every one of the small structural JSON defects came from <em>bf16</em> (small-sample noise, not a real bf16 weakness). The FP8-attention boot warning about “uncalibrated scales” produced no visible errors.</p><p><strong>Bottom line: FP8 is safe to ship.</strong> ~1.5–1.7× throughput, half the KV memory, and no measurable quality cost.</p><h3>Part 11 — What this means and how we apply it</h3><p>The point of all this wasn’t a leaderboard — it was to pick what we actually run. Here’s the takeaway, and it’s refreshingly simple after all the measuring:</p><p><strong>For live / on-demand serving → </strong><strong>quant (FP8).</strong> Best throughput, lowest p95 at every load, valid output, half the memory. The only cost is a longer cold start (~427s of on-the-fly weight quantization), which only matters on the first request after idle.</p><p><strong>For the nightly bulk enrichment → offline + FP8.</strong> → Offline batch removes the entire HTTP path (the OpenAI client, the network, Modal’s poll round-trips — the client itself was a bottleneck at high concurrency) and hands vLLM a <em>full queue from t=0</em>, so continuous batching runs maximally full the whole time. That alone bought <strong>+33% per GPU and ~30% cheaper per ticker</strong> vs the HTTP server <em>at the same engine</em> — and FP8 stacks on top of it orthogonally. We ran offline as bf16 for a clean comparison; <strong>offline + FP8 is the untested cheapest config and the obvious next experiment.</strong></p><p><strong>What to drop:</strong> fp8-ckpt (no benefit on a model this small — on-the-fly quantization is already cheap, and the checkpoint download offsets any saving), tuned (no benefit — our ~2k -token prompts already fit the default prefill budget, so the bigger budget is headroom we never use), and no-flashinfer (FlashInfer can&#39;t run on Gemma-4 at all).</p><h3>Conclusion</h3><p>The headline: <strong>we went from ~19 tok/s serialized to ~2,640 tok/s served (and ~$0.0022 → ~$0.0004 per ticker), proved every step with the real enrichment code, and confirmed FP8 costs us nothing in answer quality.</strong> Since this app doesn’t need immediate responses (it’s not a user-facing chatbot), we can use offline mode combined with FP8 quantization to get quality results at a fraction of the cost and speed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07e8de3240d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Multi-Agent AI Trading System: Technical Deep Dive into Architecture]]></title>
            <link>https://medium.com/@ishveen/building-a-multi-agent-ai-trading-system-technical-deep-dive-into-architecture-b5ba216e70f3?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ba216e70f3</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[langchain-deepagents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Tue, 30 Dec 2025 00:51:46 GMT</pubDate>
            <atom:updated>2025-12-30T00:51:46.429Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction: From Concepts to Code</h3><p>NOTE: Please review these posts on LinkedIn before you start technical deep dive and as its gives you conceptual understanding of this project.</p><ul><li><a href="https://www.linkedin.com/pulse/building-production-ready-agentic-ai-system-langchain-kharbanda-myfme/">Building Production-Ready Agentic AI System with Langchain</a></li><li><a href="https://www.linkedin.com/pulse/beyond-agent-product-vs-platform-thinking-ai-kharbanda-zf0re">Beyond the Agent: Product vs. Platform Thinking in AI Architecture</a></li><li><a href="https://www.linkedin.com/pulse/ai-multi-agent-architectural-patterns-langchain-v10-from-kharbanda-lggte">AI Multi-Agent Architectural Patterns with LangChain V1.0: From Router to Subagents</a></li></ul><p><strong>Prerequisites:</strong> Familiarity with Python, async programming, LangChain basics, and LangGraph concepts. This is advanced material for developers building production multi-agent systems.</p><p><strong>Architecture Overview: Three Agent Patterns, One System</strong></p><p>This app combines three distinct agent implementation patterns, each chosen for its specific strengths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_tjaxH_PTi0fRaODtrEbIg.png" /><figcaption>MultiAgent Architecture Pattern with Agent &amp; Subagents as Tools</figcaption></figure><p><strong>Main Agent (Autonomous):</strong> Built with LangChain’s create_agent, this is a fully autonomous agent that sees all available tools and decides what to invoke. It handles tool composition, parallel execution, sequential reasoning, and result synthesis. It acts as an orchestrator.</p><p><strong>Debate Workflow (LangGraph Custom):</strong> A deterministic multi-step workflow for investment analysis. Bull agent presents optimistic case, Bear agent presents pessimistic case, Judge synthesizes into recommendation, and Human approves trades. Built with LangGraph’s StateGraph for explicit control over flow.</p><p><strong>Deep Agent (deepagents Library):</strong> A research coordinator that spawns four specialized analyst subagents in parallel: fundamental, technical, sentiment, macro. Uses the deepagents library for file system tools, planning, and subagent coordination.</p><p><strong>Key Architectural Decision:</strong> We use different patterns where each excels. Autonomous agent for flexible orchestration and simple questions. Custom workflow for deterministic investment logic. Deep analysis for research coordination. The workflows become tools the main agent invokes.</p><p>Let’s see how each is built.</p><h4><strong>Main Agent</strong>: Autonomous Orchestration with create_agent</h4><p><em>learn more about it in their documentation: </em><a href="https://docs.langchain.com/oss/python/langchain/agents"><em>https://docs.langchain.com/oss/python/langchain/agents</em></a></p><p>The main agent is built using LangChain’s create_agent function. This creates a fully autonomous agent that can reason about which tools to use, compose multiple tools in one turn, and maintain conversation context.</p><pre># Conceptual structure - actual implementation simplified<br>def create_main_agent(model, tools, middleware, persistence):<br>    &quot;&quot;&quot;<br>    Create autonomous agent with:<br>    - Model: LLM for reasoning<br>    - Tools: All 46 tools (direct, MCP, workflow wrappers)<br>    - Middleware: Cross-cutting concerns (prompts, guardrails, HITL)<br>    - Persistence: PostgreSQL for state + memory<br>    &quot;&quot;&quot;<br>    <br>    agent = create_agent(<br>        model=model,<br>        tools=all_tools,              # Agent sees everything<br>        middleware=middleware_stack,   # Intercept at key points<br>        checkpointer=postgres_saver,   # Conversation state<br>        store=postgres_store,          # Cross-conversation memory<br>        context_schema=UserContext,    # Runtime config type<br>        state_schema=AgentState,       # Conversation state type<br>    )<br>    <br>    return compiled_agent_graph</pre><p><strong>What this gives us →</strong> An agent that autonomously decides which tools to invoke based on the user’s question. It can call multiple tools in one turn, use output from one tool as input to another, and synthesize results into coherent responses.</p><h3>Context Schema: Runtime Configuration</h3><p>The context schema defines what configuration the agent receives on every invocation. This isn’t part of conversation state, but it has runtime parameters that can change per request.</p><pre># Context: Runtime configuration passed on each invocation<br>class UserContext(TypedDict):<br>    user_id: str<br>    trading_level: str  # beginner | intermediate | advanced<br>    max_position_size: int<br>    risk_tolerance: str  # conservative | moderate | aggressive<br>    can_place_orders: bool</pre><p><strong>Why TypedDict →</strong> LangChain’s create_agent requires TypedDict for context schema. This provides type hints while remaining compatible with LangChain&#39;s runtime injection system.</p><p><strong>How it’s used →</strong> Middleware accesses runtime.context to adjust behavior. Tools receive context via ToolRuntime[UserContext] parameter. This separates conversation state (what was said) from runtime config (who&#39;s saying it and their preferences).</p><h3>State Schema: Conversation Memory</h3><p>The state schema defines what’s stored in conversation history between turns.</p><pre># State: What&#39;s saved between conversation turns<br>class AgentState(TypedDict):<br>    messages: List[BaseMessage]  # Conversation history<br>    user_id: str                 # For memory lookup</pre><p><strong>Design choice →</strong> Keep state minimal. Messages contain the conversation. User ID enables memory lookup. Everything else is runtime context or derived data.</p><p><strong>How messages work →</strong> The add_messages annotation tells LangGraph to append new messages rather than replace. This builds conversation history naturally.</p><p>Invocation Pattern</p><pre># How the agent is invoked<br>async def handle_request(user_message, user_id, thread_id):<br>    agent = get_main_agent()  # Pre-compiled at startup<br>    <br>    # Input: Current state<br>    input_state = {<br>        &quot;messages&quot;: [UserMessage(content=user_message)],<br>        &quot;user_id&quot;: user_id<br>    }<br>    <br>    # Config: Runtime parameters + persistence<br>    config = {<br>        &quot;configurable&quot;: {<br>            &quot;thread_id&quot;: thread_id,  # Conversation identifier<br>        },<br>        &quot;context&quot;: {  # Runtime configuration<br>            &quot;user_id&quot;: user_id,<br>            &quot;trading_level&quot;: &quot;intermediate&quot;,<br>            &quot;max_position_size&quot;: 100,<br>            &quot;risk_tolerance&quot;: &quot;moderate&quot;<br>        }<br>    }<br>    <br>    # Agent decides what tools to call, composes results<br>    result = await agent.ainvoke(input_state, config)<br>    <br>    return result[&quot;messages&quot;][-1].content</pre><p><strong>Key points →</strong> The agent is pre-compiled once at startup (covered later). Input provides current state. Config provides thread ID for persistence and context for runtime behavior. Agent autonomously handles everything else (Thanks LangChain!)</p><h3>Debate Workflow: LangGraph Custom Pattern</h3><p>The Debate workflow is a custom LangGraph implementation. It’s deterministic → the Bull always runs before Bear, Judge always synthesizes, Human always approves trades. This gives us explicit control over the investment analysis process.</p><h3>Workflow Structure</h3><pre># Conceptual structure of debate workflow<br>def create_debate_workflow(model, tools):<br>    workflow = StateGraph(state_schema=DebateState)<br>    <br>    # Create specialized nodes<br>    bull_agent = create_agent_node(model, BULL_PERSONA)<br>    bear_agent = create_agent_node(model, BEAR_PERSONA)<br>    judge = create_judge_node(model)<br>    approval = create_human_approval_node()<br>    <br>    # Add nodes to graph<br>    workflow.add_node(&quot;bull&quot;, bull_agent)<br>    workflow.add_node(&quot;bear&quot;, bear_agent)<br>    workflow.add_node(&quot;judge&quot;, judge)<br>    workflow.add_node(&quot;approval&quot;, approval)<br>    <br>    # Define deterministic flow<br>    workflow.set_entry_point(&quot;bull&quot;)<br>    workflow.add_edge(&quot;bull&quot; → &quot;bear&quot;)<br>    workflow.add_edge(&quot;bear&quot; → &quot;judge&quot;)<br>    workflow.add_edge(&quot;judge&quot; → &quot;approval&quot;)<br>    workflow.add_edge(&quot;approval&quot; → END)<br>    <br>    # Compile with persistence<br>    return workflow.compile(<br>        checkpointer=postgres_checkpointer,<br>        store=postgres_store<br>    )</pre><p><strong>Why deterministic flow →</strong> Investment analysis benefits from <strong><em>structured reasoning</em></strong>. Bull presents optimistic case with supporting evidence. Bear presents risks and concerns. Judge weighs both sides with user’s risk tolerance. Human reviews if orders are involved. This structure ensures complete analysis.</p><h3>State Schema: Rich Domain Model</h3><p>Unlike the main agent’s minimal state, the debate workflow has rich a domain-specific state.</p><pre># Debate state: Investment analysis domain model<br>class DebateState(TypedDict):<br>    topic: str                    # What we&#39;re analyzing<br>    messages: List[BaseMessage]   # Agent exchanges<br>    current_ticker: str           # Stock symbol<br>    <br>    # Analysis outputs<br>    bull_score: float             # 0-10 optimism score<br>    bear_score: float             # 0-10 pessimism score<br>    <br>    # Decision outputs<br>    recommendation: TradeRecommendation  # Structured decision<br>    final_report: str             # Formatted report for user<br>    <br>    # Human-in-loop fields<br>    needs_approval: bool<br>    approved: bool<br>    pending_order: dict</pre><p><strong>Why rich state → </strong>Each node updates specific fields. Bull updates bull_score and adds messages. Bear updates bear_score and adds messages. Judge synthesizes into recommendation and final_report. State flows through nodes, accumulating analysis.</p><h3>Node Implementation Pattern</h3><p>Each node is an async function that receives state and config, performs its logic, and returns state updates.</p><pre># Conceptual node implementation<br>async def bull_agent_node(state: DebateState, config):<br>    &quot;&quot;&quot;Bull agent: Present optimistic investment case.&quot;&quot;&quot;<br>    <br>    # Extract inputs from state<br>    ticker = state[&quot;current_ticker&quot;]<br>    messages = state[&quot;messages&quot;]<br>    <br>    # Extract context from config<br>    trading_level = config[&quot;configurable&quot;][&quot;trading_level&quot;]<br>    <br>    # Build prompt with persona and context<br>    prompt = f&quot;&quot;&quot;<br>    {BULL_PERSONA}<br>    <br>    Analyze {ticker} for {trading_level} investor.<br>    Find growth catalysts, competitive advantages, bullish signals.<br>    Provide score 0-10 and supporting evidence.<br>    &quot;&quot;&quot;<br>    <br>    # Invoke model with tools<br>    response = await model_with_tools.ainvoke([prompt] + messages)<br>    <br>    # Extract score from response<br>    bull_score = extract_score(response.content)<br>    <br>    # Return state updates<br>    return {<br>        &quot;messages&quot;: [response],  # Append to conversation<br>        &quot;bull_score&quot;: bull_score  # Update analysis<br>    }</pre><p><strong>Pattern → </strong>Node receives state and config → Builds persona-specific prompt → Invokes model with tools → Extracts structured information → Returns state updates → Next node receives updated state.</p><p><strong>Why this works →</strong> Each agent has clear role (bull/bear), access to tools, and focused task. Judge synthesizes their findings with user’s risk parameters. Flow is explicit and debuggable.</p><h3>Structured Output</h3><p>The judge node produces structured recommendations using Pydantic schemas.</p><pre># Structured recommendation schema<br>class TradeRecommendation(BaseModel):<br>    symbol: str<br>    action: str  # &quot;buy&quot; | &quot;hold&quot; | &quot;pass&quot;<br>    quantity: Optional[int]<br>    confidence: float  # 0.0 to 1.0<br>    bull_score: float<br>    bear_score: float<br>    reasoning: str<br>    risk_assessment: str  # &quot;low&quot; | &quot;medium&quot; | &quot;high&quot;</pre><pre># Judge uses structured output<br>async def judge_node(state: DebateState, config):<br>    # Extract bull and bear cases from messages<br>    bull_case = extract_bull_case(state[&quot;messages&quot;])<br>    bear_case = extract_bear_case(state[&quot;messages&quot;])<br>    <br>    # Get user risk parameters from config<br>    risk_tolerance = config[&quot;configurable&quot;][&quot;risk_tolerance&quot;]<br>    max_position = config[&quot;configurable&quot;][&quot;max_position_size&quot;]<br>    <br>    # Build synthesis prompt<br>    prompt = f&quot;&quot;&quot;<br>    Synthesize investment decision:<br>    <br>    Bull Case: {bull_case}<br>    Bear Case: {bear_case}<br>    Risk Tolerance: {risk_tolerance}<br>    Max Position: {max_position} shares<br>    <br>    Provide structured recommendation.<br>    &quot;&quot;&quot;<br>    <br>    # Use structured output<br>    structured_model = model.with_structured_output(TradeRecommendation)<br>    recommendation = await structured_model.ainvoke(prompt)<br>    <br>    # Calculate position size if buying<br>    if recommendation.action == &quot;buy&quot;:<br>        recommendation.quantity = calculate_quantity(<br>            bull_score=recommendation.bull_score,<br>            bear_score=recommendation.bear_score,<br>            confidence=recommendation.confidence,<br>            max_position=max_position<br>        )<br>    <br>    # Format for user<br>    final_report = format_report(recommendation, config)<br>    <br>    return {<br>        &quot;recommendation&quot;: recommendation,<br>        &quot;final_report&quot;: final_report<br>    }</pre><p><strong>Why structured output → </strong>Type safety, validation, downstream processing. The recommendation object can be logged, analyzed, or passed to other systems. The formatted report gives users readable analysis: both from one LLM call.</p><h3>Context Passing Pattern</h3><p><strong>Critical detail → </strong>Debate workflow expects context values flattened in config[&quot;configurable&quot;], not as a nested object.</p><pre># When invoking debate workflow from tool wrapper<br>config = {<br>    &quot;configurable&quot;: {<br>        &quot;thread_id&quot;: f&quot;debate_{user_id}_{ticker}&quot;,<br>        &quot;trading_level&quot;: trading_level,       # Flattened<br>        &quot;max_position_size&quot;: max_position_size,  # Flattened<br>        &quot;risk_tolerance&quot;: risk_tolerance,     # Flattened<br>    }<br>}<br><br># Nodes access via config parameter<br>async def some_node(state, config):<br>    trading_level = config[&quot;configurable&quot;][&quot;trading_level&quot;]<br>    # Use directly</pre><p><strong>Why this pattern:</strong> Simple, direct access. No runtime injection needed. Works reliably with pre-compiled graphs. Matches LangGraph’s config semantics.</p><h3>Deep Agent: deepagents Library Integration</h3><p>The Deep agent uses the deepagents library, which provides built-in planning, file system tools, and subagent coordination. This is different from our custom LangGraph workflow → it’s a pre-built framework for research coordination.</p><h3>Library Integration Pattern</h3><pre># Using deepagents library<br>from deepagents import create_deep_agent<br><br>def create_deep_workflow(model, tools):<br>    &quot;&quot;&quot;Create deep analysis workflow using deepagents library.&quot;&quot;&quot;<br>    <br>    # Define analyst subagents<br>    analyst_subagents = [<br>        {<br>            &quot;name&quot;: &quot;fundamental-analyst&quot;,<br>            &quot;description&quot;: &quot;Financial analysis expert&quot;,<br>            &quot;system_prompt&quot;: FUNDAMENTAL_PROMPT,<br>            &quot;tools&quot;: tools<br>        },<br>        {<br>            &quot;name&quot;: &quot;technical-analyst&quot;,<br>            &quot;description&quot;: &quot;Price action expert&quot;,<br>            &quot;system_prompt&quot;: TECHNICAL_PROMPT,<br>            &quot;tools&quot;: tools<br>        },<br>        {<br>            &quot;name&quot;: &quot;sentiment-analyst&quot;,<br>            &quot;description&quot;: &quot;News and social sentiment expert&quot;,<br>            &quot;system_prompt&quot;: SENTIMENT_PROMPT,<br>            &quot;tools&quot;: tools<br>        },<br>        {<br>            &quot;name&quot;: &quot;macro-analyst&quot;,<br>            &quot;description&quot;: &quot;Economic context expert&quot;,<br>            &quot;system_prompt&quot;: MACRO_PROMPT,<br>            &quot;tools&quot;: tools<br>        }<br>    ]<br>    <br>    # Create deep agent using library<br>    deep_graph = create_deep_agent(<br>        model=model,<br>        system_prompt=COORDINATOR_PROMPT,<br>        subagents=analyst_subagents,<br>        tools=tools,<br>        store=postgres_store<br>    )<br>    <br>    return deep_graph</pre><p><strong>What deepagents provides → </strong>Planning tools (write_todos, check progress), file system tools (read, write, edit files), subagent spawning (where analysts are run in parallel), synthesis (combine findings). We don’t have to make these on our own because the library already handles this!</p><p><strong>Why use a library → </strong>Deep analysis is a solved pattern. It spawns specialists, let them research independently, and synthesized findings. deepagents implements this well with no need to rebuild</p><h3>Subagent Definitions</h3><p>Each analyst is defined with name, description, system prompt, and tools.</p><pre># Example: Fundamental analyst definition<br>FUNDAMENTAL_ANALYST = {<br>    &quot;name&quot;: &quot;fundamental-analyst&quot;,<br>    &quot;description&quot;: &quot;Expert in financial analysis conducting valuation and growth assessment&quot;,<br>    &quot;system_prompt&quot;: &quot;&quot;&quot;<br>    You are a FUNDAMENTAL ANALYST.<br>    <br>    Focus: Financial health, competitive position, valuation, growth prospects<br>    <br>    Workflow:<br>    1. Gather financial data using available tools<br>    2. Analyze competitive moat and market position<br>    3. Assess valuation relative to growth<br>    4. Identify key risks and catalysts<br>    <br>    Output:<br>    - FUNDAMENTAL SCORE: X/10<br>    - Financial Health: [assessment]<br>    - Competitive Position: [assessment]<br>    - Valuation: [cheap/fair/expensive]<br>    - Growth Outlook: [assessment]<br>    - Key Risks: [list]<br>    - Key Catalysts: [list]<br>    &quot;&quot;&quot;,<br>    &quot;tools&quot;: all_available_tools<br>}</pre><p><strong>Pattern:</strong> Each analyst has focused a role, specific evaluation criteria, and clear output format. The coordinator delegates tasks, the analysts execute independently, and the coordinator synthesizes.</p><h3>Coordinator System Prompt</h3><p>The main deep agent coordinates the analysts.</p><pre>COORDINATOR_PROMPT = &quot;&quot;&quot;<br>You are an INVESTMENT RESEARCH COORDINATOR.<br><br>Your capabilities:<br>1. Planning (write_todos): Break analysis into tasks<br>2. File System: Save research to /research/{ticker}/<br>3. Subagent Delegation: task analysts with specific queries<br>4. Synthesis: Combine findings into comprehensive report<br><br>Available analysts:<br>- fundamental-analyst: Financial and competitive analysis<br>- technical-analyst: Price action and momentum analysis  <br>- sentiment-analyst: News and social sentiment analysis<br>- macro-analyst: Economic and sector context analysis<br><br>Process:<br>1. Create analysis plan<br>2. Delegate to specialist analysts (can run in parallel)<br>3. Review their outputs<br>4. Synthesize comprehensive investment report<br><br>Output format:<br>- Executive Summary<br>- Multi-dimensional Scores (fundamental, technical, sentiment, macro)<br>- Recommendation (BUY/HOLD/PASS with quantity)<br>- Confidence and Risk Level<br>- Time Horizons (short/medium/long-term)<br>- Detailed Analysis<br>&quot;&quot;&quot;</pre><p><strong>How it works → </strong>Coordinator receives query, plans analysis tasks, and spawns analyst subagents with specific questions. Each analyst uses tools to research their domain. The coordinator reviews outputs and synthesizes final report. This is all orchestrated by deepagents library!</p><h3>Context Passing Pattern</h3><p><strong>Critical detail → </strong>Deep agent expects context embedded in the message text, not in config.</p><pre># When invoking deep workflow from tool wrapper<br>enhanced_message = f&quot;&quot;&quot;<br>Perform comprehensive analysis on {ticker}.<br><br>Context:<br>- Stock: {ticker}<br>- Trading Level: {trading_level}<br>- Max Position: {max_position_size} shares<br>- Risk Tolerance: {risk_tolerance}<br><br>Provide comprehensive investment analysis.<br>&quot;&quot;&quot;<br><br>initial_state = {<br>    &quot;messages&quot;: [HumanMessage(content=enhanced_message)]<br>}<br><br>config = {<br>    &quot;configurable&quot;: {<br>        &quot;thread_id&quot;: f&quot;deep_{user_id}_{ticker}&quot;,<br>        &quot;user_id&quot;: user_id  # Just primitives<br>    }<br>}<br><br>result = await deep_graph.astream(initial_state, config)</pre><p><strong>Why this pattern → </strong>the deepagents library doesn’t use LangChain’s runtime context injection. The context must be in the message where the agent can see it. This is library-specific → we adapt our tool wrapper to match the library’s expectations.</p><h3>Tools Architecture: MCP Servers and Direct Database Access</h3><p>The main agent has access to <strong>46</strong> tools across four categories: MCP server tools, direct database tools, memory tools, and workflow wrappers. This section covers the first three (workflow wrappers in next section).</p><h3>Tool Composition Strategy</h3><pre># High-level tool organization<br>SYNC_TOOLS = [<br>    get_stock_price,          # Database: Historical prices<br>    get_stock_news,           # Database: Keyword search<br>    search_past_conversations  # Memory: LangGraph store<br>]<br><br># MCP tools loaded async at startup<br>alpaca_tools = await load_alpaca_mcp()      # 38 tools<br>trading_tools = await load_trading_mcp()    # 1 tool  <br>vector_tools = await load_vector_mcp()      # 2 tools<br><br># Workflow wrappers<br>workflow_tools = [debate_analysis, deep_analysis]<br><br># Combined for main agent<br>all_tools = SYNC_TOOLS + alpaca_tools + trading_tools + vector_tools + workflow_tools<br># Total: ~46 tools</pre><p><strong>Design strategy → </strong>Mix direct database tools (fast, exact matches), MCP servers (extensible, standardized), memory tools (cross-conversation), and workflow wrappers (complex analysis). Each tool type serves different needs.</p><h3>MCP Architecture</h3><p>Model Context Protocol (MCP) is a standard for exposing tools as servers. We use three MCP servers: vendor server (Alpaca) and two custom servers (Trading, Vector Search).</p><p><strong>Why MCP:</strong> Standardized protocol, server-side tool execution, easy to add new capabilities, language-agnostic, and it can be maintained by different teams.</p><h4>MCP Server 1: Alpaca (Vendor)</h4><p>Alpaca provides official MCP server for market data. We deploy their server as-is.</p><pre># Deployment: Use Alpaca&#39;s official MCP server<br># Docker: node:18 with @modelcontextprotocol/server-alpaca<br># Endpoint: http://alpaca-mcp-server:8000/mcp<br># Transport: HTTP with JSON-RPC protocol<br><br># Client integration<br>ALPACA_CONFIG = {<br>    &quot;transport&quot;: &quot;http&quot;,<br>    &quot;url&quot;: &quot;http://alpaca-mcp-server:8000/mcp&quot;<br>}<br><br>alpaca_client = MultiServerMCPClient(ALPACA_CONFIG)<br>tools = await alpaca_client.get_tools()<br><br># Returns ~44 tools, we filter to 38 (remove option tools with null schemas)<br>safe_tools = filter_tools(tools, exclude=OPTIONS_TOOLS)</pre><p><strong>What it provides →</strong> Account info, positions, portfolio history, watchlists, asset search, latest bars/quotes/trades, snapshots, corporate actions, and more. Real-time market data through standardized interface.</p><p><strong>Why filter →</strong> Some option-related tools have schema fields with type: null which Gemini rejects. We filter these client-side for compatibility.</p><h4>MCP Server 2: Custom Trading</h4><p>For security-critical order placement, we built custom MCP server with explicit control.</p><pre># Server structure (FastAPI + MCP protocol)<br>class TradingMCPServer:<br>    &quot;&quot;&quot;Custom MCP server for trade execution.&quot;&quot;&quot;<br>    <br>    def handle_mcp_request(method, params):<br>        &quot;&quot;&quot;Process MCP JSON-RPC requests.&quot;&quot;&quot;<br>        if method == &quot;initialize&quot;:<br>            return protocol_version_and_capabilities()<br>        <br>        elif method == &quot;tools/list&quot;:<br>            return [place_trading_order_definition()]<br>        <br>        elif method == &quot;tools/call&quot;:<br>            tool = params[&quot;name&quot;]<br>            args = params[&quot;arguments&quot;]<br>            <br>            if tool == &quot;place_trading_order&quot;:<br>                # Validate inputs<br>                validate_order(args)<br>                <br>                # Execute via Alpaca<br>                order = submit_to_alpaca(args)<br>                <br>                # Return result<br>                return format_order_result(order)<br>    <br>    def place_trading_order(symbol, side, quantity):<br>        &quot;&quot;&quot;Execute market order with validation.&quot;&quot;&quot;<br>        # Input validation<br>        validate_symbol(symbol)<br>        validate_quantity(quantity)<br>        <br>        # Create Alpaca order request<br>        order_request = create_market_order(<br>            symbol=symbol,<br>            side=side,  # buy or sell<br>            quantity=quantity,<br>            time_in_force=&quot;day&quot;<br>        )<br>        <br>        # Submit to Alpaca<br>        order = alpaca_client.submit_order(order_request)<br>        <br>        return {<br>            &quot;status&quot;: &quot;success&quot;,<br>            &quot;order_id&quot;: order.id,<br>            &quot;symbol&quot;: order.symbol,<br>            &quot;quantity&quot;: order.qty,<br>            &quot;message&quot;: f&quot;Order placed successfully&quot;<br>        }</pre><p><strong>Why custom:</strong> Security, control, and logging. Order placement needs explicit validation, audit trails, rate limiting. Custom server gives us control over execution while exposing standardized MCP interface.</p><h4>MCP Server 3: Vector Search (RAG)</h4><p>For semantic search over news and Reddit, we built a vector search server using pgvector.</p><pre># Server structure (FastAPI + MCP + pgvector + Vertex AI)<br>class VectorSearchMCPServer:<br>    &quot;&quot;&quot;MCP server for semantic search using embeddings.&quot;&quot;&quot;<br>    <br>    def initialize():<br>        # Connect to Vertex AI for embeddings<br>        vertexai.init(project=GCP_PROJECT)<br>        embedding_model = TextEmbeddingModel(&quot;text-embedding-004&quot;)<br>        <br>        # Connect to PostgreSQL with pgvector<br>        db = connect_postgres_with_pgvector()<br>    <br>    def semantic_news_search(query, limit=5):<br>        &quot;&quot;&quot;Search news articles semantically.&quot;&quot;&quot;<br>        # Generate query embedding<br>        embedding = embedding_model.embed(query)<br>        <br>        # Vector similarity search (cosine distance)<br>        sql = &quot;&quot;&quot;<br>            SELECT headline, summary, source, url, symbols,<br>                   1 - (text_vector &lt;=&gt; %s) as similarity<br>            FROM news_articles<br>            WHERE text_vector IS NOT NULL<br>            ORDER BY text_vector &lt;=&gt; %s<br>            LIMIT %s<br>        &quot;&quot;&quot;<br>        <br>        results = db.query(sql, embedding, embedding, limit)<br>        <br>        return {<br>            &quot;query&quot;: query,<br>            &quot;results&quot;: results,<br>            &quot;count&quot;: len(results)<br>        }<br>    <br>    def semantic_reddit_search(query, limit=5):<br>        &quot;&quot;&quot;Search Reddit posts semantically.&quot;&quot;&quot;<br>        # Same pattern: embed query, vector search, return results<br>        embedding = embedding_model.embed(query)<br>        results = db.vector_search(&quot;social_media_posts&quot;, embedding, limit)<br>        return format_results(query, results)</pre><p><strong>Why semantic search → </strong>Keyword matching misses related content. “chip supply issues” should find articles about “semiconductor shortages” even without exact words. Embedding-based search finds semantic similarity (I also have a RAG series on embeddings and vector search if you want to learn more).</p><p><strong>Why MCP server → </strong>Encapsulates complexity (embeddings, vector search, caching). Makes semantic search available as standardized tool. Can be reused across agents.</p><h3>Direct Database Tools</h3><p>For historical data where exact matches work fine, we use direct PostgreSQL queries wrapped as LangChain tools.</p><pre># Pattern: Database query wrapped as tool<br>@tool<br>def get_stock_price(query: str) -&gt; str:<br>    &quot;&quot;&quot;Get historical stock prices from database.&quot;&quot;&quot;<br>    # Extract symbols from natural language<br>    symbols = extract_symbols_from_query(query)<br>    <br>    # Extract time period<br>    time_range = extract_time_period(query)<br>    <br>    # Query database<br>    results = {}<br>    for symbol in symbols:<br>        data = db.query(&quot;&quot;&quot;<br>            SELECT date, open, high, low, close, volume<br>            FROM stock_prices<br>            WHERE symbol = ? AND date BETWEEN ? AND ?<br>            ORDER BY date DESC<br>        &quot;&quot;&quot;, symbol, time_range.start, time_range.end)<br>        <br>        results[symbol] = data<br>    <br>    # Format for LLM<br>    return format_price_data(results)<br><br>@tool<br>def get_stock_news(query: str) -&gt; str:<br>    &quot;&quot;&quot;Get news articles from database (keyword search).&quot;&quot;&quot;<br>    # Extract topic<br>    topic = extract_topic_from_query(query)<br>    <br>    # Keyword search<br>    articles = db.query(&quot;&quot;&quot;<br>        SELECT headline, source, url, created_at<br>        FROM news_articles<br>        WHERE symbols ILIKE ? OR headline ILIKE ?<br>        ORDER BY created_at DESC<br>        LIMIT 10<br>    &quot;&quot;&quot;, f&quot;%{topic}%&quot;, f&quot;%{topic}%&quot;)<br>    <br>    return format_news(articles)</pre><p><strong>Why direct tools → </strong>Simple, fast, no extra infrastructure. For historical prices and keyword news search, direct database access is sufficient. Save MCP/embeddings for cases that need it.</p><h3>Memory Tool</h3><p>For cross-conversation memory, we implement tool that searches LangGraph store.</p><pre>@tool<br>async def search_past_conversations(<br>    query: str,<br>    runtime: ToolRuntime[UserContext]<br>) -&gt; str:<br>    &quot;&quot;&quot;Search user&#39;s conversation history across all threads.&quot;&quot;&quot;<br>    user_id = runtime.context[&quot;user_id&quot;]<br>    store = runtime.store<br>    <br>    # Search in user&#39;s namespace<br>    results = await store.asearch(<br>        namespace=(user_id, &quot;conversation_stats&quot;),<br>        query=query,<br>        limit=5<br>    )<br>    <br>    # Format results<br>    summaries = []<br>    for item in results:<br>        symbols = item.value.get(&quot;symbols_discussed&quot;, [])<br>        message_count = item.value.get(&quot;message_count&quot;, 0)<br>        summaries.append(<br>            f&quot;Conversation about {symbols} ({message_count} messages)&quot;<br>        )<br>    <br>    return format_memory_results(summaries)</pre><p><strong>How memory works:</strong> Middleware extracts insights from conversations (symbols discussed, message counts, decisions made) and saves to store. Memory tool searches these insights. Gives agent access to past context without loading full conversation histories.</p><h3>Tool Integration</h3><p>All tools combine into single list for main agent.</p><pre># Agent creation with all tools<br>async def create_main_agent():<br>    # Load all tool categories<br>    sync_tools = [get_stock_price, get_stock_news, search_past_conversations]<br>    alpaca_tools = await load_alpaca_mcp_tools()<br>    trading_tools = await load_trading_mcp_tools()<br>    vector_tools = await load_vector_mcp_tools()<br>    workflow_tools = [debate_analysis, deep_analysis]<br>    <br>    # Combine<br>    all_tools = (<br>        sync_tools + alpaca_tools + trading_tools + <br>        vector_tools + workflow_tools<br>    )<br>    <br>    # Create agent with everything<br>    agent = create_agent(<br>        model=model,<br>        tools=all_tools,  # ~46 tools total<br>        middleware=middleware_stack,<br>        persistence=persistence_config<br>    )<br>    <br>    return agent</pre><p><strong>What this enables:</strong> The main agent autonomously decides to:</p><ul><li>Get historical prices (direct database)</li><li>Search news semantically (vector MCP)</li><li>Check account positions (Alpaca MCP)</li><li>Run debate analysis (workflow wrapper)</li><li>Place trades with approval (trading MCP + HITL middleware)</li><li>Search conversation history (memory tool)</li></ul><p>And compose any combination in single turn. The tool architecture, three MCP servers, direct database, memory, workflow wrappers, gives comprehensive capabilities while maintaining separation of concerns.</p><h3>Subagent Tool Wrappers: Connecting Main Agent to Workflows</h3><p>The key to the subagents pattern is wrapping workflows as tools the main agent can invoke. These wrappers handle context engineering, graph invocation, and result extraction.</p><h3>Pre-Compilation Pattern</h3><p><strong>Critical architectural decision:</strong> Workflows are compiled once at application startup and stored globally. Tool wrappers invoke pre-compiled graphs.</p><pre># Application initialization (startup)<br>async def init_application():<br>    global debate_graph, deep_graph, main_agent<br>    <br>    # Pre-compile debate workflow<br>    debate_graph = create_debate_workflow(<br>        model=model,<br>        tools=ALL_TOOLS<br>    )<br>    <br>    # Pre-compile deep workflow<br>    deep_graph = create_deep_workflow(<br>        model=model,<br>        tools=ALL_TOOLS<br>    )<br>    <br>    # Create main agent with workflow wrappers<br>    main_agent = await create_main_agent(<br>        model=model,<br>        tools=all_tools_including_wrappers<br>    )<br><br># Global access functions<br>def get_debate_graph():<br>    return debate_graph<br><br>def get_deep_graph():<br>    return deep_graph</pre><p><strong>Why pre-compilation:</strong> Graph compilation is expensive: this creates connection pools, binds tools to model, andbuilds execution graph. Doing this on every tool call causes performance issues and resource leaks. <strong>Pre-compile once, reuse forever.</strong></p><p><strong>Performance impact → </strong>Without pre-compilation, debate tool takes 7 seconds (2 seconds compilation + 5 seconds execution). With pre-compilation, 5 seconds (just execution). Resource usage drops from creating 2+ connection pools per request to 3 total pools for entire application.</p><h3>Debate Tool Wrapper</h3><pre># Debate analysis wrapper tool<br>@tool<br>async def debate_analysis(<br>    ticker: str,<br>    runtime: ToolRuntime[UserContext]<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Perform bull/bear debate analysis for investment decisions.<br>    <br>    Use when user asks: &quot;Should I buy X?&quot; &quot;Analyze X&quot; &quot;Is X good investment?&quot;<br>    &quot;&quot;&quot;<br>    # Get pre-compiled graph<br>    debate_graph = get_debate_graph()<br>    <br>    # Extract context from runtime<br>    user_id = runtime.context[&quot;user_id&quot;]<br>    trading_level = runtime.context[&quot;trading_level&quot;]<br>    max_position = runtime.context[&quot;max_position_size&quot;]<br>    risk_tolerance = runtime.context[&quot;risk_tolerance&quot;]<br>    <br>    # Create initial state for workflow<br>    initial_state = {<br>        &quot;topic&quot;: f&quot;Investment analysis for {ticker}&quot;,<br>        &quot;messages&quot;: [HumanMessage(f&quot;Analyze {ticker}&quot;)],<br>        &quot;current_ticker&quot;: ticker,<br>        &quot;bull_score&quot;: 0.0,<br>        &quot;bear_score&quot;: 0.0<br>    }<br>    <br>    # Context engineering: Flatten for debate workflow<br>    config = {<br>        &quot;configurable&quot;: {<br>            &quot;thread_id&quot;: f&quot;debate_{user_id}_{ticker}&quot;,<br>            &quot;trading_level&quot;: trading_level,        # Flattened<br>            &quot;max_position_size&quot;: max_position,      # Flattened<br>            &quot;risk_tolerance&quot;: risk_tolerance,       # Flattened<br>            &quot;can_place_orders&quot;: False               # Flattened<br>        }<br>    }<br>    <br>    # Invoke pre-compiled workflow<br>    final_state = await debate_graph.ainvoke(initial_state, config)<br>    <br>    # Extract result<br>    final_report = final_state[&quot;final_report&quot;]<br>    <br>    return final_report</pre><p><strong>Key points:</strong></p><ol><li>Gets pre-compiled graph (not creating new one)</li><li>Extracts context from runtime.context (ToolRuntime injection)</li><li>Engineers context for workflow (flattens into config)</li><li>Invokes workflow with engineered config</li><li>Extracts and returns result</li></ol><p><strong>Context engineering →</strong>This is the critical piece. Main agent receives UserContext via runtime injection. Debate workflow expects flattened values in config. Wrapper translates between the two patterns.</p><h3>Deep Tool Wrapper</h3><pre># Deep analysis wrapper tool<br>@tool<br>async def deep_analysis(<br>    ticker: str,<br>    runtime: ToolRuntime[UserContext]<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Perform comprehensive multi-dimensional analysis.<br>    <br>    Use when user explicitly requests: &quot;Deep analysis&quot; &quot;Comprehensive report&quot; &quot;In-depth research&quot;<br>    &quot;&quot;&quot;<br>    # Get pre-compiled graph<br>    deep_graph = get_deep_graph()<br>    <br>    # Extract context from runtime<br>    user_id = runtime.context[&quot;user_id&quot;]<br>    trading_level = runtime.context[&quot;trading_level&quot;]<br>    max_position = runtime.context[&quot;max_position_size&quot;]<br>    risk_tolerance = runtime.context[&quot;risk_tolerance&quot;]<br>    <br>    # Context engineering: Embed in message for deepagents<br>    enhanced_message = f&quot;&quot;&quot;<br>    Comprehensive analysis request for {ticker}<br>    <br>    Stock: {ticker}<br>    Trading Level: {trading_level}<br>    Max Position: {max_position} shares<br>    Risk Tolerance: {risk_tolerance}<br>    <br>    Provide complete investment analysis with fundamental, technical, <br>    sentiment, and macro perspectives.<br>    &quot;&quot;&quot;<br>    <br>    # Create initial state<br>    initial_state = {<br>        &quot;messages&quot;: [HumanMessage(content=enhanced_message)]<br>    }<br>    <br>    # Simple config (deepagents doesn&#39;t use complex config)<br>    config = {<br>        &quot;configurable&quot;: {<br>            &quot;thread_id&quot;: f&quot;deep_{user_id}_{ticker}&quot;,<br>            &quot;user_id&quot;: user_id<br>        }<br>    }<br>    <br>    # Invoke pre-compiled workflow<br>    final_state = await deep_graph.ainvoke(initial_state, config)<br>    <br>    # Extract result from messages<br>    final_messages = final_state[&quot;messages&quot;]<br>    last_ai_message = find_last_ai_message(final_messages)<br>    final_report = last_ai_message.content<br>    <br>    return final_report</pre><p><strong>Key difference →</strong> Deep workflow expects context in message content, not config. This is deepagents library convention. Wrapper embeds context in message where agent can see it.</p><p><strong>Why different patterns:</strong> Different workflow types have different expectations. Debate is our custom LangGraph → we control config pattern. Deep uses deepagents library → we adapt to library’s patterns. Wrapper layer handles translation.</p><h3>Error Handling</h3><p>Tool wrappers include comprehensive error handling for graceful degradation.</p><pre>@tool<br>async def debate_analysis(ticker: str, runtime: ToolRuntime) -&gt; str:<br>    try:<br>        # Normal execution path<br>        result = await execute_debate_workflow(ticker, runtime)<br>        return result<br>        <br>    except TimeoutError:<br>        return f&quot;Analysis of {ticker} timed out. Please try again.&quot;<br>        <br>    except ValueError as e:<br>        return f&quot;Invalid data for {ticker}: {str(e)}&quot;<br>        <br>    except Exception as e:<br>        logger.error(f&quot;Debate analysis error: {e}&quot;, exc_info=True)<br>        return f&quot;Error analyzing {ticker}. Please try again later.&quot;</pre><p><strong>Why important →</strong> Workflows can fail (API timeouts, data unavailable, network issues). Errors shouldn’t crash the main agent. Wrapper catches errors, returns user-friendly messages, and logs details for debugging.</p><h3>Middleware Architecture: Cross-Cutting Concerns</h3><p>Middleware intercepts agent execution to add functionality like dynamic prompts, guardrails, human-in-the-loop, and memory extraction. These concerns cut across all tool invocations.</p><h3>Middleware Execution Points</h3><p>LangChain’s AgentMiddleware provides four interception points:</p><pre>class AgentMiddleware:<br>    &quot;&quot;&quot;Base class for middleware.&quot;&quot;&quot;<br>    <br>    def dynamic_prompt(self, request):<br>        &quot;&quot;&quot;Generate system prompt based on context.&quot;&quot;&quot;<br>        # Called before model invocation<br>        # Returns prompt string<br>        pass<br>    <br>    def wrap_model_call(self, request, handler):<br>        &quot;&quot;&quot;Intercept model calls.&quot;&quot;&quot;<br>        # Called around model invocation<br>        # Can modify request, response, or abort<br>        pass<br>    <br>    def after_agent(self, state, runtime):<br>        &quot;&quot;&quot;Run after agent completes.&quot;&quot;&quot;<br>        # Called after agent finishes turn<br>        # Can modify final state<br>        pass</pre><p><strong>How it works →</strong>Middleware stack executes in order. Each middleware can modify behavior at key points. This enables cross-cutting concerns without modifying agent or workflow code.</p><h3>Dynamic Prompt Middleware</h3><p>Adjusts system prompt based on user’s trading level.</p><pre>class TradingLevelPrompt(AgentMiddleware):<br>    &quot;&quot;&quot;Adapt prompt to user&#39;s experience level.&quot;&quot;&quot;<br>    <br>    def dynamic_prompt(self, request):<br>        trading_level = request.runtime.context[&quot;trading_level&quot;]<br>        <br>        base = &quot;You are a helpful stock market assistant.&quot;<br>        <br>        if trading_level == &quot;beginner&quot;:<br>            return f&quot;&quot;&quot;{base}<br>            <br>            User Level: BEGINNER<br>            - Explain in simple, everyday language<br>            - Avoid jargon and technical terms<br>            - Focus on basic principles<br>            - Be educational and encouraging<br>            &quot;&quot;&quot;<br>        <br>        elif trading_level == &quot;intermediate&quot;:<br>            return f&quot;&quot;&quot;{base}<br>            <br>            User Level: INTERMEDIATE<br>            - Balance simplicity with detail<br>            - Use moderate terminology with brief explanations<br>            - Include key metrics and reasoning<br>            &quot;&quot;&quot;<br>        <br>        else:  # advanced<br>            return f&quot;&quot;&quot;{base}<br>            <br>            User Level: ADVANCED<br>            - Provide in-depth analysis<br>            - Use advanced terminology freely<br>            - Include comprehensive metrics<br>            - Discuss complex strategies<br>            &quot;&quot;&quot;</pre><p><strong>Pattern:</strong> Access runtime.context in dynamic_prompt. Return adapted prompt string. Agent gets appropriate prompt for user’s level without conditional logic in agent code.</p><h3>Guardrail Middleware</h3><p>Validates requests to prevent off-topic queries.</p><pre>class BlockNonStockQueries(AgentMiddleware):<br>    &quot;&quot;&quot;Prevent non-stock-related queries.&quot;&quot;&quot;<br>    <br>    def wrap_model_call(self, request, handler):<br>        # Extract user message<br>        user_message = get_last_user_message(request)<br>        <br>        # Check for forbidden topics<br>        forbidden = [&quot;recipe&quot;, &quot;cooking&quot;, &quot;weather&quot;, &quot;sports&quot;, &quot;politics&quot;]<br>        <br>        if any(topic in user_message.lower() for topic in forbidden):<br>            logger.warning(f&quot;Off-topic query: {user_message}&quot;)<br>            # Could block here, but we just log<br>        <br>        # Continue execution<br>        return handler(request)</pre><p><strong>Pattern →</strong> wrap_model_call intercepts before execution. Can inspect request, log warnings, modify request, or abort. Here we log but allow, but we could enforce stricter blocking.</p><h3>HITL Middleware</h3><p>Human-in-the-loop for trade approval.</p><pre>class TradingHITL(AgentMiddleware):<br>    &quot;&quot;&quot;Require human approval for trading operations.&quot;&quot;&quot;<br>    <br>    def wrap_model_call(self, request, handler):<br>        # Let model execute<br>        response = handler(request)<br>        <br>        # Check if response contains trading tool calls<br>        if has_trading_tool_call(response):<br>            # Extract tool details<br>            tool_name = extract_tool_name(response)<br>            tool_args = extract_tool_args(response)<br>            <br>            # Request human approval (blocks execution)<br>            approval = request_human_decision({<br>                &quot;tool&quot;: tool_name,<br>                &quot;args&quot;: tool_args,<br>                &quot;message&quot;: &quot;Approve this trade?&quot;<br>            })<br>            <br>            # If not approved, remove tool call<br>            if not approval[&quot;approved&quot;]:<br>                response = remove_tool_call(response, tool_name)<br>        <br>        return response</pre><p><strong>Pattern:</strong> Execute first, then check response. If trading tool found, interrupt for approval. If rejected, remove tool call. Agent continues with remaining tools. Critical for security → no trades execute without human confirmation.</p><h3>Workflow Pass-Through Middleware</h3><p>Ensures full workflow reports reach users.</p><pre>class WorkflowPassThrough(AgentMiddleware):<br>    &quot;&quot;&quot;Prevent LLM from summarizing workflow results.&quot;&quot;&quot;<br>    <br>    WORKFLOW_TOOLS = {&quot;debate_analysis&quot;, &quot;deep_analysis&quot;}<br>    <br>    def after_agent(self, state, runtime):<br>        messages = state[&quot;messages&quot;]<br>        <br>        # Find last AI message (agent&#39;s response)<br>        last_ai_msg = find_last_ai_message(messages)<br>        <br>        # Find last tool message (workflow result)<br>        last_tool_msg = find_last_tool_message(messages)<br>        tool_name = get_tool_name(last_tool_msg)<br>        <br>        # If workflow tool and agent summarized result<br>        if tool_name in self.WORKFLOW_TOOLS:<br>            tool_content = last_tool_msg.content<br>            ai_content = last_ai_msg.content<br>            <br>            # Check if agent summarized (response much shorter)<br>            if len(ai_content) &lt; len(tool_content) * 0.7:<br>                # Replace with full tool result<br>                return {<br>                    &quot;messages&quot;: [<br>                        AIMessage(content=tool_content)<br>                    ]<br>                }<br>        <br>        return None  # No changes</pre><p><strong>Pattern:</strong> after_agent runs after agent completes. Detects workflow tool calls. If agent summarized (response much shorter than tool result), it replaces with full result. This ensures users get complete formatted reports.</p><p><strong>Why needed:</strong> LLMs try to be “helpful” by summarizing. Workflow returns 500-character formatted report with scores and reasoning. LLM summarizes to “Based on analysis, recommendation is HOLD.” This defeats purpose of detailed report. Middleware enforces pass-through.</p><h3>Memory Extraction Middleware</h3><p>Extracts insights from conversations for long-term memory.</p><pre>class ExtractInsights(AgentMiddleware):<br>    &quot;&quot;&quot;Save conversation insights to long-term memory.&quot;&quot;&quot;<br>    <br>    def after_agent(self, state, runtime):<br>        messages = state[&quot;messages&quot;]<br>        user_id = runtime.context[&quot;user_id&quot;]<br>        store = runtime.store<br>        <br>        # Extract symbols discussed<br>        symbols = extract_symbols_from_messages(messages)<br>        <br>        # Build insight<br>        insight = {<br>            &quot;symbols_discussed&quot;: symbols,<br>            &quot;message_count&quot;: len(messages),<br>            &quot;last_updated&quot;: datetime.now()<br>        }<br>        <br>        # Save to store (async, non-blocking)<br>        asyncio.create_task(<br>            store.aput(<br>                namespace=(user_id, &quot;conversation_stats&quot;),<br>                key=&quot;latest&quot;,<br>                value=insight<br>            )<br>        )<br>        <br>        return None  # No state changes</pre><p><strong>Pattern:</strong> after_agent extracts data from final state. Saves to store asynchronously. Doesn’t block response. Memory tool later searches these insights.</p><h3>Middleware Stack Ordering</h3><p>Order matters — middleware executes sequentially.</p><pre>middleware_stack = [<br>    trading_level_prompt,      # 1. Set dynamic prompt<br>    select_response_format,    # 2. Configure structured output<br>    block_non_stock_queries,   # 3. Validate request<br>    prevent_data_leak,         # 4. Privacy guardrail<br>    handle_tool_errors,        # 5. Error handling<br>    trading_hitl,              # 6. Human approval (before response)<br>    workflow_pass_through,     # 7. Enforce full reports (after agent)<br>    extract_insights,          # 8. Save memory (after agent)<br>]</pre><p><strong>Why this order:</strong></p><ol><li>Prompt must be set before model call</li><li>Format config before execution</li><li>Guardrails before processing</li><li>Error handling wraps execution</li><li>HITL checks response before returning</li><li>Pass-through enforces workflow results</li><li>Memory extraction happens last</li></ol><h3>State Management &amp; Persistence</h3><p>State management has two layers: conversation state (checkpointer) and cross-conversation memory (store).</p><h3>Checkpointer: Conversation State</h3><p>Saves conversation state at each step for continuity and interruption.</p><pre># PostgreSQL checkpointer<br>from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver<br><br>checkpoint_pool = AsyncConnectionPool(<br>    conninfo=DB_URI,<br>    max_size=10,<br>    kwargs={&quot;autocommit&quot;: True}<br>)<br><br>checkpointer = AsyncPostgresSaver(checkpoint_pool)<br><br># Use in agent<br>agent = create_agent(<br>    model=model,<br>    tools=tools,<br>    checkpointer=checkpointer,  # Saves state after each step<br>    ...<br>)</pre><p><strong>How it works:</strong> After each step, LangGraph saves state snapshot to PostgreSQL with thread_id as key. Next invocation with same thread_id loads state and continues conversation.</p><p><strong>What it stores:</strong> Messages, state fields (bull_score, bear_score, recommendations), pending operations. Everything needed to resume conversation exactly where it left off.</p><h3>Store: Cross-Conversation Memory</h3><p>Key-value storage for data persisting across conversations.</p><pre># PostgreSQL store<br>from langgraph.store.postgres.aio import AsyncPostgresStore<br><br>store_pool = AsyncConnectionPool(<br>    conninfo=DB_URI,<br>    max_size=10,<br>    kwargs={&quot;autocommit&quot;: True}<br>)<br><br>store = AsyncPostgresStore(store_pool)<br><br># Use in agent<br>agent = create_agent(<br>    model=model,<br>    tools=tools,<br>    store=store,  # Cross-conversation memory<br>    ...<br>)</pre><p><strong>How it works:</strong> Middleware and tools save/retrieve data using namespaced keys.</p><pre># Saving insights<br>await store.aput(<br>    namespace=(user_id, &quot;conversation_stats&quot;),<br>    key=&quot;latest&quot;,<br>    value={&quot;symbols&quot;: [&quot;AAPL&quot;], &quot;count&quot;: 10}<br>)<br><br># Retrieving insights<br>result = await store.aget(<br>    namespace=(user_id, &quot;conversation_stats&quot;),<br>    key=&quot;latest&quot;<br>)<br><br># Searching insights<br>results = await store.asearch(<br>    namespace=(user_id, &quot;conversation_stats&quot;),<br>    query=&quot;Tesla&quot;,<br>    limit=5<br>)</pre><p><strong>Pattern:</strong> Namespace by (user_id, category). Keys identify specific insights. Search finds relevant past data. Memory tool uses this for cross-conversation recall.</p><h3>Thread ID Strategy</h3><p>Thread IDs determine conversation boundaries.</p><pre># Main agent: User-controlled persistence<br>thread_id = f&quot;conv_{user_id}_{timestamp}_{random}&quot;<br><br># Debate workflow: Analysis-specific<br>thread_id = f&quot;debate_{user_id}_{ticker}&quot;<br><br># Deep workflow: Analysis-specific<br>thread_id = f&quot;deep_{user_id}_{ticker}&quot;</pre><p><strong>Strategy:</strong></p><ul><li>Main agent threads persist across many queries (continuity)</li><li>Workflow threads scoped to specific analysis (isolation)</li><li>User can start new conversation (new thread_id)</li></ul><p><strong>Why isolation:</strong> Debate analysis for AAPL shouldn’t load history from TSLA analysis. Each analysis starts fresh. Main agent conversation maintains context across different analyses.</p><h3>Implementation Patterns &amp; Best Practices</h3><p>These are the patterns that actually work in production.</p><h3>Pattern 1: Pre-Compile Graphs at Startup</h3><p><strong>Problem:</strong> Creating graphs on every tool call is expensive and causes resource leaks.</p><p><strong>Solution:</strong> Compile once at startup, invoke from wrappers.</p><pre># Startup<br>debate_graph = create_debate_workflow(model, tools)<br>deep_graph = create_deep_workflow(model, tools)<br><br># Tool wrapper<br>@tool<br>async def debate_analysis(ticker, runtime):<br>    graph = get_debate_graph()  # Pre-compiled<br>    result = await graph.ainvoke(...)<br>    return result</pre><p><strong>Impact:</strong> 30% faster, no resource leaks, proper lifecycle.</p><h3>Pattern 2: Context Engineering Per Workflow</h3><p><strong>Problem:</strong> Different workflows expect context differently.</p><p><strong>Solution:</strong> Engineer context in tool wrapper for each workflow’s expectations.</p><pre># Debate: Flattened in configurable<br>config = {<br>    &quot;configurable&quot;: {<br>        &quot;trading_level&quot;: level,<br>        &quot;max_position&quot;: size,<br>        ...<br>    }<br>}<br><br># Deep: Embedded in message<br>message = f&quot;Context: level={level}, size={size}\nAnalyze {ticker}&quot;</pre><p><strong>Why:</strong> Each workflow has different patterns. Wrapper handles translation.</p><h3>Pattern 3: Middleware for Quality Enforcement</h3><p><strong>Problem:</strong> LLMs summarize tool results.</p><p><strong>Solution:</strong> Middleware detects and replaces summaries.</p><pre>class WorkflowPassThrough(AgentMiddleware):<br>    def after_agent(self, state, runtime):<br>        if workflow_tool_used and agent_summarized:<br>            return {&quot;messages&quot;: [full_tool_result]}</pre><p><strong>Why:</strong> Programmatic enforcement. LLM instructions aren’t reliable.</p><h3>Pattern 4: Structured Output with Pydantic</h3><p><strong>Problem:</strong> Need typed, validated data from LLM.</p><p><strong>Solution:</strong> Pydantic models with with_structured_output.</p><pre>class TradeRec(BaseModel):<br>    symbol: str<br>    action: str<br>    confidence: float<br><br>model = model.with_structured_output(TradeRec)<br>result = await model.ainvoke(prompt)<br># result is validated TradeRec instance</pre><p><strong>Why:</strong> Type safety, validation, downstream processing.</p><h3>Pattern 5: MCP for Extensibility</h3><p><strong>Problem:</strong> Adding capabilities requires code changes.</p><p><strong>Solution:</strong> MCP servers provide standardized tool interface.</p><pre># Add new capability: Deploy MCP server<br>new_server = deploy_mcp_server(capability)<br><br># Load in agent<br>new_tools = await mcp_client.get_tools(new_server_url)<br>all_tools += new_tools<br><br># Agent automatically gets new capabilities</pre><p><strong>Why:</strong> Standardized protocol. Server-side execution. Easy to extend.</p><h3>Pattern 6: Error Handling in Wrappers</h3><p><strong>Problem:</strong> Workflows can fail in various ways.</p><p><strong>Solution:</strong> Comprehensive error handling with graceful messages.</p><pre>@tool<br>async def debate_analysis(ticker, runtime):<br>    try:<br>        return await execute_workflow(...)<br>    except TimeoutError:<br>        return &quot;Analysis timed out. Try again.&quot;<br>    except ValueError as e:<br>        return f&quot;Invalid data: {e}&quot;<br>    except Exception as e:<br>        logger.error(&quot;Workflow error&quot;, exc_info=True)<br>        return &quot;Error occurred. Try again later.&quot;</pre><p><strong>Why:</strong> Main agent receives string either way. Errors don’t crash system.</p><h3>Pattern 7: Middleware Ordering</h3><p><strong>Problem:</strong> Middleware order affects behavior.</p><p><strong>Solution:</strong> Carefully order stack based on dependencies.</p><pre>[<br>    dynamic_prompt,        # Before model call<br>    guardrails,           # Before processing<br>    hitl,                 # Check response<br>    workflow_pass_through, # After agent<br>    memory_extraction,    # Last<br>]</pre><p><strong>Why:</strong> Dependencies exist. Prompt before execution. Memory after completion.</p><h3>Conclusion: Composable Agent Architecture</h3><p>We’ve built a multi-agent system combining three distinct patterns:</p><p><strong>Autonomous Main Agent:</strong> Flexible orchestration with LangChain’s create_agent. Sees all tools. Composes capabilities. Handles parallel execution and sequential reasoning.</p><p><strong>Custom LangGraph Workflows:</strong> Deterministic control for structured processes. Debate workflow ensures complete investment analysis with explicit bull/bear/judge flow.</p><p><strong>Specialized Agent Libraries:</strong> Purpose-built capabilities with deepagents. Research coordination with parallel analyst subagents and synthesis.</p><p>These patterns compose naturally. Workflows become tools. Main agent orchestrates everything. Clean separation of concerns.</p><p><strong>Key implementation details that make it work:</strong></p><ul><li>Pre-compiled graphs for performance</li><li>Context engineering per workflow type</li><li>Middleware for cross-cutting concerns</li><li>MCP servers for extensibility</li><li>Proper state management and persistence</li><li>Structured output for typed data</li><li>Comprehensive error handling</li></ul><p><strong>For developers building similar systems:</strong> Start with LangChain’s documented patterns. Use create_agent for orchestration. Use LangGraph for custom workflows. Integrate specialized libraries where they add value. Pre-compile your graphs. Engineer context per workflow. Use middleware for quality. Test thoroughly.</p><p><strong>The multi-agent future is here!</strong></p><h3>Learn More</h3><p><strong>LangChain Documentation:</strong></p><ul><li>create_agent: <a href="https://docs.langchain.com/oss/python/langchain/agents">https://docs.langchain.com/oss/python/langchain/agents</a></li><li>Multi-Agent Patterns: <a href="https://docs.langchain.com/oss/python/langchain/multi-agent">https://docs.langchain.com/oss/python/langchain/multi-agent</a></li><li>LangGraph: <a href="https://docs.langchain.com/oss/python/langgraph">https://docs.langchain.com/oss/python/langgraph</a></li><li>Middleware: <a href="https://docs.langchain.com/oss/python/langchain/agents/middleware">https://docs.langchain.com/oss/python/langchain/agents/middleware</a></li><li>Model Context Protocol: <a href="https://modelcontextprotocol.io/">https://modelcontextprotocol.io</a></li></ul><p><strong>deepagents Library:</strong></p><ul><li>GitHub: <a href="https://github.com/deepagents/deepagents">https://github.com/deepagents/deepagents</a></li><li>Documentation: <a href="https://deepagents.readthedocs.io/">https://deepagents.readthedocs.io</a></li></ul><p><strong>Our Series:</strong></p><ul><li>Part 1: <a href="https://www.linkedin.com/pulse/building-production-ready-agentic-ai-system-langchain-kharbanda-myfme/">Building Production-Ready Agentic AI Systems</a></li><li>Part 2: <a href="https://www.linkedin.com/pulse/beyond-agent-product-vs-platform-thinking-ai-kharbanda-zf0re/">Product vs. Platform Thinking in AI Architecture</a></li><li>Part 3: <a href="https://www.linkedin.com/pulse/ai-multi-agent-architectural-patterns-langchain-v10-from-kharbanda-lggte">From Router to Subagents — Architectural Journey</a></li><li>Part 4: Technical Deep Dive (this article)</li></ul><p><strong>GitHub:</strong> <a href="https://github.com/yourusername/stocktickers-ai">https://github.com/ishveen-ai/stocktickers-ai</a> (Repo final work cleanup work in progress and will be released in next couple of week — max by mid of Feb)</p><p>Follow the series as we continue exploring production patterns for multi-agent AI systems!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ba216e70f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My RAG Journey: Part 4 - Vector Databases]]></title>
            <link>https://medium.com/@ishveen/my-rag-journey-part-4-vector-databases-034bed7ee477?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/034bed7ee477</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 22:12:22 GMT</pubDate>
            <atom:updated>2025-11-23T22:12:22.242Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! I’m 15 years old, and I am trying to learn as much as I can about <strong>Generative AI</strong>. One topic that has interested me is RAG. Read my first part (<a href="https://medium.com/@ishveen/my-rag-journey-part-1-document-parsing-2fa832866ea9">click here</a>), second article (<a href="https://medium.com/@ishveen/my-rag-journey-part-2-chunking-6390b6e85ba8">click here</a>), and third (<a href="https://medium.com/@ishveen/my-rag-journey-part-3-embedding-and-vector-databases-cf17dc14ee14">click here</a>) for context. Now that we have embedded our data for an LLM to understand, <strong>how can we store these embeddings and find the most relevant documents that relate to a user’s query? </strong>Read on to learn more!</p><p>Prior to my research, I thought that vector databases just stored the embeddings of RAG documents and found documents that were the most semantically similar to a user’s query, and then this document would be used by an LLM in order to answer the user’s question. I didn’t think that there was much of a difference between different types of vector databases because they all achieved the same goal of finding the most relevant documents. After I researched this topic, I learned that vector databases are much more complex then they seem.</p><p>For reference, the PDF and Notebooks (retrieval data) were from these repositories:</p><ul><li>PDF → <a href="https://github.com/dcai-course/dcai-course">https://github.com/dcai-course/dcai-course</a></li><li>Notebook → <a href="https://github.com/dcai-course/dcai-lab/tree/master">https://github.com/dcai-course/dcai-lab/tree/master</a></li><li>The PDF lesson and corresponding notebook are from a lesson on data curation for LLMs</li></ul><h3>What are vector databases?</h3><p>Normal databases store any type of data for the sole purpose of using this data in some way. In vector databases, they are specialized to only store embeddings, and this data is retrieved so that we can perform RAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l25uvOAWmztzflCB" /><figcaption>From: <a href="https://www.pinecone.io/learn/vector-database/">https://www.pinecone.io/learn/vector-database/</a></figcaption></figure><p>How Vector Databases play a role in RAG:</p><ol><li>Our content/data is parsed, chunked, and embedded (done in previous articles)</li><li>These embeddings are stored in the vector database</li><li>A user sends a query from the application</li><li>The query is embedded</li><li>The query is compared to the documents in order to find the most relevant document that can answer the question</li><li>An LLM uses the query and the most relevant documents to answer the user’s question</li></ol><p><strong>But why must we use these vector databases? Why can’t we use traditional databases?</strong></p><p>Although some normal databases support vectors, most databases struggle because of the complexity of the vectors. Their architectures are not suitable for storing and semantic searching. Vector databases offer higher search performance with much lower costs.</p><p>The next question you may be asking is “<strong>What makes vector databases optimized for RAG applications?</strong>”</p><h3><strong>Indexes and ANN</strong></h3><p>This is the main part of vector databases that distinguish them from normal databases. We can think of an index like an index in a book — we can skim through the index, since it is an alphabetical order, so that we can find the words that we are looking for. Indexes in a database do the same thing. They are organized in a way that makes it easier for the search algorithm to skim through the data in order to find relevant data points faster. <em>Note: The vectors themselves are not stored in the index, they are stored in the DB Storage layer, known as the </em><strong><em>disk/SSD</em></strong></p><p>These indexes perform Approximate Nearest Neighbor Search (ANN). Rather than finding an exact match for the best data points, it finds an approximate best data point. Although this is not 100% accurate, it’s accuracy reduction is minuscule and it allows for significantly faster search. ANN is performed based on the type of index.</p><p>In real-time RAG applications, you have millions of data points that are embedded. For my small dataset, I don’t necessarily need to use an index because it would still go through my data quickly. For real RAG datasets, indexes are necessary in order to speed up the search process. These indexes help power ANN search, which helps find close enough matches with great speed.</p><p>Here are the different index/ANN types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SapM24WegqQB1XfSvViMww.png" /><figcaption>Summary of each method</figcaption></figure><ul><li>Flat (Brute force) → This is a search technique that is technically performed without an index. Every single data point is compared to the query in order to find an exact match. This only works well for very small datasets, because larger data sets would take too long to process</li><li>Tree based → The topmost node of the tree is referred to as the root node. After this, the root node connects to many internal nodes, each with their own keys. These nodes that are connected to the root node are created by partitioning the space into pieces, and they either connect to other internal nodes or they connect to leaf nodes, which are at the lowest level. These leaf nodes contain keys and pointers to the actual data in the database. When a query is sent, the keys of each possible node are evaluated and a node is chosen. This is repeated for each internal layer of nodes until a leaf node is reached. Tree-based structures fail for embeddings because embeddings are high-dimensional, and distance metrics become meaningless in &gt;30 dimensions (This is known as the curse of dimensionality). <em>Methods like Annoy and ScaNN utilize tree architectures in their index.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ir4pwZW7JL86e_xq0n9sPw.png" /><figcaption><a href="https://builtin.com/data-science/b-tree-index">https://builtin.com/data-science/b-tree-index</a> → Simplified version of tree based models</figcaption></figure><ul><li>Graph Based/Hierarchical Navigable Small World graphs (HNSW) → This is one of the most popular methods for storing embeddings. Nodes in this graph are connected to their “nearest neighbors” or the nodes that are most similar to them. The top layer of the graph have a small number of nodes, essentially being a sparse “summary” of the dataset. The number of nodes increases per layer as you get lower until you reach the bottom layer, which contains all of the data points densely connected. The search algorithm starts at the top with a summary and moves down the layers so that at the last layer, it only has to examine a small amount of data points (similar to tree). This method, although seeming similar to tree, is much better for embeddings because it shows how ALL data points connect, so the model stays efficient as dimensions are added. <em>Why this matters for RAG: provides low latency with high accuracy</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDtqdLr4SspyA_h1yCia0g.png" /><figcaption><a href="https://medium.com/data-science/similarity-search-part-4-hierarchical-navigable-small-world-hnsw-2aad4fe87d37">https://medium.com/data-science/similarity-search-part-4-hierarchical-navigable-small-world-hnsw-2aad4fe87d37</a> → graph based methods avoids scanning the entire dataset by navigating through layers</figcaption></figure><ul><li>Inverted File Index (IVF) → Imagine we have a two dimensional space with an x and y axis, and we split the graph into a certain number of sections. The vectors are split into these sections based off where they fall into the graph. The regions act as buckets for the vectors, and each bucket has a “centroid” point (the data point in the center of the region). During search, the query is compared to the each centroid, and the region that is closest to the query is chosen. Only data points in this region are compared to the query, reducing the amount of data points that are compared to the query. <em>Why this matters for RAG: search is fast even with millions of embeddings</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*k0me1lUorV5aSIv7.png" /><figcaption><a href="https://blog.dailydoseofds.com/p/approximate-nearest-neighbor-search">https://blog.dailydoseofds.com/p/approximate-nearest-neighbor-search</a> → each region has a centroid that is compared with the query</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*kYQH43VCsztH1mQR.png" /><figcaption><a href="https://blog.dailydoseofds.com/p/approximate-nearest-neighbor-search">https://blog.dailydoseofds.com/p/approximate-nearest-neighbor-search</a> → most similar region(s) are chosen and searched</figcaption></figure><ul><li>Product Quantization (PQ) → This method compresses the vectors in order to save memory and speed up distance computations. Each vector is split into sub-vectors and then a centroid value is computed for each sub-vector. The sub vector is then replaced with a “summary sub vector” meaning that the old sub vector and new sub vector are very similar (semantically), but this new summary sub vector is much smaller, so takes up less storage, allowing for faster computations. <em>Why this matters for RAG: reduces memory so that more documents can be stored.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*beQCXCyzoHRsSDWrLPXBIw.png" /><figcaption><a href="https://thedataquarry.com/blog/vector-db-3/">https://thedataquarry.com/blog/vector-db-3/</a> → vector is split into sub vectors, these sub vectors are quantized</figcaption></figure><p>Libraries like <strong>Facebook AI Similarity Search (FAISS)</strong> exist for the sole purpose of indexing and searching embeddings, but this is still not considered a vector database, because a database has many other common features other than indexing and searching (which are shown below). <em>NOTE: These are not all the possible functions of databases, and not all vector databases have all of these functions.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGVCCTrsFEjywe-HE1RzTw.png" /></figure><ul><li>Horizontal Sharding → Horizontal scaling refers to splitting the data across multiple machines/nodes so that each handles a subset of vectors. For example, if I had 10 million vectors and ten shards, each shard would have 1M. Then the ANN index would only need to find nearest neighbors for 1/10th of the data, but they query would have to visit all of the shards. This helps speed up search because if each node looks through this data in parallel, it would work 10x as fast.</li><li>Vertical Sharding → This is another method that scales the vector database. This is when you add more compute (CPU/RAM) to a single node, which increases the storage available. This method is much easier than horizontal scaling, but it is limited by the amount of hardware you can add to one server. Additionally, putting all of your data in one server is not a safe practice because the server can fail, which would destroy all of your data.</li><li>Replication → This creates another node that is a copy of one of the existing ones. This is useful because if one node fails, the replica is still there so the RAG system can work effectively. Many data bases use raft for metadata or shard replication</li><li>Raft → This is an algorithm that stores metadata about the clusters/collections in the database. This metadata can include data about what shards, nodes, or collections exist</li><li>Metadata filtering → Each vector can be stored along with metadata, which is data that describes data (in this case the data that describes vectors). This can include timestamps, IDS, coding languages and other tags. This helps us filter our search; for example, if I say I only want data written in the coding language Python, it would filter out data that does not have the language as Python in the metadata. This reduces search time by removing irrelevant data points quickly and efficiently</li><li>Hybrid search → This was described in the last article more in depth. This is when you have access to both sparse and dense embeddings of the same data, so you can perform semantic and keyword search. You can then combine the results of each to find the documents that both methods identified as relevant documents. Although dense and sparse is the most common pairing, it can also be considered hybrid search if you include metadata filtering.</li><li>CRUD operations → This stands for Create, Read, Update, and Delete, which are four functions of databases. It allows you to add new embeddings, retrieve them, modify them, or remove them. You can perform these actions in real time without having to rebuild the entire index.</li><li>Security → For enterprise data, vector DBs can encrypt data at rest, add authentication, and control who can query or insert data into certain collections. Some managed services allow you to deploy your data in a private network, so the vector search cannot be accessed on the public internet</li></ul><p>Now that we know how vector databases work, how do we choose our vector database?</p><h3>Vector databases</h3><p>Just like how there is an MTEB ranking for embeddings, there is also a database ranking (although it is not nearly as detailed as MTEB). This ranking focuses on the popularity of databases, NOT THE FUNCTION.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xbf55sZbg1XVIRcKdLHqSw.png" /><figcaption>Find rankings here “<a href="https://db-engines.com/en/ranking/vector+dbms">https://db-engines.com/en/ranking/vector+dbms</a>”</figcaption></figure><p>Although this ranking includes databases that support datatypes other than vectors, we will only focus on the databases marked as ONLY vector databases (but you can always use the other ones).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pUoOr8QP35RovN2C_3ccqA.png" /><figcaption>Main Vector DBs + Best use cases</figcaption></figure><ul><li>Chroma → This is the simplest of the five databases that runs with a Python library. It does not require much configuration and it is ideal for prototypes or personal projects. It is not built for large datasets, and it uses HNSW (primarily) for search. Although it is easy to implement, it lacks true scaling methods such as sharding and replication.</li><li>Pinecone → This database only works on the cloud and it emphasizes vector database features outside of indexing and search such as scalability, sharding, and low-latency. It is also the only propriety database of the five. It’s index method is propriety, but it likely uses HNSW internally. It also supports dimensions up to 16k+. This method is serverless, meaning that they manage the underlying infrastructure of servers while you (the developer) can focus on coding. This makes this method the most expensive. This is best for enterprise RAG that relies on low latency.</li><li>Milvus → This database was built specifically for large, billion-scale datasets. This uses etcd rather than Raft for metadata and allows for GPU-accelerated index. Most importantly, it supports all kinds of ANN algorithms such as Flat, IVF, IVF-PQ, HNSW, Annoy, etc. They have a managed service called Zilliz Cloud which makes deployment much easier (otherwise the developer would have to use Kubernetes for real scaling, which is very complex). This method is best for enterprises with huge datasets</li><li>Qdrant → This database is know for their advanced metadata filtering, and it also has high performance per GPU and update-able HNSW indexes. They are also extremely well known for their hybrid filtering performance (this makes sense because of their metadata filtering). This is best for e-commerce applications or any other apps that require complex metadata filtering.</li><li>Weaviate → This is an open source, Go-based vector DB that organizes data into classes with a schema, where you can define the relations between two objects. This uses an HNSW index (default), and thse indexes can be sharded or replicated. It also uses the GraphQL API, which allows for us to retrieve the exact data we want in a single request(no over/under fetching of data). This method also has a strong hybrid search, but it is more complex in setup. They have their own managed service called Weaviate Cloud. This method is best for multimodal RAG and scenarios that need to capture structured relationships.</li></ul><p>When choosing which database you want to use, you should consider things such as:</p><ul><li>Your use case</li><li>The size of your data</li><li>Your experienced with managed cloud services</li><li>Type of index you want to use</li><li>Dimensions of embeddings</li><li>Security needed</li><li>Cost</li></ul><p>We now know that when we send a query to the DB:</p><ol><li>Metadata filters are applied</li><li>The query is routed to the appropriate shards</li><li>ANN is applied to each shard</li><li>The results are merged across nodes</li><li>(Optional) the results are reranked based on similarity and returned</li></ol><p>But how do we practically store our vectors and retrieve data for RAG?</p><h3>ChromaDB</h3><p>At first, I started with Chroma’s local <strong>PersistentClient</strong>. In theory, it sounded simple: All I had to do was store my embeddings and perform retrieval. In practice, it was <strong>not</strong> that simple, and I had to actually connect to the cloud.</p><h4>1. Turning my embeddings into collections</h4><p>Collections store data into different “buckets” so I was able to store all of my chunking + embeddings pairs into separate collections. Here was my data structure in my local folders:</p><ul><li>A folder called embeddings with five folders in it</li><li>Each folder either said nbformat (notebook) or pdf + the chunking method used → pdf_chunks_A_recursive_1000_150/…pdf_chunks_B_semantic_minilm_thr084/…nbformat_chunks_A_jsonl_md900_135_code800_120/…</li><li>Inside each of those folders, they had folders of each embedding model used with a JSONL (for sparse) or parquet (for dense) file inside</li></ul><p><em>Note: I didn’t use the sparse embeddings for this because Chroma does not support inputting precomputed sparse embeddings, only their own.</em></p><p>I wrote a script that:</p><ul><li>Walked over every embeddings.parquet file.</li><li>Loaded three columns from each Parquet:</li></ul><p>id → unique chunk ID</p><p>text → the actual chunk (PDF slide text or notebook cell)</p><p>vector → the embedding</p><ul><li>Created a Chroma collection named like: &lt;chunking_name&gt;_&lt;embedding_pair&gt;</li></ul><pre>for parquet_path in parquet_paths:<br>    chunking_name = parquet_path.parents[1].name<br>    encoder_name = parquet_path.parents[0].name<br><br>    collection_name = f&quot;{chunking_name}__{encoder_name}&quot;<br>    print(f&quot;\n=== Ingesting {parquet_path} into collection &#39;{collection_name}&#39; ===&quot;)<br><br>    df = pd.read_parquet(parquet_path)<br><br>    if &quot;id&quot; not in df.columns or &quot;text&quot; not in df.columns or &quot;vector&quot; not in df.columns:<br>        raise ValueError(f&quot;File {parquet_path} is missing a required column&quot;)<br><br>    df[&quot;id&quot;] = df[&quot;id&quot;].astype(str)<br><br>    ids = df[&quot;id&quot;].tolist()<br>    docs = df[&quot;text&quot;].tolist()<br>    vectors = [to_python_list(v) for v in df[&quot;vector&quot;]] # the to_python_list function just turns the embedding into a flat Python list with MAX 4096 dimensions<br>    if not vectors:<br>        print(&quot; no embeddings found in this parquet&quot;)<br>        continue</pre><p>Then, rather than using a PersistantClient that locally, I used Chroma’s cloud, and I set up a client like this:</p><pre>client = chromadb.CloudClient(<br>  api_key=&#39;api_key_here&#39;,<br>  tenant=&#39;tenant_name_here&#39;,<br>  database=&#39;Article_test&#39; # name of my database<br>)</pre><p>For each embedding , I created the collection and stored the embeddings</p><pre>    collection = client.get_or_create_collection(<br>        name=collection_name,<br>        embedding_function=None,<br>    )<br><br>    collection.add(<br>        ids=ids,<br>        embeddings=vectors,<br>        documents=docs,<br>    )<br>    print(f&quot;Inserted {len(ids)} records into &#39;{collection_name}&#39;&quot;)</pre><h4>2. Refine embeddings</h4><p>Unfortunately, I could only have a max embedding dimension of 4096, but I had some with more dimensions, so this function would trim the vector (which likely led to less accurate results for the models that produced embeddings with &gt;4k dimensions)</p><pre>def to_python_list(vec):<br>    arr = np.asarray(vec, dtype=float).ravel()<br>    if arr.size &gt; MAX_DIM:          # MAX_DIM = 4096<br>        arr = arr[:MAX_DIM]         # truncate instead of erroring<br>    return arr.tolist()</pre><h4>3. What embeddings looked like in the UI</h4><p>Here is how my collections were shown in Chroma Cloud:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p60d_nnyEoxyMUFvAhxXJA.png" /><figcaption>Chroma Cloud UI</figcaption></figure><p>This UI made it much easier for me to manage my embeddings and collections (highly recommend to use clouds for DBs).</p><h4>4. Analyze collections</h4><p>The reason why I split up my collections like this is so that I can test the results that the LLM give based on the chunking + embedding pair. First, my code looked at the collection names in order to infer what document is was and the chunking + embedding pair.</p><pre>def infer_encoder_key(coll_name: str) -&gt; str | None:<br>    for key in [&quot;openai_openai&quot;, &quot;jina_jina&quot;, &quot;qwen_voyage&quot;]:<br>        if key in coll_name:<br>            return key<br>    return None<br><br>def infer_collection_type(coll_name: str) -&gt; str | None:<br>    if coll_name.startswith(&quot;nbformat_&quot;):<br>        return &quot;nb&quot;<br>    if coll_name.startswith(&quot;pdf_&quot;):<br>        return &quot;pdf&quot;<br>    return None</pre><h4>5. Embed Query</h4><p>I created separate queries based on whether it was a notebook or PDF, and then the queries were embeded based on the embedding model that was used for that collection. This is because the dimensions of the query and the document must be consistent in order to perform semantic search.</p><pre><br>NB_QUESTIONS = [<br>    &quot;Using the lab notebook code, how can you take a dataset with multiple annotators, detect noisy or conflicting labels, and improve label quality before training a model?&quot;,<br>    &quot;Based on the notebook, how could you use model predictions, cross-validation, and tools like cleanlab to find mislabeled examples in the dataset and decide which rows to fix or remove?&quot;,<br>]<br><br>PDF_QUESTIONS = [<br>    &quot;According to the lecture slides, how can large language models be used for data curation and cleaning in real-world applications?&quot;,<br>    &quot;From the lecture slides, why is detecting and handling PII important when curating training data for LLMs, and what approaches can be used to remove or mask PII safely?&quot;,<br>]<br><br>def embed_hf(model: str, texts: List[str], batch: int = 32) -&gt; np.ndarray:<br>    if not texts:<br>        return np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    if model not in _HF_CACHE:<br>        rev = os.getenv(&quot;JINA_V3_REV&quot;) if &quot;jina&quot; in model else None<br>        _HF_CACHE[model] = SentenceTransformer(<br>            model,<br>            trust_remote_code=True,<br>            revision=rev,<br>        )<br>    m = _HF_CACHE[model]<br>    outs = []<br>    for i in range(0, len(texts), batch):<br>        chunk = texts[i : i + batch]<br>        emb = m.encode(chunk, batch_size=batch, show_progress_bar=False)<br>        outs.append(emb)<br>    X = np.vstack(outs) if outs else np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    return l2_normalize(X) if len(X) &gt; 0 else X<br><br>def embed_openai(model: str, texts: List[str], batch: int = 128) -&gt; np.ndarray:<br>    if not texts:<br>        return np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    cli = get_oai_client()<br>    outs = []<br>    for i in range(0, len(texts), batch):<br>        chunk = texts[i : i + batch]<br>        r = cli.embeddings.create(model=model, input=chunk)<br>        emb = np.vstack([np.array(d.embedding, dtype=&quot;f4&quot;) for d in r.data])<br>        outs.append(emb)<br>    X = np.vstack(outs) if outs else np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    return l2_normalize(X) if len(X) &gt; 0 else X<br><br>def embed_voyage(model: str, texts: List[str], batch: int = 16) -&gt; np.ndarray:<br>    global _VOYAGE_CLIENT<br>    if not texts:<br>        return np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    if _VOYAGE_CLIENT is None:<br>        key = &quot;key here&quot;<br>        if not key:<br>            raise RuntimeError(&quot;VOYAGE_API_KEY is not set&quot;)<br>        _VOYAGE_CLIENT = voyageai.Client(api_key=key)<br>    outs = []<br>    for i in range(0, len(texts), batch):<br>        chunk = texts[i : i + batch]<br>        r = _VOYAGE_CLIENT.embed(chunk, model=model)<br>        outs.append(np.array(r.embeddings, dtype=&quot;f4&quot;))<br>    X = np.vstack(outs) if outs else np.zeros((0, 1), dtype=&quot;f4&quot;)<br>    return l2_normalize(X) if len(X) &gt; 0 else X</pre><h4>6. Find most relevant docs</h4><p>After the query is embedded, the code below queries the collection and gets the top documents and their distances (similarity)</p><pre>for q_idx, question in enumerate(questions, start=1):<br>            print(f&quot;\n[Q{q_idx}] {question}&quot;)<br><br>            cache_key = (question, enc_key)<br>            if cache_key not in query_cache:<br>                print(f&quot;[INFO] Embedding question for encoder={enc_key}&quot;)<br>                base_q_vec = embed_query_for_encoder(enc_key, question)<br>                query_cache[cache_key] = base_q_vec<br>            else:<br>                base_q_vec = query_cache[cache_key]<br><br>            q_vec = match_dim(base_q_vec, coll_dim)<br><br>            res = coll.query(           # db query<br>                query_embeddings=[q_vec],<br>                n_results=3,<br>                include=[&quot;documents&quot;, &quot;distances&quot;],<br>            )<br><br>            docs = res.get(&quot;documents&quot;, [[]])[0]<br>            ids = res.get(&quot;ids&quot;, [[]])[0]<br>            dists = res.get(&quot;distances&quot;, [[]])[0]<br>            hits = []<br>            for doc_id, doc, dist in zip(ids, docs, dists):<br>                snippet = (doc[:200] + &quot;...&quot;) if len(doc) &gt; 200 else doc<br>                print(f&quot;  id={doc_id}, dist={dist:.4f}&quot;)<br>                print(&quot;   &quot;, snippet.replace(&quot;\n&quot;, &quot; &quot;))<br>                hits.append((doc_id, doc, dist))<br><br>            answer = call_llm_on_hits(oai, question, coll.name, hits)</pre><h4>7. Send top docs to the LLM</h4><p>If you, look at the previous code snippet, we have a list (hits) that stores the top documents, their IDs, and their distances. It them calls the function call_llms_on_hits, which sends the docs to an LLM and returns an answer to the query</p><pre>def call_llm_on_hits(<br>    oai: OpenAIClient,<br>    question: str,<br>    collection_name: str,<br>    hits: List[Tuple[str, str, float]],<br>) -&gt; str:<br>    if not hits:<br>        return &quot;No relevant documents were found in this collection.&quot;<br><br>    blocks = []<br>    for idx, (doc_id, doc, dist) in enumerate(hits, start=1):<br>        header = f&quot;[{idx}] id={doc_id} | distance={dist:.4f}&quot;<br>        blocks.append(f&quot;{header}\n{doc}&quot;)<br><br>    context = &quot;\n\n---\n\n&quot;.join(blocks)<br><br>    messages = [<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: (<br>                &quot;You are a helpful assistant answering questions based ONLY on the provided context. &quot;<br>                &quot;If something is not in the context, say you don&#39;t know. &quot;<br>                &quot;You are looking at a single collection with a specific chunking+embedding setup.&quot;<br>            ),<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: (<br>                f&quot;Collection: {collection_name}\n&quot;<br>                f&quot;Question: {question}\n\n&quot;<br>                f&quot;Context:\n{context}\n\n&quot;<br>                &quot;Answer the question using ONLY this context.&quot;<br>            ),<br>        },<br>    ]<br><br>    resp = oai.chat.completions.create(<br>        model=CHAT_MODEL,<br>        messages=messages,<br>        temperature=0.2,<br>    )<br>    return resp.choices[0].message.content</pre><h4>8. Save answers</h4><p>Lastly, I saved the answer for each question along with the collection it was retrieved from like this</p><pre>================================================================================<br>COLLECTION: pdf_chunks_B_semantic_bge_thr084_thrOverride__qwen_voyage (encoder=qwen_voyage, type=pdf)<br>Q1: According to the lecture slides, how can large language models be used for data curation and cleaning in real-world applications?<br><br>  &gt;&gt;&gt; LLM answer:<br>Large language models (LLMs) can be used for data curation and cleaning in real-world applications by serving as powerful natural-language reasoning engines that are easy to customize for new use cases. They are also computationally inexpensive and are revolutionizing systematic text data curation.<br>----------------------------------------</pre><h3>Conclusion</h3><p>Now that I have completed researching RAG, I will begin to learn more about how to build AI agents. Stay tuned for my series on agents!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=034bed7ee477" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My RAG Journey: Part 3-Embedding and Vector Databases]]></title>
            <link>https://medium.com/@ishveen/my-rag-journey-part-3-embedding-and-vector-databases-cf17dc14ee14?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf17dc14ee14</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Sun, 26 Oct 2025 19:01:46 GMT</pubDate>
            <atom:updated>2025-10-26T19:01:46.930Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! I’m 15 years old, and I am trying to learn as much as I can about <strong>Generative AI</strong>. One topic that has interested me is RAG. Read my first part (<a href="https://medium.com/@ishveen/my-rag-journey-part-1-document-parsing-2fa832866ea9">click here</a>) and second article (<a href="https://medium.com/@ishveen/my-rag-journey-part-2-chunking-6390b6e85ba8">click here</a>) for context. This article will answer the question: “<strong>How can we make our LLM effectively <em>understand</em> our data?</strong>” Read ahead to learn what embedding methods I used!</p><p>When an LLM with a RAG system reads a user query, it looks into its database of documents to pick those that are most relevant to the query, but it cannot understand the raw text:</p><ul><li>This is where embeddings come in → they turn the words into numbers so that the computer can understand what each document says. We can think of it as coordinates because it maps where each word would be on a multi dimensional map</li><li>Similar words are closer on this map and different ones are farther away. We can measure this with methods like cosine similarity, Euclidean distance, and dot product</li><li>Not all embeddings are the same → sparse embeddings are used to find keywords in the document that are relevant in the user query while dense embeddings find how similar in meaning two documents are</li><li>Hybrid search → This is crucial for RAG systems. Sparse embeddings can be used to find keywords in documents while dense can find semantically similar documents. By combining the two methods, we can find the most relevant documents that have both key terms and semantic similariy to the query</li><li>The type of methods that we use are important. If you do not use the right embedding combination, the RAG system won’t find the most relevant documents and the answer it generates will be inaccurate.</li></ul><p>For reference, the PDF and Notebooks (retrieval data) were from these repositories:</p><ul><li>PDF → <a href="https://github.com/dcai-course/dcai-course">https://github.com/dcai-course/dcai-course</a></li><li>Notebook → <a href="https://github.com/dcai-course/dcai-lab/tree/master">https://github.com/dcai-course/dcai-lab/tree/master</a></li><li>The PDF lesson and corresponding notebook are from a lesson on data curation for LLMs</li></ul><h3>What are embeddings?</h3><p>LLMs are unable to understand the language that humans use, because they only understand numbers. Embeddings are able to translate words into numbers that a computer can understand.</p><p>An embedding is a long list of numbers, referred to as a vector, which represents what a word means. Words and sentences with similar meanings are close together while those that are different are farther apart. The dimensions are the criteria by which words are being judged.</p><p>For example, let’s look at this image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*yM-HdId0CD63_bjSEWKg2w.jpeg" /></figure><ul><li>Similar words are closer to each other → animals like wolf, cat, and dog are clustered together</li><li>Different words are far apart → words such as cat and apple are far apart because they are semantically dissimilar (one is an animal and one is a fruit)</li></ul><p>Think of embedding models like this but instead of being 3D, these models can have hundreds or even thousands of dimensions. This is what allows phrases with similar meanings to be closer together.</p><p>In RAG, embeddings help the LLM truly understand the data that it is trying to retrieve. Otherwise, it will not understand what is being retrieved and it will begin to hallucinate (make up information).</p><h3>How can we use embeddings to find similarity between words?</h3><p>In order to find the similarity between embeddings, we simply measure the distance or angle between two embeddings (words). There are three popular methods used to do this.</p><p>Here’s a TL;DR</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jq7bYxcv7CQDY182B2Cb9w.png" /></figure><ul><li><strong>Cosine similarity → </strong>This method measures the angles between two angles and ignores the length of vectors. This is best for text embeddings because words with similar meanings can be different lengths but they have a small angle in between them. <em>Here is a conceptual diagram + equation:</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*8Iu-thdWrNcUvoVBXg42vw.png" /><figcaption>Created by author</figcaption></figure><ul><li>Dot product → This method measures how well two vectors align. Higher magnitudes pointing in a similar direction have the highest values (most similar). If they form a right angle, the dot product is zero. If they point in opposite directions, the dot product is negative (less similar). These are best for neural networks because their architecture also uses dot products. <em>Here is a conceptual diagram + equation:</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*pTY_p-U5aRB-ZTWLwR7k1w.png" /><figcaption>Created by author</figcaption></figure><ul><li>Euclidean distance → This method measures raw distances between two points. This means two similar vectors can appear different if one has a larger magnitude. This is best for images since each point is marked at a certain position. <em>Here is a conceptual diagram + equation:</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*u1Y6vhFWKYo78_BLRG8XEg.png" /><figcaption>Created by author</figcaption></figure><h3>Embedding Models</h3><p>You should choose your embedding model based on your use case and how well they perform</p><h4>Sparse embedding models</h4><p>Here’s a TL;DR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZNJGWSH2nTo0GXfmSTVQw.png" /><figcaption>Created by author</figcaption></figure><p>Sparse embeddings mean that the vectors consist of mostly zeroes. These methods are very basic and they hold no semantic meaning, but they are crucial for RAG because they enable keyword search in order to find relevant documents.</p><ul><li>Basic/Outdated Methods → One-Hot Encoding (Every word in the vocabulary is marked as a zero except one word is 1), Bag of Words (BoW) (This counts how often a word appears), TF-IDF (Words are given weight based on how often they show up in a text)</li><li>BM25 → Extension the basic BoW and TF-IDF methods to create embeddings, meaning it is only capable of raw keyword search. It utilizes traditional TF-IDF method by giving each word a weight based on how often that word appears per document. It is extended by normalization so that there is no bias (longer documents will not have higher frequencies because of their length). Additionally, term-saturation is applied → if a term appears a few times, the TF-IDF increases rapidly. But, as the term frequency rises, the TF-IDF begins to plateau so that frequent words aren’t inaccurately over-rewarded like TF-IDF’s traditional linear model. There is no learning required for retrieval, making it simple to use, but it can miss words that are synonyms.</li><li>SPLADE → Stands for Sparse Lexical and Expansion model. This uses a contextual dense model (scroll down to learn about them) in order to create a vector the size of the vocabulary (number of unique words used). Although it may use a dense vector, it still outputs a sparse vector by using this dense model to assign weights to each word. So, words like “turtle” might show up in an document, but the model may also look for the word “tortoise.” It also uses an activation function (like ReLU) so that the values are positive or zero (sparse vector relies on zeros). Although this is much more effective, this method may be harder to use because it requires some training, so it may be slower. Many researchers have proposed to make this faster by separating document and query encoders (same model works in parallel to create two separate embeddings), regularization/normalization, and faster query encoders (embeddings are generated quickly with a smaller dense model)</li><li>BGE-M3 → A newer model created by the Beijing Academy for Artificial Intelligence. This model is special because it is able to use sparse, dense, and multi-vector retrieval all in one. This means it can handle semantic search and keyword search at the same time. The sparse vector is generated similarly to SPLADE, but that same document may also include and dense embedding corresponding with it. These models also support <strong>Matryoshka embeddings, </strong>meaning that you can adjust the number of dimensions that the model outputs (which allows us to trade-off between performance and cost). This model is useful because you don’t need separate models for sparse and dense embeddings, but it is complex and resource-heavy.</li></ul><h4>Static dense methods</h4><p>These methods learn how to embed words based on multi-dimensional vectors, where similar words have similar embeddings because they are closely related. These embeddings are static, meaning that they stay the same no matter what the context is. For example, the word “bark” can refer to the sound that dogs make or the bark on trees (based on the context), but the embedding will be the same. Vectors can look like → [0.4, 0.5, 0.9, -0.2, 0.1, 0.8]</p><ul><li>Word2Vec → This learns the embeddings by predicting the context so that similar words are close in space. This is small, so it can be trained fast and it is good for clustering similar words together. This works well if you have limited compute resources, but the downside is that it is static and it struggles with words out of vocabulary (OOV) and morphology of words (prefixes and suffixes). This method does not capture grammar either.</li><li>GloVe → This method uses global co-occurrence, meaning that it weighs how often words appear together. This model is much easier and faster to train because there are many more pre-trained models and pre-computed matrices. This method has the same downsides as Word2Vec. This is best if you want to use general pre-trained models for efficiency.</li><li>FastText → This method is aware of subwords (prefixes and suffixes). This means there is built in OOV handling because it can “guess” what the word means. But, this method is still static and it cannot detect grammar. This is great for text that uses a lot of words with prefixes and suffixes or any noisy text in general.</li></ul><h4>Dense + Context-aware</h4><p>These embeddings take into account of the words that surround them (because the surrounding text can change the meaning of the word). These are much more common to use (and more effective too).</p><p><strong>Basic Models:</strong></p><ul><li>ELMo → A deep-context aware model that is able to accurately capture grammar. Although this is a big step up from the other models, this model is still unable to handle OOV words. This model is best for Question &amp; Answering (Q&amp;A) and Name Entity Recognition (NER)</li><li>Sentence-BERT (SBERT) / Sentence Transformers → On top of being context-aware, this model can handle OOV words (similar to FastText) and it has a deep understanding of grammar, syntax, and structure. Models like these are best for chatbots because of their deep understanding of words.</li></ul><p>Below, we will discuss the <strong>MTEB leaderboard (</strong><a href="https://huggingface.co/spaces/mteb/leaderboard"><strong>CLICK HERE</strong></a><strong>) </strong>(hosted on Hugging Face), which is a benchmark for dense text embedding models. These models are evaluated by many tasks such as retrieval, classification, clustering, and semantic textual similarity (STS), and models are ranked by their overall performance. Here is a deep-dive of each category so that we can evaluate which model we need based on our use case.</p><ul><li>Classification → Tests how well embeddings can help sort texts into categories (such as filtering emails as span or not span). Embedding models with high scores in this category perform well on labeled data sets.</li><li>Clustering → This determines how well the embedding model can group similar texts together, showing how well it can help discover topics and trends (better at finding patterns). This is an important feature for unstructured, high-dimensional, and complex data.</li><li>Retrieval → How relevant the retrieved documents are based on the query. For example, if we ask “How do you use LangChain?”, the retrieval should be relevant because our embeddings are relevant. If we want to created a chatbot that uses RAG (like we are), this category is crucial to the chatbot’s performance in order to retrieve correct answers.</li><li>Semantic Textual Similarity (STS) → Measures how well the model can judge the similarity between two phrases/texts. Similar phrases should have high similarity and phrases with different phrases should have low similarity. This can be important for plagiarism detection or search engines, where the model must be able to accurately find similarities.</li><li>Pair Classification → This tasks compares two texts to see if they fall under the same category/meaning. A use case of this feature can be for duplicate query detection; when someone asks a question, this model can detect to see if that question has already been asked with a different wording. This is best algorithms that need to find duplicates or matches.</li><li>Reranking → After the initial retrieval step, this category evaluates how well it is able to rank the list of results by relevance. This is crucial for systems that return query results with a list of relevant items: e-commerce websites (they rank by most to least relevant products), Q&amp;A systems (finds most relevant answer for query), and web search engines (most relevant results).</li><li>Bitext Mining → Finding similarities in phrases that are in two different languages (referred to as translation pairs). Despite the language barrier, the model should be able to find similarity in phrases that mean the same thing. This feature is important if you want to create something like a chatbot that uses a translation system.</li><li>Summarization → This evaluates how well embeddings can evaluate summaries of texts. For example, if there is one AI summary and one human made summary, it should check to make sure these embeddings are semantically similar. Models that perform well here will produce embeddings that accurately capture the overall meaning of a document.</li></ul><p>Now, you choose your model based off which categories are most important for your use case. For our RAG chatbot that retrieves relevant slides and code based on the user query, sections like STS, Retrieval, and Reranking may be the most important. Remember, a model that ranks #1 overall may not be best for <strong>your</strong> specific use case.</p><h4>Code Embedding Models</h4><p>Next, we will dive into some code embedding models to use for our Jupyter Notebook. These models are specialized to understand the syntax and structure of coding languages, because using regular models can lead to confusion:</p><ul><li>VoyageCode3 → This model can have 256, 512, 1024, or 2048 dimensions for embeddings and it can offer different quantization formats (size of each number). It has a context of up to 32K tokens and it has a large dataset trained across 300+ coding languages. This model claims to also outperform OpenAI v3 large (on average).</li><li>OpenAI Text Embedding 3 Large (text-embedding-3-large) → Although this model is a high performing general purpose (<strong>ranked #17 on the MTEB leaderboard as of now</strong>), it also performs well for coding tasks with 3072 dimensions and 8191 tokens of context. It’s weakness can be that it can struggle on deep code-only tasks because it is general purpose.</li><li>jina-embeddings-v2-base-code → Open-source model, knows over 30+ coding languages, over 137 million parameters (lightweight), and 8192 tokens of context. This model is optimized for code search and you can integrate it with AWS’s SageMaker, through the API, or through HuggingFace</li><li>CodeBERT → Created by Microsoft, 768 dimensions. Open-source model. It is pre-trained and multi-lingual training on code and text pairs (which may be useful for Jupyter notebooks). But, this model is older and it has been surpassed by new ones</li></ul><p><strong>Hybrid Search</strong></p><p>Using both sparse and dense embeddings are important → Dense embeddings are used in order to find documents with similar semantic meaning while sparse embeddings help find documents with the same keywords as the query.</p><h3>Practical usage of dense embeddings</h3><p>If you look at my previous article, you can see that I used a multitude of parsing and chunking methods for a slide deck (PDF) and a Jupyter Notebook file (Python + Markdown). Here are the three embedding model combinations I used:</p><ul><li><strong>Jina text + Jina code embeddings</strong> → These models fall under the same organization, so I assumed that they would be compatible together</li><li><strong>Qwen 0.6B + VoyageCode3 </strong>→ These models are leaders in their specialization and they were easy to use</li><li><strong>OpenAI Text Embeddings 3 Large </strong>→ This model is able to understand both text and code, which makes it easier to integrate as one</li></ul><h4>Combination</h4><p>Combining the text and code embeddings ensures that the retrieval models will be able to understand both the code and markdown explanations, which is essential when working with Jupyter Notebooks. Each model has a different amount of vector dimensions, so how do we combine the vector dimensions so that we have consistent dimensions without changing the meaning of the embeddings?</p><p>In simple terms, we create a function that aligns the dimensions of both code and text vectors so that they are the same, which creates one unified vector. For example, lets say my coding embedding model has 1000 dimensions and the text model has 2000. If I use the code model, the first 1000 dimensions show the embeddings and the 2000 after that are 0’s. If we use the text model, the first 1000 dimensions are 0’s and the other 2000 show the text embeddings. <strong>This is called padding the vector.</strong></p><pre># this function combines the text and code vectors and pads as needed<br>def make_pair(name, text_vecs, code_vecs, texts, metas):<br>    n = max(len(text_vecs), len(code_vecs)) # maximum number of rows = vector with largest num of rows<br>    # determines padding based on the diference of the two vectors<br>    if len(text_vecs) &lt; n:<br>        text_vecs = np.pad(text_vecs, ((0, n - len(text_vecs)), (0, 0)))<br>    if len(code_vecs) &lt; n:<br>        code_vecs = np.pad(code_vecs, ((0, n - len(code_vecs)), (0, 0)))<br><br>    comb = np.concatenate([text_vecs, code_vecs], axis=1)   # &lt;-- join halves<br>    return rows_to_df([(t, m, v) for t, m, v in zip(texts[:n], metas[:n], list(comb))])</pre><p>But, this process distorts the distances with all of the extra zeroes, so how can we prevent this?</p><p><strong>L2 Normalization → </strong>Process by which the distance of a vector is scaled to be equal to 1. This is done by summing the squares of each component within a vector and taking the square root of this sum (let’s call this number “x”). Then, each of the components within this vector is divided by “x”, making the distance of the vector 1 no matter what, so there is no change in semantic meaning. Here is how I used it in my code:</p><pre>def l2_normalize(mat): <br>    n = np.linalg.norm(mat, axis=1, keepdims=True); n[n==0]=1; return mat/n </pre><h4>How each of the three models are paired</h4><pre>def embed_and_save_pairs(fpath,recs,batch):<br>    # create path for embedding files<br>    stem=safe_stem(fpath)<br><br>    texts=[r[&quot;text&quot;] for r in recs]; metas=[r.get(&quot;metadata&quot;,{}) for r in recs]<br>    <br>    # text and metadata together<br>    text_rows=[(t,m) for t,m in zip(texts,metas) if not is_code_chunk(t,m)]<br>    # code and metadata together<br>    code_rows=[(t,m) for t,m in zip(texts,metas) if is_code_chunk(t,m)]<br><br>    # just the text (separated becausing embedding models don&#39;t need metadata)<br>    txts_text=[t for t,_ in text_rows]; mtxts=[m for _,m in text_rows]<br>    # just the metadata (so we can join the metadata with the text afterwards)<br>    mtxts = [{&quot;cell_type&quot;: &quot;markdown&quot;}]<br><br>    txts_code=[t for t,_ in code_rows]; mcode=[m for _,m in code_rows]<br>    mcode = [{&quot;cell_type&quot;: &quot;code&quot;}]</pre><p>For all three models, we embed the text and code. Then, we use the make pair function with combines the two vectors and pads as needed. The comments in the OpenAI model’s code apply to the other embeddings too.</p><ul><li>Open AI model</li></ul><pre># although we are using one model, we are defining it as two separate models so that the routing logic is consistent<br><br># OpenAI+OpenAI<br>print(f&quot;[openai_openai] {stem}&quot;)<br># pass only the text<br>oai_text=embed_one(&quot;oai3l_text&quot;,txts_text,batch)<br>oai_code=embed_one(&quot;oai3l_code&quot;,txts_code,batch)<br># pass embeddings, text, and metadata through the make_pairs function<br>df=make_pair(&quot;openai_openai&quot;,oai_text,oai_code,txts_text+txts_code,mtxts+mcode)<br>save_pair_df(df,EMB_ROOT/stem/&quot;openai_openai&quot;)</pre><ul><li>Jina + Jina</li></ul><pre># Jina+Jina<br>print(f&quot;[jina_jina] {stem}&quot;)<br>jt=embed_one(&quot;jina_text&quot;,txts_text,batch)<br>jc=embed_one(&quot;jina_code&quot;,txts_code,batch)<br>df=make_pair(&quot;jina_jina&quot;,jt,jc,txts_text+txts_code,mtxts+mcode)<br>save_pair_df(df,EMB_ROOT/stem/&quot;jina_jina&quot;)</pre><ul><li>Qwen + Voyage</li></ul><pre># Qwen+Voyage<br>print(f&quot;[qwen_voyage] {stem}&quot;)<br>qw=embed_one(&quot;qwen_text_06b&quot;,txts_text,batch)<br>vg=embed_one(&quot;voyage_code&quot;,txts_code,batch)<br>df=make_pair(&quot;qwen_voyage&quot;,qw,vg,txts_text+txts_code,mtxts+mcode)<br>save_pair_df(df,EMB_ROOT/stem/&quot;qwen_voyage&quot;)</pre><p>Now the next question you may ask is: <strong>“How does it know whether to use the code model or the normal text model?”</strong></p><p>The code below looks into the metadata of each chunk to see if it falls under code or markdown. If it fails to do this, it looks into the chunk itself to see if there are words in the document that indicate that it is a python file (def, import, class, etc.), but this is just a fallback method.</p><pre>CODE_FENCE_RE=re.compile(r&quot;```|^%%|\bimport\s+\w+|\bdef\s+\w+\(|\bclass\s+\w+\(&quot;,re.I)<br>def is_code_chunk(text,meta):<br>    ct=(meta or {}).get(&quot;cell_type&quot;)<br>    if ct: return str(ct).lower()==&quot;code&quot;<br>    return bool(CODE_FENCE_RE.search(text or &quot;&quot;))</pre><p>In summary, if you pass two models for the code to use:</p><ul><li>It creates a folder for the embeddings to go in</li><li>It splits the data based on text vs code and then separates the content and the metadata</li><li>Embeds the code and text vectors and performs L2 normalization for both</li><li>Combines and pads the embeddings as needed</li><li>Creates a DataFrame that puts the content and the metadata back together, and the embeddings are saved in its designated folder</li></ul><h3>Practical Usage of Sparse Embeddings</h3><p>I only embedded the PDF because keyword search is crucial for finding the most relevant documents, but for code, there usually isn’t any crucial keywords (especially if variable and function names less semantic).</p><p>Methods:</p><ul><li>SPLADE for sparse embeddings</li><li>BGE-M3 for BOTH sparse and dense</li></ul><p>The embeddings are saved as JSONL files, and each line has many values. Here are some of the possible values of the output (which are in the “rec” dictionary in the code for each embedding model):</p><ul><li>id → unique number tagged to each chunk (starts at zero)</li><li>vector → only in dense, represents the dense embedding (semantic similarity)</li><li>indices → in sparse only, list of token ids that had non-zero values in the sparse vector (meaning they are significant</li><li>values → in sparse only, the weights (importance scores) associated with each token id in “indices”</li><li>dim → for dense it represents the side of the vector, for sparse embeddings it is the vocabulary size</li><li>model → name of model in Hugging Face</li></ul><h4>SPLADE</h4><ul><li><a href="https://huggingface.co/naver/splade-v3">Click here</a> to access the model</li></ul><p>Code walk through (look at comments for explanation):</p><pre># setup<br>import os, json, torch<br>from sentence_transformers import SparseEncoder<br><br>FILES = [<br>    (&quot;path_input1&quot;,<br>     &quot;path_output1&quot;),<br>    (&quot;path_input2&quot;,<br>     &quot;path_output2&quot;),<br>    (&quot;path_input3&quot;,<br>     &quot;path_output3&quot;)]<br><br># model from hugging face<br>MODEL_ID = &quot;naver/splade-v3&quot;<br><br># change to cuda if available<br>encoder = SparseEncoder(MODEL_ID, device=&quot;cpu&quot;)<br><br>       <br>for input_path, out_dir in FILES:<br>    os.makedirs(out_dir, exist_ok=True)<br><br>    # read ids and texts<br>    ids, texts = [], []<br>    with open(input_path, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for i, line in enumerate(f):<br>            if not line.strip():<br>                continue<br>            obj = json.loads(line)<br>            # get id<br>            ids.append(obj.get(&quot;id&quot;, i))<br>            # get text<br>            texts.append(obj.get(&quot;text&quot;, &quot;&quot;))<br><br>    print(f&quot;\n {len(texts)} chunks in {input_path}&quot;)<br><br>    # makes the embedding the shape --&gt; [number_of_chunks, vocab_dimensions]<br>    with torch.no_grad():<br>        emb = encoder.encode_document(texts)   # PyTorch sparse tensor (CSR/COO)<br><br>    # stores size of vocab<br>    vocab_dim = int(emb.shape[1])<br><br>    # makes the embeddings in a jsonl file (one line per chunk)<br>    out_path = os.path.join(out_dir, &quot;splade_sparse.jsonl&quot;)<br>    with open(out_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as out_f:<br>        for row_i in range(emb.shape[0]):<br>            indices, values = row_indices_values_from_torch_sparse(emb, row_i)<br>            rec = {<br>                &quot;id&quot;: ids[row_i],<br>                &quot;indices&quot;: indices,<br>                &quot;values&quot;: values,<br>                &quot;dim&quot;: vocab_dim,<br>                &quot;model&quot;: MODEL_ID<br>            }<br>            out_f.write(json.dumps(rec) + &quot;\n&quot;)</pre><h4>BGE-M3</h4><ul><li><a href="https://huggingface.co/BAAI/bge-m3">Click here</a> for access</li></ul><p>Code walk through (look at comments for explanation):</p><pre>import os, json<br>import numpy as np<br>from FlagEmbedding import BGEM3FlagModel<br><br># inputs and outputs<br>FILES = [<br>    (&quot;path_input1&quot;,<br>     &quot;path_output1&quot;),<br>    (&quot;path_input2&quot;,<br>     &quot;path_output2&quot;),<br>    (&quot;path_input3&quot;,<br>     &quot;path_output3&quot;)]<br><br># model from hugging face<br>MODEL_ID = &quot;BAAI/bge-m3&quot;<br><br># BGE-M3 model (FP16 for speed)<br>model = BGEM3FlagModel(MODEL_ID, use_fp16=True)<br>vocab_dim = model.tokenizer.vocab_size # vocab size metadata<br><br>for input_path, out_base in FILES:<br><br>    # Read texts and ids<br>    ids, texts = [], []<br>    with open(input_path, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for i, line in enumerate(f):<br>            if not line.strip():<br>                continue<br>            obj = json.loads(line)<br>            # ids<br>            ids.append(obj.get(&quot;id&quot;, i))<br>            # text<br>            texts.append(obj.get(&quot;text&quot;, &quot;&quot;))<br><br>    # Encode once to get BOTH dense and sparse<br>    out = model.encode(<br>        texts,<br>        batch_size=12,<br>        max_length=8192,<br>        return_dense=True, # dense<br>        return_sparse=True, # sparse<br>        return_colbert_vecs=False,<br>    )<br><br>    dense_vecs = out[&quot;dense_vecs&quot;] <br>    lex_weights_list = out[&quot;lexical_weights&quot;] # list of dicts:{token_id: weight}<br><br>    # Save DENSE in jsonl<br>    dense_path = os.path.join(dense_dir, &quot;bge_m3_dense.jsonl&quot;)<br>    with open(dense_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as df:<br>        for i, vec in enumerate(dense_vecs):<br>            rec = {<br>                &quot;id&quot;: ids[i],<br>                &quot;vector&quot;: np.asarray(vec, dtype=float).tolist(),<br>                &quot;dim&quot;: int(dense_vecs.shape[1]),<br>                &quot;model&quot;: MODEL_ID,<br>            }<br>            df.write(json.dumps(rec) + &quot;\n&quot;)<br><br>    # save SPARSE indices and values<br>    sparse_path = os.path.join(sparse_dir, &quot;bge_m3_sparse.jsonl&quot;)<br>    with open(sparse_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as sf:<br>        for i, lw in enumerate(lex_weights_list):<br>            # lw is a dict: {token_id: weight}<br>            if not lw:<br>                indices, values = [], []<br>                nnz = 0<br>            else:<br>                # keep as id/weight for compactness &amp; fast scoring later<br>                indices = [int(k) for k in lw.keys()]<br>                values  = [float(v) for v in lw.values()]<br>                nnz = len(indices)<br>            # create jsonl<br>            rec = {<br>                &quot;id&quot;: ids[i],<br>                &quot;indices&quot;: indices,<br>                &quot;values&quot;: values,<br>                &quot;dim&quot;: int(vocab_dim),<br>                &quot;model&quot;: MODEL_ID,<br>            }<br>            sf.write(json.dumps(rec) + &quot;\n&quot;)<br><br><br>print(&quot;\nAll BGE-M3 dense &amp; sparse embeddings created successfully.&quot;)</pre><p>In my retrieval article, I will pair each dense embedding model pairing with both of the sparse embeddings for <strong>hybrid search.</strong></p><h3>Conclusion</h3><p>I learned that embeddings are not a “one-size fits all situation” and that different situations call for different types of embedding models. Additionally, it is impossible for LLMs to understand the retrieved data without these embeddings. It is crucial to use hybrid embeddings so that the LLM can both understand the semantic similarity of multiple documents and important keywords that are relevant to the query.</p><p>In my next article, I will discuss many methods used to create vector databases for the embeddings to be stored and and decide which ones best fit for text and code embeddings. In my retrieval article, I will use hyybrid search to see which combination of sparse and dense models are the best!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf17dc14ee14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My RAG Journey: Part 2 - Chunking]]></title>
            <link>https://medium.com/@ishveen/my-rag-journey-part-2-chunking-6390b6e85ba8?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6390b6e85ba8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[chunking]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Sun, 14 Sep 2025 00:51:52 GMT</pubDate>
            <atom:updated>2025-09-14T00:51:52.200Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! I’m 15 years old, and I am trying to learn as much as I can about <strong>Generative AI</strong>. One topic that has interested me is RAG. In my first part (<a href="https://medium.com/@ishveen/my-rag-journey-part-1-document-parsing-2fa832866ea9">click here</a>), I discussed document parsing methods methods and decided that I wanted to use <strong>Unstructured Hi-Res</strong> for PDFs and <strong>nbformat</strong> for python notebooks. This article answers the question: “<strong>How can we effectively <em>chunk </em>our data for RAG?” </strong>Read on to figure out what chunking methods I learned about and which ones I tested!</p><p>NOTE: I do explain embeddings and retrieval part as examples here in this article, but I will have a separate article for these topics so I can explain them with more detail.</p><p>For reference, the PDF and Notebooks (retrieval data) were from these repositories:</p><ul><li>PDF → <a href="https://github.com/dcai-course/dcai-course">https://github.com/dcai-course/dcai-course</a></li><li>Notebook → <a href="https://github.com/dcai-course/dcai-lab/tree/master">https://github.com/dcai-course/dcai-lab/tree/master</a></li><li>The PDF lesson and corresponding notebook are from a lesson on data curation for LLMs</li></ul><h3>What are chunking and embedding?</h3><p>Chunking is how we split long documents into smaller, meaningful pieces (“chunks”) so we can search and index them. Chunking matters because LLMs can’t index or retrieve from huge blobs effectively because of its limited context window (it can only receive a limited amount of tokens at once). Good chunking preserves meaning and boundaries while bad chunking loses context, so retrieval of data is worse and the LLM can begin to hallucinate. Chunks should be big enough to contain enough information but small enough so that it fits within the context window and gives you low-latency responses.</p><p>Embeddings matter too, but I will cover that more in detail in my next article. So you can get a basic understanding of what embeddings are (they are relevant in this article too), they convert <strong>tokens </strong>(pieces of text like words and subwords) into a list of numbers that capture the semantic meaning and context of these words, allowing the LLM to understand the relationship between words and phrases.</p><h3>Chunking methods</h3><p>There are many chunking methods available and they all serve a different purpose based on you data. This section is very lengthy because it covers many different subtopics based on how your data is structured.</p><h4><strong>Here are some simple/core chunking methods:</strong></h4><ul><li><strong>Fixed-Size Chunking</strong> → You pick a number of tokens to split your text by and each chunk is equally sized.<em> </em>It uses a tokenizer to convert words into tokens and then makes chunks into a certain amount of tokens (such as 500). This is great for uniform and structured data such as logs or chat transcripts. This is the most simple choice and great for prototyping, but it ignores the meaning of data and can break chunks in the middle of sentences. <em>You can use LangChain (which is a wrapper that simplifies these processes rather than coding everything from scratch) too for this method.</em></li></ul><pre># langchain fixed-size chunking<br>from langchain.text_splitter import CharacterTextSplitter<br><br># splitter variable for fixed-size chunking<br>fixed_splitter = CharacterTextSplitter.from_tiktoken_encoder( # from_tiktoken_encoder lets me choose how I tokenize data<br>    model_name=&quot;gpt-4&quot;, # tokenizer <br>    separator=&quot; &quot;, # character it will split on (not in the middle of words)<br>    chunk_size=50,  # number of characters for each chunk<br>    chunk_overlap=0  # zero overlap<br>)<br><br># this is a placeholder value you will most likely upload the text file<br>text = &quot;&quot;&quot;Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. <br>Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. <br>Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.&quot;&quot;&quot;<br><br># split the text <br>fixed_size_chunks = fixed_splitter.split_text(text)<br><br># this will return a list where each chunk is an item in the list<br>print(&quot;Fixed-size Chunks:\n&quot;, fixed_size_chunks)<br><br>&quot;&quot;&quot; <br>output: <br>Fixed-size Chunks:<br> [&#39;Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et&#39;, &#39;dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut&#39;, &#39;aliquip ex ea commodo consequat. \nDuis aute irure dolor in reprehenderit in voluptate velit esse cillum&#39;, &#39;dolore eu fugiat nulla pariatur. \nExcepteur sint occaecat cupidatat non proident, sunt in culpa qui&#39;, &#39;officia deserunt mollit anim id est laborum.&#39;] <br>&quot;&quot;&quot;</pre><ul><li><strong>Sliding window/overlap</strong> → This method overlaps chunks by taking part of the previous chunk. This improves recall by keeping context, but it increases storage taken up because of the repetitive data. This is great for boosting recall and queries that may need information from multiple chunks/topics. Too much overlap can be redundant, so we often overlap by 10–20% of the chunk size. <em>We can implement this with the same tool as the fixed-size splitter from LangChain</em></li></ul><pre># langchain overlap chunking<br>from langchain.text_splitter import CharacterTextSplitter<br><br># splitter variable for chunking<br>overlap_splitter = CharacterTextSplitter.from_tiktoken_encoder( # from_tiktoken_encoder lets me choose how I tokenize data<br>    model_name=&quot;gpt-4&quot;, # tokenizer <br>    separator=&quot; &quot;,   # character it will split on (not in the middle of words)<br>    chunk_size=50,    # number of characters for each chunk<br>    chunk_overlap=10     # slight overlap<br>)<br><br># this is a placeholder value you will most likely upload the text file<br>text = &quot;&quot;&quot;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&quot;&quot;&quot;<br><br># split the text <br>overlap_chunks = overlap_splitter.split_text(text)<br><br># this will return a list where each chunk is an item in the list<br>print(&quot;overlap chunks:\n&quot;, overlap_chunks)<br>&quot;&quot;&quot;<br>output:<br><br>overlap chunks:<br> [&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat.&#39;, &#39;pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam&#39;, &#39;eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas.&#39;, &#39;metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel&#39;, &#39;hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos&#39;, &#39;conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex&#39;, &#39;Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis.&#39;, &#39;tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus&#39;, &#39;vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc&#39;, &#39;lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent&#39;, &#39;Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing&#39;, &#39;amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi&#39;, &#39;In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar&#39;, &#39;tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada&#39;, &#39;massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu.&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet&#39;, &#39;ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In&#39;, &#39;sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna&#39;, &#39;sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa&#39;, &#39;Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti&#39;, &#39;vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum&#39;, &#39;himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque&#39;, &#39;sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean&#39;, &#39;Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum&#39;, &#39;nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit&#39;, &#39;Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos&#39;, &#39;conubia nostra inceptos himenaeos.&#39;]<br>&quot;&quot;&quot;<br></pre><ul><li><strong>Sentence-level chunking</strong> → Each chunk is a sentence and it splits data based on punctuation (“.”, “!”, “?”, etc.). This keeps chunks grammatically coherent by not splitting in the middle of sentences. This method is great for data with clear punctuation, but it may group things together if there is no clear punctuation and the chunks can be too short for proper retrieval. <em>To make this process easier, we can use LlamaIndex (another wrapper because LangChain does not cover this).</em></li></ul><pre># llama_index_sentence_parser<br>from llama_index.core.node_parser import SentenceWindowNodeParser<br>from llama_index.core.node_parser import SentenceSplitter<br><br># create the sentence window node parser w/ default settings<br>node_parser = SentenceWindowNodeParser.from_defaults(<br>    window_size=3,<br>    window_metadata_key=&quot;window&quot;,<br>    original_text_metadata_key=&quot;original_text&quot;,<br>)<br><br># base node parser is a sentence splitter<br>sentence_splitter = SentenceSplitter()<br><br>from llama_index.core import Settings<br><br>Settings.text_splitter = sentence_splitter<br><br>from llama_index.core import SimpleDirectoryReader<br><br># my sample PDF (there is not a lot of clear punctuation to split on)<br>documents = SimpleDirectoryReader(<br>    input_files=[&quot;path_to_pdf_here&quot;]<br>).load_data()<br><br>base_nodes = sentence_splitter.get_nodes_from_documents(documents)<br><br>for i in range(5):<br>    print(&quot;/n chunk: &quot;, i)<br>    print(base_nodes[i].text)<br><br>&quot;&quot;&quot;<br>Output:<br><br>/n chunk:  0<br>Introduction to <br>data-centric ai<br>Lecture 7:<br>Data Curation and LLMs<br>https://dcai.csail.mit.edu<br>/n chunk:  1<br>Background: Large Language Models (LLMs)<br>• You&#39;ve probably heard of ChatGPT, GPT-4, Llama, ...<br>• LM: seq2seq model trained to predict the next word (token)<br>• <br>• Large model capacity<br>• Unsupervised pre-training on an internet-scale corpus<br>• Solve a variety of NLP tasks<br>P(xi ∣ xi−1, xi−2, … , x0)<br>/n chunk:  2<br>Application: zero-shot prompting<br>/n chunk:  3<br>Application: few-shot prompting<br>/n chunk:  4<br>Comparison of LLM prompting/training techniques <br>Source: Andrej Karpathy (2023)<br>&quot;&quot;&quot;<br></pre><ul><li><strong>Paragraph</strong> → This splits data by paragraph. This is great for general information like essays, article, blogs, reports, etc. It is even better if this data has consistent paragraphs so that the chunks are of similar length. This method may not be best if the chunk size varies too much because then retrieval would be inconsistent. If paragraphs are too big, they could surpass the token limit. Some paragraphs may contain too much information while others could contain too little, so it all depends on how your data’s paragraphs are structured. <em>I couldn’t find any wrappers that are specifically meant for paragraph splitting, but we can easily use LangChain’s CharacterTextSplitter to split on blank lines</em></li></ul><pre># langchain paragraph chunking (using \n\n as the separator instead of)<br>from langchain.text_splitter import CharacterTextSplitter<br><br># splitter variable for fixed-size chunking<br>paragraph_splitter = CharacterTextSplitter.from_tiktoken_encoder( # from_tiktoken_encoder lets me choose how I tokenize data<br>    model_name=&quot;gpt-4&quot;, # tokenizer <br>    separator=&quot;\n\n&quot;,   # character it will split on (not in the middle of paragraphs)<br>    chunk_size=500,    # number of characters for each chunk<br>    chunk_overlap=0     # no overlap<br>)<br><br># this is a placeholder value you will most likely upload the text file locally<br>text = &quot;&quot;&quot;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&quot;&quot;&quot;<br><br># split the text <br>paragraph_chunks = paragraph_splitter.split_text(text)<br><br># this will return a list where each chunk is an item in the list<br>print(&quot;chunks:\n&quot;, paragraph_chunks)<br><br>&quot;&quot;&quot;<br>output:<br><br>chunks:<br> [&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.\n\nLorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;, &#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;]&quot;&quot;&quot;</pre><ul><li><strong>Recursive chunking</strong> → This is best for uneven data because it begins by chunking by a bigger method (like paragraphs) and if tokens are too big, it falls back to a smaller chunking method (such as sentences). <em>We can use LangChain’s RecursiveCharacterTextSplitter for this method</em></li></ul><pre># langchain recursive splitting<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br># splitter variable for recursive chunking<br>recursive_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder( # from_tiktoken_encoder lets me choose how I tokenize data<br>    model_name=&quot;gpt-4&quot;, # tokenizer <br>    chunk_size=100,    # MAX number of characters for each chunk<br>    chunk_overlap=0,     # no overlap (usually would but this is just an example)<br>    is_separator_regex=False,<br><br>)<br><br># this is a placeholder value you will most likely upload the text file locally<br>text = &quot;&quot;&quot;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&quot;&quot;&quot;<br><br># split the text <br>recursive_chunks = recursive_splitter.split_text(text)<br><br># this will return a list where each chunk is an item in the list<br>print(&quot;chunks:\n&quot;, recursive_chunks)<br><br>&quot;&quot;&quot;<br>output:<br><br>chunks:<br> [&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;, &#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;, &#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;, &#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;, &#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent&#39;, &#39;taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;]<br>&quot;&quot;&quot;</pre><h4>Here are some structure-aware splitters:</h4><ul><li><strong>Markdown header-aware</strong> → These are best for formatted docs, READMEs, tech blogs, etc. You can define when headers can split (usually basic MD formatting) and then the subsequent text is grouped with that header. Be sure that MD is properly formatted before using this method. <em>We can use LangChain’s MarkdownHeaderTextSplitter, which returns the header as the metadata and the subsequent text as the content.</em></li></ul><pre>from langchain_text_splitters import MarkdownHeaderTextSplitter<br>splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[(&quot;#&quot;,&quot;H1&quot;),(&quot;##&quot;,&quot;H2&quot;),(&quot;###&quot;,&quot;H3&quot;)])<br><br>md_text = &quot;&quot;&quot;<br># HEADER 1<br><br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br><br>## Header 2<br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br>### Header 3<br>Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.<br><br><br><br>&quot;&quot;&quot;<br>md_chunks = splitter.split_text(md_text)<br>print(md_chunks)<br><br>&quot;&quot;&quot;<br>output:<br><br>[Document(metadata={&#39;H1&#39;: &#39;HEADER 1&#39;}, page_content=&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;), Document(metadata={&#39;H1&#39;: &#39;HEADER 1&#39;, &#39;H2&#39;: &#39;Header 2&#39;}, page_content=&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;), Document(metadata={&#39;H1&#39;: &#39;HEADER 1&#39;, &#39;H2&#39;: &#39;Header 2&#39;, &#39;H3&#39;: &#39;Header 3&#39;}, page_content=&#39;Lorem ipsum dolor sit amet consectetur adipiscing elit. Quisque faucibus ex sapien vitae pellentesque sem placerat. In id cursus mi pretium tellus duis convallis. Tempus leo eu aenean sed diam urna tempor. Pulvinar vivamus fringilla lacus nec metus bibendum egestas. Iaculis massa nisl malesuada lacinia integer nunc posuere. Ut hendrerit semper vel class aptent taciti sociosqu. Ad litora torquent per conubia nostra inceptos himenaeos.&#39;)]<br><br>&quot;&quot;&quot;</pre><ul><li><strong>Layout/OCR-aware (PDFs)</strong> → This is best for scanned documents, slides, and other visually complex data. This can be split by page, by section, or by image/table based on what your data requires (Ex: Table and graph heavy data might chunk based on these figures and the text surrounding and describing it). This can be done through document parsing too (which I covered in my last article).</li></ul><h4>Semantic-aware splitters (meaning-driven):</h4><p>These usually use embeddings and similar methods to cut where topics shift.</p><ul><li><strong>Sentence-similarity boundaries </strong>→ This uses embeddings to find the embedding vectors of each sentence (the meaning). Then, it groups sentences together if they are similar and cuts the chunk if the sentences are too different because it signals that it is going into the next topic. <em>LangChain also offers this feature but I used LlamaIndex to show different approaches.</em></li></ul><pre># semantic embedding<br>from llama_index.core.node_parser import SemanticSplitterNodeParser<br>from transformers import AutoTokenizer, AutoModel<br>from llama_index.core.schema import Document<br>from llama_index.embeddings.huggingface import HuggingFaceEmbedding<br><br># embedding model <br>model=HuggingFaceEmbedding(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)<br><br># semantic splitter <br>splitter = SemanticSplitterNodeParser(<br>buffer_size=1, # number of sentences to group together when evaluating semantic similarity<br>breakpoint_percentile_threshold=75, # threshold that must be met to not split<br>embed_model=model<br>)<br><br># sample text<br>AI_text=&quot;&quot;&quot;Irrespective of your personal thoughts or philosophies on the state of AI or its developmental trajectory, AI is here to stay. Not only have the largest adtech companies like Google, Meta, Amazon and Microsoft embraced and integrated their versions of the technology into their tools, but there is also a massive race to democratize AI so small and medium-sized businesses can realize its potential. As we’ve all seen over the past several months, AI has transitioned rapidly from a trendy conversation topic into a key tool supporting many of the digital tools that we use.<br>In much the same vein, AI has also shifted away from being thought of as a pillar of marketing or business strategy, and towards a more sensible role of an embedded tool enabling tactful execution. The issue with approaching AI as a strategy is that instead of finding a solution for a specific problem, you’re now looking for a problem that your pre-determined solution can fix. A rollout of AI without understanding your root business problems is akin to giving every employee a smartphone and magically hoping that productivity and creativity will improve overnight. At the root of it, AI is a tool that unlocks efficiencies, but only if we can identify where and how AI can be applied beneficially.<br>Given the rapid pace of development in the AI field, its not surprising that keeping up with the latest information is difficult. This may cause undue fear of the technology’s potential risks or alienate marketers who want to understand AI’s promise. For better or for worse, the best way to overcome these concerns is to start testing and dabbling with the technology in safe ways. You could ask a tech-savvy friend or family member to give you a walkthrough of ChatGPT or Midjourney. Or ask an AI tool to provide information on a topic you already know about to understand its limitations. However you approach it, keep an open mind and stay curious, and you’ll quickly see what these tools can do and where they ultimately falter. After all, without a firm understanding of what AI is able to offer, you may find yourself unable to guide your team or your business to tailor the technology to your needs, while your competition continues to test, adopt and grow.<br>&quot;&quot;&quot;<br><br><br>document=[Document(text=AI_text)]<br>nodes= splitter.get_nodes_from_documents(document)<br>for i in range(0,3):<br>    print(nodes[i])<br><br>&quot;&quot;&quot;<br>output:<br><br>2025-09-12 23:11:28,819 - INFO - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2<br>Node ID: b78466f2-0872-4814-91f4-29d679bba8f1<br>Text: Irrespective of your personal thoughts or philosophies on the<br>state of AI or its developmental trajectory, AI is here to stay. Not<br>only have the largest adtech companies like Google, Meta, Amazon and<br>Microsoft embraced and integrated their versions of the technology<br>into their tools, but there is also a massive race to democratize AI<br>so small an...<br>Node ID: af56846f-f079-4978-a678-23dafee7e4ca<br>Text: This may cause undue fear of the technology’s potential risks or<br>alienate marketers who want to understand AI’s promise.<br>Node ID: 76d51644-eec5-44f2-90cb-3c0563d087a1<br>Text: For better or for worse, the best way to overcome these concerns<br>is to start testing and dabbling with the technology in safe ways. You<br>could ask a tech-savvy friend or family member to give you a<br>walkthrough of ChatGPT or Midjourney. Or ask an AI tool to provide<br>information on a topic you already know about to understand its<br>limitations.<br>&quot;&quot;&quot;</pre><ul><li><strong>Graph-based segmentation </strong>→ This uses a LLM to create a knowledge graph based on the relationship of words (it uses unstructured data to form its own relationships). Rather than focusing on grouping related sentences, graph-based chunking builds a complex representation of how words connect to each other. This is great for structured content where sentences are strongly related, and this can be stronger than normal semantic chunking (in some cases). But, the cost will be extremely high if you want to effectively carry out this method. <em>This is more experimental so there are no real wrappers for thsi method.</em></li></ul><h4>Code-aware splitters</h4><ul><li><strong>Function/Class based</strong> → This is best for coding languages that modularize their data into classes and functions (Python, Java, C++, etc). This is achieved by sorting code into a tree-like format so it can easily identify functions and classes in programs. Then, each function/class/method are their own chunk so that code doesn’t split in the middle and lose context. <em>The following code uses LangChain’s recursive splitter (but we must define the language); the code splits the previous code block into chunks.</em></li></ul><pre># Code-aware splitters<br><br># previous code example<br>code_1=&quot;&quot;&quot;# semantic embedding<br>from llama_index.core.node_parser import SemanticSplitterNodeParser<br>from transformers import AutoTokenizer, AutoModel<br>from llama_index.core.schema import Document<br>from llama_index.embeddings.huggingface import HuggingFaceEmbedding<br><br># embedding model <br>#tokenizer = AutoTokenizer.from_pretrained(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)<br>#model = AutoModel.from_pretrained(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)<br>model=HuggingFaceEmbedding(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)<br><br># semantic splitter <br>splitter = SemanticSplitterNodeParser(<br>buffer_size=1, # number of sentences to group together when evaluating semantic similarity<br>breakpoint_percentile_threshold=75, # threshold that must be met to not split<br>embed_model=model<br>)<br><br>AI_text=&quot;Irrespective of your personal thoughts or philosophies on the state of AI or its developmental trajectory, AI is here to stay. Not only have the largest adtech companies like Google, Meta, Amazon and Microsoft embraced and integrated their versions of the technology into their tools, but there is also a massive race to democratize AI so small and medium-sized businesses can realize its potential. As we’ve all seen over the past several months, AI has transitioned rapidly from a trendy conversation topic into a key tool supporting many of the digital tools that we use.<br>&quot;<br><br>document=[Document(text=AI_text)]<br>nodes= splitter.get_nodes_from_documents(document)<br>for i in range(0,3):<br>    print(nodes[i])&quot;&quot;&quot;<br><br><br>from langchain_text_splitters import (<br>    Language,<br>    RecursiveCharacterTextSplitter,<br>)<br><br># splitter<br>python_splitter = RecursiveCharacterTextSplitter.from_language(<br>    language=Language.PYTHON, chunk_size=100, chunk_overlap=0<br>)<br><br>python_docs = python_splitter.create_documents([code_1])<br>for i in range (1,10):<br>    print(python_docs[i])<br><br>&quot;&quot;&quot;<br>output:<br>page_content=&#39;from transformers import AutoTokenizer, AutoModel<br>from llama_index.core.schema import Document&#39;<br>page_content=&#39;from llama_index.embeddings.huggingface import HuggingFaceEmbedding&#39;<br>page_content=&#39;# embedding model&#39;<br>page_content=&#39;#tokenizer = AutoTokenizer.from_pretrained(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)&#39;<br>page_content=&#39;#model = AutoModel.from_pretrained(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)&#39;<br>page_content=&#39;model=HuggingFaceEmbedding(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)&#39;<br>page_content=&#39;# semantic splitter <br>splitter = SemanticSplitterNodeParser(&#39;<br>page_content=&#39;buffer_size=1, # number of sentences to group together when evaluating semantic similarity&#39;<br>page_content=&#39;breakpoint_percentile_threshold=75, # threshold that must be met to not split<br>embed_model=model<br>)&#39;<br>&quot;&quot;&quot;</pre><ul><li><strong>Comment and Code</strong> → This method groups the code block and the doc-string above it (since this doc-string likely describes the). This is great for Jupyter Notebooks because the Markdown cell likely desribes the code cell that follows it. This may not be great for normal code files (in that case you would use the previous method).</li></ul><p>Now that we have reviewed the theory, let’s see this in practice.</p><h3>PDF Chunking</h3><p>For the PDF, I came up with two good chunking methods I wanted to test, and I created two embeddings for each chunking method (one with each model)</p><ul><li><strong>Pair A</strong> → Respects the structure of the slides (such as sections and paragraphs), if a block is over 1000 tokens it makes the chunk smaller, it overlaps by around 150 tokens; why: <em>cheap, stable, simple, and makes sense logically based on how we read slides</em></li></ul><pre># pdf_pair_a<br>from pathlib import Path<br>import json, re<br>from sentence_transformers import SentenceTransformer<br>import numpy as np<br><br>SRC = Path(&quot;unstructered_markdown_path_here&quot;)<br>OUT = Path(&quot;outputs/pdf_pair_a.jsonl&quot;); OUT.parent.mkdir(parents=True, exist_ok=True)<br><br>def tokenize(s):<br>    return re.findall(r&quot;\S+&quot;, s or &quot;&quot;)<br><br>def split_recursive(text, max_tokens=1000):<br>    # try paragraphs -&gt; sentences -&gt; words until &lt;= max_tokens<br>    paras = [p.strip() for p in (text or &quot;&quot;).split(&quot;\n\n&quot;) if p.strip()]<br>    chunks = []<br>    for p in paras:<br>        if len(tokenize(p)) &lt;= max_tokens:<br>            chunks.append(p); continue<br>        # sentences<br>        sents = re.split(r&quot;(?&lt;=[.!?])\s+&quot;, p)<br>        cur = []<br>        for s in sents:<br>            if len(tokenize(&quot; &quot;.join(cur+[s]))) &lt;= max_tokens:<br>                cur.append(s)<br>            else:<br>                if cur: chunks.append(&quot; &quot;.join(cur)); cur = [s]<br>        if cur: chunks.append(&quot; &quot;.join(cur))<br>    return chunks<br><br>def add_overlap(chunks, overlap_tokens=150):<br>    out = []<br>    prev_tail = []<br>    for ch in chunks:<br>        toks = tokenize(ch)<br>        merged = &quot; &quot;.join(prev_tail + toks)<br>        out.append(merged)<br>        prev_tail = toks[-overlap_tokens:] if len(toks) &gt; overlap_tokens else toks<br>    return out<br><br># read unstructured markdown (jsonl per page)<br>records = []<br>with SRC.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for line in f:<br>        obj = json.loads(line)<br>        if obj.get(&quot;metadata&quot;, {}).get(&quot;element_type&quot;) != &quot;PageMarkdown&quot;:<br>            continue<br>        page = obj[&quot;metadata&quot;].get(&quot;page&quot;)<br>        text = obj.get(&quot;text&quot;, &quot;&quot;)<br>        # recursive split and overlap<br>        base = split_recursive(text, max_tokens=1000)<br>        final = add_overlap(base, overlap_tokens=150)<br>        for i, ch in enumerate(final):<br>            records.append({&quot;text&quot;: ch, &quot;metadata&quot;: {&quot;page&quot;: page, &quot;splitter&quot;: &quot;A_recursive&quot;}})<br><br># saves chunks<br>with OUT.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for r in records:<br>        f.write(json.dumps(r, ensure_ascii=False) + &quot;\n&quot;)<br><br># embed<br>def embed_all(texts, model_name):<br>    model = SentenceTransformer(model_name)<br>    vecs = model.encode(texts, normalize_embeddings=True, convert_to_numpy=True)<br>    return vecs.astype(&quot;float32&quot;)<br><br>texts = [r[&quot;text&quot;] for r in records]<br>minilm = &quot;sentence-transformers/all-MiniLM-L6-v2&quot;<br>bge    = &quot;BAAI/bge-small-en-v1.5&quot;<br><br>vecs_minilm = embed_all(texts, minilm)<br>vecs_bge    = embed_all(texts, bge)<br><br>print(&quot;chunks=&quot;, len(texts), &quot;, MiniLM shape=&quot;, vecs_minilm.shape, &quot;, BGE shape:&quot;, vecs_bge.shape)</pre><ul><li><strong>Pair B</strong> → Reads each sentence sing an embedding model and chunks sentences together if their <strong>cosine distance </strong>(similarity) is high. Once similarity is too low, a new chunk is started. It caps at 1200 tokens so chunks are not too big. Pages that the document parser found too hard to read (look at my previous part!) have a higher threshold to meet for similarity; Why: <em>groups topics together, able to bypass messy OCR and unreliable layout</em></li></ul><pre># pdf_pair_b<br>from pathlib import Path<br>import json, re, numpy as np<br>from sentence_transformers import SentenceTransformer<br>from numpy.linalg import norm<br><br>SRC = Path(&quot;path_to_unstructured_markdown&quot;)<br>OUT = Path(&quot;outputs/pdf_pair_b.jsonl&quot;); OUT.parent.mkdir(parents=True, exist_ok=True)<br><br>HARD = {9,17,28,22,13,12,11,14,19,3,21}  # pages you found tricky<br>THRESH_HARD = 0.18   # lower distance = tighter semantic grouping<br>THRESH_EASY = 0.22<br>MAX_TOK = 1200<br><br>tok = lambda s: re.findall(r&quot;\S+&quot;, s or &quot;&quot;)<br><br>def cos_dist(a, b):<br>    a = a / (norm(a) + 1e-9); b = b / (norm(b) + 1e-9)<br>    return 1.0 - float(np.dot(a, b))<br><br>enc = SentenceTransformer(&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)  # use one model to *decide* boundaries (original code uses both one at a time)<br><br>def chunk_page(text, page):<br>    th = THRESH_HARD if page in HARD else THRESH_EASY<br>    sents = re.split(r&quot;(?&lt;=[.!?])\s+&quot;, text or &quot;&quot;)<br>    live, live_vec = [], None<br>    chunks = []<br>    for s in sents:<br>        s = s.strip()<br>        if not s: continue<br>        s_vec = enc.encode([s], normalize_embeddings=True, convert_to_numpy=True)[0]<br>        if not live:<br>            live, live_vec = [s], s_vec<br>            continue<br>        cand = &quot; &quot;.join(live + [s])<br>        # cut if semantic shift OR size too big<br>        if cos_dist(live_vec, s_vec) &gt; th or len(tok(cand)) &gt; MAX_TOK:<br>            chunks.append(&quot; &quot;.join(live))<br>            live, live_vec = [s], s_vec<br>        else:<br>            live.append(s)<br>            live_vec = (live_vec + s_vec) / 2.0<br>    if live: chunks.append(&quot; &quot;.join(live))<br>    return chunks<br><br>recs = []<br>with SRC.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for line in f:<br>        obj = json.loads(line)<br>        if obj.get(&quot;metadata&quot;, {}).get(&quot;element_type&quot;) != &quot;PageMarkdown&quot;:<br>            continue<br>        page = int(obj[&quot;metadata&quot;].get(&quot;page&quot;, 0) or 0)<br>        text = obj.get(&quot;text&quot;, &quot;&quot;)<br>        for ch in chunk_page(text, page):<br>            recs.append({&quot;text&quot;: ch, &quot;metadata&quot;: {&quot;page&quot;: page, &quot;splitter&quot;: &quot;B_semantic&quot;}})<br><br>with OUT.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for r in recs:<br>        f.write(json.dumps(r, ensure_ascii=False) + &quot;\n&quot;)<br><br># embed<br>def embed_all(txts, model):<br>    m = SentenceTransformer(model)<br>    return m.encode(txts, normalize_embeddings=True, convert_to_numpy=True).astype(&quot;float32&quot;)<br><br>T = [r[&quot;text&quot;] for r in recs]<br>vecs_minilm = embed_all(T, &quot;sentence-transformers/all-MiniLM-L6-v2&quot;)<br>vecs_bge    = embed_all(T, &quot;BAAI/bge-small-en-v1.5&quot;)<br>print(&quot;B: chunks:&quot;, len(T), &quot;, MiniLM:&quot;, vecs_minilm.shape, &quot;, BGE:&quot;, vecs_bge.shape)</pre><h3>Notebook chunking</h3><p>These are the two chunking methods I used:</p><ul><li><strong>Pair A</strong> → Markdown headers are grouped with the code cells they are explaining, keeps outputs in chunk to show what code is supposed to do; Why: <em>improves quality and relevance of chunks</em></li></ul><pre># nb_pair_a.py<br>from pathlib import Path<br>import json, re<br>from sentence_transformers import SentenceTransformer<br>import numpy as np<br><br>SRC = Path(&quot;nbformat_jsonl_path&quot;)<br>OUT = Path(&quot;outputs/nb_pair_a.jsonl&quot;); OUT.parent.mkdir(parents=True, exist_ok=True)<br><br>def is_header(md):<br>    return bool(re.match(r&quot;^\s{0,3}#{1,6}\s&quot;, md or &quot;&quot;))<br><br># collect cells by (type, text)<br>cells = []<br>with SRC.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for line in f:<br>        obj = json.loads(line)<br>        text = obj.get(&quot;text&quot;,&quot;&quot;); meta = obj.get(&quot;metadata&quot;,{})<br>        cells.append((meta.get(&quot;cell_type&quot;), text))<br><br># groups header markdown + next code/markdown until next header<br>chunks = []<br>buffer = []<br>for ctype, text in cells:<br>    if ctype == &quot;markdown&quot; and is_header(text):<br>        if buffer: chunks.append(&quot;\n\n&quot;.join(buffer)); buffer = []<br>    buffer.append(text)<br>if buffer: chunks.append(&quot;\n\n&quot;.join(buffer))<br><br># embed<br>with OUT.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for ch in chunks:<br>        f.write(json.dumps({&quot;text&quot;: ch, &quot;metadata&quot;: {&quot;splitter&quot;: &quot;NB_A_struct&quot;}}, ensure_ascii=False) + &quot;\n&quot;)<br><br>model_names = [&quot;sentence-transformers/all-MiniLM-L6-v2&quot;, &quot;BAAI/bge-small-en-v1.5&quot;]<br>for name in model_names:<br>    m = SentenceTransformer(name)<br>    vecs = m.encode(chunks, normalize_embeddings=True, convert_to_numpy=True).astype(&quot;float32&quot;)<br>    print(&quot;NB-A --&gt; &quot;, name, vecs.shape)</pre><ul><li><strong>Pair B</strong> → It uses the sliding window method, meaning it groups together three cells and a time and overlaps one cell each time so the context stays for each chunk</li></ul><pre># nb_pair_b.py<br>import argparse, json, re<br>from pathlib import Path<br><br>SRC  = &quot;path_to_nbformat_jsonl&quot;<br>OUTS = &quot;outputs&quot;; OUTS.mkdir(parents=True, exist_ok=True)<br><br># token counter <br>try:<br>    import tiktoken<br>    enc = tiktoken.get_encoding(&quot;cl100k_base&quot;)<br>    count_tokens = lambda s: len(enc.encode(s or &quot;&quot;))<br>except Exception:<br>    count_tokens = lambda s: len(re.findall(r&quot;\S+&quot;, s or &quot;&quot;))<br><br>CODE_FENCE = &quot;```&quot;<br>def looks_like_code(s: str) -&gt; bool:<br>    h = (s or &quot;&quot;)[:400]<br>    return &quot;```&quot; in h or re.search(r&quot;\b(def |class |import |from |for\s+\w+\s+in |if __name__ == [&#39;\&quot;]__main__[&#39;\&quot;])&quot;, h)<br><br>def fence_if_code(s: str) -&gt; str:<br>    return s if CODE_FENCE in (s or &quot;&quot;) else (f&quot;```python\n{s}\n```&quot; if looks_like_code(s) else s)<br><br>def load_cells(p: Path):<br>    cells = []<br>    with p.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for line in f:<br>            obj = json.loads(line)<br>            cells.append({&quot;text&quot;: obj.get(&quot;text&quot;,&quot;&quot;) or &quot;&quot;, &quot;meta&quot;: obj.get(&quot;metadata&quot;,{}) or {}})<br>    return cells<br><br>def sliding_windows(cells, n, s):<br>    i, out = 0, []<br>    while i &lt; len(cells):<br>        win = cells[i:i+n]<br>        if not win: break<br>        out.append(win)<br>        if i + n &gt;= len(cells): break<br>        i += s<br>    return out<br><br>def join_cells(win):<br>    return &quot;\n\n&quot;.join(fence_if_code(c[&quot;text&quot;]) for c in win).strip()<br><br>def fallback_per_cell(win, max_toks):<br>    out = []<br>    for c in win:<br>        t = fence_if_code(c[&quot;text&quot;])<br>        if count_tokens(t) &lt;= max_toks:<br>            out.append(t); continue<br>        # split paragraphs then lines if needed<br>        for para in [p for p in re.split(r&quot;\n{2,}&quot;, t) if p.strip()]:<br>            if count_tokens(para) &lt;= max_toks: out.append(para); continue<br>            buf = []<br>            for ln in [x for x in para.splitlines() if x.strip()]:<br>                cand = (&quot;\n&quot;.join(buf+[ln])).strip()<br>                if count_tokens(cand) &lt;= max_toks: buf.append(ln)<br>                else:<br>                    if buf: out.append(&quot;\n&quot;.join(buf))<br>                    buf = [ln]<br>            if buf: out.append(&quot;\n&quot;.join(buf))<br>    return out<br><br>def run(N, S, K):<br>    cells = load_cells(SRC)<br>    wins = sliding_windows(cells, N, S)<br>    out_path = OUTS / f&quot;nbformat_chunks_B_jsonl_win{N}s{S}_max{K}.jsonl&quot;<br>    w = 0<br>    with out_path.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for widx, win in enumerate(wins):<br>            merged = join_cells(win)<br>            if count_tokens(merged) &lt;= K:<br>                rec = {<br>                    &quot;text&quot;: merged,<br>                    &quot;metadata&quot;: {<br>                        &quot;splitter&quot;: &quot;NB_B_window&quot;,<br>                        &quot;cells_per_chunk&quot;: N, &quot;stride&quot;: S, &quot;window_index&quot;: widx,<br>                        &quot;cell_indices&quot;: [c[&quot;meta&quot;].get(&quot;cell_index&quot;) for c in win],<br>                        &quot;cell_types&quot;: [c[&quot;meta&quot;].get(&quot;cell_type&quot;) for c in win],<br>                        &quot;source_path&quot;: str(SRC),<br>                    },<br>                }<br>                f.write(json.dumps(rec, ensure_ascii=False) + &quot;\n&quot;); w += 1<br>            else:<br>                for small in fallback_per_cell(win, K):<br>                    rec = {<br>                        &quot;text&quot;: small,<br>                        &quot;metadata&quot;: {<br>                            &quot;splitter&quot;: &quot;NB_B_window_fallback_per_cell&quot;,<br>                            &quot;cells_per_chunk&quot;: 1, &quot;stride&quot;: S, &quot;window_index&quot;: widx,<br>                            &quot;source_path&quot;: str(SRC),<br>                        },<br>                    }<br>                    f.write(json.dumps(rec, ensure_ascii=False) + &quot;\n&quot;); w += 1<br>    print(f&quot;[B-window] Wrote {w} chunk(s) -&gt; {out_path}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    ap = argparse.ArgumentPars</pre><h3>Conclusion</h3><p>This part of my RAG journey taught me plenty of things I didn’t know. I had no idea there were so many steps to create an effective RAG model and I discovered there were many options for each step of the process.</p><p>In my <strong>next article, </strong>I will cover some embedding strategies and my final article will test each individual method I used to see which one works the best. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6390b6e85ba8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My RAG Journey: Part 1 — Document Parsing]]></title>
            <link>https://medium.com/@ishveen/my-rag-journey-part-1-document-parsing-2fa832866ea9?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fa832866ea9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 02:03:16 GMT</pubDate>
            <atom:updated>2025-08-20T02:03:16.553Z</atom:updated>
            <content:encoded><![CDATA[<h3>My RAG Journey: Part 1 — Document Parsing</h3><p>Hello! I’m 15 years old, and I am trying to learn as much as I can about <strong>Generative AI</strong>. One topic that has interested me is RAG. Before you can build great RAG systems, you need clean text from your source files. This article answers the question: “<strong>How do we reliably parse PDFs and Python notebooks before we feed them to an LLM?</strong>” Read on to learn more on which parsers worked for me!</p><p>Goal: Turn PDFs + .ipynb notebooks into clean text/markdown for RAG or analysis, and then <strong>measure</strong> which tool does best.</p><ul><li>For the PDF, I used the following parsers: Unstructured (fast and hi-res), LlamaParse, PyMuPDF, and PyPDF</li><li>For the .ipynb notebook, I used: nbformat and Jupytext</li></ul><h3><strong>What is a document parser?</strong></h3><p>When you give an LLM (or any other AI) and PDF or code notebook, it can’t see it like any human can. It needs a <strong>parser </strong>that gives structured text (layouts and content) so you can feed it to an LLM, index it for searching, or analyze it. Good parsers should capture all the text, order the text properly, and (ideally) mark some structure (like headings, lists, tables, paragraphs, etc.)</p><h3>PDF Parsers Analysis</h3><ul><li>PyPDF → It is a pure-Python library for working with PDFs. There is no computer vision involved, it only looks from the PDF’s internal text objects. Although the lack of computer vision makes it light, fast, and easy to use, it can often miss smaller text and is not compatible with complex PDFs. Text on images may be missed and the reading order may be inconsistent. It is the most simple one.</li></ul><pre>with open(&quot;output_file&quot;, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for i, page in enumerate(reader.pages, start=1):  # loops through each page<br>        txt = page.extract_text() or &quot;&quot;   # NOTE: might miss text in images (no OCR)<br>        f.write(json.dumps(rec(txt, {. # create json line<br>            &quot;source_path&quot;: str(PDF_FILE),<br>            &quot;parser&quot;: &quot;pypdf&quot;,<br>            &quot;page&quot;: i,<br>            &quot;block_index&quot;: 0,  # no blocks since we only get page-level text<br>            &quot;created_at&quot;: int(time.time()),<br>            &quot;hash&quot;: fhash,<br>        }), ensure_ascii=False) + &quot;\n&quot;)</pre><ul><li>PyMuPDF (fitz<em> </em>library) → The library MuPDF is a C library known for rendering PDF documents. This is a Python binding for MuPDF, meaning it enables Python to utilize libraries written in other programming languages. It does not utilize Optical Character Recognition (OCR) or computer vision (like PyPDF) but it extracts text with coordinates that create “boxes.” It is often more complete and has a better layout than PyPDF, but it still is not true OCR/computer vision</li></ul><pre>with open(&quot;output_file&quot;, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for pno in range(len(doc)):  <br>        page = doc[pno]<br>        # each block: (x0, y0, x1, y1, text, block_no, ...)<br>        for b_idx, block in enumerate(page.get_text(&quot;blocks&quot;)):<br>            x0, y0, x1, y1, text, *_ = block<br>            f.write(json.dumps(rec(text, {.  <br>                &quot;source_path&quot;: str(PDF_FILE),<br>                &quot;parser&quot;: &quot;pymupdf&quot;,<br>                &quot;page&quot;: pno + 1,<br>                &quot;block_index&quot;: b_idx,<br>                &quot;coordinates&quot;: [x0, y0, x1, y1], # x0, y0 is top left corner of block while x1, y1 is bottom-right corner<br>                &quot;created_at&quot;: int(time.time()),<br>                &quot;hash&quot;: fhash,<br>            }), ensure_ascii=False) + &quot;\n&quot;)</pre><ul><li>Unstructured (fast) → There are two modes: “fast” and “hi-res.” Fast mode uses simple text extraction (likely uses PyPDF or PyMuPDF underneath) and splits PDF into elements (Title, Lists, etc.). It is easy to install and use, but it can miss tiny text and is less accurate on complex text. This is more still pretty fast but it does not perform as well as hi-res mode</li></ul><pre># gives with markdown formatting<br>with open(OUT_MD_JSONL, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for i, md in enumerate(per_page_md, start=1): # per_page_md = markdown<br>            f.write(json.dumps(rec(md, {<br>                &quot;source_path&quot;: str(PDF_FILE),<br>                &quot;file_type&quot;: &quot;pdf&quot;,<br>                &quot;parser&quot;: &quot;unstructured_fast_markdown&quot;,<br>                &quot;element_type&quot;: &quot;PageMarkdown&quot;,<br>                &quot;page&quot;: i,<br>                &quot;block_index&quot;: 0,<br>                &quot;created_at&quot;: now,<br>                &quot;hash&quot;: fhash,<br>            }), ensure_ascii=False) + &quot;\n&quot;)<br><br># gives raw text (no formatting)<br>with open(OUT_TEXT_JSONL, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for i, txt in enumerate(per_page_txt, start=1): # per_page_text = plain text<br>        f.write(json.dumps(rec(txt, {<br>            &quot;source_path&quot;: str(PDF_FILE),<br>            &quot;file_type&quot;: &quot;pdf&quot;,<br>            &quot;parser&quot;: &quot;unstructured_fast_text&quot;,<br>            &quot;element_type&quot;: &quot;PageText&quot;,<br>            &quot;page&quot;: i,<br>            &quot;block_index&quot;: 0,<br>            &quot;created_at&quot;: now,<br>            &quot;hash&quot;: fhash,<br>        }), ensure_ascii=False) + &quot;\n&quot;)<br># both md and text are creating to make evaluation of parsers easier<br>OUT_MD.write_text(&quot;\n\n&quot;.join(per_page_md), encoding=&quot;utf-8&quot;)</pre><ul><li>Unstructured (hi-res) → This version uses a vision model too. It uses a layout detection model on images and OCR for text. It gives a much better structure than the fast version and is able to parse smaller text, but it is much heavier and slower.</li></ul><pre>elements = partition_pdf(<br>    str(PDF_FILE),<br>    strategy=&quot;hi_res&quot;,<br>    infer_table_structure=True, # reconstruct tables if present<br>)<br><br>#...<br><br>def el_to_md(e):<br>  # makes elements into markdown based on tool output<br>  if et in (&quot;Title&quot;, &quot;Header&quot;):<br>        return f&quot;# {t}&quot;<br>  # ...<br><br>with open(OUT_MD_JSONL, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for i, md in enumerate(per_page_md, start=1):<br>        f.write(json.dumps(rec(md, {<br>            &quot;source_path&quot;: str(PDF_FILE),<br>            &quot;file_type&quot;: &quot;pdf&quot;,<br>            &quot;parser&quot;: &quot;unstructured_fast_markdown&quot;,<br>            &quot;element_type&quot;: &quot;PageMarkdown&quot;,<br>            &quot;page&quot;: i,<br>            &quot;block_index&quot;: 0,<br>            &quot;created_at&quot;: now,<br>            &quot;hash&quot;: fhash,<br>        }), ensure_ascii=False) + &quot;\n&quot;)<br><br>with open(OUT_TEXT_JSONL, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for i, txt in enumerate(per_page_txt, start=1):<br>        f.write(json.dumps(rec(txt, {<br>            &quot;source_path&quot;: str(PDF_FILE),<br>            &quot;file_type&quot;: &quot;pdf&quot;,<br>            &quot;parser&quot;: &quot;unstructured_fast_text&quot;,<br>            &quot;element_type&quot;: &quot;PageText&quot;,<br>            &quot;page&quot;: i,<br>            &quot;block_index&quot;: 0,<br>            &quot;created_at&quot;: now,<br>            &quot;hash&quot;: fhash,<br>          }), ensure_ascii=False) + &quot;\n&quot;)</pre><ul><li>LlamaParse → All the other parsers are open-source, but LlamaParse is the only propriety parser (but it offers a free tier). It has it’s own server-side models and you use an API to use these models to parse your PDF. It returns clean markdown or raw JSON. The per-page text layout makes it compatible with RAG and is the most complete and clean, but it can cost money (which can be costly when parsing thousands of documents). You must also have a stable internet connection since you will be using an API.</li></ul><pre>parser = LlamaParse(   # configure LlamaParse client<br>        api_key=api_key,<br>        num_workers=1, # gentler on network<br>        verbose=True,<br>        language=&quot;en&quot;,<br>        max_timeout=180, # wait long enough for job to finish<br>    )<br># ...<br><br>pages = getattr(result, &quot;pages&quot;, None) or []<br>print(f&quot;LlamaParse reported pages: {len(pages)}&quot;)  # just in case nothing is returned<br><br># ...<br><br>md_docs = result.get_markdown_documents(split_by_page=True) or \<br>              [type(&quot;Doc&quot;, (), {&quot;text&quot;: (getattr(p, &quot;md&quot;, &quot;&quot;) or &quot;&quot;)}) for p in pages]<br><br>with open(OUT_MD_JSONL, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for i, d in enumerate(md_docs, start=1):<br>        f.write(json.dumps(rec(d.text, {<br>            &quot;source_path&quot;: str(PDF_FILE),<br>            &quot;file_type&quot;: &quot;pdf&quot;,<br>            &quot;parser&quot;: &quot;llamaparse_markdown&quot;,<br>            &quot;element_type&quot;: &quot;PageMarkdown&quot;,<br>            &quot;page&quot;: i,<br>            &quot;block_index&quot;: 0,<br>            &quot;created_at&quot;: now,<br>            &quot;hash&quot;: fhash,<br>        }), ensure_ascii=False) + &quot;\n&quot;)<br><br>OUT_MD.write_text(&quot;\n\n&quot;.join((d.text or &quot;&quot;).strip() for d in md_docs), encoding=&quot;utf-8&quot;)</pre><p>Based on this surface level research:</p><ul><li>PyMuPDF, PyPDF, and Unstructured (fast) are best if you want quick parsing, but they may not give the best results if your PDFs are too complex</li><li>Unstructured (hi-res) and LlamaParse much slower, but they yield the best results.</li><li>PyPDF is best for pure-python and basic parsing, PyMuPDF is best for fast, layout aware tasks, Unstructured (fast) is best for fast and structured elements, Unstructured (hi-res) is best for complex docs, and LlamaParse is best for clean Markdown for each page</li></ul><h3>Notebook Parser Analysis</h3><ul><li>nbformat → It is the official notebook parser. Jupyter Notebook files are just JSON documents that store the code, Markdown, and output. This parser splits the markdown, code, and outputs into JSONL format (One JSON object per line). It is better for human context since it includes outputs.</li></ul><pre>with open(&quot;output_file&quot;, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>    for idx, cell in enumerate(nb.cells):  # iterates over each cell<br>        body = cell.source or &quot;&quot;<br>        # shows outputs for code cells<br>        if cell.cell_type == &quot;code&quot;:<br>            outs = []<br>            for o in cell.get(&quot;outputs&quot;, []):<br>                if o.get(&quot;output_type&quot;) == &quot;stream&quot;:<br>                    outs.append(o.get(&quot;text&quot;, &quot;&quot;))    # grab printed text<br>                elif o.get(&quot;output_type&quot;) in {&quot;execute_result&quot;, &quot;display_data&quot;}:<br>                    txt = o.get(&quot;data&quot;, {}).get(&quot;text/plain&quot;)  # get plain text<br>                    if isinstance(txt, str): outs.append(txt)<br>            if outs: body = body.rstrip() + &quot;\n\n&quot; + &quot;\n&quot;.join(map(str, outs))<br>        f.write(json.dumps(rec(body, {<br>            &quot;source_path&quot;: str(IPYNB_FILE),<br>            &quot;parser&quot;: &quot;nbformat&quot;,<br>            &quot;cell_type&quot;: cell.cell_type,<br>            &quot;cell_index&quot;: idx,<br>            &quot;execution_count&quot;: cell.get(&quot;execution_count&quot;),  # Execution count (In [#]) may be none if never run<br>            &quot;hash&quot;: fhash,<br>        }), ensure_ascii=False) + &quot;\n&quot;)</pre><ul><li>jupytext → Very similar to nbformat, but it does not return outputs of cells (making it better for version control and lighter). It returns a text-based output (my code made two JSONL files for Python and Markdown split based on the text output)</li></ul><pre>md_rows, py_rows = [], []. # makes lines for markdown and python seperately<br>    for idx, cell in enumerate(nb.cells):<br>        base_meta = {<br>            &quot;source_path&quot;: str(IPYNB),<br>            &quot;file_type&quot;: &quot;ipynb&quot;,<br>            &quot;cell_index&quot;: idx,<br>            &quot;cell_type&quot;: cell[&quot;cell_type&quot;],<br>            &quot;language&quot;: lang if cell[&quot;cell_type&quot;] == &quot;code&quot; else None, # only needed for code cells<br>            &quot;execution_count&quot;: cell.get(&quot;execution_count&quot;),<br>            &quot;hash&quot;: fhash,<br>            &quot;created_at&quot;: created,<br>            &quot;parser&quot;: &quot;jupytext&quot;,<br>        }<br>        md_rows.append(make_rec(render_cell_md(cell, lang), base_meta | {&quot;variant&quot;: &quot;md&quot;}))<br>        py_rows.append(make_rec(render_cell_py_percent(cell), base_meta | {&quot;variant&quot;: &quot;py&quot;}))</pre><p>Field meanings:</p><ul><li>source_path: absolute/relative path of the original file.</li><li>parser: the tool or mode used: &quot;pypdf&quot;, &quot;pymupdf&quot;, &quot;unstructured_fast_text&quot;, &quot;unstructured_hires_markdown&quot;, &quot;llamaparse_markdown&quot;, &quot;nbformat&quot;, &quot;jupytext&quot;, etc.</li><li>page: 1-based page number (PDFs only).</li><li>block_index: index of the block within a page (or 0 if you store just page-level).</li><li>coordinates: [x0, y0, x1, y1] bounding box in PDF <strong>points</strong> (PyMuPDF).</li><li>element_type: semantic label like &quot;PageMarkdown&quot;, &quot;PageText&quot;, or Unstructured categories (&quot;Title&quot;, &quot;ListItem&quot;, etc.) when relevant.</li><li>cell_type: &quot;markdown&quot; / &quot;code&quot; (notebooks).</li><li>execution_count: Jupyter execution order (notebooks).</li><li>language: &quot;python&quot; for code cells if applicable (useful for multi-language notebooks).</li><li>hash: hash of the source file to detect input changes.</li><li>created_at: Unix timestamp when the record was generated.</li></ul><h3>The Evaluation Process</h3><p>For reference, the PDF and Notebooks were from these repositories:</p><ul><li>PDF → <a href="https://github.com/dcai-course/dcai-course">https://github.com/dcai-course/dcai-course</a></li><li>Notebook → <a href="https://github.com/dcai-course/dcai-lab/tree/master">https://github.com/dcai-course/dcai-lab/tree/master</a></li><li>The PDF lesson and corresponding notebook are from a lesson on data curation for LLMs</li></ul><p>Instead of comparing the parsers to some “golden standard,” I decided to compare them with each other. The evaluation code for the PDF parsers found the similarities between two parsers and showed the differences. For the top 12 hardest pages (most disagreement), a central (medoid) parser was chosen (the parser that is most similar to all the others). I could see the differences and similarities between two parsers for each of these pages, and I would compare it with the original PDF to see which is most accurate. As for the notebooks, they were compared to each other.</p><p>Below is the full code for the evaluation of all the PDF parsers:</p><pre># pdf_eval.py<br># Compare multiple PDF parsers to each other by page (no baseline).<br># Outputs:<br>#   - page_scores.csv: per-page disagreement + who’s medoid<br>#   - spot_check.md: a short human-friendly report + unified diffs for the most disputed pages<br><br>import json, csv, difflib, math<br>from pathlib import Path<br>from collections import defaultdict<br><br>try:<br>    from rapidfuzz.distance import Levenshtein<br>    HAVE_RAPIDFUZZ = True<br>except Exception:<br>    HAVE_RAPIDFUZZ = False<br><br># label -&gt; path to JSONL output (one record per page) for each parser<br>CANDIDATES = {<br>    &quot;pypdf&quot;: &quot;parsed_docs/pypdf/pdf_pypdf.jsonl&quot;,<br>    &quot;pymupdf&quot;: &quot;parsed_docs/pymupdf/pdf_pymupdf.jsonl&quot;,<br>    &quot;unstructured-fast-text&quot;: &quot;parsed_docs/unstructured/fast/unstructured_fast_text.jsonl&quot;,<br>    &quot;unstructured-hires-text&quot;: &quot;parsed_docs/unstructured/hi-res/unstructured_text.jsonl&quot;,<br>    &quot;llamaparse-text&quot;: &quot;parsed_docs/llamaparse/llamaparse_text.jsonl&quot;,<br>}<br>OUT_DIR = Path(&quot;eval&quot;); OUT_DIR.mkdir(parents=True, exist_ok=True)<br>PAGE_SCORES_CSV = OUT_DIR / &quot;page_scores.csv&quot;<br>SPOT_CHECK_MD = OUT_DIR / &quot;spot_check.md&quot;<br>TOP_K_PAGES = 8   # how many “most-disagreed” pages to include in spot_check.md<br># --------------------------------------------------------<br><br>def norm_text(s: str) -&gt; str:<br>    # Light normalization: lowercase + collapse whitespace.<br>    return &quot; &quot;.join((s or &quot;&quot;).lower().split())<br><br>def read_pages(jsonl_path: str) -&gt; dict[int, str]:<br>    &quot;&quot;&quot;Return {page_number: text} from a JSONL file that has {&quot;text&quot;:..., &quot;metadata&quot;:{&quot;page&quot;:...}}.&quot;&quot;&quot;<br>    page_text = {}<br>    p = Path(jsonl_path)<br>    if not p.exists():<br>        return page_text<br>    with p.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        for line in f:<br>            obj = json.loads(line)<br>            text = obj.get(&quot;text&quot;, &quot;&quot;)<br>            meta = obj.get(&quot;metadata&quot;, {})<br>            page = int(meta.get(&quot;page&quot;, 0) or 0)<br>            if page &gt; 0:<br>                page_text[page] = norm_text(text)<br>    return page_text<br><br>def sim(a: str, b: str) -&gt; float:<br>    &quot;&quot;&quot;Return similarity in [0,1].&quot;&quot;&quot;<br>    if HAVE_RAPIDFUZZ:<br>        # normalized distance in [0,1]; convert to similarity<br>        d = Levenshtein.normalized_distance(a, b)  # 0=same, 1=totally different<br>        return 1.0 - d<br>    # fallback: difflib ratio (quick &amp; built-in)<br>    return difflib.SequenceMatcher(None, a, b).ratio()<br><br>def pairwise_dist(texts: list[str]) -&gt; list[list[float]]:<br>    &quot;&quot;&quot;Return NxN distance matrix (1-sim).&quot;&quot;&quot;<br>    n = len(texts)<br>    D = [[0.0]*n for _ in range(n)]<br>    for i in range(n):<br>        for j in range(i+1, n):<br>            s = sim(texts[i], texts[j])<br>            d = 1.0 - s<br>            D[i][j] = D[j][i] = d<br>    return D<br><br>def medoid_index(D: list[list[float]]) -&gt; int:<br>    &quot;&quot;&quot;Index with minimal average distance to all others (the medoid).&quot;&quot;&quot;<br>    scores = [sum(row)/max(1, len(row)-1) for row in D]<br>    return min(range(len(scores)), key=lambda i: scores[i])<br><br>def unified_diff(a: str, b: str, from_label: str, to_label: str, n=3) -&gt; str:<br>    return &quot;\n&quot;.join(difflib.unified_diff(<br>        a.splitlines(), b.splitlines(),<br>        fromfile=from_label, tofile=to_label, lineterm=&quot;&quot;, n=n)<br>    )<br><br>def main():<br>    # 1) Load per-page texts for all parsers<br>    per_label_pages: dict[str, dict[int, str]] = {lbl: read_pages(path) for lbl, path in CANDIDATES.items()}<br>    all_pages = sorted(set().union(*[set(d.keys()) for d in per_label_pages.values()]))<br>    labels = list(per_label_pages.keys())<br><br>    # 2) Compute disagreement per page and pick medoid parser for that page<br>    rows = []<br>    worst_pages = []  # keep (page, disagreement_score, details)<br>    for page in all_pages:<br>        texts = []<br>        present_labels = []<br>        for lbl in labels:<br>            if page in per_label_pages[lbl]:<br>                present_labels.append(lbl)<br>                texts.append(per_label_pages[lbl][page])<br><br>        if len(texts) &lt; 2:<br>            # don’t evaluate pages where fewer than 2 parsers produced output<br>            continue<br><br>        D = pairwise_dist(texts)<br>        midx = medoid_index(D)<br>        medoid_lbl = present_labels[midx]<br>        # how far is each parser from the medoid?<br>        dists_from_medoid = [(present_labels[j], D[midx][j]) for j in range(len(present_labels))]<br>        dists_from_medoid.sort(key=lambda t: t[1], reverse=True)<br>        far_lbl, far_dist = dists_from_medoid[0]<br>        mean_dist = sum(D[midx]) / (len(D[midx]) - 1)<br><br>        rows.append({<br>            &quot;page&quot;: page,<br>            &quot;medoid_label&quot;: medoid_lbl,<br>            &quot;mean_dist_from_medoid&quot;: round(mean_dist, 3),<br>            &quot;farthest_label&quot;: far_lbl,<br>            &quot;farthest_dist&quot;: round(far_dist, 3),<br>            &quot;num_candidates&quot;: len(present_labels),<br>        })<br>        # simple “disagreement score”: max distance to medoid<br>        worst_pages.append((page, far_dist, {<br>            &quot;present_labels&quot;: present_labels,<br>            &quot;medoid_label&quot;: medoid_lbl,<br>            &quot;farthest_label&quot;: far_lbl,<br>            &quot;texts&quot;: texts,<br>            &quot;midx&quot;: midx,<br>        }))<br><br>    # 3) Save the per-page summary CSV<br>    with PAGE_SCORES_CSV.open(&quot;w&quot;, newline=&quot;&quot;, encoding=&quot;utf-8&quot;) as f:<br>        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))<br>        w.writeheader()<br>        w.writerows(rows)<br><br>    # 4) Create a human-friendly spot-check file with diffs for the most disagreed pages<br>    worst_pages.sort(key=lambda t: t[1], reverse=True)<br>    with SPOT_CHECK_MD.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>        f.write(&quot;# PDF spot-check: biggest disagreements\n\n&quot;)<br>        f.write(f&quot;_Top {TOP_K_PAGES} pages by disagreement (no ground truth, medoid = most central parser)._  \n\n&quot;)<br>        for page, score, detail in worst_pages[:TOP_K_PAGES]:<br>            pres = detail[&quot;present_labels&quot;]; midx = detail[&quot;midx&quot;]<br>            medoid_lbl = detail[&quot;medoid_label&quot;]<br>            f.write(f&quot;## Page {page} — disagreement score (max dist to medoid): {score:.3f}\n&quot;)<br>            f.write(f&quot;- Present: {&#39;, &#39;.join(pres)}  \n&quot;)<br>            f.write(f&quot;- Medoid: **{medoid_lbl}**  \n\n&quot;)<br>            medoid_text = detail[&quot;texts&quot;][midx]<br>            # show a diff for each non-medoid<br>            for j, lbl in enumerate(pres):<br>                if j == midx: <br>                    continue<br>                other = detail[&quot;texts&quot;][j]<br>                diff = unified_diff(medoid_text, other, medoid_lbl, lbl, n=3)<br>                f.write(&quot;&lt;details&gt;\n&lt;summary&gt;diff vs &quot; + lbl + &quot;&lt;/summary&gt;\n\n&quot;)<br>                f.write(&quot;```diff\n&quot; + diff + &quot;\n```\n&lt;/details&gt;\n\n&quot;)<br><br>    print(f&quot;Wrote:\n- {PAGE_SCORES_CSV}\n- {SPOT_CHECK_MD}\n&quot;)<br>    print(&quot;Tip: the medoid is a consensus anchor, **not** guaranteed to be the most accurate. Use the diffs to eyeball truth on tricky pages.&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>The Results: Which Parser Won??</h3><p><strong>For notebooks:</strong></p><ul><li>Both tools gave very similar outputs for both Markdown and Python</li><li>nbformat preserved outputs while Jupytext kept only text and code</li></ul><p><strong>For PDFs:</strong></p><ul><li>For the easier pages, all the parsers had very similar outputs</li><li>For the harder pages, the simpler parsers (PyPDF, PyMuPDF, and Unstructured-fast) only captured basic text while LlamaParse and Unstructured hi-res were able to capture text in images and other small details</li><li>LlamaParse was often good, but it added noisy OCR to heavier pages and blended the text in nearby areas. The text did not flow like how you would normally read it. A lot of the text was jumbled together in a random order. It is still a strong choice if you want polished Markdown and if you are ok with API costs, but I would personally use <strong>Unstructured (hi-res).</strong></li></ul><p><strong>Example: this PDF page and the parser outputs for medoid (Unstructured fast), Unstructured hi-res, and LlamaParse</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvnLG68Yrl5ntxWCX0jcJg.png" /></figure><ul><li>Unstructured fast (the central parser for this page) only returned :</li></ul><pre><br>```text<br>Use a more powerful LLM to evaluate<br><br>GPT-3.5<br><br># GPT-4<br>```</pre><ul><li>Unstructured (hi-res) returned:</li></ul><pre>``text<br># Use a more powerful LLM to evaluate<br><br>aN) You Consider the following product review: &quot;I&#39;m a professional wildlife photographer. I&#39;ve been shooting with this camera for six months, oat . and it&#39;s been amazing! If you want to hear more from me, follow me on Instagram @matthewramirez73&quot; Please identify whether not the above text contains any personally . a . . ae . -o. identifiable information. This includes, but is not limited to, Pll such as aname, location, or credit card number. Just reply with &quot;true&quot; if the example contains Pll, otherwise answer with &quot;false&quot;. &lt; 4/T &gt; ChatGPT False<br><br># GPT-3.5<br><br>tO aaggress tne users Question OF INSTTUCTION WILNOUT any Irrelevant sentences. The answer provides high-quality content, demonstrating expert knowledge in the area, is very well written, logical, easy-to-follow, engaging, and insightful. Please first provide a brief reasoning you used to derive …<br>```</pre><ul><li>LlamaParse returned</li></ul><pre>```text<br># Use a more powerful LLM to evaluate<br><br>auuless sentences. The answer provides high-quality content, demonstrating expert knowledge in the area, is very well written, logical, easy-to-follow; engaging, and insightful:<br><br>Consider the following product review: &quot;I&#39;m a professional wildlife photographer: I&#39;ve been shooting with this camera for six months; and it&#39;s been amazing! If you want to hear more from me; follow me on Instagram @matthewramirez73&quot;<br><br>Please first provide brief reasoning you used to derive the rating score and then write &quot;Score: &amp;#x3C;rating&gt;&quot; in the last line:<br><br>Input:<br><br>Consider the following product review: &quot;I&#39;m a professional wildlife photographer: I&#39;ve been shooting with this camera for six months; and it&#39;s been amazing! If you want to hear more from me; follow me on Instagram @matthewramirez73&quot;<br><br>Please identify whether or not the above text contains any personally identi…<br>```</pre><p>In summary:</p><ul><li>For notebooks: I would use <strong>nbformat </strong>since it provides you the choice of preserving outputs. If you do not want to preserve cell outputs, it would most likely not make a difference.</li><li>For PDFs: I would use <strong>Unstructured fast</strong> for easy pages and <strong>Unstructured hi-res</strong> for complex pages where the most important information is</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fa832866ea9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Journey Learning LangGraph at 14: Building a Reflective AI Agent]]></title>
            <link>https://medium.com/@ishveen/my-journey-learning-langgraph-at-14-building-a-reflective-ai-agent-2d0ef2b557ae?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d0ef2b557ae</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 04:22:53 GMT</pubDate>
            <atom:updated>2025-03-03T04:22:53.937Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi! I’m 14 years old, and lately, I’ve been obsessed with artificial intelligence (AI). My dad is really into AI, so we talk about it every day — listening to podcasts, following the news, or discussing his work stories. Recently, my dad challenged me to explore a new platform called <a href="https://langchain-ai.github.io/langgraph/tutorials/#rag">LangGraph</a>. He believes LangGraph will totally change how AI agents are built in the future.</p><p>To get started, he signed me up for an online class called <a href="https://academy.langchain.com/">“Introduction to LangGraph”</a> on LangChain Academy. The course was exciting at first, but after completing 72%, I felt I had learned enough theory and was ready to do something practical instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*EEFBXEeQ-i59ub9GyQJwWA.png" /><figcaption><a href="https://academy.langchain.com/enrollments">https://academy.langchain.com/enrollments</a></figcaption></figure><h3>Starting Point: LangChain Agents</h3><p>Initially, my dad gave me a project that involved LangChain for loading and retrieving data. We used LangChain loaders to import content from PDFs and GitHub repositories. Then, using a LangChain agent, we connected this loaded content to an AI model that could dynamically answer questions based on the retrieved information.</p><p>Here’s a basic LangChain agent setup snippet my dad gave me:</p><pre># Load PDF documents<br>pdf_files = glob(&quot;./pdfs/*.pdf&quot;)<br>pdf_documents = []<br>for pdf_file in pdf_files:<br>    loader = PyPDFLoader(pdf_file)<br>    pdf_documents.extend(loader.load())<br><br># Load GitHub repository documents<br>github_loader = GithubFileLoader(<br>    repo=&quot;langchain-ai/langgraph&quot;,<br>    branch=&quot;main&quot;,<br>    github_api_url=&quot;https://api.github.com&quot;,<br>    access_token=&quot;&lt;YOUR_ACCESS_TOKEN&gt;&quot;,<br>    file_filter=lambda file_path: file_path.startswith(&quot;examples/multi_agent&quot;) and file_path.endswith(&quot;.ipynb&quot;)<br>)<br>github_documents = github_loader.load()</pre><pre># Initialize LangChain agent<br>tools = [<br>    Tool(<br>        name=&quot;pdf_search&quot;,<br>        func=lambda q: pdf_retriever.invoke(q),<br>        description=&quot;Searches PDF documents for answers.&quot;<br>    ),<br>    Tool(<br>        name=&quot;github_search&quot;,<br>        func=lambda q: github_retriever.invoke(q),<br>        description=&quot;Searches GitHub repo examples.&quot;<br>    )<br>]<br><br>agent = initialize_agent(<br>    tools,<br>    llm,<br>    agent=AgentType.OPENAI_FUNCTIONS,<br>    verbose=True,<br>)</pre><p>This basic agent worked fine, but it had limitations — it didn’t really reflect or improve upon its answers.</p><h3>Choosing LangGraph Architecture</h3><p>To upgrade my chatbot, I chose LangGraph’s advanced architecture called <a href="https://langchain-ai.github.io/langgraph/tutorials/#rag">“Multi-Agent Systems with Reflection &amp; Critique”</a>. This system is smarter because it includes self-improvement and reflection capabilities. It means my chatbot could now check and enhance its own answers.</p><h3>1. Building a Smarter Supervisor</h3><p>The first step was creating a smarter supervisor to choose the best agent (PDF or GitHub) based on the question type.</p><p>Here’s my improved supervisor function:</p><pre># Supervisor decides the best agent based on the question<br>def supervisor(state: AgentState) -&gt; AgentState:<br>    prompt = ChatPromptTemplate.from_template(&quot;&quot;&quot;<br>    Choose the agent for answering this question:<br>    - pdf: theory notes or PDF documents<br>    - github: practical code examples<br><br>    Question: {question}<br><br>    Answer exactly one word (pdf/github):<br>    &quot;&quot;&quot;)<br>    decision = llm.invoke(prompt.format(question=state[&quot;question&quot;]))<br>    next_agent = decision.content.strip().lower()<br>    return {&quot;next&quot;: next_agent if next_agent in [&quot;pdf&quot;, &quot;github&quot;] else &quot;pdf&quot;}</pre><h3>2. Reflection and Critique</h3><p>Next, I implemented reflection and critique functions, allowing the chatbot to evaluate and improve its own answers. After the initial response, the chatbot critiques itself to ensure the answers are clear, accurate, and practical.</p><p>Here’s how my reflection function looked:</p><pre># Reflection to critique and improve chatbot answers<br>def reflect(state: AgentState) -&gt; AgentState:<br>    iteration = state.get(&quot;iteration&quot;, 0) + 1<br>    state[&quot;iteration&quot;] = iteration<br><br>    last_response = state[&quot;messages&quot;][-1].content<br>    critique_prompt = f&quot;&quot;&quot;<br>        Critique this response:<br>        {last_response}<br><br>        Provide exactly three improvements:<br>        1. Factual correction or clarification<br>        2. Structure or readability improvement<br>        3. Practical applicability suggestion<br>    &quot;&quot;&quot;<br><br>    critique = llm.invoke(critique_prompt)<br><br>    refine_prompt = f&quot;&quot;&quot;<br>        Original answer:<br>        {last_response}<br><br>        Critique points:<br>        {critique.content}<br><br>        Incorporate these improvements into a refined answer.<br>    &quot;&quot;&quot;<br><br>    refined = llm.invoke(refine_prompt)<br><br>    if refined.content.strip() == last_response.strip() or iteration &gt;= 2:<br>        next_step = &quot;finalize&quot;<br>    else:<br>        next_step = &quot;reflect&quot;<br><br>    return {&quot;messages&quot;: state[&quot;messages&quot;] + [AIMessage(refined.content)], &quot;next&quot;: next_step}</pre><h3>3. Observability with LangSmith</h3><p>I then connected my chatbot to LangSmith, a powerful tool that tracks every action of the chatbot. With LangSmith, I could watch the chatbot’s entire thought process in real-time — from initial answers to detailed reflections.</p><h3>What I Learned</h3><p>During this journey, I learned about:</p><ul><li><strong>Semantic Chunking</strong>: Splitting big documents into smaller, searchable pieces.</li><li><strong>Vector Databases</strong>: Efficiently storing and retrieving information chunks.</li><li><strong>Reflection &amp; Critique</strong>: AI agents that improve their answers automatically.</li><li><strong>Containerization with Docker</strong>: Using Docker allowed me and my dad to easily collaborate and run our chatbot consistently, ensuring we both worked on the same libraries and software setups without any compatibility issues.</li></ul><h3>LangSmith Observability</h3><p>Observing my chatbot through LangSmith was exciting. I could clearly see each step the chatbot took, understanding better how AI agents think, reflect, and improve.</p><p>Here’s how my chatbot responded and how LangSmith traced those responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFVSuOiTGWiXFq4LplemmA.png" /><figcaption>Chatbot Response- Github Agent</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*niMHFenK2pMH_sDnnr88uQ.png" /><figcaption>LangSmith Traceability with tracking of each Steps</figcaption></figure><p>Checkout out my Youtube for Demo of Chatbot &amp; detailed responses:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FROsVSYNPG5w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DROsVSYNPG5w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FROsVSYNPG5w%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6aef4406c06b66a988b17bd01647462b/href">https://medium.com/media/6aef4406c06b66a988b17bd01647462b/href</a></iframe><p>Check out my GitHub repository to explore the complete project setup:</p><p><a href="https://github.com/ishveen-ai/learnaiagents">GitHub - ishveen-ai/learnaiagents</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d0ef2b557ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Journey with Deep Learning: Building a Food Classifier and Speeding It Up with NVIDIA’s CUDA-X…]]></title>
            <link>https://medium.com/@ishveen/my-journey-with-deep-learning-building-a-food-classifier-and-speeding-it-up-with-nvidias-cuda-x-350970269197?source=rss-75aefbf1f3d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/350970269197</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ishveen Kaur]]></dc:creator>
            <pubDate>Sat, 09 Nov 2024 16:43:36 GMT</pubDate>
            <atom:updated>2024-11-09T16:43:36.446Z</atom:updated>
            <content:encoded><![CDATA[<h3>My Journey with Deep Learning: Building a Food Classifier and Speeding It Up with NVIDIA’s CUDA-X AI/TensorRT Tools</h3><p>Hey everyone! I’m 14 years old and recently, I have been on a journey to learn deep learning (DL) and tried to build an image classification model. I want to share my experience in case it helps other beginners or anyone curious about deep learning and NVIDIA tools like TensorRT, which is covered in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*F9paU8GOrTd3gkidZk4_Vw.png" /><figcaption>My Final Output</figcaption></figure><h3>Starting with the Basics: The NVIDIA DL Course</h3><p>It all started with an NVIDIA course on deep learning. This course was my intro to the world of DL, where I learned about image models, the basics of neural networks, and how to create models from scratch. I learned how to work with different tools and libraries, including the basics of PyTorch, one of the popular DL frameworks, and got introduced to one of NVIDIA’s tools that helped make DL even better on their GPUs.</p><p>This tool is called <strong>TensorRT</strong>. After fully training your model, the next step is to test it, which is called the <strong>inference phase</strong>. TensorRT’s job is to make the model classify the food faster while still keeping the accuracy, which is good when you want immediate, real-time responses. It is able to reduce latency specifically on NVIDIA GPUs.</p><h3>Finding a Dataset: The Food101 Challenge</h3><p>Once I was comfortable with the basics, I thought that I should try to make a model with a real dataset. I found the <strong>Food101 dataset</strong> on Hugging Face, which has images of 101 different kinds of food. My goal was to build a model that could recognize food categories from an image, like pizza, sushi, or cupcakes.</p><h3>Setting Up the Data</h3><p>Here’s the code I used to set up my dataset:</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;) <br><br># Step 2: Setup directories and dataset<br>import os<br>from datasets import load_dataset<br><br># Define the output directory path in Google Drive<br>output_dir = &#39;/content/drive/MyDrive/food101_mini_samples&#39;<br><br># Dictionary to store the number of saved images for each label<br>label_image_count = {}<br><br># Check if the directory already exists and has content<br>if os.path.exists(output_dir) and any(os.scandir(output_dir)):<br>    print(&quot;Found existing dataset directory. Loading existing images...&quot;)<br><br>    # Count existing images in each label directory<br>    for label in os.listdir(output_dir):<br>        label_dir = os.path.join(output_dir, label)<br>        if os.path.isdir(label_dir):<br>            num_images = len([f for f in os.listdir(label_dir) if f.endswith(&#39;.jpg&#39;)])<br>            label_image_count[label] = num_images<br><br>    print(f&quot;Loaded {len(label_image_count)} existing categories&quot;)<br><br>else:<br>    print(&quot;No existing dataset found. Creating new dataset...&quot;)<br><br>    # Create the main directory<br>    os.makedirs(output_dir, exist_ok=True)<br><br>    # Load the Food101 dataset<br>    dataset = load_dataset(&quot;food101&quot;)<br>    print(&quot;Dataset structure:&quot;)<br>    print(dataset)<br><br>    # Save images for all labels<br>    for example in dataset[&#39;train&#39;]:<br>        label = dataset[&#39;train&#39;].features[&#39;label&#39;].int2str(example[&#39;label&#39;])<br>        image = example[&#39;image&#39;]<br><br>        # Create a subdirectory for the label<br>        label_dir = os.path.join(output_dir, label)<br>        os.makedirs(label_dir, exist_ok=True)<br><br>        # Save the image<br>        image_save_path = os.path.join(label_dir, f&quot;{label_image_count.get(label, 0)}.jpg&quot;)<br>        image.save(image_save_path)<br><br>        # Update image count<br>        label_image_count[label] = label_image_count.get(label, 0) + 1<br><br># Print summary<br>print(&quot;\nDataset Summary:&quot;)<br>print(f&quot;Total categories: {len(label_image_count)}&quot;)<br>print(f&quot;Categories: {list(label_image_count.keys())}&quot;)<br>print(&quot;\nImages per category:&quot;)<br>for label, count in label_image_count.items():<br>    print(f&quot;{label}: {count} images&quot;)<br><br>CATEGORIES = list(label_image_count.keys())</pre><p>This setup prepared my data with categories and images per category, such as:</p><pre>strawberry_shortcake: 750 images<br>beef_tartare: 750 images<br>...</pre><p>Building and Fine-Tuning the Model with PyTorch</p><p>Instead of building a model from scratch, I used a <strong>pretrained ResNet50</strong> model and fine-tuned it for my dataset. I customized the classifier layer to fit my categories and used data augmentation to improve training.</p><pre>import ipywidgets as widgets<br>from IPython.display import display, Image as IPImage<br>import torch<br>import numpy as np<br>from PIL import Image<br>import io<br>import time<br>import torchvision.transforms as transforms<br><br><br>class PyTorchInferenceTest:<br>    def __init__(self, model_path, device):<br>        self.device = device<br><br>        # Initialize PyTorch TensorRT model<br>        self.pytorch_engine = TensorRTInference(model_path, device)<br><br>        # Create upload widget<br>        self.upload_widget = widgets.FileUpload(<br>            accept=&#39;image/*&#39;,<br>            multiple=False,<br>            description=&#39;Upload Image&#39;<br>        )<br><br>        # Create output widgets<br>        self.image_widget = widgets.Output()<br>        self.results_widget = widgets.Output()<br><br>        # Register callback<br>        self.upload_widget.observe(self.handle_upload, names=&#39;value&#39;)<br><br>        # Create transforms<br>        self.transform = transforms.Compose([<br>            transforms.Resize((224, 224)),<br>            transforms.ToTensor(),<br>            transforms.Normalize(<br>                mean=[0.485, 0.456, 0.406],<br>                std=[0.229, 0.224, 0.225]<br>            )<br>        ])<br><br>        # Display widgets<br>        display(widgets.VBox([<br>            self.upload_widget,<br>            self.image_widget,<br>            self.results_widget<br>        ]))<br><br>    def preprocess_image(self, image_data):<br>        &quot;&quot;&quot;Preprocess image for PyTorch inference&quot;&quot;&quot;<br>        # Convert to PIL Image<br>        image = Image.open(io.BytesIO(image_data))<br>        if image.mode != &#39;RGB&#39;:<br>            image = image.convert(&#39;RGB&#39;)<br><br>        # Apply transforms<br>        tensor_image = self.transform(image)<br>        return tensor_image.unsqueeze(0).to(self.device)<br><br>    def format_predictions(self, predictions):<br>        &quot;&quot;&quot;Format predictions to show top 5 results&quot;&quot;&quot;<br>        if isinstance(predictions, torch.Tensor):<br>            predictions = predictions.cpu().numpy()<br><br>        top5_idx = np.argsort(predictions.flatten())[-5:][::-1]<br>        return [(CATEGORIES[idx], float(predictions.flatten()[idx])) for idx in top5_idx]<br><br>    def handle_upload(self, change):<br>        &quot;&quot;&quot;Handle image upload&quot;&quot;&quot;<br>        if not change.new:<br>            return<br><br>        # Get uploaded file data<br>        image_data = next(iter(change.new.values()))[&#39;content&#39;]<br><br>        # Display uploaded image<br>        with self.image_widget:<br>            self.image_widget.clear_output()<br>            display(IPImage(image_data))<br><br>        # Preprocess image<br>        processed_image = self.preprocess_image(image_data)<br><br>        # Run inference<br>        results = self.pytorch_engine.compare_inference(processed_image)<br><br>        # Display results<br>        with self.results_widget:<br>            self.results_widget.clear_output()<br><br>            print(&quot;PyTorch Inference Results:&quot;)<br>            print(f&quot;Native PyTorch Time: {results[&#39;pytorch_time&#39;]:.2f}ms&quot;)<br>            print(f&quot;TensorRT Time: {results[&#39;tensorrt_time&#39;]:.2f}ms&quot;)<br>            print(f&quot;Speedup: {results[&#39;pytorch_time&#39;]/results[&#39;tensorrt_time&#39;]:.2f}x&quot;)<br><br>            print(&quot;\nTop 5 Predictions:&quot;)<br>            for label, prob in self.format_predictions(results[&#39;tensorrt_pred&#39;]):<br>                print(f&quot;{label}: {prob*100:.2f}%&quot;)<br><br># Usage<br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>pytorch_test = PyTorchInferenceTest(&#39;best_model.pth&#39;, device)</pre><p>After running the training process for 30 epochs, here are some of the results I observed:</p><pre>Epoch: 1, Train Loss: 3.055 | Train Acc: 25.28% | Val Acc: 45.10%<br>...<br>Epoch: 30, Train Loss: 0.684 | Train Acc: 80.52% | Val Acc: 69.02%</pre><p>Optimizing with TensorRT for Faster Inference</p><p>After training, I used <strong>TensorRT</strong> to improve the speed of my model during inference. Here’s the PyTorch inference code, along with TensorRT, to compare speeds.</p><pre>import ipywidgets as widgets<br>from IPython.display import display, Image as IPImage<br>import torch<br>import torch_tensorrt as trt  <br>import numpy as np<br>from PIL import Image<br>import io<br>import time<br>import torchvision.transforms as transforms<br><br><br>class PyTorchInferenceTest:<br>    def __init__(self, model_path, device):<br>        self.device = device<br><br>        # Load the PyTorch model and compile it with TensorRT<br>        model = torch.load(model_path, map_location=device)<br>        model.eval()<br>        self.pytorch_engine = trt.compile(model, inputs=[trt.Input((1, 3, 224, 224))], enabled_precisions={torch.float32})<br><br>        # Create upload widget<br>        self.upload_widget = widgets.FileUpload(<br>            accept=&#39;image/*&#39;,<br>            multiple=False,<br>            description=&#39;Upload Image&#39;<br>        )<br><br>        # Create output widgets<br>        self.image_widget = widgets.Output()<br>        self.results_widget = widgets.Output()<br><br>        # Register callback<br>        self.upload_widget.observe(self.handle_upload, names=&#39;value&#39;)<br><br>        # Create transforms<br>        self.transform = transforms.Compose([<br>            transforms.Resize((224, 224)),<br>            transforms.ToTensor(),<br>            transforms.Normalize(<br>                mean=[0.485, 0.456, 0.406],<br>                std=[0.229, 0.224, 0.225]<br>            )<br>        ])<br><br>        # Display widgets<br>        display(widgets.VBox([<br>            self.upload_widget,<br>            self.image_widget,<br>            self.results_widget<br>        ]))<br><br>    def preprocess_image(self, image_data):<br>        &quot;&quot;&quot;Preprocess image for PyTorch inference&quot;&quot;&quot;<br>        image = Image.open(io.BytesIO(image_data))<br>        if image.mode != &#39;RGB&#39;:<br>            image = image.convert(&#39;RGB&#39;)<br><br>        tensor_image = self.transform(image)<br>        return tensor_image.unsqueeze(0).to(self.device)<br><br>    def format_predictions(self, predictions):<br>        &quot;&quot;&quot;Format predictions to show top 5 results&quot;&quot;&quot;<br>        if isinstance(predictions, torch.Tensor):<br>            predictions = predictions.cpu().numpy()<br><br>        top5_idx = np.argsort(predictions.flatten())[-5:][::-1]<br>        return [(CATEGORIES[idx], float(predictions.flatten()[idx])) for idx in top5_idx]<br><br>    def handle_upload(self, change):<br>        &quot;&quot;&quot;Handle image upload&quot;&quot;&quot;<br>        if not change.new:<br>            return<br><br>        image_data = next(iter(change.new.values()))[&#39;content&#39;]<br><br>        with self.image_widget:<br>            self.image_widget.clear_output()<br>            display(IPImage(image_data))<br><br>        processed_image = self.preprocess_image(image_data)<br><br>        # Run inference<br>        start_time = time.time()<br>        results = self.pytorch_engine.compare_inference(processed_image)<br>        pytorch_time = (time.time() - start_time) * 1000  # Convert to ms<br><br>        # Display results<br>        with self.results_widget:<br>            self.results_widget.clear_output()<br><br>            print(&quot;PyTorch Inference Results:&quot;)<br>            print(f&quot;Native PyTorch Time: {results[&#39;pytorch_time&#39;]:.2f}ms&quot;)<br>            print(f&quot;TensorRT Time: {results[&#39;tensorrt_time&#39;]:.2f}ms&quot;)<br>            print(f&quot;Speedup: {results[&#39;pytorch_time&#39;]/results[&#39;tensorrt_time&#39;]:.2f}x&quot;)<br><br>            print(&quot;\nTop 5 Predictions:&quot;)<br>            for label, prob in self.format_predictions(results[&#39;tensorrt_pred&#39;]):<br>                print(f&quot;{label}: {prob*100:.2f}%&quot;)<br><br># Usage<br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>pytorch_test = PyTorchInferenceTest(&#39;best_model.pth&#39;, device)</pre><p>After running the inference test, I got the following results:</p><pre>PyTorch Inference Results:<br>Native PyTorch Time: 9.97ms<br>TensorRT Time: 1.88ms<br>Speedup: 5.30x<br><br>Top 5 Predictions:<br>onion_rings: 99.29%<br>bread_pudding: 0.19%<br>...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*F9paU8GOrTd3gkidZk4_Vw.png" /></figure><p>This comparison showed that the model could run an inference much faster with TensorRT than without it!</p><h3>Diving Into CUDA-X AI: RAPIDS, NeMo, and Triton Inference Server</h3><p>Exploring <strong>CUDA-X AI</strong> introduced me to tools beyond TensorRT:</p><ul><li><strong>RAPIDS</strong> for speeding up ML workflows, similar to how TensorRT optimizes DL.</li><li><strong>NeMo</strong> for working with generative AI, especially for tasks like natural language processing.</li><li><strong>Triton Inference Server</strong> for hosting/deploying optimized models so they’re ready to make fast predictions in real-time.</li></ul><h3>Lessons Learned and What’s Next</h3><p>Through this project, I learned about everything from building models to using NVIDIA’s tools for optimization. Although my model’s accuracy wasn’t perfect, I’m excited to keep exploring, especially by running for more epochs or testing different architectures, parameters, and optimizers.</p><p>If you’re new to DL, my advice would be to dive in and start experimenting. There’s so much you can learn, and tools like TensorRT can make a huge difference in how you optimize your projects!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=350970269197" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>