<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[OLarry - Medium]]></title>
        <description><![CDATA[Leveraging Ai for tax automation and insights. - Medium]]></description>
        <link>https://medium.com/olarry?source=rss----8520b8c773b7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>OLarry - Medium</title>
            <link>https://medium.com/olarry?source=rss----8520b8c773b7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:03:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/olarry" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Prompt Hacking: How DSPy + MIPRO Brings Real Optimization to LLM Workflows]]></title>
            <link>https://medium.com/olarry/beyond-prompt-hacking-how-dspy-mipro-brings-real-optimization-to-llm-workflows-f69242488ee8?source=rss----8520b8c773b7---4</link>
            <guid isPermaLink="false">https://medium.com/p/f69242488ee8</guid>
            <category><![CDATA[dspy]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Dean Chen]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 02:48:29 GMT</pubDate>
            <atom:updated>2025-04-17T02:48:29.852Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIIGAsJLUjh3NIAxRoGr2g.png" /></figure><h3>The Pain of Current Prompt Crafting</h3><p>If you’ve worked with large language models (LLMs), you’ve probably spent countless hours tweaking prompt strings: moving words around, adding examples, removing fluff, retrying responses, and hoping for the best. Prompt engineering has become a mix of intuition, trial-and-error, and sheer luck.</p><p>While this can be effective for one-off tasks or early prototyping, it becomes unmanageable as applications grow. Manual prompts are hard to maintain, hard to scale, and impossible to optimize systematically.</p><p>Here is a familiar example: imagine you want to summarize an email.</p><ul><li>You ask: “Summarize this.” → LLM gives you a weird, long summary.</li><li>You try again: “Give me a one-sentence summary of this email.” → Better, but maybe misses the point.</li><li>You try again: “Read this email and tell me the main action item in one sentence.” → Closer!</li></ul><p>This constant fiddling with how you phrase your request (the “prompt”) is annoying, slow, and unreliable. What works for one assistant (say, GPT-3.5) might fail with another (GPT-4), or even fail on a slightly different email. This is known as “prompt hacking.”</p><h3><strong>Enter</strong>: DSPy (Moving From Prompts to Programs)</h3><p><strong>DSPy</strong> (Declarative Self-improving Python) is a framework from Stanford NLP research that allows you to build LLM applications by programming rather than just prompting. It introduces structure, modularity, and optimization into the world of LLM app development.</p><p>Let’s start with an example, summarizing email:</p><p>In DSPy, you define a signature in a declarative way:</p><pre>class SummarizeEmail(Signature):<br>    email_body: str             # input<br>    one_line_summary: str       # output</pre><p>DSPy knows:</p><ul><li><strong>Input</strong> = email_body</li><li><strong>Output</strong> = one_line_summary</li></ul><p>And when you use it in a module:</p><pre>from dsp import Predict<br><br>summarize = Predict(SummarizeEmail) # DSPy module<br>output = summarize(email_body=&quot;Hey, let&#39;s meet next Monday at noon.&quot;)<br>print(output.one_line_summary)<br><br># &quot;Meeting scheduled for Monday at noon.&quot;</pre><p>DSPy generates the full prompt (e.g., few-shot examples) and the LLM call behind the scenes to convert input to output.</p><p>DSPy determines inputs and outputs in a Signature class based on what you provide at runtime. i.e., <strong>you provide only the inputs</strong>, and DSPy infers that everything else is output to be predicted.</p><p>DSPy infers<strong>:</strong></p><ul><li>email_body = <strong>input</strong> (you provided it)</li><li>one_line_summary = <strong>output</strong> (you didn’t provide it, but it’s in the Signature)</li></ul><p>This example is just a starting point, but it shows that instead of winging it with plain verbal requests, DSPy helps you provide your LLM assistant with the following:</p><ol><li>Clear job descriptions (called “Signatures”), so instead of a fuzzy request like in the earlier example, you will write a simple description like the following:</li></ol><ul><li><strong>Input</strong>: Email Body → <strong>Output</strong>: One-Sentence Summary</li><li><strong>Input</strong>: News Article → <strong>Output</strong>: Bullet Points of Key Facts</li></ul><p>This tells the assistant what you want (the task) without you needing to figure out the exact magic words to ask. Basically, it&#39;s an input and output intent that specifies the “signature” of the job.</p><p>2. Follow this with standard tools/steps (Called “Modules”). This allows you to tell the assistant to use standard procedures. Think of building blocks like:</p><ul><li><strong>Predict</strong>: Just do the task described in the job description directly.</li><li><strong>ChainOfThought</strong>: Think step-by-step before answering (like showing your work).</li></ul><p>You can combine these tools as well. For example: “First, Extract Key Facts from the article, then Summarize Those Facts.”</p><p>3. Followed up with the optimizer (Like a Smart Manager). This is the most interesting bit. DSPy has a “manager” (called an Optimizer or Teleprompter) that helps your assistant get really good at their job for you.</p><p>You give the manager examples of good work (e.g., sample emails and the perfect summaries you wanted). Then you tell the manager what “good” means (e.g., “the summary must be short and accurate”).</p><p>The manager then automatically figures out the best “magic words” (the best prompts) for your specific assistant (LLM) to use with its Job Descriptions (Signatures) and Tools (Modules) to get the results you want consistently. It tries different phrasings, different examples in the prompt, etc., and picks the best combination based on your examples.</p><p><strong>So,</strong> combining all three parts, DSPy can help you structure your prompt engineering:</p><ul><li>Stop guessing the magic words to make the LLM understand you.</li><li>Tell DSPy what you want (Inputs → Outputs) using simple “Signatures”.</li><li>Use DSPy’s building blocks (“Modules”) to structure the task.</li><li>Let DSPy’s “Optimizer” automatically find the best magic words (prompts) for your specific LLM and your specific task, based on examples you provide.</li></ul><p>DSPy moves away from the fragile, string-based prompt engineering model and toward a Pythonic programming model. Think of it as defining what you want your LLM to do — like in functional programming — without specifying how to say it in prompt text. It uses a declarative style, meaning you define what you want the LLM to do, rather than how to do it. It also features self-improvement capabilities, allowing you to optimize prompts and weights automatically, improving the reliability and adaptability of your applications.</p><p>This allows you to spend less time fiddling with prompts, gives you more reliable results, and actually improves your LLM application automatically when you give it examples. It turns messy “prompt hacking” into something more like structured programming and automated tuning.</p><p>Here is a <strong>real-world</strong> DSPy’s Python code example.</p><p>Imagine you’re building a smart tax assistant to help CPAs pre-screen clients. A client writes:</p><p>“My spouse and I have two kids, ages 5 and 9. Our joint income last year was around $180k. We live in California.”</p><p>You want to extract structured info, check eligibility rules, and suggest next steps.</p><p>This is an excellent case for DSPy: you define the logic once, give a few training examples, and let it auto-tune itself.</p><pre>from dsp import Signature, Example, ChainOfThought, BootstrapFewShot, Program<br><br># Signature — What info should the model extract and compute?<br>class TaxCreditCheck(Signature):<br>    &quot;&quot;&quot;Determine eligibility for Child Tax Credit based on a client note.&quot;&quot;&quot;<br>    client_note: str<br>    num_children: int<br>    joint_income: str<br>    income_threshold_met: bool<br>    qualified_for_ctc: bool<br>    recommendation: str<br><br># Module — Use Chain-of-Thought for rule-based reasoning<br>class TaxEligibilityProgram(Program):<br>    def __init__(self):<br>        super().__init__()<br>        self.cot = ChainOfThought(TaxCreditCheck)<br><br>    def forward(self, client_note):<br>        return self.cot(client_note=client_note)<br><br># Few-shot examples to guide the optimization<br>examples = [<br>    Example(<br>        client_note=(<br>            &quot;We have one child, she’s 3 years old. We earned $150,000 jointly last year. We’re based in New York.&quot;<br>        ),<br>        num_children=1,<br>        joint_income=&quot;$150,000&quot;,<br>        income_threshold_met=True,<br>        qualified_for_ctc=True,<br>        recommendation=&quot;Client qualifies for full Child Tax Credit. Include in return.&quot;<br>    ),<br>    Example(<br>        client_note=(<br>            &quot;My wife and I have 3 kids: 2, 6, and 8 years old. Our joint income last year was around $410,000.&quot;<br>        ),<br>        num_children=3,<br>        joint_income=&quot;$410,000&quot;,<br>        income_threshold_met=False,<br>        qualified_for_ctc=False,<br>        recommendation=&quot;Client exceeds income threshold for Child Tax Credit. Do not apply.&quot;<br>    ),<br>    Example(<br>        client_note=(<br>            &quot;We have two children, ages 4 and 6, and our total household income last year was $180,000. We live in Illinois.&quot;<br>        ),<br>        num_children=2,<br>        joint_income=&quot;$180,000&quot;,<br>        income_threshold_met=True,<br>        qualified_for_ctc=True,<br>        recommendation=&quot;Client qualifies for full Child Tax Credit. Include in return.&quot;<br>    )<br>]<br><br># Run optimizer to find best prompting strategy<br>teleprompter = BootstrapFewShot(<br>    signature=TaxCreditCheck,<br>    examples=examples,<br>    metric=&quot;accuracy&quot;,  # Or create custom logic for eval<br>    max_bootstrapped_examples=2<br>)<br><br># Compile the optimized version<br>optimized_tax_program = teleprompter.compile(TaxEligibilityProgram())<br><br># Test it out<br>test_note = (<br>    &quot;Hey, we made about $180,000 last year and have two kids, ages 5 and 9. We&#39;re in California.&quot;<br>)<br><br>result = optimized_tax_program(client_note=test_note)<br><br># Show results<br>print(&quot;Number of Children:&quot;, result.num_children)<br>print(&quot;Income:&quot;, result.joint_income)<br>print(&quot;Income Threshold Met:&quot;, result.income_threshold_met)<br>print(&quot;Eligible for CTC:&quot;, result.qualified_for_ctc)<br>print(&quot;Recommendation:&quot;, result.recommendation)</pre><p><strong>Note</strong>: <em>When crafting examples, be careful of unintended correlations. In this case, we added a third example to ensure the model doesn’t mistakenly associate having more children with ineligibility — the true determining factor is income.</em></p><p>The following diagram compares traditional manual prompting with DSPy’s structured approach to LLM workflows. On the right, manual prompting relies on hand-crafted strings and ad-hoc parsing, which is brittle and hard to scale. On the left, DSPy introduces a modular, programmatic framework where inputs and outputs are defined via Signatures, executed through reusable modules, and optimized using a Teleprompter. This enables consistent, tunable, and more maintainable LLM applications.</p><figure><img alt="highlevel overview DSPy comparing to manual prompting" src="https://cdn-images-1.medium.com/max/967/0*EGHupW-6Px-f3ZFA" /><figcaption>DSPs vs. Manual Prompting</figcaption></figure><p>Overall, DSPy’s benefit is that it;</p><ul><li>Abstracts away prompt engineering to accelerate iteration.</li><li>Improves the reliability of prompt engineering by creating consistent pipelines with clearly defined logic.</li><li>Enhances the adaptability and scalability of the engineering process with easily updating or extending functionality without rewriting prompts.</li></ul><p>DSPy can be applied to a wide range of use cases, including question answering, summarization, code generation, and custom NLP tasks.</p><p>One of DSPy’s biggest strengths is its <strong>model-agnostic prompt generation</strong>. You define what you want using Signatures and Examples, and DSPy handles how to ask for it, even tuning differently for different LLMs.</p><blockquote><em>So far, we’ve seen how DSPy helps structure and modularize LLM programs. But what about performance? How do you get better results from limited training data?</em></blockquote><blockquote><em>Enter </em><strong><em>MIPRO</em></strong><em> — a built-in DSPy optimizer that learns how to choose better instructions and examples automatically. In this section, we’ll walk through a common scenario: mediocre results with a manually written prompt, and how MIPRO improves it.</em></blockquote><h3>What is MIPRO? (Auto Optimizer, Optimization That Actually Learns)</h3><p><strong>MIPRO</strong> stands for <strong>Multi-Iteration Prompt Optimization</strong>. DSPy has a built-in optimizer that automatically refines your prompts and few-shot examples using real evaluation metrics.</p><p>Think of it as, instead of you manually trying to figure out the perfect way to phrase your instructions (which can be hard!), MIPRO does it for you. You tell MIPRO what you want the LLM to do (the task). For example, “summarize this article.” Then you give MIPRO some examples of what a good result looks like (training data). For instance, you might show a few news articles and good summaries of them. You also tell MIPRO how to judge if the AI is doing a good job (a metric). This could be something like “How accurate is the summary?”</p><p>Then, MIPRO goes to work like this:</p><ol><li>It tries out different ways of phrasing the instructions (prompts). It doesn’t just pick one; it tries many variations.</li><li>It also figures out good examples to show the LLM to help it learn (few-shot examples). It can even create these examples automatically from the training data you provided.</li><li>It tests how well the LLM performs with each set of instructions and examples, using the “good job” metric you gave it to determine which ones work best.</li><li>It keeps repeating this process—trying new instructions and examples, evaluating them, and improving them over and over again—providing a constant cycle of learning and refinement until it finds the best possible instructions for the AI to do the task well.</li></ol><p>So it lets you:</p><ul><li>Define your task (e.g., extract information from text)</li><li>Provide training data and a metric (e.g., accuracy)</li><li>Let DSPy generate, evaluate, and improve prompts over multiple iterations</li></ul><p>It employs a bootstrapping technique to generate few-shot examples from training data automatically. It also uses a “proposer” model that crafts instruction candidates based on dataset summaries, module code, and other heuristics.</p><p>MIPRO then performs optimization using techniques like Bayesian Optimization with minibatching * (<em>much like neural network hyperparameter tuning — but targets the space of prompt designs (e.g., instruction wording, example selection), using minibatching to evaluate multiple candidate prompts efficiently and guide the search for optimal configurations.)</em> to explore instruction/example combinations efficiently.</p><p>However, there are a few things to keep in mind:</p><ul><li>If you don’t have many examples to show MIPRO, it might become too focused on those specific examples and not work well in other situations. This is like the dog only learning to sit when you use a particular treat. I.e., overfitting on small datasets if over-optimized</li><li>This process can take a lot of computer power and time, especially if the LLM is very complex or you have a lot of data.</li><li>If you don’t have a good way to measure if the LLM is doing a good job, MIPRO might optimize for the wrong thing. For example, if you only care about the length of the summary and not the accuracy, MIPRO might only make very short summaries, even if they miss important information. I.e., Sensitive to evaluation metrics and the quality of validation data</li><li>Sometimes, you might still need to manually adjust some of MIPRO’s settings to get the best results. It’s like needing to fine-tune your dog training techniques based on the dog’s personality. Manual tuning of parameters like trials, minibatch size, and temperature may still be needed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*_VNqmSBGXjnngYTB" /><figcaption>MIPRO flow</figcaption></figure><p>This diagram illustrates the MIPRO optimization workflow in DSPy. The process begins when a user defines a task and provides training examples and an evaluation metric. MIPRO then generates variations of prompts and few-shot examples using a proposer model and bootstrapping. These candidates are evaluated using <strong>Bayesian Optimization with minibatching</strong>, which is an iterative refining prompt based on performance. Once the best combination of instructions and examples is found, it’s used in the DSPy program to guide the LLM in performing the task with greater accuracy and consistency.</p><h3>Code Walkthrough — Real Task: Customer Support Classification with MIPRO</h3><p>Let’s walk through a more realistic and compelling use case: classifying customer support messages into categories like <strong>Billing</strong>, <strong>Technical Issue</strong>, <strong>Shipping</strong>, and <strong>Account</strong>.</p><h4>Task</h4><p>Classify incoming customer messages into predefined categories using DSPy + MIPRO for prompt and example optimization.</p><h4>Input Example</h4><p>“I was charged $29.99 even after I canceled my subscription. This is outrageous.”</p><h4>Define DSPy Program</h4><p>Using DSPy is an <strong>iterative process</strong>. To maximize performance:</p><ul><li>Define your task and provide a few example inputs (labels are optional at the start)</li><li>Choose the right built-in layers (DSPy modules)</li><li>Assign a Signature to each layer to specify input/output</li><li>Write your program using regular Python code</li><li>Optimize with a DSPy optimizer like MIPRO</li></ul><p>In our case, we’re defining a message classification task with one module and one signature.</p><pre>from dspy import Signature, Predict, MIPRO, ExactMatch<br><br>class ClassifySupportMessage(Signature):<br>    message = str<br>    category = str  # Billing, Technical Issue, Shipping, Account<br><br>class SupportClassifier(Predict):<br>    signature = ClassifySupportMessage<br><br>model = SupportClassifier()<br><br>trainset = [<br>    {&quot;message&quot;: &quot;I want to update my credit card details&quot;, &quot;category&quot;: &quot;Billing&quot;},<br>    {&quot;message&quot;: &quot;App crashes every time I log in&quot;, &quot;category&quot;: &quot;Technical Issue&quot;},<br>    {&quot;message&quot;: &quot;My order never arrived&quot;, &quot;category&quot;: &quot;Shipping&quot;},<br>    {&quot;message&quot;: &quot;Please delete my account permanently&quot;, &quot;category&quot;: &quot;Account&quot;},<br>]</pre><h4>Optimize with MIPRO</h4><pre>mipro = MIPRO(metric=ExactMatch(), max_iters=10)<br>optimized_model = mipro(model, trainset)</pre><h4>Evaluate Performance</h4><pre>testset = [<br>    {&quot;message&quot;: &quot;I canceled last week but was still charged&quot;, &quot;category&quot;: &quot;Billing&quot;},<br>    {&quot;message&quot;: &quot;The app doesn&#39;t load on my phone&quot;, &quot;category&quot;: &quot;Technical Issue&quot;},<br>    {&quot;message&quot;: &quot;My package arrived late and damaged&quot;, &quot;category&quot;: &quot;Shipping&quot;},<br>    {&quot;message&quot;: &quot;How do I deactivate my profile?&quot;, &quot;category&quot;: &quot;Account&quot;},<br>]<br><br>correct = 0<br>for example in testset:<br>    pred = optimized_model.predict(message=example[&quot;message&quot;])<br>    if pred.category.lower() == example[&quot;category&quot;].lower():<br>        correct += 1<br><br>accuracy = correct / len(testset)<br>print(f&quot;Accuracy: {accuracy:.2f}&quot;)</pre><h4>Before vs. After MIPRO Optimization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*FZVnXu-zW7evqsAFlORR1g.png" /></figure><p>DSPy and MIPRO move LLM development from intuition-based prompt fiddling to structured, testable workflows, just like writing production code. If you’re building serious LLM apps, thinking this way is worth learning.</p><p>In a real-world production pipeline, this could be integrated as follows:</p><ol><li><strong>LLM Program Definition</strong>: Define task signatures and modules using DSPy.</li><li><strong>Offline MIPRO Optimization</strong>: Run prompt optimization regularly on validation data (e.g., nightly retrain).</li><li><strong>Metrics Monitoring</strong>: Track performance drift using real queries.</li><li><strong>Deployment</strong>: Export optimized programs or prompts into your backend classifier or routing logic.</li><li><strong>Continuous Learning</strong>: Optionally retrain as feedback is gathered.</li></ol><p>DSPy + MIPRO becomes the <strong>training-time optimizer</strong> to continuously improve your model’s behavior before shipping updates.</p><h3>From Signatures to Agents: DSPy for Agentic Systems</h3><p>One of the most potent ideas in DSPy is how Signatures replace manual prompt templates.</p><p>Instead of writing:</p><pre>&quot;Classify this message: {message}&quot; → category</pre><p>You define:</p><pre>class ClassifySupportMessage(Signature):<br>    message = str<br>    category = str</pre><p>This abstraction opens the door to composable, self-improving agents.</p><p>To illustrate how DSPy enables the development of <strong>agentic behavior</strong> through structured programming and <strong>orchestration</strong>, let’s walk through a lightweight assistant that performs two tasks:</p><ol><li>Extracts relevant topics from user questions.</li><li>Answers the question with contextual awareness.</li></ol><p>We begin by defining a <strong>Signature</strong> to structure the input and output for topic extraction:</p><pre>class ExtractTopicsSignature(dspy.Signature):<br>    question = dspy.InputField()<br>    topics = dspy.OutputField(format=list)</pre><p>And a module that utilizes this <strong>Signature</strong>:</p><pre>class ExtractTopics(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.extractor = dspy.Predict(ExtractTopicsSignature)<br><br>    def forward(self, question):<br>        return self.extractor(question=question).topics</pre><p>Here, we define the ExtractTopics module, which uses dspy. Predict along with our ExtractTopicsSignature to instruct the language model to identify the topics relevant to the input question.</p><p>Next, we define a second <strong>Signature</strong> and module for generating a context-aware response:</p><pre>class AnswerWithContextSignature(dspy.Signature):<br>    question = dspy.InputField()<br>    topics = dspy.InputField(format=list)<br>    answer = dspy.OutputField()<br><br>class AnswerWithContext(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.responder = dspy.Predict(AnswerWithContextSignature)<br><br>    def forward(self, question, topics):<br>        return self.responder(question=question, topics=topics).answer</pre><p>Now, we combine these individual modules into a single <strong>orchestration</strong> module, the Assistant:</p><pre>class Assistant(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.topic_extractor = ExtractTopics()<br>        self.answerer = AnswerWithContext()<br><br>    def forward(self, question):<br>        topics = self.topic_extractor(question)<br>        return self.answerer(question=question, topics=topics)</pre><p>This Assistant module demonstrates a basic form of <strong>orchestration</strong>, where the output of one module (ExtractTopics) is used as the input for another (AnswerWithContext), creating a simple but functional agentic workflow.</p><h3>Automatic Optimization with DSPy</h3><p>The true strength of DSPy lies in its ability to <strong>optimize</strong> these structured programs automatically. Instead of manually tuning prompts, you provide training data and a metric, and DSPy takes care of the rest. For our simple Assistant, the optimization process might look like this:</p><pre>trainset = [dspy.Example(question=q, answer=a) for q, a in training_data]<br>optimizer = dspy.teleprompt.LabeledFewShot() # Utilizes an optimizer like MIPRO<br>optimized_agent = optimizer.compile(student=Assistant(), trainset=trainset)</pre><p>Here, we use LabeledFewShot, a teleprompter in DSPy that employs an <strong>optimizer</strong> (such as MIPRO) to automatically refine the prompts and potentially the module configurations within our Assistant based on the provided trainset. This means DSPy compiles and <strong>optimizes</strong> across all the defined modules, freeing you from the burden of manual prompt engineering for each individual component.</p><p>What we see here, <strong>DSPy + MIPRO,</strong> represents a significant evolution in how we build LLM-powered software.</p><p>No more hacking prompts in a notebook. Instead, define structured programs with natural inputs/outputs and let DSPy optimize them for you. It’s systematic, testable, and most importantly, scalable.</p><p>The era of prompt engineering may not be dead, but it’s certainly evolving. In that transformation, DSPy and MIPRO are leading the way.</p><h4>Takeaways:</h4><ul><li>DSPy lets you structure LLM applications like functions, not strings.</li><li>It uses Signatures, Modules, and Optimizers to formalize prompt design.</li><li>MIPRO builds on this by automatically learning optimal instructions + examples using Bayesian Optimization.</li><li>The combo is a scalable alternative to trial-and-error prompting.</li></ul><h4>References:</h4><p><strong>Goel, K., Manning, C. D., Hashimoto, T. B., et al.</strong> (2024). <em>DSPy: Deep Learning for Prompt Programming with LLMs. </em><a href="https://arxiv.org/abs/2401.01939">arXiv:2401.01939</a></p><p><strong>Snoek, J., Larochelle, H., &amp; Adams, R. P.</strong> (2012). <em>Practical Bayesian Optimization of Machine Learning Algorithms. </em><a href="https://arxiv.org/abs/1206.2944">arXiv:1206.2944</a></p><p><strong>What is DSPy? Declarative, Self-Improving Language Programs</strong> (2024). <a href="https://medium.com/codex/what-is-dspy-declarative-self-improving-language-programs-93c6abc30aa7">Medium — Codex</a></p><p><strong>Taking Prompt Optimization from an Art into a Science</strong> (2024). <a href="https://medium.com/vvus/taking-prompt-optimization-from-an-art-into-a-science-80094f2747b2">Medium — VVUS</a></p><p><strong>Brown, T. B., Mann, B., Ryder, N., et al.</strong> (2020). <em>Language Models are Few-Shot Learners. </em><a href="https://arxiv.org/abs/2005.14165">arXiv:2005.14165</a></p><p><strong>Perez, E., Kiela, D., &amp; Cho, K.</strong> (2021). <em>True Few-Shot Learning with Language Models. </em><a href="https://arxiv.org/abs/2105.11447">arXiv:2105.11447</a></p><p><strong>Wei, J., Wang, X., Schuurmans, D., et al.</strong> (2022). <em>Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. </em><a href="https://arxiv.org/abs/2201.11903">arXiv:2201.11903</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f69242488ee8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/olarry/beyond-prompt-hacking-how-dspy-mipro-brings-real-optimization-to-llm-workflows-f69242488ee8">Beyond Prompt Hacking: How DSPy + MIPRO Brings Real Optimization to LLM Workflows</a> was originally published in <a href="https://medium.com/olarry">OLarry</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Difference Between RAG and AI Agents]]></title>
            <link>https://medium.com/olarry/understanding-the-difference-between-rag-and-ai-agents-10df56b35e02?source=rss----8520b8c773b7---4</link>
            <guid isPermaLink="false">https://medium.com/p/10df56b35e02</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Dean Chen]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 15:38:53 GMT</pubDate>
            <atom:updated>2025-03-07T15:38:53.382Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QuqUYKVLqkxPNjbbq78gwQ.png" /><figcaption>NYC Nighttime</figcaption></figure><p>So, everyone’s working on enhancing those powerful language models — LLMs — without the need for constant retraining. Two really interesting ways people are tackling this are called <strong>RAG (Retrieval-Augmented Generation)</strong> and <strong>AI Agents</strong>. Now, they might seem a bit alike at first glance, but they’re actually quite different. Each has its own strengths and weaknesses, and they’re best suited for different kinds of tasks.</p><p>Alright, let’s get down to brass tacks with a simple example. Picture this: you’re hanging out with friends, and someone asks, “Hey, what’s the average temperature in New York during summer, and should I pack light or bring a jacket?” You’ve got a couple of ways to figure that out.</p><p><strong>With RAG</strong>, it’s like having a research assistant. Instead of just relying on what it already knows, it quickly scans through a massive library of historical weather data, picks out the most relevant information, and uses that to give you a precise answer. For example, it might tell you that the average summer temperature in New York during the day is around 85°F (29°C) and pull a list of recommendations from your clothing data. It’s not just guessing; it’s pulling from reliable, up-to-date sources to make sure you get the most accurate response.</p><p><strong>But with an AI Agent</strong>, it’s like having a super-smart personal travel planner. It doesn’t just retrieve information — it actively plans and makes decisions for you. Here’s how it works:</p><ol><li><strong>Retrieves Historical Data</strong>: It uses RAG to find the average summer temperature in New York.</li><li><strong>Checks Real-Time Data</strong>: It pulls the current weather forecast using a weather API.</li><li><strong>Analyzes Your Preferences</strong>: It remembers you prefer packing light but hate being cold.</li><li><strong>Makes a Recommendation</strong>: Based on the average temperature, current forecast, and your preferences, it suggests, ‘Pack light, but bring a light jacket for cooler evenings.’</li></ol><p>Intrigued? Now let’s dive deeper.</p><h3>What is Retrieval-Augmented Generation (RAG)?</h3><p><strong>Retrieval-Augmented Generation (RAG)</strong> is a technique that enhances the response of LLMs by retrieving relevant information from an external knowledge base before generating an answer. Unlike standard LLMs that rely solely on pre-trained knowledge, RAG provides more up-to-date and domain-specific context to improve accuracy and reliability.</p><h3>How RAG Works</h3><ol><li><strong>Async Preprocessing Step:</strong> Before retrieval can happen, the information or documents must first be converted into vector format using an embedding model and stored in a Vector Database (VD). This step happens asynchronously as new documents are ingested into the system.</li><li><strong>User Query:</strong> When a user submits a query, the input is also converted into vector format using the same embedding model.</li><li><strong>Retrieval Step:</strong> The system searches the Vector Database to find the most semantically similar documents to the input query by leveraging the vector space properties.</li><li><strong>Conversion Back to Text:</strong> Once the most relevant documents are retrieved from the Vector Database, they are converted back to text before being fed into the LLM.</li><li><strong>Augmentation Step:</strong> The retrieved text data is injected into the LLM’s context to enhance its response.</li><li><strong>Response Generation:</strong> The LLM processes both the retrieved data and its pre-trained knowledge to generate a more accurate and relevant response.</li></ol><p>This method is useful in applications such as<strong> </strong>customer support, medical research, and financial analysis,<strong> </strong>where real-time and contextually rich responses are critical.</p><p>Here is a diagram illustrating how RAG functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*WFKs6Jb1gfNv3NQV" /></figure><ul><li><strong>Data Set:</strong> The knowledge source that the LLM can retrieve information from.</li><li><strong>Vector Database (VD):</strong> Stores preprocessed documents as vector representations for efficient retrieval.</li><li><strong>User Query:</strong> The prompt asking for information (e.g., “What phone plan is best?”).</li><li><strong>Embedding Model:</strong> Converts both documents and user queries leveraging Encoder model into vector format for comparison.</li><li><strong>Retrieval Step:</strong> The system searches the Vector Database for semantically similar documents.</li><li><strong>Conversion Back to Text:</strong> The retrieved vectors are mapped back to their original textual content.</li><li><strong>Pre-trained LLM:</strong> The model combines retrieved knowledge with its pre-trained understanding.</li><li><strong>Final Output:</strong> A response that is more informed and context-aware.</li></ul><p>Here is an example of RAG code snippet: (Using LangChain with FAISS for Document Retrieval)</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain.vectorstores import FAISS<br>from langchain.embeddings.openai import OpenAIEmbeddings<br>from langchain.document_loaders import TextLoader<br>from langchain.chains import RetrievalQA<br>import os<br><br># Set API Keys<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;your_openai_api_key&quot;<br><br># Load and Embed Documents<br>documents = TextLoader(&quot;knowledge_base.txt&quot;).load()<br>vectorstore = FAISS.from_documents(documents, OpenAIEmbeddings())<br><br># Initialize LLM and Retrieval QA<br>llm = ChatOpenAI(model=&quot;gpt-4&quot;, temperature=0)<br>retriever = vectorstore.as_retriever()<br>qa_chain = RetrievalQA.from_chain_type(llm=llm, chain_type=&quot;stuff&quot;, retriever=retriever)<br><br># Ask a Question<br>question = &quot;What are the key takeaways from the document?&quot;<br>retrieved_docs = retriever.get_relevant_documents(question)<br><br>for i, doc in enumerate(retrieved_docs):<br>    print(f&quot;Document {i+1}:\n{doc.page_content}\n&quot;)<br><br>#Augmentation and Response Generation (Pre-trained LLM + Retrieved Context)<br>response = qa_chain.run(question)<br>print(response)</pre><p><strong>Limitations of RAG: Context Size Constraints</strong></p><p>While RAG is a powerful technique for enhancing LLMs with external knowledge, it’s not without its limitations. One of the most significant challenges is its dependency on the LLM’s context window size.</p><ul><li><strong>Context Window Size</strong>: LLMs have a fixed context window (e.g., 4,096 tokens for GPT-3.5 or 8,192 tokens for GPT-4, 128k for GPT-4o, Gemini 1.5 Pro 2 million)*. This limits the amount of information that can be retrieved and processed in a single query.</li><li><strong>Retrieval vs. Context</strong>: RAG retrieves relevant documents from a knowledge base, but if the combined size of the retrieved documents and the user query exceeds the LLM’s context window, the system must truncate or omit information. This can lead to incomplete or suboptimal responses.</li><li><strong>Workarounds for context size limitation: </strong>Breaking down large documents into smaller, manageable chunks that fit within the context window. Furthermore, using a two-step retrieval process — first retrieving a broad set of documents and then refining the selection to the most relevant ones. Summarizing or compressing retrieved documents to reduce their size while retaining key information.</li></ul><h3>The Power of AI Agents</h3><p>In contrast, AI Agents has a few key abilities that make it so effective:</p><ol><li><strong>Tools</strong> — These could be anything from weather APIs and search engines to even other AI systems, like a RAG system, for pulling specific document-based information. Leverage RAG to research historical weather patterns, then using a real-time API for the current temperature.</li><li><strong>Memory</strong> — This allows it to remember past clues and conversations, so it can build context and understand the bigger picture.</li></ol><ul><li><strong>Short-term Memory:</strong> Stores recent interactions within a single conversation session. This helps maintain context within a limited timeframe.</li><li><strong>Long-term Memory:</strong> Persists across multiple interactions, enabling AI Agents to recall historical information from previous conversations.</li><li><strong>Near Real-Time/Streaming Memory:</strong> Enables AI Agents to process and retain information dynamically from continuous streams of input, such as real-time chat logs, sensor data, or event-driven systems.</li><li><strong>Passive Memory:</strong> Passively stores and retrieves relevant historical information when needed, rather than continuously updating in real time.</li></ul><p><strong>3. Planning</strong> — AI Agents analyze the problem, figure out the best sequence of actions to solve it, and decide which tools to use and when. It’s not about random actions — it’s about strategic thinking and careful analysis to get the job done right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*xPBVpDafcde0p5YP" /></figure><h4>What Kind of Analysis Does an AI Agent Perform?</h4><p>Some of the analysis performed by the AI Agents:</p><ul><li><strong>Dependency Analysis:</strong> Determining what tasks rely on each other and sequencing them logically. For example, if you ask an AI Agent to book a flight, it may first check your existing calendar, ensure passport validity, and only then proceed with booking.</li><li><strong>Constraint Handling:</strong> Taking into account rules, budgets, deadlines, and preferences. If you tell an AI Agent to find the best flight, it can cross-check price limits, layover preferences, and airline loyalty programs.</li><li><strong>Optimization:</strong> It doesn’t just provide options but optimizes them based on multiple factors. A meeting planner AI Agent might suggest a conference room that minimizes travel time between your meetings while ensuring all required attendees can join.</li><li><strong>Decision Making:</strong> AI Agents can proactively suggest actions, not just provide data. For example, instead of just giving you the weather report, it might say, <em>“You have an outdoor event tomorrow, and there’s a 70% chance of rain. Should I reschedule it or find an indoor venue?”</em></li></ul><h4>AI Agents Go Beyond Information Retrieval — Action</h4><p>An AI Agent doesn’t stop at retrieving information — it can execute actions on your behalf if you allow it. For example:</p><ul><li><strong>Personal Calendar Planning:</strong> Imagine telling an AI Agent, “I have a gym session tomorrow, a client call in the morning, and I need to book a flight in the afternoon. Plan my day for me.” The AI Agent would: Retrieve your existing commitments. Suggest the best slot for your flight booking, avoiding conflicts. Automatically reschedule flexible tasks like gym sessions. And set reminders based on your preferences.</li><li><strong>Smart Home Automation:</strong> AI Agents can interact with IoT devices to execute actions. For example, <em>“If the room temperature exceeds 75°F, turn on the air conditioner.”</em></li><li><strong>Business Workflow Automation:</strong> AI Agents can automate workflows by integrating with CRM systems, managing customer queries, or handling order processing based on real-time data.</li></ul><p>This ability to plan, analyze, and take actions dynamically is what makes AI Agents significantly different from RAG-based retrieval systems.</p><p>This concept is also known as <strong>ReAct</strong> (Reasoning and Acting), it is the ability of an AI model to both reason and take actions</p><p>Following is an example of AI Agent code snippet: (simulate Customer support use case)</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain.agents import Tool, AgentExecutor, create_openai_tools_agent<br>from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder<br>from langchain.memory import ConversationBufferMemory<br>import os<br><br># Set API Keys<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;your_openai_api_key&quot;<br><br># Define Tools<br>def search_knowledge_base(query: str) -&gt; str:<br>    return f&quot;Here are some articles that might help: [Article 1, Article 2, Article 3]&quot;<br><br>def check_order_status(order_id: str) -&gt; str:<br>    return f&quot;Order {order_id} is out for delivery.&quot;<br><br>tools = [<br>    Tool(<br>        name=&quot;SearchKnowledgeBase&quot;,<br>        func=search_knowledge_base,<br>        description=&quot;Useful for finding customer support articles.&quot;<br>    ),<br>    Tool(<br>        name=&quot;CheckOrderStatus&quot;,<br>        func=check_order_status,<br>        description=&quot;Useful for checking the status of an order.&quot;<br>    )<br>]<br><br># Initialize LLM and Memory<br>llm = ChatOpenAI(model=&quot;gpt-4&quot;, temperature=0)<br>memory = ConversationBufferMemory(memory_key=&quot;chat_history&quot;, return_messages=True)<br><br># Create Agent<br>prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a helpful customer support AI. Use tools and memory to assist the user.&quot;),<br>    MessagesPlaceholder(variable_name=&quot;chat_history&quot;),<br>    (&quot;user&quot;, &quot;{input}&quot;)<br>])<br><br>agent = create_openai_tools_agent(llm, tools, prompt)<br>agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True)<br><br># Simulate a Customer Interaction<br>user_query = &quot;Hi, I need help with my order #12345.&quot;<br>response = agent_executor.invoke({&quot;input&quot;: user_query})<br><br>print(&quot;AI Response:&quot;, response[&quot;output&quot;])<br><br># Follow-up Query (Demonstrates Memory)<br>follow_up_query = &quot;Can you also suggest some articles for troubleshooting?&quot;<br>response = agent_executor.invoke({&quot;input&quot;: follow_up_query})<br><br>print(&quot;AI Response:&quot;, response[&quot;output&quot;])</pre><h4>The Role of Multi-Agent Systems</h4><p>The field of AI Agents is increasingly focusing on <strong>Multi-Agent Systems (MAS)</strong>, where multiple agents collaborate to achieve complex goals. Multi-Agent Systems leverage the strengths of different AI Agents specialized in specific tasks, allowing for scalability, modularity, and enhanced reasoning.</p><p>Several frameworks are driving this multi-agent paradigm, including:</p><ul><li><strong>LangGraph</strong> — A framework for building multi-agent workflows, enabling agents to interact with structured logic*.</li><li><strong>CrewAI</strong> — A system designed for orchestrating and managing multiple AI agents working on different tasks*.</li><li><strong>AWS Bedrock Agents</strong> — A platform offering agent-based AI solutions integrated into AWS infrastructure*.</li></ul><p>These frameworks support connected and enterprise-scale AI Agents that can process inputs in parallel or sequential workflows, depending on the complexity of the tasks.</p><h4><strong>Expanded Frameworks for Multi-Agent Systems</strong></h4><p>Furthermore, the AI Agent ecosystem is rapidly growing, with several frameworks emerging to support multi-agent collaboration. In addition to LangGraph, CrewAI, and AWS Bedrock Agents, here are a few other notable frameworks:</p><ul><li><strong>Llama Index</strong>: A framework designed for building and managing multi-agent systems, particularly focused on efficient data retrieval and context management*.</li><li><strong>Langroid</strong>: A lightweight and flexible framework for orchestrating multi-agent workflows, with a focus on natural language interactions and task delegation*.</li><li><strong>Google’s Multi-Agent Framework</strong>: Recently announced, this framework aims to provide a runtime environment for managing multi-agent systems at scale, including support for SaaS integrations*.</li></ul><p>These frameworks are pushing the boundaries of what’s possible with multi-agent systems, enabling more complex and scalable AI-driven applications.</p><p>Example: Multi-Agent Code Generation with LangGraph</p><pre>from langgraph.graph import Graph<br>from langchain.agents import Tool<br>from langchain.chat_models import ChatOpenAI<br>from langchain.agents import initialize_agent<br>from langchain.prompts import PromptTemplate<br><br># Define the CodeGeneratorAgent<br>def code_generator(query: str) -&gt; str:<br>    llm = ChatOpenAI(model=&quot;gpt-3.5-turbo&quot;)<br>    prompt = PromptTemplate(<br>        input_variables=[&quot;query&quot;],<br>        template=&quot;Generate Python code for: {query}&quot;<br>    )<br>    chain = prompt | llm<br>    return chain.invoke({&quot;query&quot;: query}).content<br><br># Define the CodeReviewerAgent<br>def code_reviewer(code: str) -&gt; str:<br>    llm = ChatOpenAI(model=&quot;gpt-3.5-turbo&quot;)<br>    prompt = PromptTemplate(<br>        input_variables=[&quot;code&quot;],<br>        template=&quot;Review the following Python code and provide feedback:\n\n{code}&quot;<br>    )<br>    chain = prompt | llm<br>    return chain.invoke({&quot;code&quot;: code}).content<br><br># Define the Multi-Agent Workflow using LangGraph<br>def create_multi_agent_workflow():<br>    # Create a graph<br>    workflow = Graph()<br><br>    # Add nodes for each agent<br>    workflow.add_node(&quot;generate_code&quot;, code_generator)<br>    workflow.add_node(&quot;review_code&quot;, code_reviewer)<br><br>    # Define the edges (flow of the workflow)<br>    workflow.add_edge(&quot;generate_code&quot;, &quot;review_code&quot;)<br><br>    # Set the entry point<br>    workflow.set_entry_point(&quot;generate_code&quot;)<br><br>    # Compile the workflow<br>    return workflow.compile()<br><br># Run the Multi-Agent Workflow<br>def run_workflow(query: str):<br>    workflow = create_multi_agent_workflow()<br>    result = workflow.invoke({&quot;query&quot;: query})<br>    return result<br><br># Example Usage<br>if __name__ == &quot;__main__&quot;:<br>    user_query = &quot;a function to calculate the factorial of a number&quot;<br>    print(&quot;Generating code and reviewing it...&quot;)<br>    output = run_workflow(user_query)<br>    print(&quot;Generated Code:\n&quot;, output[&quot;generate_code&quot;])<br>    print(&quot;\nCode Review:\n&quot;, output[&quot;review_code&quot;])<br></pre><h4><strong>Structuring Multi-Agent Business Logic with Prompting Parameters</strong></h4><p>For Multi-Agent Systems to function effectively, business logic and input structures must be well-defined. This includes:</p><ul><li><strong>Prompting Parameters</strong> — Inputs that guide AI Agent behavior, similar to API parameters.</li><li><strong>Business Logic Encoding</strong> — Defining rules that dictate how AI Agents interact with tools and memory.</li><li><strong>Standardized Input Models</strong> — Using structured frameworks such as Python’s Pydantic to enforce input validation and ensure data integrity when passing information between agents.</li></ul><h4><strong>The Evolution of RAG: Agentic RAG</strong></h4><p>While traditional RAG focuses on retrieving and augmenting information, <strong>Agentic RAG</strong> takes this a step further by integrating the retrieval process with AI Agent capabilities. In this advanced pattern, the system doesn’t just retrieve information — it actively decides how to use it.</p><p>For example, an Agentic RAG system might:</p><ul><li>Dynamically select which knowledge sources to query based on the task.</li><li>Refine its retrieval strategy in real-time using feedback from the LLM or user.</li><li>Combine retrieved information with other tools (e.g., APIs or databases) to generate more comprehensive responses.</li></ul><p>This approach is particularly useful in complex scenarios where static retrieval isn’t enough. For instance, in legal research, an Agentic RAG system could retrieve case law, analyze precedents, and even suggest arguments based on the retrieved data.</p><p>Agentic RAG represents the next evolution of retrieval-augmented systems, further blurring the lines between retrieval and action.</p><h4>Limitations of AI Agents</h4><p>While AI Agents excel at decision-making, planning, and executing tasks, they are not without their challenges. Here are some of the key limitations:</p><ul><li><strong>Development Complexity</strong>: Building AI Agents requires significant expertise in AI, machine learning, and software engineering. Designing agents that can plan, reason, and interact with multiple tools is inherently complex.</li><li><strong>Tool Reliability and Integration Challenges</strong>: AI Agents rely heavily on external tools (e.g., APIs, databases) to perform tasks. If these tools are unavailable, slow, or unreliable, the agent’s performance can suffer. Connecting AI Agents to diverse tools and systems can be technically challenging, especially when dealing with proprietary or poorly documented APIs.</li><li><strong>Context Window Constraints</strong>: Like RAG, AI Agents are limited by the LLM’s context window size. This restricts the amount of historical data or conversation context the agent can retain.</li><li><strong>LLM Hallucinations</strong>: AI Agents powered by LLMs can sometimes generate incorrect or nonsensical responses, especially when dealing with ambiguous or incomplete information.</li><li><strong>Surface-Level Reasoning</strong>: While AI Agents can simulate reasoning and planning, they lack true understanding or common sense. Their decisions are based on patterns in data rather than deep comprehension.</li><li><strong>Brittleness</strong>: AI Agents may struggle with tasks that require nuanced understanding or adaptability outside their training data.</li><li><strong>Workarounds</strong>: Developing standardized interfaces and protocols for tool integration can reduce complexity and improve reliability. With advances in LLMs with larger context windows (e.g., 100K+ tokens) can mitigate memory limitations. In the meantime, incorporating human oversight can help catch errors and ensure ethical decision-making. Furthermore, developing methods to make AI Agents’ decision-making processes more transparent and interpretable.</li></ul><h3>So, What’s the Bottom Line?</h3><p>Knowing the difference between RAG and AI Agents is like choosing the right tool for the job.</p><ul><li><strong>RAG</strong> gives LLMs a boost with extra knowledge, like adding research notes to a paper.</li><li><strong>AI Agents</strong> are more like project managers, making decisions and taking action.</li></ul><p>Currently, ReAct (Reasoning and Acting) is a leading concept for AI Agents, enabling them to think through complex tasks and act strategically. But the future looks even more exciting. We’re moving toward Multi-Agent Systems, where teams of AI Agents collaborate to tackle large-scale projects — think of it like a superhero squad!</p><p>Frameworks like LangGraph, CrewAI, AWS Bedrock, and Llama Index are already paving the way for this shift, and new, emerging frameworks are poised to push the boundaries even further. By structuring agent inputs with prompting parameters and validated schemas, we can create more reliable, scalable AI-driven applications.</p><p>Now, I’m curious — are any of you diving into Multi-Agent Systems? How do you think this will change the AI landscape? Share your thoughts or projects in the comments below — I’d love to hear what you’re working on!</p><p>Here are some misconceptions about RAG and AI Agents:</p><p><strong>Is RAG itself an agent?</strong></p><ul><li>No. RAG is a retrieval-based augmentation technique, while AI Agents perform structured execution beyond retrieval. As a matter of fact, RAG can be used as one of AI-Agent’s tools (more commonly used tools are APIs, but RAG is currently part of that tool set, a.k.a <strong>Agentic RAG</strong></li></ul><p><strong>How do AI Agents decide which tools to use?</strong></p><ul><li>Agents use predefined logic or reinforcement learning (e.g., ReAct framework, ToolFormer) to determine tool usage dynamically.</li></ul><p><strong>Why does LLM-generated output sometimes differ from retrieved documents?</strong></p><ul><li>LLMs reframe, summarize, or hallucinate based on prompt design. Proper <strong>retrieval validation</strong> is critical.</li></ul><p><strong>Is history-aware retrieval the same as an AI Agent?</strong></p><ul><li>No. While history-aware retrievers improve contextual grounding, they lack autonomous decision-making abilities.</li></ul><p><strong>Why do we need memory in AI Agents?</strong></p><ul><li>To handle multi-turn queries effectively. Without memory, LLMs lose the conversation context.</li></ul><p><strong>How can we validate RAG-generated responses?</strong></p><ul><li>To make sure RAG-generated responses are accurate and relevant, we can use metrics like retrieval precision, embedding similarity, and human checks. These help confirm that the information retrieved is on point and the final answer makes sense in context.</li></ul><p><strong>Is zero-shot reasoning sufficient for AI Agents?</strong></p><ul><li>Not really. While zero-shot reasoning lets AI Agents handle tasks without prior examples, it often falls short in complex situations. For tasks that need multi-step planning or deeper context, AI Agents usually rely on memory or external reasoning tools to work effectively.</li></ul><p><strong>References</strong></p><ul><li>LangChain. (n.d.). <em>ReAct Agent</em>. Retrieved from <a href="https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/">https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/</a></li><li>Hugging Face. (n.d.). <em>Retrieval-Augmented Generation (RAG)</em>. Retrieved from <a href="https://huggingface.co/docs/transformers/model_doc/rag">https://huggingface.co/docs/transformers/model_doc/rag</a></li><li>AWS. (n.d.). <em>Agents for Amazon Bedrock</em>. Retrieved from <a href="https://aws.amazon.com/bedrock/agents/">https://aws.amazon.com/bedrock/agents/</a></li><li>CrewAI. (n.d.). <em>CrewAI: Orchestrating Multi-Agent Workflows</em>. Retrieved from <a href="https://docs.crewai.com/introduction">https://docs.crewai.com/introduction</a></li><li>Llama Index. (n.d.). <em>Llama Index: A Framework for Multi-Agent Systems</em>. Retrieved from <a href="https://gpt-index.readthedocs.io/">https://gpt-index.readthedocs.io/</a></li><li>Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv preprint arXiv:2302.04761 <a href="https://ai.meta.com/research/publications/toolformer-language-models-can-teach-themselves-to-use-tools/">https://ai.meta.com/research/publications/toolformer-language-models-can-teach-themselves-to-use-tools/</a></li><li>Optimizing Token Utilization: Considerations and Settings for OpenAI Models <a href="https://www.linkedin.com/pulse/optimizing-token-utilization-considerations-settings-openai-russell-z5b3c/">https://www.linkedin.com/pulse/optimizing-token-utilization-considerations-settings-openai-russell-z5b3c/</a></li><li>Google’s Gemini 1.5 Pro Will Have 2 Million Tokens. Here’s What That Means <a href="https://www.cnet.com/tech/services-and-software/googles-gemini-1-5-pro-will-have-2-million-tokens-heres-what-that-means/">https://www.cnet.com/tech/services-and-software/googles-gemini-1-5-pro-will-have-2-million-tokens-heres-what-that-means/</a></li><li>Chain of Agents: Large language models collaborating on long-context tasks <a href="https://research.google/blog/chain-of-agents-large-language-models-collaborating-on-long-context-tasks/">https://research.google/blog/chain-of-agents-large-language-models-collaborating-on-long-context-tasks/</a></li><li>Google AI Introduces PlanGEN: A Multi-Agent AI Framework Designed to Enhance Planning and Reasoning in LLMs through Constraint-Guided Iterative Verification and Adaptive Algorithm Selection <a href="https://www.marktechpost.com/2025/02/28/google-ai-introduces-plangen-a-multi-agent-ai-framework-designed-to-enhance-planning-and-reasoning-in-llms-through-constraint-guided-iterative-verification-and-adaptive-algorithm-selection/">https://www.marktechpost.com/2025/02/28/google-ai-introduces-plangen-a-multi-agent-ai-framework-designed-to-enhance-planning-and-reasoning-in-llms-through-constraint-guided-iterative-verification-and-adaptive-algorithm-selection/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10df56b35e02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/olarry/understanding-the-difference-between-rag-and-ai-agents-10df56b35e02">Understanding the Difference Between RAG and AI Agents</a> was originally published in <a href="https://medium.com/olarry">OLarry</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[QSBS stock tax exemption: The entrepreneurs’ advantage]]></title>
            <link>https://medium.com/olarry/qsbs-stock-tax-exemption-the-entrepreneurs-advantage-eea9f4cd85a3?source=rss----8520b8c773b7---4</link>
            <guid isPermaLink="false">https://medium.com/p/eea9f4cd85a3</guid>
            <category><![CDATA[taxes]]></category>
            <category><![CDATA[rsu]]></category>
            <category><![CDATA[qsbs]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[entrepreneur]]></category>
            <dc:creator><![CDATA[Becky Hobbs]]></dc:creator>
            <pubDate>Tue, 09 Jul 2024 14:13:40 GMT</pubDate>
            <atom:updated>2024-07-09T14:14:57.306Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Entreprenure closing on the sale of her company using QSBS" src="https://cdn-images-1.medium.com/max/1024/1*KcDpv8C208jie58vzssl5w@2x.png" /></figure><p>This story was originally published at: <a href="https://olarry.com/qsbs/">https://olarry.com/qsbs/</a></p><p>Entrepreneurs often speak in a language that can be confusing if you are not fluent in ‘founder’. Here, we will help distill and demystify one of the best tax strategies for preparing to start a new business: QSBS. Making the tax code work for you, you could see upwards of a $10M capital gains exclusion and, in fact, more if you choose to stack your QSBS exclusion.</p><ul><li>QSBS allows a shareholder to exclude gains arising from the sale of a company’s stock from taxable income.</li><li>You can exclude up to $10M in capital gains or 10 times your original investment.</li><li>To be eligible for the federal tax exclusion, the company must be a U.S.-based C corporation. State eligibility depends on the state, as not all states allow the exclusion.</li><li>Serial entrepreneurs should consider rolling their QSBS over into new QSBs to accelerate the holding period and increase the exclusion gain.</li></ul><h3>What exactly is what is QSBS?</h3><p>Qualified Small Business Stock (QSBS) refers to shares (stock) in a Qualified Small Business (QSB). It was created as a tax incentive to encourage founders, early employees, and investors to risk joining, founding, or investing in a start-up, which encourages innovation and entrepreneurship. The specific Internal Revenue Code (IRC) that pertains to QSBS is Section 1202 and was originally enacted as part of the Budget Reconciliation Act of 1993, at which time gain exclusion was limited to 50–75% of the total eligible gain, depending on the date of the stock sale. In September 2010, these limits were amended to current-day levels, with the most recent update in 2015 making current exclusionary limits permanent.</p><h3>How does QSBS exclusion work?</h3><p>Under Section 1202, shareholders of QSBS stock acquired after 9/27/2010 (the date the limit increased) can exclude the larger of either $10m in capital gains or 10 times their original investment from federal taxation upon the sale of the stock. If they originally invested $2M, they could exclude up to $20M in capital gains. If they invested $1K, they could exclude up to $10M in capital gains. Prior to 9/27/2010 a portion of QSBS excluded gain needed to be added back for AMT (Alternative Minimum Tax) purposes. However, any QSBS acquired after 9/27/10 is also excluded from alternative minimum taxation.</p><p>Any gains excluded from capital gains tax due to QSBS treatment are also excluded from the 3.8% net investment income tax.</p><h4>How it breaks down</h4><p>The qualified small business company you work for has just sold for $500M, and you own 2% of the company — making your share of the sales proceeds $10M. Let’s assume your cost basis (the amount you originally invested) is zero to keep the math simpler.</p><p>If your stock did not qualify for QSBS treatment, you would owe the following taxes:</p><ul><li>$10M X 20% capital gains tax rate = $2M in federal capital gain taxes.</li><li>$10M X 3.8% net investment income tax rate = 380K in net investment income taxes</li><li>Total taxes would be $2,380,000.</li></ul><p>However, if the sale of your stock does qualify for the QSBS exclusion, you would save $2.38M in federal taxes. Note: states like California do not conform to QSBS treatment. California’s tax rate is up to 13.3%, which means you could owe up to $1.33M in taxes in this scenario.</p><h3>Effective tax planning to minimize taxation for QSBS</h3><p>There are 3 different strategies that can be leveraged when anticipating a gain from QSBS stock.</p><ol><li><strong>83B election: </strong>This strategy must be employed when the stock is granted. It allows the taxpayers to elect to be taxed on the fair market value of the restricted stock at the time of grant rather than at the time of vesting, which can greatly reduce the amount of tax owed. This election also triggers the holding period for QSBS to be the date of grant rather than the vesting date.</li><li><strong>Exclusion stacking: </strong>If a QSBS holder anticipates that the stock gain will exceed the exclusion cap, they can gift the QSBS to their children either directly or via trusts. The children can then also claim the QSBS exclusion on their tax returns upon the sale of the stock. QSBS is generally viewed as being a per-taxpayer exclusion. However, arguments have been successfully made that holders of QSBS can gift shares to their spouse, who can then also exclude capital gains, effectively increasing the exclusion to $20M on a married filing jointly tax return.</li><li><strong>1045 rollover</strong>: If the QSBS stock has not reached the 5-year holding threshold, the 1045 rollover allows the holder to roll the proceeds from the sale of the QSBS over to reinvest into another QSB. Allowing the holder to retain the original holding period, potentially resulting in an increase in the amount of gain excluded.</li></ol><h3>What if my company was acquired?</h3><p>QSBS eligibility can also roll forward. Let’s say your company is acquired, with a portion of the purchase being made in cash and the remainder in stock in the acquiring company. Even if the acquiring company is not a QSB, your stock in the acquirer would carry over QSB status — allowing you to exclude gains further down the road should you sell. However, any additional appreciation in the acquired stock would be subject to taxation.</p><h4>How it breaks down:</h4><p>In our previous example, your company, of which you hold 2%, sold for $500M cash. Instead, let’s assume the $500M sale was split between $250M in cash and $250M in stock in the acquiring company; let’s call the acquiring company “Company B”.</p><p>In this scenario, only the $250M in cash would be taken into account for gain calculation purposes in the year of the sale. The $250M in stock received in the acquiring company would not trigger an income event until you sold that stock further down the line.</p><ul><li>$250M in cash X 2% = $5M — your share in the cash proceeds. Excluded from taxes due to QSBS treatment</li><li>$250M in stock of Company B x 2% = $5M in fair market value on the date of the sale.</li></ul><p>2 years later, Company B is now worth double what it was on the date of the transaction, and you sell your shares for $10M</p><ul><li>$10M in proceeds <em>less </em>$0 cost basis = $10M Gain</li><li>$10M Gain <em>less </em>than $5M of gain attributed to your original company’s stock, which qualifies for QSBS exclusion, results in only $5M in taxable gain.</li><li>Total QSBS exclusion is $10M: 5M in year 0 related to the cash portion of the sale. $5M in year 2 after the sale of Company B’s stock</li></ul><h3>How do you qualify for QSBS?</h3><p>To determine whether your stock options qualify for QSBS, you must consider the <strong>corporation’s eligibility requirements</strong> and the requirements of the <strong>individual shareholder</strong> who acquired the shares.</p><h4>Corporation Eligibility Requirements to be a Qualified Small Business (QSB)</h4><ul><li>Must be a U.S.-based C Corporation. Being a C-corporation also allows founders to take advantage of the TCJA corporate income tax rates, which reduced them from 35% to a flat 21%.</li><li>Aggregate gross assets must not exceed $50m before and immediately after stock issuance</li><li>Must be an active business, not a passive investment company, meaning at least 80% of the corporation’s assets must be used actively in a qualified trade or business during substantially all of the shareholders holding period of the stock.</li><li>No more than 10% of the corporation’s assets can be stocks or securities of other companies in which it does not own a majority interest.</li><li>Generally, the following industries are excluded from qualifying for QSBS treatment: health, law, engineering, architecture, accounting, actuarial services, performing arts, athletics, financial services, brokerage services, and consulting.</li></ul><h4>Stock Shareholder Requirements</h4><ul><li>The shareholder must have acquired the (QSBS) stock at its original issuance and paid for it in money or property, or it must have been given as compensation for services provided. This means the stock must have been acquired directly from the issuing corporation and not via a secondary market transaction.</li><li>The stock must have been held for at least 5 years before selling. If the company is acquired and stock is issued by the acquirer in exchange for the QSBS, the holding period still carries over. (similar to QSBS status mentioned above).</li><li>Must be actual stock, held for the required holding period, not stock options, meaning the options must be exercised first in order to start the holding period.</li></ul><p><em>Any tax advice herein is not intended or written to be used.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eea9f4cd85a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/olarry/qsbs-stock-tax-exemption-the-entrepreneurs-advantage-eea9f4cd85a3">QSBS stock tax exemption: The entrepreneurs’ advantage</a> was originally published in <a href="https://medium.com/olarry">OLarry</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>