<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Frank Wittkampf on Medium]]></title>
        <description><![CDATA[Stories by Frank Wittkampf on Medium]]></description>
        <link>https://medium.com/@frankw_usa?source=rss-556a60f9c64b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NKA6NcV6ulN5ingXiivgKQ.png</url>
            <title>Stories by Frank Wittkampf on Medium</title>
            <link>https://medium.com/@frankw_usa?source=rss-556a60f9c64b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:04:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@frankw_usa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spoiler Alert: The Magic of RAG Does Not Come from AI]]></title>
            <link>https://medium.com/data-science/spoiler-alert-the-magic-of-rag-does-not-come-from-ai-8a0ed2ad4800?source=rss-556a60f9c64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a0ed2ad4800</guid>
            <category><![CDATA[enterprise-technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[applied-ai]]></category>
            <dc:creator><![CDATA[Frank Wittkampf]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 14:03:08 GMT</pubDate>
            <atom:updated>2024-11-19T03:36:14.006Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why retrieval, not generation, makes RAG systems magical</h4><h4>Quick POCs</h4><p>Most quick proof of concepts (POCs) which allow a user to explore data with the help of conversational AI simply blow you away. It feels like pure magic when you can all of a sudden talk to your documents, or data, or code base.</p><p>These POCs work wonders on small datasets with a limited count of docs. However, as with almost anything when you bring it to production, you quickly run into problems at scale. When you do a deep dive and you inspect the answers the AI gives you, you notice:</p><ul><li>Your agent doesn’t reply with complete information. It missed some important pieces of data</li><li>Your agent doesn’t reliably give the same answer</li><li>Your agent isn’t able to tell you how and where it got which information, making the answer significantly less useful</li></ul><p>It turns out that the <strong>real magic in RAG</strong> does not happen in the generative AI step, but in the process of retrieval and composition. Once you dive in, it’s pretty obvious why…</p><p><em>* RAG = Retrieval Augmented Generation — </em><a href="https://en.wikipedia.org/wiki/Retrieval-augmented_generation"><em>Wikipedia Definition of RAG</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*JKXH3z6q5S_CIrofFt37pw.png" /><figcaption>RAG process — Illustration</figcaption></figure><h3>So, how does a RAG-enabled AI agent answer a question?</h3><p>A quick recap of how a simple RAG process works:</p><ol><li>It all starts with a <strong>query</strong>. The user asked a question, or some system is trying to answer a question.</li><li>A <strong>search</strong> is done with the query. Mostly you’d embed the query and do a similarity search, but you can also do a classic elastic search or a combination of both, or a straight lookup of information</li><li>The search result is a set of <strong>documents</strong> (or document snippets, but let’s simply call them documents for now)</li><li>The documents and the essence of the query are combined into some easily readable <strong>context</strong> so that the AI can work with it</li><li>The <strong>AI interprets</strong> the question and the documents and generates an answer</li><li>Ideally this answer is <strong>fact checked</strong>, to see if the AI based the answer on the documents, and/or if it is appropriate for the audience</li></ol><h3>Where’s the magic?</h3><p>The dirty little secret is that the essence of the RAG process is that you have to provide the answer to the AI (before it even does anything), so that it is able to give you the reply that you’re looking for.</p><p>In other words:</p><ul><li>the work that the AI does (step 5) is <strong>apply judgement, and properly articulate the answer</strong></li><li>the work that the engineer does (step 3 and 4) is <strong>find the answer and compose it such that AI can digest it</strong></li></ul><p>Which is more important? The answer is, of course, it depends, because if judgement is the critical element, then the AI model does all the magic. But for an endless amount of business use cases, finding and properly composing the pieces that make up the answer, is the more important part.</p><h3>What are the typical engineering problems to solve if you want proper RAG process?</h3><p>The first set of problems to solve when running a RAG process are the data ingestion, splitting, chunking, document interpretation issues. I’ve written about a few of these in <a href="https://medium.com/@frankw_usa">prior articles</a>, but am ignoring them here. For now let’s assume you have properly solved your data ingestion, you have a lovely vector store or search index.</p><p>Typical challenges:</p><ul><li><strong>Duplication </strong>— Even the simplest production systems often have duplicate documents. More so when your system is large, you have extensive users or tenants, you connect to multiple data sources, or you deal with versioning, etc.</li><li><strong>Near duplication</strong> — Documents which largely contain the same data, but with minor changes. There are two types of near duplication: <br> — Meaningful — E.g. a small correction, or a minor addition, e.g. a date field with an update<br> — Meaningless — E.g.: minor punctuation, syntax, or spacing differences, or just differences introduced by timing or intake processing</li><li><strong>Volume</strong> — Some queries have a very large relevant response data set</li><li><strong>Data freshness vs quality</strong> — Which snippets of the response data set have the most high quality content for the AI to use vs which snippets are most relevant from a time (freshness) perspective?</li><li><strong>Data variety</strong> — How do we ensure a variety of search results such that the AI is properly informed?</li><li><strong>Query phrasing and ambiguity</strong> — The prompt that triggered the RAG flow, might not be phrased in such a way that it yields the optimal result, or might even be ambiguous</li><li><strong>Response Personalization</strong> — The query might require a different response based on who asks it</li></ul><p>This list goes on, but you get the gist.</p><h3>Sidebar: Don’t unlimited context windows solve this?</h3><p>Short answer: no.</p><p>The cost and performance impact of using extremely large context windows shouldn’t be underestimated (you easily 10x or 100x your per query cost), not including any follow up interaction that the user/system has.</p><p>However, putting that aside. Imagine the following situation.</p><blockquote>We put Anne in room with a piece of paper. The paper says: *patient Joe: complex foot fracture.* Now we ask Anne, does the patient have a foot fracture? Her answer is “yes, he does”.<br><br>Now we give Anne a hundred pages of medical history on Joe. Her answer becomes “well, depending on what time you are referring to, he had …”<br><br>Now we give Anne thousands of pages on all the patients in the clinic…</blockquote><p>What you quickly notice, is that how we define the question (or the prompt in our case) starts to get very important. <strong>The larger the context window, the more nuance the query needs.</strong></p><p>Additionally, <strong>the larger the context window,</strong> <strong>the universe of possible answers grows.</strong> This can be a positive thing, but in practice, it’s a method that invites lazy engineering behavior, and is likely to reduce the capabilities of your application if not handled intelligently.</p><h3>Suggested approaches</h3><p>As you scale a RAG system from POC to production, here’s how to address typical data challenges with specific solutions.</p><h4>Duplication</h4><p>Duplication is inevitable in multi-source systems. By using fingerprinting (hashing content), document IDs, or semantic hashing, you can identify exact duplicates at ingestion and prevent redundant content. However, consolidating metadata across duplicates can also be valuable; this lets users know that certain content appears in multiple sources, which can add credibility or highlight repetition in the dataset.</p><pre># Fingerprinting for deduplication<br>def fingerprint(doc_content):<br>    return hashlib.md5(doc_content.encode()).hexdigest()<br><br># Store fingerprints and filter duplicates, while consolidating metadata<br>fingerprints = {}<br>unique_docs = []<br>for doc in docs:<br>    fp = fingerprint(doc[&#39;content&#39;])<br>    if fp not in fingerprints:<br>        fingerprints[fp] = [doc]<br>        unique_docs.append(doc)<br>    else:<br>        fingerprints[fp].append(doc)  # Consolidate sources</pre><h4>Near Duplication</h4><p>Near-duplicate documents (similar but not identical) often contain important updates or small additions. Given that a minor change, like a status update, can carry critical information, freshness becomes crucial when filtering near duplicates. A practical approach is to use cosine similarity for initial detection, then retain the freshest version within each group of near-duplicates while flagging any meaningful updates.</p><pre>from sklearn.metrics.pairwise import cosine_similarity<br>from sklearn.cluster import DBSCAN<br>import numpy as np<br><br># Cluster embeddings with DBSCAN to find near duplicates<br>clustering = DBSCAN(eps=0.1, min_samples=2, metric=&quot;cosine&quot;).fit(doc_embeddings)<br><br># Organize documents by cluster label<br>clustered_docs = {}<br>for idx, label in enumerate(clustering.labels_):<br> if label == -1:<br> continue<br> if label not in clustered_docs:<br> clustered_docs[label] = []<br> clustered_docs[label].append(docs[idx])<br><br># Filter clusters to retain only the freshest document in each cluster<br>filtered_docs = []<br>for cluster_docs in clustered_docs.values():<br> # Choose the document with the most recent timestamp or highest relevance<br> freshest_doc = max(cluster_docs, key=lambda d: d[&#39;timestamp&#39;])<br> filtered_docs.append(freshest_doc)</pre><h4>Volume</h4><p>When a query returns a high volume of relevant documents, effective handling is key. One approach is a **layered strategy**:</p><ul><li><strong>Theme Extraction</strong>: Preprocess documents to extract specific themes or summaries.</li><li><strong>Top-k Filtering</strong>: After synthesis, filter the summarized content based on relevance scores.</li><li><strong>Relevance Scoring</strong>: Use similarity metrics (e.g., BM25 or cosine similarity) to prioritize the top documents before retrieval.</li></ul><p>This approach reduces the workload by retrieving synthesized information that’s more manageable for the AI. Other strategies could involve batching documents by theme or pre-grouping summaries to further streamline retrieval.</p><h4>Data Freshness vs. Quality</h4><p>Balancing quality with freshness is essential, especially in fast-evolving datasets. Many scoring approaches are possible, but here’s a general tactic:</p><ul><li><strong>Composite Scoring</strong>: Calculate a quality score using factors like source reliability, content depth, and user engagement.</li><li><strong>Recency Weighting</strong>: Adjust the score with a timestamp weight to emphasize freshness.</li><li><strong>Filter by Threshold</strong>: Only documents meeting a combined quality and recency threshold proceed to retrieval.</li></ul><p>Other strategies could involve scoring only high-quality sources or applying decay factors to older documents.</p><h4>Data Variety</h4><p>Ensuring diverse data sources in retrieval helps create a balanced response. Grouping documents by source (e.g., different databases, authors, or content types) and selecting top snippets from each source is one effective method. Other approaches include scoring by unique perspectives or applying diversity constraints to avoid over-reliance on any single document or perspective.</p><pre># Ensure variety by grouping and selecting top snippets per source<br><br>from itertools import groupby<br><br>k = 3  # Number of top snippets per source<br>docs = sorted(docs, key=lambda d: d[&#39;source&#39;])<br><br>grouped_docs = {key: list(group)[:k] for key, group in groupby(docs, key=lambda d: d[&#39;source&#39;])}<br>diverse_docs = [doc for docs in grouped_docs.values() for doc in docs]</pre><h4>Query Phrasing and Ambiguity</h4><p>Ambiguous queries can lead to suboptimal retrieval results. Using the exact user prompt is mostly not be the best way to retrieve the results they require. E.g. there might have been an information exchange earlier on in the chat which is relevant. Or the user pasted a large amount of text with a question about it.</p><p>To ensure that you use a refined query, one approach is to ensure that a RAG tool provided to the model asks it to rephrase the question into a more detailed search query, similar to how one might carefully craft a search query for Google. This approach improves alignment between the user’s intent and the RAG retrieval process. The phrasing below is suboptimal, but it provides the gist of it:</p><pre>tools = [{ <br>  &quot;name&quot;: &quot;search_our_database&quot;, <br>  &quot;description&quot;: &quot;Search our internal company database for relevant documents&quot;, <br>  &quot;parameters&quot;: { <br>   &quot;type&quot;: &quot;object&quot;, <br>   &quot;properties&quot;: { <br>    &quot;query&quot;: { <br>     &quot;type&quot;: &quot;string&quot;, <br>     &quot;description&quot;: &quot;A search query, like you would for a google search, in sentence form. Take care to provide any important nuance to the question.&quot; <br>    } <br>   }, <br>   &quot;required&quot;: [&quot;query&quot;] <br>  } <br>}]</pre><h4>Response Personalization</h4><p>For tailored responses, integrate user-specific context directly into the RAG context composition. By adding a user-specific layer to the final context, you allow the AI to take into account individual preferences, permissions, or history without altering the core retrieval process.</p><p>By addressing these data challenges, your RAG system can evolve from a compelling POC into a reliable production-grade solution. Ultimately, the effectiveness of RAG relies more on careful engineering than on the AI model itself. While AI can generate fluent answers, the real magic lies in how well we retrieve and structure information. So the next time you’re impressed by an AI system’s conversational abilities, remember that it’s likely the result of an expertly designed retrieval process working behind the scenes.</p><p>I hope this article provided you some insight into the RAG process, and why the magic that you experience when talking to your data isn’t necessarily coming from the AI model, but is largely dependent on the design of your retrieval process.</p><p>Please comment with your thoughts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a0ed2ad4800" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/spoiler-alert-the-magic-of-rag-does-not-come-from-ai-8a0ed2ad4800">Spoiler Alert: The Magic of RAG Does Not Come from AI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Next-Level Agents: Unlocking the Power of Dynamic Context]]></title>
            <link>https://medium.com/data-science/next-level-agents-unlocking-the-power-of-dynamic-context-68b8647eef89?source=rss-556a60f9c64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/68b8647eef89</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[architecture-agent]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Frank Wittkampf]]></dc:creator>
            <pubDate>Wed, 15 May 2024 01:12:19 GMT</pubDate>
            <atom:updated>2024-05-15T01:12:19.115Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How to improve accuracy, speed, and token usage of AI agents</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cot84CsTvC4nC2omybk_Yw.png" /><figcaption><em>Most AI agents differ significantly from each other (image generated with MidJourney)</em></figcaption></figure><h3>Introduction</h3><p>The behavior of an AI agent is defined by two things: (1) the model it runs on and (2) the context that you provide to the model. <strong>How</strong> you provide this context determines the quality of the output of the agent. You can even say that the behavior that distinguishes two AI agents running on the same model is fully defined by their context. So, what do we mean with context for an agent? See the <em>Types of Context (</em>image) below for examples.</p><p>This article delves into more advanced methods to enhance AI agent efficiency and accuracy by selectively tailoring context according to the needs of an agent. First I will describe 5 tactics, and then some additional implementation tips. The learnings in the article come from extensive experience in working with multi-agent teams in a production environment, but are just as applicable to single agent systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQkhkuo6zII49KWH15LxmA.png" /><figcaption>Types of Context for AI agents (image by author)</figcaption></figure><h3>Understanding Contextual Needs</h3><p>Most online examples and tutorials simplify their approach to context (e.g. applying a fixed cutoff length). Real AI applications need a more sophisticated approach.</p><p>Different agents have different contextual needs depending on the task at hand. Some agents may only require just the prior message, while other agents might need extensive historical data to respond accurately. These differences suggest that a tailored approach to context management for each agent is needed.</p><p>Consider the following example of an ongoing conversation between a user and 2 agents (a SW developer and a SW reviewer):</p><pre>1 <strong>User</strong>: “Please improve the sorting function I wrote from the uploaded file”<br>2 <strong>System_prompt_for_coder</strong>: “You are a SW developer, you help the user to <br>  develop …”<br>3 <strong>Coding_agent</strong>: “read_file_function”<br>4 <strong>Function_response</strong>: “coding snippet: ```some returned code```”<br>5 <strong>System_prompt_for_coder</strong>: “You are a SW developer, you help the user to<br>  develop …”<br>6 <strong>Coding_agent</strong>: “I’ve improved your code by adding … and … to your code, <br>  here is the new function …”<br>7 <strong>System_prompt_for_reviewer</strong>: “you are a SW code reviewer, you …”<br>8 <strong>Code_reviewer_agent</strong>: “After reviewing the improved code, there <br>  are a few more things that I would add to the code …&quot;<br>9 <strong>System_prompt_for_coder</strong>: “you are a SW developer, you help the user to <br>  develop …”<br>10 <strong>Coding_agent</strong>: “ … ”</pre><p>The context, as defined in this example is obviously repetitive and inefficient. Many lines are repeated (e.g. the system prompts), and each agent is getting more context than it needs. Let’s review a few ways to improve the context handling.</p><h3>Tactic 1: Message Labeling</h3><p>Message labeling is a critical tool for managing and optimizing the interaction between AI agents and their tasks. By assigning metadata to each message in the conversation, we can smartly select the information that is most relevant to the agent’s task at hand. This tactic involves several key strategies:</p><p><strong>Relevance Labeling</strong>: Each message should be tagged with labels that reflect its relevance to ongoing and future interactions. This process involves analyzing the content of the message and determining its potential utility for the agent’s decision-making processes. For example, messages that contain questions, decisions or insights should be marked as highly relevant.</p><p><strong>Permanence Labeling</strong>: It is vital to categorize messages based on their longevity and usefulness over time. Some messages, such as those containing foundational decisions or milestone communications, hold long-term value and should be retained across sessions. In contrast, system messages might only be needed once in a specific moment. These should be excluded from the agent’s memory once their immediate relevance has passed.</p><p><strong>Source and Association Labeling</strong>: This involves identifying the origin of each message, whether it be from a specific agent, a user, function, or other process. This labeling helps in constructing a structured and easily navigable history that allows agents to efficiently retrieve and reference information based on source or task relevance.</p><p>Applying smart labels to the metadata of a message allows you to use smart selection. Keep reading for some examples.</p><h3>Tactic 2: Agent-specific context requirements</h3><p>Different agents have different requirements. Some agents can operate on very little information, while others need a lot of context to operate correctly. This tactic builds on the labeling we just discussed.</p><p><strong>Critical Context Identification</strong>: It is crucial to identify which messages are critical for each specific agent and focus on these to streamline processing and enhance response accuracy. Let’s look at line 8 in the context above. The code reviewer only needs a limited amount of context to be able to accurately do its work. We can even say with some certainty that it will produce a worse answer if we give it more than the necessary context.</p><p>So what context <em>does</em> it need? Take a quick look, and you’ll infer that the code reviewer only needs its own system prompt, and it needs the last agent message before it, containing the latest iteration of the code (line 6).</p><p>Therefore, each agent should be configured such that it selects only the history that it needs. The code reviewer only looks at the last 2 messages, while the code writer needs a longer history.</p><h3>Tactic 3: Optimization of System Prompts</h3><p><strong>Placement:</strong> When you do a quick search on agents and system prompts, it’s clear that placement of a system prompt for an agent matters. Should it be the first message in the chain, the last message? Opinions and results vary, depending on the use case. For example, which gives a better result?</p><pre>1) user: &quot;I visited dr. Fauci on Thursday, and got diagnosed with …&quot;<br>2) system: &quot;Extract all medically relevant info from the user prompt&quot;</pre><p>Or</p><pre><br>1) system: &quot;Extract all medically relevant info from the user prompt&quot;<br>2) user: &quot;I visited dr. Fauci on Thursday, and got diagnosed with …&quot;</pre><p>If you test this with a larger and more complex history, you’ll find that different placement of the same instruction produces different results. What is clear is that system prompts should be intentionally placed in a specific position, and that this depends on the agent and use case.</p><p>Note: in my experience it depends on the LLM to know if the system prompt should be the last message in the chain. Often putting it as the last message is a good choice (with definite exceptions, e.g. information extraction).</p><p><strong>Permanence:</strong> Secondly, in addition to placement of system prompts, in most use cases you should ensure that a system prompt is labeled as “fleeting”. What I mean with a fleeting message, is that it does not remain in the chat history after it has been used.</p><p>For example, agent1 might get a prompt that it should step back and reflect on the problem. One turn later, agent1 is asked to use its reflection to find a solution to the problem. In that second turn, the initial prompt (to reflect) will only confuse the agent and result in a worse answer. It would have been better if that initial system prompt was not permanent, and would not have been provided to the agent again. In the example below, this means that line #2 should be removed when you make your second LLM call to the agent.</p><pre>#1 <strong>User</strong>: <br>  &quot;What is the meaning of the universe?&quot;<br>#2 <strong>System</strong>: <br>  &quot;You&#39;re a philosopher; before you answer, step back and <br>  reflect on what the user&#39;s question implies.&quot;<br>#3 <strong>Agent_philo</strong>: <br>  &quot;Reflecting on this question, one can consider the universe <br>  through various philosophical lenses - existentialism views <br>  it as a canvas for creating personal meaning, while cosmology <br>  sees it as a playground for understanding physical laws. This <br>  interplay between human perception and scientific exploration <br>  is central to understanding our query.&quot;<br>#4 <strong>System</strong>: <br>  &quot;You&#39;re a philosopher, and you just reflected deeply on the <br>  question posed by the user. Use that insightful reflection to <br>  craft a comprehensive answer.&quot;<br>#5 <strong>Agent_philo</strong>: <br>  &quot;From an existentialist perspective, the meaning of the universe <br>  is a personal journey, where each individual constructs their own <br>  meaning through choices and life experiences. In contrast, from a <br>  cosmological viewpoint, it is about exploring and understanding <br>  the vast, intricate laws that govern the cosmos. These perspectives <br>  help us grasp the multifaceted nature of our universe, blending our <br>  existential quests with scientific inquiry.&quot;</pre><p>This approach ensures that any old system prompt is removed from the chain, and that each prompt is positioned optimally. Now we have an uncluttered conversation history, allowing for more precise and predictable interactions.</p><h3>Tactic 4: Reducing redundancy in your RAG</h3><p>We could dedicate a dozen articles to optimizing your agent by improving how you do RAG, but will keep it contained to a few paragraphs here. The sheer volume of tokens that can come from using RAG is so large that we have to mention a few techniques on how to manage it. If you haven’t already, this is a topic you should spend considerable time researching.</p><p>Basic tutorials on RAG mostly assume that the documents that you or your user uploads are simple and straightforward. However, in practice most documents are complex and unpredictable. My experience is that a lot of documents have repetitive information. For example, the same information is often repeated in the intro, body, and conclusion of a PDF article. Or a medical file will have repetitive doctor updates with (almost) the same information. Or logs are repeated over and over. Also, especially in production environments, when dealing with retrieval across a large body of files, the content returned by a standard RAG process can be extremely repetitive.</p><p><strong>Dealing with Duplicates</strong>: A first step to optimize your RAG context, is to identify and remove exact and near duplicates within the retrieved document snippets to prevent redundancy. Exact duplicates are easy to identify. Near duplicates can be detected by semantic similarity, by looking at diversity of vector embeddings (diverse snippets have vectors that have a larger distance from each other), and many other techniques. How you do this will be extremely dependent on your use case. Here are a <a href="https://www.perplexity.ai/search?q=show+me+best+practices+on+how+to+remove+near+duplicates+in+my+RAG+results">few examples (by perplexity)</a></p><p><strong>Diversity in Responses:</strong> Another way to ensure diversity of RAG responses by smartly grouping content from various files. A very simple, but effective approach is to not just take the top N documents by similarity, but to use a GROUP BY in your retrieval query. Again, if you employ this depends highly on your use case. Here’s an <a href="https://www.perplexity.ai/search?q=show+me+a+query+that+selects+the+top+N+similar+results+from+a+vector+database+but+groups+it+by+file+so+that+you+ensure+the+diversity+of+snippets+come+from+several+files+instead+of+just+one+or+two">example (by perplexity)</a></p><p><strong>Dynamic Retrieval</strong>: So, given that this article is about dynamic context, how do you introduce that philosophy into your RAG process? Most RAG processes retrieve the top N results, e.g. the top 10 most similar document snippets. However, this is not how a human would retrieve results. When you search for information, you go to something like google, and you search until you find the right answer. This could be in the 1st or 2nd search result, or this could be in the 20th. Of course, depending on your luck and stamina ;-). You can model your RAG the same way. We can allow the agent to do a more selective retrieval, only giving it the top few results, and have the agent decide if it wants more information.</p><p>Here’s a suggested approach. Don’t just define one similarity cutoff, define a high, medium and low cutoff point. For example, the results of your search could be 11 very similar, 5 medium, and 20 somewhat similar docs. If we say the agent gets 5 docs at a time, now you let the agent itself decide if it wants more or not. You tell the agent that it has seen 5 of the 11 very similar docs, and that there are 25 more beyond that. With some prompt engineering, your agent will quickly start acting much more rationally when looking for data.</p><h3>Tactic 5: Advanced Strategies for Context Processing</h3><p>I’ll touch upon a few strategies to take dynamic context even a step further.</p><p><strong>Instant Metadata</strong>: As described in tactic 1, adding metadata to messages can help you to preselect the history that a specific agent needs. For most situations, a simple one word text label should be sufficient. Knowing that something comes from a given function, or a specific agent, or user allows you to add a simple label to the message, but if you deal with very large AI responses and have a need for more optimization, then there is a more advanced way to add metadata to your messages: with AI.</p><p>A few examples of this are:</p><ul><li>A simple way to label a history message, is to make a separate AI call (to a cheaper model), which generates a label for the message. However, now you’re making 2 AI calls each time, and you’re introducing additional complexity in your flow.</li></ul><p>A more elegant way to generate a label is to have the original author of a message generate a label at the same time as it writes its response.</p><ul><li>Have the agent give you a response in JSON, where one element is its normal response, and the other element is a label of the content.</li><li>Use multi-function calling, and provide the agent a function that it’s required to call, which defines the message label.</li><li>In any function call that the agent makes, reserve a required parameter which contains a label.<br>In this way, you instantly generate a label for the function contents.</li></ul><p>Another advanced strategy to optimize context dynamically is to pre-process your RAG..</p><p><strong>Dual processing for RAG: </strong>To optimize your RAG flow, you might consider using a cheaper (and faster) LLM to condense your RAG results before they are provided into your standard LLM. The trick when using this approach, is to use a very simple and non-disruptive prompt that condenses or simplifies the original RAG results into a more digestible form.</p><p>For example, you might use a cheaper model to strip out specific information, to reduce duplication, or to only select parts of the document that are relevant to the task at hand. This does require that you know what the strengths and weaknesses of the cheaper model are. This approach can save you a lot of cost (and speed) when used in combination with a more powerful model.</p><h3>Implementation</h3><p>OK, so does all the above mean that each of my agents needs pages and pages of custom code to optimize its performance? How do I generalize these concepts and extend them?</p><p><strong>Agent Architecture</strong>: The answer to these questions is that there are clean ways to set this up. It just requires some foresight and planning. Building a platform that can properly run a variety of agents requires that you have an Agent Architecture. If you start with a set of clear design principles, then it’s not very complicated to make use of dynamic context and have your agents be faster, cheaper, and better. All at the same time.</p><p>Dynamic Context Configuration is one of the elements of your Agent Architecture.</p><p><strong>Dynamic Context Configuration</strong>: As discussed in this article, each agent has unique context needs. And managing these needs can come down to managing a lot of variation across all possible agent contexts (see the image at the top of the article). However, the good news is that these variations can easily be encoded into a few simple dimensions. Let me give you an example that brings together most of the concepts in this article.</p><p>Let’s imagine an agent who is a SW developer who first plans their actions, and then executes that plan. The context configuration for this agent might be:</p><ul><li>Retain the initial user question</li><li>Retain the plan</li><li>Forget all history except for the last code revision and the last message in the chain</li><li>Use RAG (on uploaded code files) without RAG condensation</li><li>Always set system prompt as last message</li></ul><p>This configuration is saved in the context configuration of this agent. So now your definition of an AI agent is that it is more than a set of prompt instructions. Your agent also a has a specific context configuration.</p><p>You’ll see that across agents, these configurations can be very meaningful and different, and that they allow for a great abstraction of code that otherwise would be very custom.</p><h3><em>Rounding up</em></h3><p>Properly managing Dynamic context not only enhances the performance of your AI agents but also greatly improves accuracy, speed, and token usage… Your agents are now faster, better, and cheaper, all at the same time.</p><p>Your agent should not only be defined by its prompt instructions, it should also have its own context configuration. Using simple dimensions that encode a different configuration for each agent, will greatly enhance what you can achieve with your agents.</p><p>Dynamic Context is just one element of your Agent Architecture. Invite me to discuss if you want to learn more. Hit me up in the comments section with questions or other insights, and of course, give me a few clicks on the claps or follow me if you got something useful from this article.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68b8647eef89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/next-level-agents-unlocking-the-power-of-dynamic-context-68b8647eef89">Next-Level Agents: Unlocking the Power of Dynamic Context</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering has changed forever]]></title>
            <link>https://medium.com/data-science/engineering-has-changed-forever-4d2fcc4f3841?source=rss-556a60f9c64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d2fcc4f3841</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Frank Wittkampf]]></dc:creator>
            <pubDate>Mon, 20 Nov 2023 20:15:16 GMT</pubDate>
            <atom:updated>2023-11-21T17:00:37.723Z</atom:updated>
            <content:encoded><![CDATA[<h4>Recent advancements in AI are upending traditional architectures. Companies that fail to pick the right one will be left behind.</h4><blockquote>“Almost everything we know about good software architecture has to do with making software easy to change.”</blockquote><blockquote>— Mary Poppendieck, Software Development Expert and Author</blockquote><p>In the last 6 months, the foundation of engineering has undergone a dramatic transformation, a change so profound that many organizations are only beginning to understand and adapt to its implications. Most companies have started to take a small incremental step to sprinkle some AI into their software, instead of diving deep into the possibilities.</p><p>In this article, let’s focus on the architectural implications of AI. Broadly, this shift encompasses a move from traditional, rule-based architectures to more dynamic, AI-centric models, deeply altering the role of AI in software systems and challenging our foundational understanding of software architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gSN-tWmYLijXyBlA9ocaDg.png" /><figcaption>Diagram: shift towards AI Engineering (image by Author)</figcaption></figure><p>You can describe the architectural shift in 4 phases:</p><ol><li>Classic Software Services Architecture</li><li>AI-Enabled Software Services Architecture</li><li>AI Software Service Architecture</li><li>AI Engineering</li></ol><p>Each phase differs significantly from the others in implementation and outcomes. Some companies will stumble from one phase into the next, other companies skip ahead. Regardless, most have yet to make a conscious choice among the options. Never mind that this choice could be the critical difference between surviving or being outpaced by their competition.</p><blockquote>Note: in this article, when I refer to AI, I generally mean generative AI, however some readers might want to classify certain Machine Learning applications as such as well. An example from my own experience would be how FinTech companies use ML for credit decisioning and payment timing.</blockquote><h4><strong>1. Classic software services architecture</strong></h4><p>We’re wildly simplifying, but for purposes of this article we’ll group a wide variety of software services architectures into the “classic” bucket. Predictable and precise, frequently adhering to a rule-based, service-oriented approach. Each component within this system serves a specific, predetermined function. This type of architecture is mostly deterministic and modular, with the general purpose of being predictable and scalable.</p><p>The majority of software engineering in enterprise lives in this category. Each company is looking to enable their software with more AI functionality, and will mostly look one step ahead to incorporate some AI, which lands them in an…</p><h4><strong>2. AI-enabled software services architecture</strong></h4><p>AI-enabled software services architecture mixes a classic software architecture with select AI enhancements. While the core remains a service-oriented structure, certain services are augmented with AI capabilities, introducing elements of machine learning, predictive analytics, or generative AI. This integration retains the system’s foundational modularity and predictability but adds a layer of adaptability, enabling it to respond dynamically to evolving data and requirements. The architecture strikes a balance, maintaining the reliability of traditional software services while embracing the flexibility and new functionality offered by AI.</p><p>Examples of enabling traditional services with AI are: transforming unstructured data into structure (extract consumer sentiment into a database), simple chatbot services, or simple AI/ML decisioning services (should we provide credit to this person). However, when you try to go deep to unleash the power of AI you soon discover that managing AI within old software constructs is strongly limiting its potential. Putting an orchestration service on top of AI restricts much of its use cases to input-output translation services.</p><h4><strong>3. AI Software Service Architecture</strong></h4><p>AI software service architecture embeds at least one fully AI-driven service within a traditional software framework. Here, the classic platform manages overall functions but utilizes an AI-specific service for complex, adaptive tasks. This AI service, autonomously processing inputs like text or data events, enables advanced decision-making by the system. Even though the core of the architecture remains rooted in classic software methodologies, this integration of an AI-centric service enhances functionality. It combines the structured orchestration of traditional software with AI’s dynamic problem-solving capabilities.</p><p>Using AI in this fashion unlocks broader AI use cases. An example of an AI service would be one that can perform a mixed process of input transformation, reflection, decisioning, and action. For example, when a new file arrives, the AI takes a look at the file, compares it to its instruction set, determines it is an unstructured file, reads the first lines of content, determines that it needs to be manually inspected, and forwards it to a supervisory department.</p><h4><strong>4. AI Engineering</strong></h4><p>In truly native AI engineering, the spotlight shifts entirely to AI as the core software, diverging significantly from traditional and hybrid architectures. In this paradigm, engineering efforts focus on empowering the AI itself, enhancing its capabilities and autonomy. The AI isn’t just an element within the system; it <strong>is</strong> the system, tasked with platform-like responsibilities. Engineers work to expand the functionalities of the AI, not on orchestrating the AI. Examples of engineering work would be: integrating multimodal inputs, broadening its decision-making scope, and equipping it with a diverse range of tools. This approach is about nurturing the AI to seamlessly integrate into various workflows and real-world applications, ensuring it evolves into a more sophisticated, self-reliant entity. Unlike previous models where AI supplements or enhances traditional structures, here AI is the bedrock, redefining the very concept of software engineering with its boundless potential and adaptability.</p><p>Examples to imagine in this architecture category would be adaptive learning loops, end to end AI process automation, multi-modal decision scenarios with feedback, and an endless amount of other examples, with the only boundary being your imagination.</p><p>Architecture phases 3 and 4 have similarities, but they differ significantly. Let’s clarity the difference a bit more. In AI engineering..</p><ul><li>… the focus of an engineer shifts to enabling the AI</li><li>… the AI is the orchestrator of most activity, any necessary orchestration is very light with a main purpose to enable the user to AI interaction</li><li>… SW services are tools/enablers for the AI instead of the AI being a tool for the SW service</li></ul><p>An example of a true AI engineered service: “Ingest financial market changes, reflect on my portfolio, decide if any additional investment should be made, and then autonomously take action on the recommendation (e.g. selling stock ABC)”.</p><h4><strong>Reflection</strong></h4><p>At the moment a lot of companies are still figuring out what AI means for them and they are experimenting with how to extract value. When companies move on to build real solutions, it is vital that they reflect on what engineering paradigm aligns best with their goals. Incrementally enabling AI solutions is where most companies will start, but in doing so they will miss out on the true capabilities that AI has to offer.</p><p>Soon, a new generation of companies emerges. Old and new companies which have deeply embraced AI native philosophies. Their development speed will far outpace that of any traditional player.</p><p>So choose wisely. In this high-stakes world of AI permeated engineering, your next move could set the course for success or obsolescence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d2fcc4f3841" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/engineering-has-changed-forever-4d2fcc4f3841">Engineering has changed forever</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Smarter Approach to Boundaries]]></title>
            <link>https://medium.com/@frankw_usa/post-3-a-smarter-approach-to-boundaries-4df97dad7d41?source=rss-556a60f9c64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4df97dad7d41</guid>
            <category><![CDATA[ai-product-management]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Frank Wittkampf]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 17:04:05 GMT</pubDate>
            <atom:updated>2023-11-20T20:31:43.793Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Understanding document chunking in vector stores</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*er_6Y4n9MvQEFYzK8HT3lQ.png" /></figure><p>Continuing from my last post, storing documents in a vector store requires us to divide our content into chunks. But how do you determine where to slice? Being smart about boundaries provides more comprehensive/self containing pieces of data (find better words here). Imagine slicing a cake: Do you leave the cherry in the middle or cut it in two?”</p><p>The most basic way to chunk a document is to pick a static length. E.g. you choose to cut it at 4000 characters.However, that means you’re most likely cutting right in the middle of a word. A slightly more evolved version would cut in the whitespace between words or sentences. If you proceed with that line of thought, you quickly land on the concept of <strong>using the document’s native logic structures</strong> to slice the content. Almost any document has a native structure to it. Most text documents contain straightforward structures like paragraphs, chapters, tables, pages, headers, captions. An Electronic Medical Record (EMR) is an example of this. XML and HTML documents often contain nested structures with clear starts and ends.</p><p>Intelligently breaking a document into its native pieces, and then combining these native pieces into the desired chunks gives us a much more intuitive and meaningful chunk of data. For example, in an EMR, ensuring that a full observation from a doctor ends up together in one chunk is very meaningful for downstream processes. The <strong>complication is that most documents have a nested structure</strong>. This means that in an EMR, a doctor’s observation can be found both at deep and shallow levels in such a nested structure, and their length can greatly vary.</p><p>To properly define boundaries requires a flexible approach to slicing.I’ve posted a simplified example below. Simply copy and paste the ‘sample code’ section below into your favorite LLM to see some sample code. The example shows a simple way to traverse the native document structure, <strong>feeling out the boundaries, and slicing where possible</strong>, while trying to not exceed the desired chunk size. It ignores a lot of subtleties, but it sets up the gist of what we’re trying to do.</p><pre><strong>Sample code (paste into an LLM):<br></strong><br>Generate code in [python] to split a document into chunks of maxChunkSize <br>by using the following strategy:<br><br>- Parse an EMR (or any other XML)<br>- For an upcoming node (XML tag) determine its content length.<br>- If its length, plus that of any previous processed nodes, is less than the desired chunk size, add it to the working chunk<br>- Apply a 95% chunkSize tolerance<br>- If the length exceeds the desired chunksize, proceed to its child-node, and repeat the above<br>- Finish the working chunk once you have enough content in it, or when you run out of children<br>- Proceed to make the next chunk</pre><p>In addition to being deliberate about where to set your chunk boundary, <strong>it’s common to apply a chunk overlap</strong>. This in essence applies a fuzziness to the boundary. Make one chunk slightly overlap the other, such that there is a higher chance that essential topics are not missed in either chunk. This comes at the cost of double storage, and the waste of sometimes supplying the AI with double context. Depending on your tolerance for repetition, you add a fuzziness factor.</p><p>There are many subtleties to apply to the example above. Which structures do you give preference, what boundaries do you set as hard boundaries that shouldn’t be crossed, what chunkSize fits the type of document and task (refer to my last post for some thoughts on that), and how do you approach nested structures that are of a totally different format (images, tables, ..). Too much to discuss in a short post. Maybe for a later day.</p><p>I hope the above gave you some food for thought. If you have experimented with different chunking strategies yourself, or if you have any questions, please share your insights and questions below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4df97dad7d41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chunk Size in AI: Why Size Matters]]></title>
            <link>https://medium.com/@frankw_usa/chunk-size-in-ai-why-size-matters-4465a8f41e37?source=rss-556a60f9c64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4465a8f41e37</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Frank Wittkampf]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 20:25:35 GMT</pubDate>
            <atom:updated>2023-08-25T20:25:35.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mPoGHAfVN8M0ipV11bQpzg.png" /></figure><p>When it comes to storing documents in AI vector stores, how we slice and dice the content can greatly influence our results. Before storing a doc in a vector store, you split it into document snippets (chunks). These snippets are then embedded, and both the doc snippet and its embedding are saved in the vector store. Later, when a user asks a question, the embedding is used to select the right snippet, and the snippet itself is used as context for the AI, so it can give a proper answer.</p><p>It turns out that size matters. How do you decide how large these snippets should be? You could cut a doc into a lot of small snippets, or some very large ones, or you could vary the size dynamically. For a proper implementation, you’ll need to use a mix of these. Let’s consider what impact the size has. We’ll need to consider the embedding, and the digestion of the snippet.</p><p><strong>Embedding tradeoff </strong>— If you use a larger snippet, then embedding it captures a representation of the general gist of the content. For a smaller snippet, the embedding is a more precise representation. Chunk size greatly influences the selection of snippets that are sent to the AI model, and therefore how it can answer the user’s question.</p><p><strong>Digestion tradeoff</strong> — Digestion in AI refers to how the model processes and understands the provided snippets. Snippets are digested by the AI when it gets sent a question accompanied by a number of snippets. Would it be better to send it 10 small snips or 2 big ones? We have to choose, because every AI limits you to a maximum length. A large snippet gives the AI lots of context on that specific part of the document. A lot of small snippets gives the AI more, but narrower data, which might be cut off exactly so they miss critical information.</p><p><strong>Consequences of our choices </strong>— Both the embedding and digestion tradeoffs are about specificity. Whether you want to find a set of very specific snippets or few large pieces of context depends on a range of factors: question type, data structure, industry/vertical, answer expectation, …</p><p>For example, a list of medications requires more specificity, a descriptive observation by a doctor requires more context.</p><p>So, you wonder, what do I do now? I’ll talk more about the implications of these tradeoffs, and a few others, in my next posts. Stay tuned if these topics interest you..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4465a8f41e37" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>