<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Gutsch on Medium]]></title>
        <description><![CDATA[Stories by David Gutsch on Medium]]></description>
        <link>https://medium.com/@david.gutsch0?source=rss-ffafbe525bbf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FJtRjgBCXGbyqWe2ypOFzg.jpeg</url>
            <title>Stories by David Gutsch on Medium</title>
            <link>https://medium.com/@david.gutsch0?source=rss-ffafbe525bbf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:51:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@david.gutsch0/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evolving Language Model Prompting: Prompting Strategies for Enhanced Language Model Performance…]]></title>
            <link>https://medium.com/@david.gutsch0/evolving-language-model-prompting-prompting-strategies-for-enhanced-language-model-performance-3352172e0d60?source=rss-ffafbe525bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3352172e0d60</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[ai-tools]]></category>
            <dc:creator><![CDATA[David Gutsch]]></dc:creator>
            <pubDate>Tue, 24 Oct 2023 11:44:10 GMT</pubDate>
            <atom:updated>2023-10-24T11:44:10.980Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evolving Language Model Prompting: Prompting Strategies for Enhanced Language Model Performance (part 2)</h3><figure><img alt="prompt engineering AI" src="https://cdn-images-1.medium.com/max/1024/1*FSPAJyX-ls80n_y4zoJhkQ.png" /></figure><p>In part 1 of this series: “<a href="https://medium.com/@david.gutsch0/tackling-language-model-limitations-a-dual-approach-to-compositionality-and-modular-reasoning-3fa0e3c86b80">Tackling Language Model Limitations: A Dual Approach to Compositionality and Modular Reasoning</a>,” we introduced some inherent shortcomings of language models, and explored some of the methods by which we can augment models to overcome those shortcomings. In this article I will introduce two more prompting strategies and their respective tradeoffs with the intention of providing enough general context on prompting strategies to enable you to choose the right strategy for the LLM agents you’re building today!</p><p>In part one the two main strategies to augment LLMs shortcomings were: 1 the utility of using external tools to enhance the language model’s access to information. 2) elicitive prompting strategies such as Chain of Thought or Self-Ask which provide a methodology by which we can improve the reasoning skills of LLM agents. Today we’ll take a look at the ReAct prompting strategy which uses both of the aforementioned strategies to provide better answers to users. The second strategy we’ll introduce is Plan and Solve Prompting which only uses elicitive prompting, but I have a conjecture that it could also use external tools to improve on all three of the prompting strategies for narrow use cases.</p><h3>ReAct</h3><p>ReAct uses both external tools to provide access to information the LLM itself does not have, as well as providing a, “reason to act and act to reason,” strategy to synthesize answers to questions the LLM cannot on it’s own synthesize. In the words of the authors of the original article:</p><blockquote><em>ReAct prompts LLMs to generate both verbal reasoning traces and actions pertaining to a task in an interleaved manner, which allows the model to perform dynamic reasoning to create, maintain, and adjust high-level plans for acting (reason to act), while also interact with the external environments (e.g. Wikipedia) to incorporate additional information into reasoning (act to reason).</em></blockquote><p>What I find unique about this strategy is what the authors refer to as interleaving, where the prompt asks questions, looks up information, then makes observations about the queried information before using both queried information and previous observations to ask another round of questions going through another reason to act cycle again. This method gives the model better interoperability between information retrieval and observation over the retrieved information.</p><p>Let’s take a look at an example of ReAct in action so we can get a better understanding of how the method works.</p><figure><img alt="ReAct example (reason to act)" src="https://cdn-images-1.medium.com/max/1024/0*h3o_HL2R_X55PVq7" /><figcaption>1d illustrates an example of ReAct</figcaption></figure><p>We see in the example above the agent thinks, queries information, then makes observations on the new information before starting the cycle again. Then once sufficient knowledge and reasoning has been performed the agent returns a final answer. The following describes the way the agent is configured in the experiment for the paper:</p><p><em>Action Space: We design a simple Wikipedia web API with three types of actions to support interactive information retrieval: (1) search[entity], which returns the first 5 sentences from a corresponding entity wiki page, if it exists, or else suggests top-5 similar entities from the wiki search engine. (2) lookup[string] which would return the next sentence in the page containing string, simulating ctrl+F functionality on the browser. (3) finish[answer], which would finish the current task with an answer. The purpose is to simulate how humans would interact with Wikipedia, and force models to retrieve via explicit reasoning in language.</em></p><p>For this experiment the agent was configured to mimic the strategy that we humans would use to browse the web to answer a question. In this case the agent is using Wikipedia to inform itself on the question for which it is answering; However, we could substitute any knowledge base over which we can define search and lookup operations, where “search” is used to find semantic similarity, and “lookup” is used to find the next lexical occurrence of a keyword. We humans do this without realizing what we are doing. We first find an article that contains information which most closely matches what we are attempting to learn about (i.e. the semantic meaning). Then once we have enough context on the subject we narrow down our search to a keyword that will better inform us on the specifics of the information we need to learn (i.e. a lexical search). In the example included above the ReAct agent performs a semantic search on “Apple Remotes” and discovers it uses the “Front Row (software),” on which it proceeds to perform a lexical search. Notice when the agent is unable to find a Wikipedia match for “Front Row,” it receives a list of similar articles it then observes which one will be most useful for answering the initial question. This is a powerful strategy that we humans can use to increase our time to solution when researching a new or unfamiliar concept.</p><p>Finally let’s compare ReAct to some of the other prompting and information retrieval mechanisms that we’ve looked at so far. The MRKL system we first looked at is technique aimed directly at information retrieval, while powerful for information retrieval, it doesn’t have a reasoning mechanism akin to ReAct. Now Self-Ask is very similar to ReAct, but there is a defining features that make it different. For ReAct each cycle of reasoning to act and acting to reason is grounded in the information store that we have provided the model. The research from the ReAct article found that when compared to other prompting and information retrieval agents, ReAct keeps itself more grounded in the information provided. To be fair to Self-Ask they did integrate a search engine for answering sub-questions in a part of their study, but at the core of their strategy was prompting and less so knowledge augmentation. There are important tradeoffs in play, but it lends itself well to questions that need to be grounded in your information store. In the article they use a Wikipedia API, but that could be substituted with any knowledge retrieval system.</p><h4>Limitations</h4><p>Now that we know some differentiating features of ReAct let’s take a look at the limitations.</p><blockquote><em>While interleaving reasoning, action, and observation steps improves ReAct’s groundedness and trustworthiness, such a structural constraint also reduces its flexibility in formulating reasoning steps, leading to more reasoning error rate than CoT… We note that there is one frequent error pattern specific to ReAct, in which the model repetitively generates the previous thoughts and actions, and we categorize it as part of “reasoning error” as the model fails to reason about what the proper next action to take and jump out of the loop.</em></blockquote><blockquote><em>For ReAct, successfully retrieving information knowledge via search is critical. Non-informative search, which counts for 23% of the error cases, derails the model reasoning and gives it a hard time to recover and reformulate thoughts. This is perhaps an expected trade-off between factuality and flexibility, which motivates our proposed strategies of combining two methods.</em></blockquote><p>A ReAct agent’s ability to ground itself in data from your knowledge store can limit its ability to reason. In such cases Self-Ask and even Chain of Thought (CoT) Reasoning may be better strategies. If the agent cannot find the information it needs to inform its next set of observations and actions, it stands to reason that it will limit the agent’s ability to converge on the correct final answer. Given this limitation of ReAct let’s take a look at another prompting strategy that can be more flexible.</p><h3>Plan-and-Solve Prompting</h3><p>Now on to our final prompting strategy, Plan and Solve prompting is an elicitive prompting strategy which is different from all three of our previously introduced strategies because it does not have access to external information. Let’s introduce the strategy as defined by it’s authors then enumerate it’s advantages:</p><blockquote><em>Plan-and-Solve Prompting consists of two components: first, devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan. Replace “let’s think step by step” with: “Let’s first understand the problem and devise a plan to solve the problem. Then, let’s carry out the plan and solve the problem step by step” (see figure 2(b)).</em></blockquote><figure><img alt="Plan and Solve prompting example" src="https://cdn-images-1.medium.com/max/1024/0*YW7BhpxD1oqPoaD4" /><figcaption>(b) Illustrates an example of Plan and Solve prompting</figcaption></figure><p>Sounds pretty similar to Self-Ask or chain of thought prompting right? While true there are some important improvements that make this prompting strategy useful. Today’s LLMs are sensitive to phrasing and expressions in prompts, so in addition to planning and solving the problem, we can add more specific language to the prompt to better identify the specific problems it needs to solve. In this article the authors were attempting to improve the ability of LLMs to perform arithmetic, so to achieve this they added some specific instructions to help the LLM focus on important intermediary steps required to do math.</p><ul><li>“pay attention to calculation”</li><li>“extract relevant variables and the corresponding numerals”</li></ul><p>While this strategy is simple, it was devised to solve two of the three errors that commonly plague Chain of Thought (CoT) reasoning. Calculation errors: These are errors in the calculation leading to wrong answers, and Missing step errors: these occur when some intermediate reasoning step(s) is missed especially when there are many steps involved [2]. These two kinds of errors compound: if the LLM leaves out the relevant and important variables, it is more likely to miss out on relevant reasoning steps, and that is supported by the results of the paper. “It is observed that both variable definition and plan existences have a negative correlation with calculation errors and missing-reasining-step errors” [2]. In other words the existence of a plan and an explicit focus on defining useful variables in the prompt decrease the likelihood of missing reasoning step errors.</p><h4>Limitations</h4><p>There are some limitations to this strategy. While improving calculation errors and missing reasoning steps, this methodology cannot solve the problem of semantic misunderstandings of an LLM. In addition there is an additional cost to using this method, which is the cost of devising a prompt that will be able to help the LLM focus on the details specific to the problem at hand. In my estimation this will limit the utility of this prompting strategy to narrow problem sets, which is supported by the use case the authors chose for the article, performing arithmetic.</p><h4>Why am I excited by this method</h4><p>As I’m sure you’ve noticed Plan and Solve Prompting is the only strategy I have introduced that is ONLY an elicitive prompt. Now given the wonderful tools that langchain provides us, I believe one or more tools can be made available to this strategy to augment its lack of information access. Hopefully the question is ruminating, David you’ve introduced three other valuable and generalizable strategies to improve the utility of LLM agents, why do I need another one? Well since we can write specific prompts to help an LLM focus on a narrow problem, I hypothesize that using plan and solve prompting in conjunction with an information retrieval mechanism to which the other three prompts had access; plan and solve prompting with information retrieval will give us the ability to better solve specific narrow problems that other prompting strategies do not have the specificity and focus to solve.</p><p>I am currently working on a project to test this hypothesis and I will report back soon with the results! As always thank you for choosing to spend your time and attention with me, I will always do my upmost to ensure I’m providing maximal value in exchange for it!</p><p>If you enjoyed this series, and have not read my 3 part series on <a href="https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92">vector databases,</a> I recommend you take a look!</p><h3>References</h3><ol><li><a href="https://arxiv.org/pdf/2210.03629.pdf">https://arxiv.org/pdf/2210.03629.pdf</a></li><li><a href="https://arxiv.org/pdf/2305.04091.pdf">https://arxiv.org/pdf/2305.04091.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3352172e0d60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tackling Language Model Limitations: A Dual Approach to Compositionality and Modular Reasoning…]]></title>
            <link>https://medium.com/@david.gutsch0/tackling-language-model-limitations-a-dual-approach-to-compositionality-and-modular-reasoning-3fa0e3c86b80?source=rss-ffafbe525bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fa0e3c86b80</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[David Gutsch]]></dc:creator>
            <pubDate>Wed, 04 Oct 2023 12:29:17 GMT</pubDate>
            <atom:updated>2023-10-24T11:56:35.427Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tackling Language Model Limitations: A Dual Approach to Compositionality and Modular Reasoning (part 1)</h3><figure><img alt="pondering AI agent" src="https://cdn-images-1.medium.com/max/1000/1*w1Q9Vz4mGEUIUMVzPgu7Ow.jpeg" /></figure><p>Welcome back to a new series where we will be introducing inherent shortcomings of language models, and exploring some of the methods by which we can augment models to overcome those shortcomings. Today you might hear from your colleagues or on the interwebs that Large Language Models (LLMs) are the new hammer you need for every nail, or that they are the slippery slope that will procreate the singularity which will enslave us to our computer overlords. I don’t have any interest in toying with either of these ideas, but I do want to talk about some deficiencies of current language models, as well as ways we can create systems that workaround their inherent problems. I will be breaking down two insightful and pragmatic articles that you can apply to your LLM systems to make them more reliable and less likely to hallucinate without needing to dive into the scientific literature <a href="https://arxiv.org/abs/2205.00445">[1]</a> <a href="https://arxiv.org/abs/2210.03350">[2]</a>.</p><h3>Overview</h3><h4>The problems</h4><ul><li>Lack of access to current and proprietary information</li><li>Lack of reasoning ability</li><li>Failure to compose final answers when data doesn’t appear together in training data</li><li>Data missing from the training set confuses the LM often leading to hallucinations</li></ul><h4>The Solutions</h4><ul><li>Modular Extensibility: giving our language models tools to explain data on which they have not been trained</li><li>Elicitive Prompting: where we give a language model a set of steps to use that improve its ability to compose a correct answer to the input question</li></ul><h3>Constraint of Machine Learning</h3><p>The fundamental constraint in machine learning that language models are not immune to is the way models react to inputs that fall outside of their training data. In the general case if you give any model an input of information that it was not trained on it will not be able to give you a sufficient answer. In the case of language models it does much worse, it will hallucinate an answer that does not exist. You only need to think for a few seconds about various fields where these tools are being applied before you can come up with a use case where this would produce catastrophic results. All of the following problems we will address today are merely different manifestations of this fundamental constraint of machine learning. What is fascinating are the new ways it rears its ugly head as our models get larger and more useful… maybe I should take those singularity doomers more seriously 😅.</p><h3>Information Access</h3><p>One way that language models struggle to be useful for some practical use cases are their relatively limited access to information. I’m sure you’ve gotten the message in chat GPT or other LLMs that say something to the effect of: “I have no information after some date in 2021 and thus cannot tell you what the weather is today in Hyderabad.” These multi-billion parameter models have been trained on what is effectively the entirety of the index-able internet, which is hella expensive and consequently is not done every couple of weeks to make sure you can figure out who won the champions league this year. Another information access problem is that of proprietary data that you or your clients need in order to accomplish their goals with these tools. Whether it be an internal knowledge base the LLM needs access to in order to augment a customer service agent’s knowledge gap in your business, or the need to feed your calendar to the chatbot booking your appointments, having access to proprietary real time data is a great limitation to language model adoption in industry.</p><h3>Reasoning Ability</h3><p>A separate issue that also stems from the general constraint of machine learning is the inability for language models to reason. There are a myriad of manifestations of this problem, one of which was the focus of the empirical study in the MRKL Systems article <a href="https://arxiv.org/abs/2205.00445">[1]</a>. They used GPT-3 and Jurassic-1 to perform basic arithmetic; It performed okay on 2-digit arithmetic, but performed pretty poorly on 4-digit numbers. We know computers do math well so why can’t LLMs? Well this stems from the fact that they are trained to “understand” the written word which is using heuristics to compare written language. When we ask them to do math they guess based on their training data, and due to the statistical nature of those guesses, it is wildly inconsistent.</p><h3>Failure to Compose Final Answers</h3><p>Arithmetic is not the only domain where language models fail to reason well; they also fail to compose data that are very simple yet disparate observed facts that LLMs fail to compose into simple solutions. This is a horribly vague way to introduce this problem so let’s talk about it in terms of the research that was performed in the article: “Measuring and Narrowing the Compositionality Gap in Language Models” <a href="https://arxiv.org/abs/2210.03350">[2]</a>. They ran an experiment on a dataset referred to as Compositional Celebrities (CC), which is a set of 8700 2-hop questions that combined: “frequently stated facts in improbable ways (e.g., “Who won the Masters Tournament the year Justin Bieber was born?”), allowing us to disentangle memorization and reasoning”. Another example of one of these questions was: “what is the calling code of the birthplace of Frida Kahlo?”. So in their words this is how they define this gap:</p><blockquote><em>We introduce the term Compositionality Gap to describe the fraction of compositional questions that the model answers incorrectly out of all the compositional questions for which the model answers the sub-questions correctly…</em></blockquote><p>So they measured the rate at which a language model answered the question wrong when they got both of the sub-problems correct. So for example in the first example they want to know how often the final answer was wrong when the language model could figure out that Justin Bieber was born in 1994, and José María Olazábal won the masters in 1994. The results of their study are staggering using GPT-3 with davinci-002 embeddings, on the hardest categories (Birth Year/Literature Nobel Prize Winner) GPT-3 only answers 1.2% of the questions correctly, while answering 80% of the sub-questions correctly. So from this study they concluded: <em>“as the GPT-3 model size grows, it knows more about the world, but its ability to compose this knowledge increases at a slower rate.”</em> At the time this study was performed GPT-3 was the largest available LLM to test on, and it fundamentally failed to compose correct answers to questions that required it to compose two simple facts. Since it could determine the individual facts, the real fault was its inability to reason and compose them. The takeaway here is not that we’ve come a long way since GPT-3, rather models will continually memorize better and retain more knowledge without improving their reasoning skills at a comparable rate.</p><p>Now that we have enumerated some of the problems with LLMs whether they be problems with information access or their relative inability to compose facts, let’s take a look at some of the solutions that these two papers propose to improve the efficacy of these models in the wild.</p><h3>MRKL Systems pronounced “Miracle”</h3><p>Allow me to introduce the “Miracle” system in the words of the authors and then we’ll break down the solution in terms of the problems addressed above:</p><blockquote>A MRKL system consists of an extendable set of modules, which we term ‘experts’, and a router that routes every incoming natural language input to a module that can best respond to the input (the output of that module can be the output of the MRKL system, or be routed to another module. These modules can be:</blockquote><blockquote>Neural, including the general-purpose huge language model as well as other smaller, specialized LMs.</blockquote><blockquote>Symbolic, for example a math calculator, a currency converter or an API call to a database.</blockquote><p>The MRKL system provides a systematic solution to both the information retrieval and reasoning problems. By introducing a set of modules that our LLMs can route the input question or sub-questions to, it provides the language model a capacity to augment its weaknesses with external tools. The MRKL paper describes the router as a simple neural net that is easy to retrain on a new set of tools; However, if you are using <a href="https://python.langchain.com/docs/modules/agents/how_to/custom_mrkl_agent">Langchain</a> to build your agents it is even simpler than that. You can simply define your tools with a name, a description, and the function to run when the language model decides the respective tool is the best for the question proposed as seen below.</p><pre>search = SerpAPIWrapper()<br>tools = [<br>    Tool(<br>        name=&quot;Search&quot;,<br>        func=search.run,<br>        description=&quot;useful for when you need to answer questions about current events&quot;,<br>    )<br>]</pre><p>The authors suggest a substantial list of benefits to using said methodology:</p><blockquote>Safe fallback: In case the input doesn’t match any existing expert module, the router sends the input directly to the general-purpose huge LM.</blockquote><blockquote>Robust extensibility: Since each expert is trained independently we are able to cheaply add new capabilities while guaranteeing that they do not compromise the performance of existing ones. The only component that requires retraining is the router which is a relatively lightweight task.</blockquote><blockquote>Interpretability: When the router invokes a specific module, that often has the side benefit of providing a rationale for the MRKL system’s output (“1 + 1 = 2 because the calculator said so”); such explanations are crucially lacking in existing language models.</blockquote><blockquote>Up-to-date information: The integration of external APIs allows the MRKL system to hook into dynamic knowledge bases, and correctly answer inputs that static models cannot.</blockquote><blockquote>Proprietary knowledge: Access to proprietary databases and other information sources.</blockquote><blockquote>Compositionality: By routing compounded multi-hop inputs to different experts we are able to naturally integrate their responses and correctly address complex inputs.</blockquote><p>Let’s unpack a few of these so we can better understand the power MRKL provides us. A counterpoint to Safe fallback is that we can prompt our LLM agent to NOT provide an answer rather than feeding the question directly to the LLM when none of the MRKL experts are helpful. This is an important strategy when the risk of providing a wrong answer outweighs the benefits of providing an answer that is not derived from the experts provided to it. Robust extensibility is one of the strongest arguments for using this method, and using langchain it does not even require us to retain the router. Up to date information and proprietary knowledge are pretty self explanatory, but are likely the most profound when it comes to using LLMs at industry. Finally the interpretability and the compositionality arguments are further supported by the Compositionality gap paper. They suggest: <em>“the probability of an LM being able to compose two facts grows as its confidence about each fact rises”</em> <a href="https://arxiv.org/abs/2210.03350">[2]</a>. So being able to provide reasoning the language model can interpret increases the ability of the model to compose a correct final answer. The MRKL system is a powerful idea that researchers and practitioners alike will be building on for a long time to come. Now shifting gears we are going to build on this final idea of compositionality in our next section.</p><figure><img alt="MRKL system design" src="https://cdn-images-1.medium.com/max/616/0*CzIpmO25-ybjPYt3" /><figcaption>MRKL system with some example experts</figcaption></figure><h3>Elicitive Prompting and Self-Ask</h3><p>Elicitive prompting is the process of crafting prompts that extract more informative, accurate, or nuanced responses by posing questions or statements in creative ways to guide the language model towards the desired answer. The first and canonical enumeration of this strategy is the chain of thought prompting technique, where you tell the prompt, “Let’s think step by step,” to prompt the model to use a step by step process to generate an answer [3]. This method is useful, but can be improved. The Self-Ask prompting strategy is defined as follow:</p><blockquote>Our method builds on chain of thought prompting, but, instead of outputting a continuous undemarcated chain-of-thought, our prompt has the model explicitly state the next follow-up question it wants to ask before answering it…</blockquote><blockquote>Self-ask (depicted in Figure 3) requires a one or few-shot prompt that demonstrates how to answer the questions… We then insert a phrase “Are follow up questions needed here:” at the end of the prompt since we found that doing so slightly improves results.</blockquote><figure><img alt="Self-Ask, Chain of Thought, and direct Prompting" src="https://cdn-images-1.medium.com/max/1024/0*1v_YWYOo3L_uAY-F" /><figcaption>Illustrates direct, Chain of Thought, and Self-Ask Prompting</figcaption></figure><p>Self-Ask extends chain of thought by forcing the model to be more explicit in breaking down the question, and composing the sub-answers before returning a final answer to a question. This aims to solve some of the failures of chain of thought. Chain of thought can lose context of what the original question was in building up a final answer due to the length of the context window of the language model. Meanwhile self ask will provide the entire context from question, sub-questions, and intermediate answers, all the way through the reasoning process up to the final answer. Furthermore when chain of thought provides bad logic in an intermediate step it will likely go off the rails and provide a final answer that has strayed from the original question. This is still possible with self-ask; However, by breaking down sub-questions and intermediate answers in greater detail it is far less likely for this mistake in reasoning to occur.</p><p>The authors of “compositionality gap” put it more concisely than I can:</p><blockquote>We Hypothesize that the advantage of self-ask over chain of thought is that it disentangles the decomposition of the full question (by formulating sub-questions) from the actual answers to those sub-questions. In addition, the rigid scaffolding self-ask provides makes it easier for the model to state the final answer in a concise, parseable way.</blockquote><p>As it pertains to both the problems of reasoning and composition of final answers the self-ask method clearly provides a much more robust prompting strategy to increases the language model’s confidence in the sub-answers which are used to compose the final answer. This is supported by experimental evidence as well. On the Compositional Celebrities dataset introduced earlier, chain of thought reasoning obtained 45.7% accuracy while Self-Ask obtained 79.6% accuracy which is no small improvement between the two methodologies <a href="https://arxiv.org/abs/2210.03350">[2]</a>.</p><h3>Where do we go from here?</h3><p>So the question remains: what can we take away from these two papers and the methodologies they introduce? The MRKL System article provides us with two takeaways: 1) external tools provide us the ability to augment the language model’s reasoning deficiencies in domains poorly suited to LMs. 2) external tools also provide information retrieval to reason with information to which it does not have access. From the Compositionality Gap article we learned that LLMs are getting better at memorizing the world much quicker than they are improving at reasoning about it, and consequently we need to employ creative prompting strategies to provide adequate context and strategy to enable them to address the queries they’re given.</p><p>If we treat language models as the everything tool, we will inevitably end up trying to hammer a screw. But if we use language models for what they are good at, and provide tools and guard rails to augment them where they are weak, we can build systems and agents that can solve increasingly complex problems with greater levels of accuracy.</p><p>Thank you for your attention reading this article, it is your greatest commodity and for that I am grateful! If you enjoyed this article check out part 2 in the series: Evolving Language Model Prompting: <a href="https://medium.com/@david.gutsch0/evolving-language-model-prompting-prompting-strategies-for-enhanced-language-model-performance-3352172e0d60">Prompting Strategies for Enhanced Language Model Performance</a>. If you haven’t seen it yet, you might enjoy my three part series on vector databases: <a href="https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92">Vector DBs the secret Sauce</a>, <a href="https://medium.com/@david.gutsch0/the-art-of-embeddings-transforming-text-for-vector-databases-926738443e70">the art of embeddings</a>, and <a href="https://medium.com/@david.gutsch0/vector-databases-understanding-the-algorithm-part-3-bc7a8926f27c">Understanding the Algorithm</a>.</p><h3>Sources</h3><p>[1] <a href="https://arxiv.org/abs/2205.00445">https://arxiv.org/abs/2205.00445</a></p><p>[2] <a href="https://arxiv.org/abs/2210.03350">https://arxiv.org/abs/2210.03350</a></p><p>[3] <a href="https://arxiv.org/abs/2201.11903">https://arxiv.org/abs/2201.11903</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fa0e3c86b80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Databases: Understanding the Algorithm (part 3)]]></title>
            <link>https://medium.com/@david.gutsch0/vector-databases-understanding-the-algorithm-part-3-bc7a8926f27c?source=rss-ffafbe525bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc7a8926f27c</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[David Gutsch]]></dc:creator>
            <pubDate>Thu, 03 Aug 2023 13:24:38 GMT</pubDate>
            <atom:updated>2023-08-03T13:24:38.687Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the <a href="https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92">part one</a> and <a href="https://medium.com/@david.gutsch0/the-art-of-embeddings-transforming-text-for-vector-databases-926738443e70">part two</a> of this series, we embarked on a journey through the fascinating world of vector databases and embeddings. We started by exploring the concept of vector databases, a revolutionary technology that transforms data into vectors in a multi-dimensional space, enabling a nuanced understanding of data and the ability to compare almost anything. We then delved deeper into the art of embeddings, a fundamental concept in deep learning that allows us to capture rich context in ones and zeros and transform raw data into a machine-readable and indexable format. These two components, vector databases and embeddings, are the building blocks of a powerful AI tool that can process and understand data in ways that were once the exclusive domain of humans. The final piece of the puzzle to understand how vector databases work is the way vector embeddings are stored and retrieved from the database.</p><p>As we move into the third part of this series, we turn our attention to the algorithms that compose our comparable embeddings into a database. The choice of algorithm used in a vector database is a critical factor that can significantly impact its performance. Different algorithms have different strengths and weaknesses, and the choice of algorithm can have a profound effect on: the latency, throughput, build time, accuracy, and re-indexing capabilities of a vector database. Understanding these algorithms is not just a matter of academic interest; it’s a practical necessity for anyone looking to get the best performance out of their vector database. Different products use different algorithms, and the performance of these products can vary widely depending on the size and shape of your data. In the following sections, we will delve into these algorithms, explore their performance trade-offs, and provide you with the knowledge you need to make an informed choice when evaluating vector database and vector index solutions.</p><h3>Introducing the families of Algorithms</h3><p>Now that we know the importance of choosing the right algorithm for our vector database, let’s introduce the types of algorithms we have at our disposal. Each family of algorithms: tree-based, quantization, hashing, clustering, and graph methods — comes with its own set of performance trade-offs. These trade-offs can significantly impact the latency, throughput, build time, accuracy, and re-indexing capabilities of a vector database. Note: Some argue that there are only three distinct categories; trees, hashes, and graphs. I purposefully added clustering and quantization methods as well since I think they are unique enough to merit their own explanation, but know that some folks would categorize quantization as a hashing algorithm and clustering as a graphing algorithm which is also true.</p><p>Tree-based methods are efficient for low-dimensional data and offer exact nearest neighbor search. However, their performance generally degrades in high-dimensional spaces due to the “curse of dimensionality”. They also require substantial memory and are less efficient for large datasets, leading to longer build times and higher latency.</p><p>Quantization methods are memory-efficient and offer fast search times by compressing vectors into compact codes. However, this compression can lead to a loss of information, potentially reducing search accuracy. Additionally, these methods can be computationally expensive during the training phase, increasing build time.</p><p>Hashing methods are fast and relatively memory-efficient, mapping similar vectors to the same hash bucket. They perform well with high-dimensional data and large-scale datasets, offering high throughput; However, the quality of search results can be lower due to hash collisions leading to false positives and negatives. Choosing the right number of hash functions and the number of hash tables are imperative as they significantly affect performance.</p><p>Clustering methods can expedite search operations by narrowing down the search space to a specific cluster, but the accuracy of the search results can be influenced by the quality of the clustering. Clustering is typically a batch process, which means it’s not well-suited to dynamic data where new vectors are constantly being added, leading to frequent re-indexing.</p><p>Graph methods provide a good balance between accuracy and speed. They are efficient for high-dimensional data and can provide high-quality search results. However, they can be memory-intensive, as they need to store the graph structure, and the construction of the graph can also be computationally expensive.</p><p>The choice of algorithm in a vector database depends on the specific requirements of the task, including the size and dimensionality of the dataset, the available computational resources, and the acceptable trade-off between accuracy and efficiency. It’s also worth noting that many modern vector databases use hybrid methods, combining the strengths of different methods to achieve high speed and accuracy. Understanding these algorithms and their trade-offs is crucial for anyone looking to get the best performance out of their vector database. Now let’s take a look at an example of a popular algorithm in each one of these categories.</p><h3>Annoy (Approximate Nearest Neighbor Oh Yeah)</h3><p>Among the tree-based algorithms used in vector databases, one stands out for its efficiency and simplicity: Annoy, or “Approximate Nearest Neighbors Oh Yeah”. Annoy is a C++ library with Python bindings that is designed by Spotify to search for points in space that are close to a given query point. It creates large read-only, file-based data structures that is memory-mapped into memory, which allows multiple processes to share the same data. I admit I am kind of cheating including this algorithm as it is a vector index rather than an algorithm in a vector database. THat being said, I found it to be the most compelling tree based algorithm that also scales very well for a tree based algorithm.</p><p>Annoy operates by using a forest of random projection trees to perform efficient approximate nearest neighbor search. The algorithm projects points onto random hyperplanes and partitions the space according to which side of the hyperplane the points fall on. This process is repeated recursively, resulting in a binary tree of partitions. A forest of trees is created each with a different random seed. When a query point is received, the algorithm traverses down each tree in the forest to find the leaf node where the point would belong. The nearest neighbors are then approximated by collecting a list of points in the leaf nodes found in all trees, and returning the top-k points from this list that are closest to the query point.</p><p>Annoy is particularly well-suited for high-dimensional data, where exact nearest neighbor search can be prohibitively expensive. It is used in production at Spotify for music recommendations, where it helps find similar songs based on their audio features. It’s worth noting that Annoy provides approximate nearest neighbors, which means it may not always return the exact nearest neighbors, but it does offer great accuracy for an approximation method. That being said, accuracy can be influenced by the number of trees in the forest and the number of points inspected during the search, both of which can be tuned and adjusted based on the specific requirements of the task.</p><p>The efficiency and memory-efficiency of Annoy make it a strong choice for handling high-dimensional data and large databases. Despite being an approximate method, Annoy often provides high-quality results that are close to the true nearest neighbors. There are a few trade-offs to consider. Building the index can take a significant amount of time, especially for large datasets. Because Annoy uses a random forest partitioning algorithm, the indices cannot be updated with new data and must be rebuilt from scratch. Depending on the size of your data set or how often your data changes, retraining the index might be prohibitively expensive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HB2EeBHpubIo1lu9" /><figcaption>Vizualization of the Random Forest Structure</figcaption></figure><h3>Product Quantization</h3><p>Quantization is a process used to transform large, complex datasets into simpler, more manageable forms. Product Quantization (PQ) is a method introduced by Hervé Jégou, Matthijs Douze, and Cordelia Schmid in 2011. In the context of vector databases, Product Quantization (PQ) is used to efficiently store and search high-dimensional vectors. While technically PQ is a technique for creating an index rather than a full-fledged vector database algorithm, its importance and effectiveness in handling high-dimensional data make it a crucial player in the field and a part of many hybrid algorithms. It’s particularly notable for its ability to provide a balance between memory usage and search quality, making it particularly well-suited for large-scale datasets where memory usage is a concern.</p><p>Product Quantization operates by dividing the high-dimensional space into low-dimensional spaces using Cartesian products, and then quantizing each of these subspaces independently. The original high-dimensional vector space is divided into distinct subspaces. For each of these subspaces, a separate codebook is learned. Each codebook contains centroids that are representative of the data distribution in that subspace. High-dimensional vectors are then represented as a concatenation of codes where each code corresponds to the nearest centroid in the respective subspace. This results in a compact and efficient representation of the high-dimensional vectors. PQ is known as an inverted index because it maps high-dimensional vectors to a compact code, and these codes can be used as keys in an index to quickly retrieve the vectors during a search operation.</p><p>Product Quantization is particularly well-suited for large-scale datasets in high-dimensional spaces where memory usage and speed are most important. On the positive side, it provides a very compact representation of high-dimensional vectors, reducing memory usage by up to 97–98%. It allows for fast nearest neighbor search, especially when combined with an inverted file system which further optimizes storage and lookup accuracy. On the downside, PQ, like other methods for approximate nearest neighbor search, is known to have lower recall and accuracy. Learning the codebooks can be computationally expensive, especially for large datasets, which can increase preprocessing time and requires re-indexing with sufficient data drift. There is a tradeoff between the quality of the results and the compactness of the representation — using more centroids will generally give better results but will also require more memory. Additionally, the division of the original space into subspaces is fixed, which may not always be optimal for a data distribution.</p><p>Product Quantization is a powerful method for approximate nearest neighbor search in high-dimensional spaces, offering a tradeoff biased towards memory efficiency and speed with marginally lower search quality. Although it may not be the best choice for all scenarios, the specific requirements of the task should be considered when choosing this method. It is often used as one component of a more complex hybrid algorithm in vector databases, for instance PQ with Inverted file indexes is an optimization used in HNSW.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZw-YlEd2zbI-npDGc4mTw.png" /><figcaption>Product Quantization visual</figcaption></figure><h3>Local Sensitivity Hashing</h3><p>In the realm of hashing algorithms utilized in vector databases, one shines for its efficiency and simplicity: Locality-Sensitive Hashing, or LSH. LSH is a method designed to handle high-dimensional data by hashing input items in such a way that similar items map to the same “buckets” with a high probability, while dissimilar items map to different buckets with a high probability. It’s a popular technique in machine learning, data mining, and information retrieval, especially when dealing with large-scale and high-dimensional data. It is important and effective in handling high-dimensional data, making it a crucial player in the field. It’s particularly notable for its scalability and ability to provide approximate nearest neighbor search results efficiently in high-dimensional spaces.</p><p>In the process of transforming high-dimensional data for efficient similarity search, several steps are involved. First text is encoded into sparse matrices using a combination of shingling and one-hot encoding. Then the MinHashing algorithm is used to transform the sparse matrices into dense matrix embeddings called signatures. Finally these signatures are partitioned into sub-vectors which are each passed through hash functions into buckets. If multiple partitions of different signatures are hashed into the same buckets they are considered candidate pairs. A candidate pair simply refers to a pair of vectors that can be considered as potential nearest neighbors.</p><p>As far as algorithmic trade offs are concerned, LSH is designed to handle large-scale and high-dimensional data, making it suitable for large datasets. It can provide a significant speedup over traditional methods for nearest neighbor search, especially in high-dimensional spaces. LSH provides approximate results, which means there is a marginal tradeoff of lower accuracy. The performance of LSH depends heavily on the choice of parameters, such as the number of hash functions and the size and number of buckets. Choosing these parameters can be challenging and may require domain knowledge and experimentation. Additionally, LSH can require a large amount of memory to store the hash tables, especially for large datasets and high-dimensional data.</p><p>In terms of latency, LSH can provide fast query times due to its hashing mechanism, though throughput can be degraded by the number of hash tables and the size of the buckets. The build time is typically fast, but it can increase with the number of hash functions used. The accuracy of LSH is often lower than exact methods, but it can be improved by using more hash tables or adjusting the bucket size. Lastly, re-indexing in LSH can be done by adding new items to the existing hash tables, but if the data distribution changes significantly, the hash functions may need to be redefined, which requires rebuilding the entire index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*GOiP1VT3IHfK6kk_zqr2dg.png" /><figcaption>LSH hashing algorithm</figcaption></figure><h3>DBSCAN (Density-Based Spatial Clustering of Applications with Noise)</h3><p>DBSCAN is a popular clustering algorithm used in machine learning and data mining. Unlike many other clustering algorithms, DBSCAN does not require the user to specify the number of clusters in advance and can discover clusters of arbitrary shape. It’s particularly useful in situations where there is a lot of noise or outliers in the data, and where the clusters are density-defined, meaning there are dense regions of data points separated by regions of lower density. It works by defining a cluster as the maximal set of density-connected points.</p><p>DBSCAN operates on the idea of density reachability and density connectivity. It starts with an arbitrary point in the dataset, and if there are at least ‘minPts’ within a given radius ‘eps’ from that point, a new cluster is created. Eps stands for epsilon which is a user defined input denoting the maximum distance between two points to be considered in a cluster, while minPts refer to the minimum number of data points required to form a cluster. The algorithm then iteratively adds all directly reachable points within the ‘eps’ radius to the cluster. This process continues until no further points can be added to the cluster. Then, the algorithm proceeds to the next unvisited point in the dataset and repeats the process until all points have been visited. The key parameters in DBSCAN are ‘eps’ and ‘minPts’, which define the cluster of points and the minimum density of points required to form a cluster respectively. DBSCAN does have its trade-offs. In terms of latency and throughput, DBSCAN can be relatively efficient for smaller datasets, but its performance can degrade for very large datasets as it needs to compute the distance between all pairs of points. The build time can be quite high for large datasets and high-dimensional data. In terms of accuracy, DBSCAN can accurately discover clusters of arbitrary shape, but its performance is sensitive to the setting of ‘eps’ and ‘minPts’. If these parameters are not set properly, DBSCAN might fail to identify clusters correctly. Lastly, re-indexing or updating the model with new data can be challenging with DBSCAN. If new data points are added, the entire clustering process needs to be rerun, which may not be practical for dynamic datasets that are updated frequently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*s02-fMLG5Hx0ZpJ9" /><figcaption>DBSCAN clustering visualization</figcaption></figure><h3>HNSW (Hierarchical Navigable Small World)</h3><p>Hierarchical Navigable Small World (HNSW) is a graph-based algorithm used for efficient nearest neighbor search in high-dimensional spaces, and is among the top-performing indexes for vector similarity search. It’s a part of a broader family of Navigable Small World (NSW) graphs, which are designed to provide a balance between search efficiency and accuracy. HNSW is particularly useful in situations where high accuracy is required, speed is preferred, and some computational expense can be afforded. HNSW works by constructing a layered graph structure, where each layer is a subset of the previous one, and search starts from the top layer and navigates down to the bottom layer.</p><p>HNSW constructs a hierarchical graph where each layer of the graph corresponds to a subset of the data points, with the topmost layer containing the fewest points. The algorithm starts by randomly assigning each point to a layer. Then, point by point, it connects it to its nearest neighbors in the same and lower layers. This results in a graph where each node is connected to other nodes that are close in the high-dimensional space. When a query comes in, the algorithm starts from a node in the topmost layer and navigates down the layers, always moving towards nodes that decrease the distance to the query point. This process continues until the algorithm reaches the bottom layer, where it performs an exhaustive search among the remaining nodes.</p><p>In terms of latency, HNSW can provide fast query times due to its hierarchical structure, which allows it to quickly narrow down the search space. Throughput can be affected by the number of layers and the number of connections per node. The build time can be quite high, especially for large datasets, as the algorithm needs to construct the graph and determine the connections between nodes. It is still much more efficient than clustering and tree based algorithms. In terms of accuracy, HNSW can provide high-quality results, but its performance can be sensitive to the choice of parameters, such as the number of layers and the number of connections per node. In General HNSW strikes a great balance between perfect accuracy and low enough latency to make it thrive in production. Lastly, re-indexing or updating the model with new data can be challenging with HNSW. With significant data drift, the entire graph may need to be reconstructed, which may not be practical for extremely dynamic datasets that are updated frequently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*GVlLj63pE57SK2KgqNPinA.png" /><figcaption>HNSW visualization</figcaption></figure><h3>Part 3 Recap</h3><p>In this article we navigated the various algorithms that power vector databases. We emphasized the importance of choosing the right algorithm for a vector database, as it can significantly impact its performance, including latency, throughput, build time, accuracy, and re-indexing requirements. We introduced the families of algorithms available for vector databases, namely tree-based, quantization, hashing, clustering, and graph methods. Each family of algorithms comes with its own set of performance trade-offs, which can significantly impact the performance of a vector database. Then we highlighted that many modern vector databases use hybrid methods, combining the strengths of different methods to achieve high speed and accuracy. Finally we took an in-depth look at several specific algorithms including: Annoy, Product Quantization, Local Sensitivity Hashing, DBSCAN, and HNSW, comparing how their tradeoffs will affect their use on production datasets. While each algorithm has its strengths and weaknesses, the choice of algorithm should depend on the specific requirements of the task, including the size and dimensionality of the dataset, the available computational resources and budget, and the acceptable trade-off between accuracy and efficiency.</p><h3>Conclusion</h3><p>Well we’ve reached the end of our series on vector databases, and if you are still reading I want to profusely thank you for the attention you’ve given this series. As a thank you I’d like to boil down this topic into the most concise explanation I can offer. A vector database is comprised of 3 parts:</p><ul><li>A procedure for encoding text (or other data for that matter) into vector embeddings that our computers can compare and “reason about”.</li><li>A mechanism for comparing these vector embeddings between one another (I.E. similarity measures).</li><li>An algorithm that allows us to use these comparisons to efficiently search, store, and retrieve our vector embeddings for use in production systems.</li></ul><p>In this three-part series on vector databases, we’ve embarked on a comprehensive journey through the fascinating world of vector databases and embeddings. We began with the foundational concept of what a vector database is, what is a vector embedding, and how do we use similarity search to compare them. Then dove deep into the art of creating embeddings by tokenizing words and using transformer models to translate the tokens into rich multi-dimensional embeddings. In the final part, we turned our attention to the intricate algorithms that enable us to search and store our embeddings. We’ve aimed to demystify these procedures and provide practical insights for those looking to leverage vector databases in their work. From the basics of vector transformation to the nuanced trade-offs of various embeddings and algorithms, we’ve strived to offer a valuable guide for both newcomers and seasoned professionals in the field.</p><p>I wrote this series to be a practical explanation of “the How” for vector databases. I’m not done writing on this topic, nor this year’s advances in AI. So If you are interested in the Why, other topics within AI, or the practical applications therein: PLEASE drop me a comment and let me know what you’re interested in.</p><p>Keep learning! I’ll see you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc7a8926f27c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Art of Embeddings: Transforming Text for Vector Databases (Part 2)]]></title>
            <link>https://medium.com/@david.gutsch0/the-art-of-embeddings-transforming-text-for-vector-databases-926738443e70?source=rss-ffafbe525bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/926738443e70</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformer-model]]></category>
            <dc:creator><![CDATA[David Gutsch]]></dc:creator>
            <pubDate>Fri, 28 Jul 2023 13:42:15 GMT</pubDate>
            <atom:updated>2023-08-03T13:49:41.626Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92">first part of this series</a>, we embarked on a journey through the cosmos of vector databases. We explored how these databases transform data into vectors in a multi-dimensional space, enabling a nuanced understanding of data and the ability to compare almost anything. Now, as we continue our exploration, we’ll delve deeper into a crucial component of this process: embeddings. Then in the last segment of our series, we’ll take a look at the algorithms that store, organize, and search these embeddings in the database itself.</p><p>Embeddings are a fundamental concept in deep learning that enable us to capture rich context in ones and zeros. They are powerful, flexible, and while their implementations are necessarily complex, their purpose is beautifully simple. As Roy Keyes succinctly puts it, <a href="https://roycoding.com/blog/2022/embeddings.html">“Embeddings are learned transformations to make data more useful.” </a>This definition encapsulates three key aspects of embeddings: they are learned, they transform data, and they make data more useful. They are learned usually via some variation of a neural network. They are transforming raw data, whether it be: image, text, entity data, product data, or audio. And they make the data more useful by capturing meaning and context in a machine readable and index-able format.</p><p>In this article we will cover the following topics:</p><ul><li>How text is tokenized starting with the Word2Vec method, compare it with newer sub-word methods, and analyze their trade offs along the way.</li><li>We’ll then introduce how transformer models are used to take these tokenized words and phrases and transform them into their final embeddings that are stored in vector databases.</li><li>Finally we will explore the relationship between tokenization and embeddings, namely how the tokenization method chosen will influence the size and efficacy of the transformer model’s embeddings on search.</li></ul><h3>Tokenization</h3><p>The process of transforming text into embeddings begins with tokenization, which is the process of breaking down text into smaller parts, or “tokens.” These tokens can be as small as individual characters or as large as entire sentences, However, in most cases they represent individual words or sub-words. A pioneering method that has evolved this process is Word2Vec, which was <a href="https://arxiv.org/abs/1301.3781">developed at google in 2013.</a> It operates by grouping the vectors of similar words together in a vector space. This is achieved by creating dense vector representations of word features, such as the context of individual words. Given enough data and a variety of contexts, Word2Vec can make accurate predictions about a word’s meaning based on its past appearances. For instance, it can infer that “man” is to “boy” what “woman” is to “girl” based on the contexts in which these words appear.</p><p>Word2Vec uses a neural network to train words against other words that neighbor them in the input corpus. It does this in one of two ways: either using the context to predict a target word, a method known as Continuous Bag of Words (CBOW), or using a word to predict a target context, which is called Skip-Gram. For example, in the sentence “The quick brown fox jumps over the lazy dog,” the CBOW model would take “The,” “quick,” “brown,” “fox,” “over,” “the,” “lazy,” “dog” as context words to predict the target word “jumps.” Conversely, the Skip-Gram model would use “jumps” to predict the surrounding context words. When the feature vector assigned to a word cannot accurately predict that word’s context, the components of the vector are adjusted, refining the model’s understanding of semantic relationships. This iterative process of adjustment and refinement is at the heart of Word2Vec’s power and effectiveness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gog3EDeM25i9ASmghmRDww.png" /><figcaption>CBOW</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdcwIx9O4RDFPqt0UNzmhA.png" /><figcaption>skip gram</figcaption></figure><p>Despite the novel approach, Word2Vec has some limitations. It cannot handle polysemy, which is when a single word or phrase has multiple meanings (e.g. river “bank”, money “bank”), which prevents it from differentiating between multiple meanings of a word based on context. Additionally it must store a vector for every unique word in the vocabulary which causes the size of the model to grow with the size of the corpus vocabulary, becoming a limiting factor for larger data sets. It also struggles with handling out-of-vocabulary words, or words that were not present in the training corpus which can lead to inaccurate representations. Lastly, Word2Vec does not account for morphological variations of words. For instance, it treats “run,” “runs,” and “running” as entirely separate words with no inherent relationship, which can lead to a loss of semantic understanding.</p><p>You might be wondering if Word2Vec doesn’t scale and cannot make simple delineations between words and contexts that we perform so easily as humans, how do we better encode word context into our embeddings? We do this with sub-word tokenization, which is a hybrid approach between word-level and character-level tokenization. It is based on the principle that frequently used words should not be split into smaller sub-words, but rare words should be decomposed into meaningful sub-words. For instance, a rare word like “annoyingly” might be decomposed into “annoying” and “ly”. This is especially useful in agglutinative languages such as Turkish, where you can form (almost) arbitrarily long complex words by stringing together sub-words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*6pCPhH7mn0MmUmI9x_MoUA.png" /><figcaption>simple sub-word tokenization example</figcaption></figure><p>Sub-word tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful context-independent representations. For instance BERT and GPT-2 limit the vocabulary size to <a href="https://arxiv.org/abs/1810.04805">30,000</a> to <a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf">50,000</a> tokens by using WordPiece and Byte Pair Encodings respectively. In addition, sub-word tokenization enables the model to process words it has never seen before, by decomposing them into known sub-words. For instance, if a model trained with sub-word tokenization encounters the word “unseenword”, it could potentially break it down into known sub-words like “un”, “seen”, and “word”. Now there are a number of different methodologies that use the sub-word approach to tokenize words. Too many I’m afraid to address in this article, but for those of you interested I will include them in an appendix at the end of the article.</p><h3>Transformer Models</h3><p>Now that we have a deeper understanding of the mechanisms by which words are tokenized we need to address how text is transformed into their final embeddings, while preserving the semantic meaning of the text on a larger scale. Forgive me in advance for there will be some hand waving at some of the intricacies of transformer models in the interest of focusing on how they enable vector databases to preserve meaning.</p><p>Vector databases generally use encoder-only transformer models; an example of this would be <a href="https://en.wikipedia.org/wiki/BERT_(language_model)">BERT</a> (Bidirectional Encoder Representations from Transformers). We only need to encode the text so that it can be compared with the other embedded bodies of text in the database. Once we know which embeddings are most similar we can use their unique ids to look up the original raw text. These models leverage the power of self-attention mechanisms and positional encodings to understand the context and semantics of words in a sentence. Let’s break down this process into its key steps:</p><ol><li><strong>Tokenization</strong>: We’ve already addressed this part of the process in depth. The number of tokens fed to the model at one time can range anywhere from the size of a sentence, a paragraph, all the way up to a small document, for instance GPT-3 currently can accommodate a maximum of 4k tokens.</li><li><strong>Embedding Lookup</strong>: Once the text is tokenized, each token is mapped to an initial embedding. These embeddings are not random but are pre-trained representations learned during the pere-training phase of the Transformer model. They serve as the starting point for understanding the semantics of each token.</li><li><strong>Positional Encoding</strong>: Transformers, by design, lack an inherent understanding of the order of tokens. To overcome this, positional encodings are added to the initial embeddings. These encodings provide information about the position of each token within the sequence, enabling the model to understand the order of words, while freeing us from the constraint of sequential processing of a text that limited processing in pre-transformer NLP models like RNNs.</li><li><strong>Self-Attention Mechanism</strong>: The next step involves the application of the self-attention mechanism. This mechanism allows each token to ‘look’ at other tokens in the sequence and weigh their influence based on their relevance. Essentially, it enables the model to determine which tokens contribute significantly to the meaning of each individual token.</li><li><strong>Aggregation</strong>: Following the self-attention step, the outputs for each token are aggregated, typically by summing them up. This aggregation results in a new set of embeddings for each token. These embeddings capture both the individual meanings of the tokens and their context within the sequence. The aggregation step combines the context-aware embeddings from the self-attention mechanism into a single vector.</li><li><strong>Feed-Forward Neural Network</strong>: The final step in the process involves passing these aggregated embeddings through a feed-forward neural network. This network processes the embeddings to produce the final set of embeddings and is shared across all positions. The feed-forward neural network further transforms these embeddings, enabling the model to learn more abstract representations, and it helps to generalize the model to unseen data.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*PCPPVL4zVCNd-49NmZ8bZg.png" /><figcaption>One Layer of the original Transformer</figcaption></figure><p>The resulting embeddings are rich in semantic and contextual information, making them incredibly useful for a wide range of natural language processing tasks. In the context of vector databases, these embeddings serve as the high-dimensional vectors that are stored and queried to retrieve semantically similar results. Now let’s take a look at how the tokenization method chosen for the embedding process influences the performance of the embedding process as well as quality and utility of the embeddings relative to one another.</p><h3>Effect of Tokenization on Embedding Efficacy</h3><p>The choice of tokenization method can greatly influence the size and effectiveness of a model’s embeddings. There are three areas which are most affected: vocabulary size, handling of out-of-vocabulary words, and the overall quality and usefulness of the embeddings.</p><p>Choice in tokenization method leads to varying vocabulary sizes. A larger vocabulary equates to more embeddings, which in turn increases the model’s size and the computational resources needed for training and inference. This is why models like BERT and GPT use various sub-word tokenization methods in order to train on a huge corpus of text, while keeping the number of tokens relatively low.</p><p>The issue of out-of-vocabulary words can also impact the quality of the embeddings. This is where sub-word tokenization comes into play, as it allows the model to construct representations for unseen words from the sub-word units it has encountered. Furthermore, certain tokenization methods may be more suitable for specific tasks or languages. This can result in more accurate embeddings and improved performance on tasks such as search. If you’re interested in the specific trade offs of sub-word tokenization methods check out the appendix at the end of the article.</p><p>Tokenization method can significantly affect the size and effectiveness of a transformer model’s embeddings. It’s a crucial consideration when designing and training these models, and it should be guided by the specific requirements of the task and the characteristics of the language of the text. Many vector databases make this determination for you, but depending on your use case you may achieve superior performance in vector search by experimenting with different tokenization methods and transformer models.</p><h3>Recap</h3><p>Today we introduced how word tokenization was twice revolutionized first by Word2Vec allowing us to transform words into machine readable vectors, then once more by breaking down text into sub-words to cut down on the vocabulary size and provide better context.</p><p>Then we introduced the encoder only transformer models which are fundamental to transforming the tokenized words into indexable and comparable context of a larger corpus of text. Through a series of steps — tokenization, embedding lookup, positional encoding, self-attention mechanism, aggregation, and a feed-forward neural network — these models create embeddings that capture both the semantic meaning of each token and the context in which it appears in the sequence.</p><p>Lastly we refined our understanding of the effect our choice in tokenization method has on the overall utility of the embeddings the transformer models will produce. By choosing the right tokenization method we can create a nuanced understanding of text that captures both the meaning of individual words and the relationships between them. We’ve seen how this choice can affect the vocabulary size, handling of out-of-vocabulary words, and the overall quality and usefulness of the embeddings.</p><h3>Conclusion</h3><p>In this comprehensive guide to embeddings, we’ve journeyed through the intricate process of transforming raw text into useful embeddings, the key component for transforming human readable text into a format about which computers may reason. As we conclude this part of our series, we hope you’ve gained a deeper understanding of the embedding processes that underpin vector databases. These processes, while complex, are fundamental to the AI revolution, enabling machines to understand and process language in ways that were once the exclusive domain of humans.</p><p>Zooming out from this article, in our first article we explained how these embeddings can be compared to produce a query result of similar text data, in this article we learned about how the vector embeddings themselves are created, so there is one last component which we left unexplored: the Algorithm that stores and searches vector databases.</p><p>Tune in next week for <a href="https://medium.com/@david.gutsch0/vector-databases-understanding-the-algorithm-part-3-bc7a8926f27c">Part 3</a>, the last installment of this series where we address the Algorithms themselves that allow us to store and query these embeddings!</p><h3>Appendix: Specific sub-word tokenization methods</h3><ol><li><strong>Byte Pair Encoding (BPE)</strong></li></ol><ul><li><strong>How it works</strong>: BPE starts with a vocabulary of individual characters and iteratively merges the most frequent pair of symbols to produce a new symbol. This process continues until a predefined number of merges have been made.</li><li><strong>Advantages</strong>: BPE can handle out-of-vocabulary words and morphological variations. It’s flexible and can adapt to the specificities of the language it’s trained on.</li><li><strong>Disadvantages</strong>: BPE can sometimes produce strange splits of words, especially for languages with complex morphology. It also requires a two-step process of first learning the BPE merges and then training the model.</li><li><strong>Used in</strong>: GPT-2, RoBERTa.</li><li><strong>Example</strong>: Given the word “lowers” and the most frequent pair is (“o”, “w”), BPE will merge them into a new symbol “ow”. The word “lowers” will be tokenized into “l”, “ow”, “e”, “r”, “s”.</li></ul><p><strong>2. Byte-Level Encoding</strong></p><ul><li><strong>How it works</strong>: Byte-level encoding uses a vocabulary of all possible byte values (256 unique bytes). It can handle any string of bytes, making it particularly useful for multilingual models or models that need to handle non-text inputs.</li><li><strong>Advantages</strong>: Byte-level encoding can handle any kind of input and doesn’t require any special handling for out-of-vocabulary words. It’s also very memory-efficient.</li><li><strong>Disadvantages</strong>: Byte-level encoding can sometimes produce very long sequences for languages that use multi-byte characters (like Chinese or Japanese).</li><li><strong>Used in</strong>: GPT-3.</li><li><strong>Example</strong>: The word “hello” will be tokenized into the corresponding byte values of each character: 104, 101, 108, 108, 111.</li></ul><p><strong>3. Word Piece</strong></p><ul><li><strong>How it works</strong>: Word Piece is similar to BPE but it prefers to keep whole words intact. It starts with a base vocabulary of individual characters and then learns a fixed number of merges, similar to BPE.</li><li><strong>Advantages</strong>: Word Piece can handle out-of-vocabulary words and it’s less likely to split words in strange ways compared to BPE.</li><li><strong>Disadvantages</strong>: Word Piece can still produce unexpected splits and it requires a two-step process of first learning the merges and then training the model.</li><li><strong>Used in</strong>: BERT, DistilBERT.</li><li><strong>Example</strong>: Given the word “lowers” and the most frequent pair is (“low”, “ers”), Word Piece will merge them into a new symbol “lowers”.</li></ul><p><strong>4. Unigram</strong></p><ul><li><strong>How it works</strong>: Unigram tokenization is a subword regularization method that learns a subword vocabulary by minimizing the loss of the likelihood of the training data.</li><li><strong>Advantages</strong>: Unigram can handle out-of-vocabulary words and it’s more flexible than BPE or Word Piece because it allows for multiple possible segmentations of a word.</li><li><strong>Disadvantages</strong>: Unigram can sometimes produce unexpected splits and it requires a two-step process of first learning the merges and then training the model.</li><li><strong>Used in</strong>: SentencePiece.</li><li><strong>Example</strong>: Given the word “I love machine learning” [“I”, “ “, “a”, “d”, “o”, “r”, “e”, “ “, “machine”, “ “, “learning”]</li></ul><p><strong>5. SentencePiece</strong></p><ul><li><strong>How it works</strong>: SentencePiece is a language-independent subword tokenizer and detokenizer. It treats the input as a raw input string, so you don’t need to pre-tokenize the text. SentencePiece implements both BPE and unigram language model with the extension of direct training from raw sentences.</li><li><strong>Advantages</strong>: SentencePiece allows for the flexibility of BPE and unigram language model while also being able to handle multiple languages in one model. It doesn’t require any pre-tokenization.</li><li><strong>Disadvantages</strong>: SentencePiece can sometimes produce unexpected splits, and the choice between BPE and unigram may not be clear for every application.</li><li><strong>Used in</strong>: Multilingual BERT, T2T (Tensor2Tensor).</li><li><strong>Example</strong>: Given the sentence “This is a test.”, SentencePiece might tokenize it into [“▁This”, “▁is”, “▁a”, “▁test”, “.”], where “▁” represents a space.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=926738443e70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Databases: The Secret Sauce of the AI Revolution]]></title>
            <link>https://medium.com/@david.gutsch0/vector-databases-the-unseen-powerhouse-of-the-ai-revolution-part-1-6685653abd92?source=rss-ffafbe525bbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/6685653abd92</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[vector-database-for-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <dc:creator><![CDATA[David Gutsch]]></dc:creator>
            <pubDate>Tue, 11 Jul 2023 12:39:31 GMT</pubDate>
            <atom:updated>2023-08-03T13:48:43.764Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1</h4><h4><strong>Introduction</strong></h4><p>Ever wondered how your favorite music streaming service seems to read your mind, suggesting songs that perfectly fit your mood? Or how your online shopping platform knows just what you need, even before you do? Ever marveled at the extensible models and applications that are being built on top of LLMs like ChatGPT? The secret behind these modern marvels is not magic, but a powerful tool in the realm of databases: vector databases. Let’s embark on a journey to unravel the mysteries of these unsung heroes of the AI revolution.</p><h4><strong>What are Vector Databases?</strong></h4><p>You’ve been around the block with traditional databases, haven’t you? Rows, columns, tables, foreign keys — the whole SQL shebang. But let’s add a twist to this tale, shall we?</p><p>A Vector Database is a high-dimensional twist on traditional databases. Unlike a Relational Database that stores data in structured tables, or a NoSQL database that handles unstructured data; a Vector Database transforms data into vectors in a multi-dimensional space. Similar to a Graph Database that represents data relationships, a Vector Database represents data similarity, with closer vectors indicating more similar data. It’s a powerful tool for machine learning and AI applications, enabling efficient similarity searches and clustering in high-dimensional data.</p><p>Imagine a database that doesn’t just store data but comprehends it. Intriguing, isn’t it? Welcome to the world of vector databases. These aren’t your run-of-the-mill data warehouses; they’re more like data transformers. They take your data, wave their magic wand, and voila — your data is now a vector, a point in a multi-dimensional space.</p><p>Think of it like this: you’re not just building a collection of data structures anymore. You’re crafting an entire cosmos where each data point (now a star) has its own unique position, determined by its features. The closer the stars, the more similar they are. It’s like navigating through a galaxy of data, where the constellations are your clusters of similar data points. How’s that for a database upgrade?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*qulHQrzjOaYwkn_eBj3AmA.png" /></figure><h4><strong>Understanding Vector Embeddings</strong></h4><p>Now, you might be wondering, why am I talking about constellations, and more importantly what’s a vector in this context? Well, imagine you’re at a party. You don’t know anyone, so you start introducing yourself. “Hi, I’m Alex, I’m a software engineer, I love rock music and hiking.” That can be encoded as a vector (i.e. a star in our metaphor). It’s a list of features that describes you. In a vector database, every piece of data gets a similar introduction, but instead of hobbies and professions, it might be color histograms for images, word frequencies for text, or user ratings for products.</p><p>Now back to the constellations. When we are encoding these vector embeddings we are storing them in a multi-dimensional space that is most comparable to space. Alex the software Engineer is closer to Sally the Data scientist, who are both farther away from Derek the Financial advisor. There are many different mechanisms by which these embeddings may be encoded and stored including, optimized search through hashing, quantization, or tree and graph-based searches. These algorithms are my favorite part, though beyond the scope of this article, tune in for part two if this too is interesting to you.</p><p>But wait, there’s more. We have two types of vectors: input vectors and query vectors. Input vectors are like the guests at the party, each with their own set of features. Query vectors, on the other hand, are like a description of the person you want to meet. “I want to meet someone who loves rock music and hiking.” The database’s job is to find the input vectors that match the query vector. Our database will use some fancy data structures and algorithms to look through the cosmos of the database and find the input vectors most similar to the query vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*qqt-96m-nIq6I2l7k3re0g.png" /></figure><h4><strong>Similarity Measures in Vector Databases</strong></h4><p>So how does the database decide which vectors are similar? It uses something called a similarity measure. It’s like a digital measuring tape of sorts that can tell how alike two vectors are, or more precisely the distance between the two in mutli-dimensional space. In vector databases similarity measures such as: cosine similarity, Euclidean distance, and dot product distance assess vector likeness. Cosine similarity gauges similarity via the angle between vectors, while Euclidean distance uses straight-line distance in multidimensional space, and dot product distance uses the product of corresponding vector values. Each measure is suited to specific data types and applications.</p><p>If that all made sense to you that’s great feel free to skip to the next section! I found some of these measures tricky so I’m providing an additional explanation for the other plebeians like myself. The Euclidean distance measure is the simplest of the three, and simply charts a straight-line path between two planets in our cosmos. Cosine similarity is like gauging the angle between two stars; the smaller the angle, the closer and more similar they are. Then there’s the dot product distance measure, which is like the gravitational pull between celestial bodies, influenced by their individual masses and distance apart. The dot product of the matrices between the vectors measure their distance between one another across all the vectors features. It is like the cosine distance except that it takes into account the magnitude of the distance as well as the angle. Each of these similarity measures guides us differently through the data universe, helping us navigate based on our destination and the cosmic conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*X74ID-9t9A_f5Nmnitlwhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*x4Kvbs6EJrS0iC_9DVDrNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*p2B6AVuXIdjfNYDhVibkqA.png" /><figcaption>Cosine, Euclidean, and Dot Product respectively</figcaption></figure><h4><strong>Applications and Use Cases of Vector Databases</strong></h4><p>Now, let’s talk about where vector databases shine. You know that friend who always knows the perfect movie for movie night? That’s what a vector database does for Netflix. Or that personal shopper who always knows what’s in style? That’s a vector database for your online shopping platform. From personalized recommendations to image recognition, vector databases are the secret sauce that makes modern applications so… well, modern.</p><p>Let’s not forget the exciting world of Language Models and Generative AI, where vector databases play a pivotal role. Behind the scenes of GPT, it’s using vector databases or at least vector indexes to store and retrieve the embeddings of words and sentences. These embeddings capture the semantic meaning of words, allowing the model to understand language in a way that’s eerily similar to how we humans do.</p><h4><strong>Recap</strong></h4><p>As we journeyed through the cosmos of vector databases, we’ve uncovered some of the hidden mechanisms that are rapidly reshaping our world.<strong> </strong>To recap: Vector databases, unlike traditional ones, transform data into vectors in a multi-dimensional space, enabling a nuanced understanding of data and the ability to compare almost anything. We’ve also explored vector embeddings, which encode data into unique vectors, and similarity measures enable us to compare the distances between each of these vectors.</p><h4>Conclusion</h4><p>The profound applications of vector databases in AI are truly awe-inspiring. From personalized recommendations on your favorite streaming service to the semantic understanding of language in models like GPT, vector databases are the unseen powerhouse behind these marvels. They are the secret sauce that makes modern applications so intuitive and responsive, transforming the way we interact with technology.</p><p>The exponential growth of vector embeddings in fields such as NLP, computer vision, LLMS, and other AI applications has led to the rise of vector databases. These databases are specialized to tackle the challenges that arise when managing vector embeddings in production, they have significant benefits over traditional databases and scale better than standalone vector indexes. Most importantly, these databases are enabling normal application developers to create extensible models build on top of LLMs or generative AI models, thus allowing us to customize models for our customers.</p><p>As we push the boundaries of AI, the role of vector databases will only become more crucial. They are not just a tool for storing and retrieving data, but a fundamental component in our quest to make machines understand and interact with the world in a human-like way. So, stay tuned, keep exploring, and remember: in the world of data, there’s always more to learn.</p><p>If you enjoyed this come back next week for <a href="https://medium.com/@david.gutsch0/the-art-of-embeddings-transforming-text-for-vector-databases-926738443e70">part 2</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6685653abd92" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>