<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[lastmile ai — blog - Medium]]></title>
        <description><![CDATA[Use lastmile ai to experiment with generative AI, personalize it for your use cases, and integrate it in your application. For enterprises and developers - Medium]]></description>
        <link>https://medium.com/lastmile-ai?source=rss----518566bd75e9---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>lastmile ai — blog - Medium</title>
            <link>https://medium.com/lastmile-ai?source=rss----518566bd75e9---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:02:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/lastmile-ai" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Guide to Evaluating Retrieval-Augmented Generation (RAG) systems]]></title>
            <link>https://medium.com/lastmile-ai/the-guide-to-evaluating-retrieval-augmented-generation-rag-systems-01568ed02d0e?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/01568ed02d0e</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Andrew Hoh]]></dc:creator>
            <pubDate>Wed, 24 Jul 2024 18:17:00 GMT</pubDate>
            <atom:updated>2024-07-24T18:17:00.433Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVxbvQvTi8_tzjtexFfpGw.png" /><figcaption>The journey from RAG POC to Production</figcaption></figure><p>Over the past several months, I’ve worked with startups and companies stuck in the proof-of-concept stage. These companies were in a loop of identifying an inaccuracy, applying a one-off fix, and repeating, in the hopes of hitting their accuracy goal. From these conversations and collaborations, it’s become clear that a good testing and evaluation framework was the key to finally transitioning a POC into production. I’m sharing my learnings and takeaways for others who may be encountering similar problems.</p><h4>The Problem</h4><p>As of today, RAG application builders are able to get a functional proof-of-concept complete with relative ease. But, in order to meet accuracy goals to deploy from proof-of-concept into production, developers are encountering many challenges. I’ve seen the regression in performance after a developer adds a <em>“response should be brief”</em> to the system prompt. I’ve seen the chunking nightmares where it starts with <em>the chunk size just needs to be a few token larger</em> and ends up with complex dynamic chunking sizes based on document classifications. At times it feels like a hopeless journey questioning whether an accuracy goal of 95% is even possible.</p><p>Taking a page from traditional ML, setting up a good evaluation test is important in navigating the decision making on how to incremental make progress. I’ll talk about the ideal evaluation (<em>unfortunately, there are limitations</em>) and the practical setup to evaluations.</p><h4>Ideal Solution</h4><p>Yes, there is a solution to this problem to definitively evaluate these RAG systems. The perfect evaluation framework is <strong>people</strong>. <strong>People</strong> are the only ones who can definitively judge whether a RAG system is responding appropriately or not, and there are two approaches to incorporating people into the system:</p><ol><li><a href="https://cloud.google.com/discover/human-in-the-loop?hl=en">Human-in-the-loop</a> — introducing people into the system to review and/or validate the results from the RAG or LLM application.</li><li><a href="https://hdsr.mitpress.mit.edu/pub/aj31wj81/release/1">Online Experimentation</a> — typically as an A/B test, driving a portion of production traffic from real users to the RAG or LLM application. Measuring the success of the experiment based on key metrics.</li></ol><p><em>If both human-in-the-loop and online experimentation are solutions to evaluating RAG applications, why is evaluation still an unsolved problem?</em></p><p>Well, the challenges that most companies are encountering are that in most cases, it’s not feasible to instrument these solutions into their RAG applications. The reasons being:</p><ol><li>Human-in-the-loop typically requires subject matter experts (SMEs) to review and validate responses. SMEs can precisely determine whether an LLM response is correct or incorrect, especially in more nuanced cases. However, SMEs’ times are valuable and it’s not always feasible to have SMEs continuously review responses. You can reduce the time needed by randomly sampling a small portion of the LLM responses for review, but this increases the likelihood of edge cases accidentally being released to production.</li><li>Online experimentation is a very powerful approach to confidently understanding whether a RAG application is positively moving success metrics. However, companies have concerns with pushing LLMs to production without a stronger guarantee on how misinformation won’t be generated (e.g. <a href="https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/"><em>What Air Canada Lost in ‘Remarkable Lying AI Chatbot Case</em></a>)</li></ol><h4>Combination of Evaluations</h4><p>In absence of the ideal solution, I’ve found that a combination of evaluation techniques serves as the approximate for evaluating the end-to-end RAG application. The best approach consists of three categories: <em>1. the vibe-check; 2. evaluators; and 3. collecting user feedback.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6T9CRAONndnNXDfsLGg_NA.png" /></figure><ol><li><strong>Vibe-checks</strong> — although this isn’t the most scientific approach, I have found that spot checking from SMEs (engineers, PMs, biz ops, etc.) is a viable and necessary step in improving a RAG application performance. The difference between those who have already productionized a RAG application vs. those who have not is that the experienced people are organized with their “spot checking”. They neatly label the input and output with a <a href="https://en.wikipedia.org/wiki/Ground_truth">ground truth</a> label. They persist this data point to be used for evaluation runs, integration tests, or regression tests. My recommendation is to invest time in a good tool to help version control and keep track of your ground truth data. You don’t want to regret it later on when you find that you don’t have any ground truth data and you have to start from scratch.</li><li><strong>Evaluators</strong> — Functions to measure the performance of your application. Evaluators can take inputs, outputs, and optionally ground truth data to provide meaningful metrics on whether your RAG application is improving across the metrics you care about. Across the applications I’ve worked with, it’s been difficult to derive a single composite metric that can accurately measure whether an LLM response with context yields better performance or not, I have found that evaluators are great indicators on whether the performance you care about is improving. There are many ways to segment the different types of evaluators (e.g. <em>with/without labels, NLP / CV , etc.</em> ), but for this document, I’ll segment off of three types of implementations of evaluators: <em>heuristic-based</em>, <em>evaluator models</em>, <em>LLM prompting<br> <br>- </em><strong>Heuristic-based</strong> — traditional NLP functions (BLEU, ROGUE, etc.) to measure the accuracy of output text against ground truth. I have found that these metrics have been less useful of late to measure performance of a RAG application’s output.<br> <br>- <strong>Evaluator model</strong> — smaller, specialized classification models to validate the performance (<em>faithfulness, answer relevancy, context relevancy, semantic similarity, etc.</em>). I have found that this approach doesn’t sacrifice on performance compared to LLM prompting, but is significantly more cost and latency efficient. Given the benefits, we, here at <a href="https://lastmileai.dev/">LastMile AI</a>, released our own evaluator models (<a href="https://blog.lastmileai.dev/harder-better-faster-stronger-llm-hallucination-detection-for-real-world-rag-part-i-949248f0ad94">p-faithful</a>).<br> <br>- <strong>LLM prompting</strong> — the current state of the art is to prompt current state-of-the-art LLMs to judge the performance of the RAG application. This approach is often-times too expensive, high latency, and imprecise (i.e. the numerical significance of the response from the LLM lacks statistical significance).</li><li><strong>User feedback</strong> — A far cry from online experimentation against external users, but almost all RAG applications go through a dogfooding phase or internal testing. Integrating RAG applications to collect well-organized labels from internal users or dogfooders is an important way to acquire high-quality labels to further improve your application.</li></ol><p>With a combination of these 3 techniques, I’ve found that companies and users have the best framework in place to get to the performance that meets their bar for production.</p><h4>Conclusion</h4><p>Evaluation for RAG systems is still an unsolved problem, but I’m betting that with more automation of these techniques and personalization, the state-of-the-art will advance further. If you’re interested in automating or fine-tuning your evaluations, feel free to connect with me at <a href="/service/mailto:andrew@lastmileai.dev">andrew@lastmileai.dev</a> and check us out @ <a href="https://lastmileai.dev/">LastMile AI</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01568ed02d0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/the-guide-to-evaluating-retrieval-augmented-generation-rag-systems-01568ed02d0e">The Guide to Evaluating Retrieval-Augmented Generation (RAG) systems</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Workbench Beta Launch]]></title>
            <link>https://medium.com/lastmile-ai/rag-workbench-beta-launch-f68776117dea?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/f68776117dea</guid>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Sarmad Qadri]]></dc:creator>
            <pubDate>Thu, 20 Jun 2024 13:26:32 GMT</pubDate>
            <atom:updated>2024-08-08T15:15:35.594Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>tl;dr</strong>: LastMile AI is proud to announce <a href="http://www.lastmileai.dev/ragworkbench">RAG Workbench</a> Beta, a dedicated developer tool for evaluating, debugging, and optimizing RAG applications. RAG Workbench provides deeper transparency into your RAG app, a rich debugging tool to go from insight -&gt; action, and enables unparalleled customization by harnessing your data to fine-tune evaluators for each of your use cases.</p><p><a href="https://form.typeform.com/to/d6sCnK6z"><strong>**Sign up for early access**</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DknoxRpmcr8AAeKf" /><figcaption>RAG Workbench experiments view</figcaption></figure><h4>Background</h4><p>Innovations in LLMs have drastically outpaced the tools needed to <em>test, evaluate</em>, and<em> debug</em> these models and their applications. We’ve surveyed over 200 engineering leaders across industries including technology, finance, energy in enterprises of all sizes who have surfaced the need for better testing harnesses and evaluations to guide the development of LLM applications, and get to production faster.</p><h4>Core Problems Today</h4><blockquote>- RAG ≠ Enterprise RAG</blockquote><p>The Enterprise RAG architecture today has become more complex than the basic RAG architecture (<em>embeddings, vector store, retrieval func and LLM generation</em>). The Enterprise RAG includes ingestion pipelines from various data sources (with users and permissions), dynamic chunking strategies, reranking models, output guardrails and much more. Root causing the origin of inaccuracies has become more challenging.</p><blockquote>- Evaluators are not aligned with Business KPIs</blockquote><p>Determining if an LLM application is performing well is a hard problem. Data science metrics (e.g.<em> BLEU, ROGUE, Perplexity</em>) are not translating to what engineering leaders care about (the Business KPIs). Each use case is unique and requires customization to more precisely measure progress. The current state of evals is more vibes based than science based.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DEr7wtG9mVK4xxED" /></figure><h3>RAG Workbench Beta</h3><p>We are proud to announce <a href="https://lastmileai.dev/products/ragworkbench">RAG Workbench</a>, a developer tool for debugging, evaluating, and optimizing Enterprise Retrieval Augmented Generated (RAG) pipelines. The tool has three core capabilities:</p><ol><li><em>Distributed Tracing</em></li><li><em>Fine-tuned Evaluators</em></li><li><em>Debugging &amp; Experimentation</em></li></ol><p><strong>Distributed Tracing</strong>: RAG Workbench has first-class support for tracing both retrieval <em>and</em> data ingestion pipelines, allowing you to bridge offline/online systems and identify anomalies with greater accuracy. The tracing SDK is compatible with <a href="https://opentelemetry.io/">OpenTelemetry</a>, an open standard protocol for tracing and observability.</p><p><strong>Fine-tuned Evaluators</strong>: LastMile AI has built in-house evaluator models that represent a step-function jump from the current state-of-the-art, which measure metrics specific to RAG applications, such as hallucination rates, response relevancy and more. The zero-shot models are 1/1000th the cost of GPT-4, and <a href="https://blog.lastmileai.dev/harder-better-faster-stronger-llm-hallucination-detection-for-real-world-rag-part-i-949248f0ad94">outperform GPT-4 base evaluations</a>.</p><p>As part of RAG Workbench, we are expanding support for custom use cases and fine-tuning evaluator models. We would love to help get the right evaluators specifically trained for you and your unique use case.</p><p><strong>Debugging &amp; Experimentation</strong>: RAG Workbench also has an interactive companion to help developers debug and iterate on any RAG application. Inspect retrieved context, rerun prompts to LLMs within the debugger and have an audit log of calls to your RAG application.</p><blockquote>We are continuously releasing improvements to help developers systematically identify the best-performing configuration for their RAG pipelines.</blockquote><p>RAG Workbench is <strong>self-hostable</strong> and designed for Enterprises. For the Beta launch, we are opening up registration.</p><blockquote>Sign up for <a href="http://www.lastmileai.dev/ragworkbench">RAG Workbench</a> here: <a href="https://lastmileai.dev/products/ragworkbench">https://lastmileai.dev/products/ragworkbench</a></blockquote><p><strong>Getting Started</strong></p><ol><li><a href="https://form.typeform.com/to/d6sCnK6z"><strong>Request access</strong></a> to <a href="https://lastmileai.dev/products/ragworkbench">RAG Workbench</a> to receive your API key.</li><li>Install the lastmile-eval package from PyPI, and follow the <a href="https://rag.lastmileai.dev/docs/quick-start">Quick Start Tutorial</a> to try it out.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHkvHs4A5xWU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHkvHs4A5xWU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHkvHs4A5xWU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/785cff39956c5c18ddb02e0ace56e34d/href">https://medium.com/media/785cff39956c5c18ddb02e0ace56e34d/href</a></iframe><p>We are constantly iterating to improve the daily lives of AI engineers, and we value every piece of feedback. Please <a href="https://discord.com/invite/xBhNKTetGx">reach out to the team on Discord </a>to get in touch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f68776117dea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/rag-workbench-beta-launch-f68776117dea">RAG Workbench Beta Launch</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harder, Better, Faster, Stronger:  LLM Hallucination Detection for Real-World RAG, Part I]]></title>
            <link>https://medium.com/lastmile-ai/harder-better-faster-stronger-llm-hallucination-detection-for-real-world-rag-part-i-949248f0ad94?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/949248f0ad94</guid>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Jonathan]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 18:56:44 GMT</pubDate>
            <atom:updated>2024-03-12T21:12:06.306Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing a very small, specialized, yet accurate hallucination detector for RAG evaluation.</h4><h3>TL;DR</h3><ul><li>We show that a small, specialized model (p-faithful-v0) can classify LLM outputs as faithful/hallucinations with accuracy similar to or better than much larger baseline models.</li><li>p-faithful-v0 is a tiny fraction of the size of the LLMs backing the baseline algorithms, unlocking large inference cost savings.</li><li>We introduce a new hallucination detection benchmark tailored for the development of real-world detector models.</li><li>We show an additional 23% (70% -&gt; 93%) accuracy improvement with modest fine-tuning.</li></ul><p><strong>We are extending early access to our hallucination detection <br>(p-faithful-v0) model to test on a diverse set of use cases. If you need help evaluating your production RAG application, we would love to work with you.</strong></p><p><a href="https://cdn.forms-content.sg-form.com/e8b9b4d4-dda0-11ee-85ff-daa2c5cfc593"><strong>Request Early Access.</strong></a></p><h3>Introduction</h3><p>We’ve all figured out by now that our LLMs are lying to us. If not all the time, still with worrying frequency, appalling confidence, and an infuriating lack of shame. LastMile is fighting back: we present a small, specialized language model (“p-faithful-v0”) that achieves very high accuracy for its size on several hallucination detection benchmarks.</p><p>In this post, we show how LLM responses generated by a RAG (Retrieval Augmented Generation) application can be accurately and cheaply classified as either faithful or hallucinatory. The model has comparable accuracy to the baselines, with less than 1% of the parameters. We also present an additional benchmark, “Fully-Synthetic V2” (FSV2), a dataset of synthetic inputs and labels which aims to address many of the issues with the other benchmarks. We built this benchmark in order to more confidently make inferences about <em>real industrial applications. </em>We provide additional analysis on this synthetic data, showing that p-faithful-v0 can easily jump from 70% to 93% with modest fine-tuning.</p><h3>Motivation &amp; Problem Statement</h3><p>LLMs can hallucinate even when they are directly fed ground truth data as part of the prompt, as is typical in a RAG setup. We want to be able to accurately, quickly, and cheaply detect when an LLM is hallucinating vs. when it is <em>faithful</em> to this ground-truth input data. Such detection can be used in application logic, e.g. to suppress high-risk responses, for batch offline analytics, for A/B testing, etc.</p><p>To benchmark detector accuracy, we followed the lead of the baseline detectors (discussed in more detail under “Baselines and Benchmarks” below). Each detector is a probabilistic binary classifier</p><p><em>p(faithful | question, data, response)</em></p><p>scoring input text from 0 to 1. The score estimates the faithfulness of the response to the reference, where 0 means certainly a hallucination. In RAG applications, where annotated reference answers are scarce if they exist at all, one can set the reference to be the data fed to the LLM (composed of the chunks retrieved for a particular query).</p><p>The benchmarks consist of rows, each with a question, data, and two answers. By construction, one of the two answers is faithful and the other is not (i.e. it is hallucinatory). The task is to predict which answer is which. This is done by scoring each triplet <br><em>(question, data, response1)</em> and<strong><em> </em></strong><em>(question, data, response2)</em><strong><em> <br></em></strong>from 0 to 1 using the classifier described above. The answer with the higher score is the predicted faithful answer, and the other one is the predicted hallucination.</p><p>For a given row, an evaluator is considered correct if it correctly identifies which answer is the faithful one (1 point). In the case of a tie (i.e. it scores the two answers the same) it is considered half correct (0.5 points).</p><h3>Baselines &amp; Benchmarks</h3><p>We looked around the world 🤖<em> </em>for some of the best open source LLM hallucination detectors, and used the benchmark datasets they cited. Specifically, we looked at <a href="https://docs.ragas.io/en/stable/">RAGAS</a> (w/ GPT-3.5) on <a href="https://huggingface.co/datasets/explodinggradients/WikiEval">WikiEval</a> and <a href="https://docs.arize.com/phoenix/evaluation/running-pre-tested-evals/hallucinations">Arize’s Phoenix</a> (w/ GPT-4) on <a href="https://github.com/RUCAIBox/HaluEval">HaluEval</a>. We did our best to reproduce their reported results as baselines (detailed below).</p><h4>Fully-Synthetic V2 (FSV2): a New Synthetic Benchmark</h4><p>Both existing benchmarks were interesting, but turned out to have a few central issues:</p><p><strong>1.</strong> <strong>The tasks are too easy</strong>. By manual inspection and model analysis, we found that existing benchmarks consist of simple, often short text with easily-distinguishable faithful/hallucinatory pairs. Even if a detector does well on these benchmarks, we are not comfortable inferring that it would do well in the real world, where data is messier and hallucinations might be more varied, less clear, etc. <strong>Easy example from HaluEval:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4OvWY4tdbVZR86YqklydTQ.png" /><figcaption><em>Example pair from HaluEval (relevant words bolded). The contrived data and question, and answers are short and straight-forward. The hallucinated answer is obviously wrong. In contrast, real-world examples are likely to be much longer and messier, and may involve much more difficult comprehension and reasoning.</em></figcaption></figure><p><strong>2.</strong> <strong>The datasets are small and bounded by manual labeling</strong>, especially in the case of WikiEval, limiting measurement confidence. WikiEval contains only 50 pairs. HaluEval is somewhat larger at several thousand examples.</p><p><strong>3. LLM training data leak</strong>: Benchmarks generally use actual LLMs like GPT-4 to generate both positive and negative answers. If the data was seen during GPT-4’s training, then answers that were expected to be hallucinatory may in fact be in agreement with the data. If GPT-4 does get lucky 🤖 — so to speak — then a negative will be incorrectly labeled as a positive in the benchmark. We found cases like this in WikiEval, and suspect this problem exists in other datasets as well.</p><p>To address all 3 problems, we developed a data synthesis pipeline that automatically generates new automatically-labeled faithful / hallucinatory pairs. We used this flexible algorithm to generate FSV2, which is one of our key <em>in-the-lab</em> datasets. “Fully” synthetic means that it does not use any externally-available data, i.e. it does not suffer from the training data leak issue. The algorithm also makes it easier to generate large datasets. Finally, the task is hard: the hallucinations are by design as subtle as possible. They are pretty difficult to label even by hand (we did a lot of spot checking).</p><h4>Limitations and Follow-Ups</h4><p>Given that the data is fully synthetic, it does not <em>directly</em> simulate any real-world domain. However, if raw, unlabeled seed data are available from a production RAG system, the algorithm is flexible enough to simulate pairs that would be generated by that system. Such a “partially-synthetic” dataset would be bounded by the seed dataset, but would serve as a more realistic benchmark. We will move from the lab to the field using this setup in future work.</p><p>Next, how do we know the synthesized pairs are correct? There is an apparent circularity problem. We know LLMs hallucinate, so how can we generate correctly-labeled faithful/hallucinated answer pairs using LLMs? Typically, benchmarks rely on human annotation at some step, which breaks the cycle, but takes time, effort, and money.</p><p>In our case, we simply accept that we will get some hallucinations and design the algorithm around that. We loosen the requirements and only care that the answer <em>pairs</em> are <em>good enough</em>: i.e. that, with high probability, each answer pair resembles closely enough a hallucinated/faithful <em>pair</em> you might see in a golden labeled dataset.</p><p>This pairing property is essential to the dataset design, and we believe analytically (and by a healthy degree of spot checking) that the algorithm works better than alternatives for this purpose. While existing public benchmarks are useful as general-purpose comparisons across many approaches, we’re optimizing ours for specific production-grade RAG system evaluation.</p><p>Nevertheless, we will keep improving on FSV2 and empirically scrutinize its correctness in a future post. I’ll give you a minute for your eyebrows to return to your forehead before proceeding.</p><h3>Results: Out of Domain Testing</h3><p>For each baseline algorithm (RAGAS and Phoenix), we used their default hallucination detection settings, underlying LLM, and benchmarks (WikiEval and HaluEval, respectively). We attempted to reproduce their results using their code, and then used the same general procedure to test <br>p-faithful-v0. In the next two tables, all results are completely out-of-domain: no fine-tuning, no in-context-learning. (This is similar to, but technically a bit different from zero-shot classification. We’re not too worried about terminology, but just in case you were wondering.)</p><p><strong>Baseline: RAGAS, GPT-3.5 (175B parameters)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vxqmx5PZls8OaG45ue8s4w.png" /></figure><p><strong>Baseline: Phoenix, GPT-4 (175B+ parameters)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aen_nwjJGGgGNGvpBCVkAw.png" /><figcaption><em>Our experimental results (pairwise accuracy). <br>The test set was used where data was sufficient to create one, else the full dataset.</em></figcaption></figure><blockquote><em>*</em>The authors report 95%, but we were only able to reproduce 85% at best, even using their code off the shelf. This could be due to the fact that they have not released their ground-truth labels, and we used a proxy task in this case. <br>Of course, we did this consistently across the detectors.</blockquote><h3>Results: Supervised Domain Adaptation via Fine-Tuning</h3><p>We then did a deeper analysis using FSV2. Specifically, we wanted to see whether performance could be raised to a strong absolute level on our hardest benchmark using a small labeled training set (under 1000 examples in the target distribution). We fine-tuned for a few epochs with mostly default hyperparameters.</p><p><strong>Baseline: p-faithful-v0 with no in-distribution fine-tuning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BOWXeU0drdtLlGYH5OcBw.png" /></figure><p>We plan to repeat this on other hard benchmarks, e.g. HaluEval 2.0. Note that for WikiEval, out-of-domain performance is already at 98%, so there is no point in fine-tuning on it.</p><h4>FSV2 Accuracy vs. Other Benchmarks</h4><p>We validated the hardness of the FSV2 task by comparing accuracy with other benchmarks. Every detector (including p-faithful-v0), does much worse on FSV2 than on the baseline benchmark. Here, we have just merged and reshaped the previous tables to control for detector and directly compare benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Wky0C23HivL_HpxSakRfQ.png" /></figure><h3>Analysis &amp; Takeaways</h3><p>First, let’s notice that p-faithful-v0 (out-of-domain) does about as well as the baseline — or better — in each configuration. It’s also interesting that the default model for RAGAS is GPT-3.5, whereas Phoenix recommends GPT-4. This may explain why p-faithful-v0 performs similarly to Phoenix but better than RAGAS. In future posts, we will only look at the strongest available baselines.</p><p>We can also note the unsurprising but useful result that in-distribution fine-tuning (on only a few hundred examples) <em>dramatically</em> improves the performance of p-faithful-v0. This took only a few epochs on an A100 and involved very minimal hyperparameter tuning.</p><p>Finally, we should emphasize that we were able to empirically validate that FSV2 is a hard task. Every out-of-domain detector does much worse on that dataset than either WikiEval or HaluEval. FSV2 is not, however, completely noise: there is some meaningful distribution to learn (we would expect 50% accuracy for complete noise). Keep in mind that these accuracy numbers <em>are</em> <em>directly comparable</em> across datasets when controlling for detector. Each number is just the proportion of the rows for which a given detector got the answer pair correct.</p><p>In our pursuit to reliably and cheaply detect hallucinations in LLM responses from real-world RAG applications, we find that p-faithful-v0 performs similarly to (or better than) much more expensive alternatives. Furthermore, we present analyses using our new more realistic benchmark, FSV2. We bring p-faithful-v0 performance up from 70% to 93% with a little fine-tuning on a modest training set.</p><p><strong>Key takeaway: In these experiments, the tiny p-faithful-v0 detector could reliably detect even the subtlest hallucinations we could find.</strong></p><h3>Get Early Access</h3><p>We are expanding access to our hallucination detection model to test on diverse real world application data. <strong>If you are building RAG applications for production use cases, and want to evaluate them, we’d love to work with you.</strong></p><h4>What to expect?</h4><ul><li>You will get access to the model end-point for ad hoc testing (pre-trained model is available for out-of-domain inference).</li><li>We will work closely with you and your team to help you prep your data, and set up your experiments (including adapting the classifier for your domain).</li><li>We will work with your team if your use case requires the model to run on-prem/in a private environment.</li></ul><p><a href="https://cdn.forms-content.sg-form.com/e8b9b4d4-dda0-11ee-85ff-daa2c5cfc593"><strong>Request Early Access</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=949248f0ad94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/harder-better-faster-stronger-llm-hallucination-detection-for-real-world-rag-part-i-949248f0ad94">Harder, Better, Faster, Stronger: 🤖 LLM Hallucination Detection for Real-World RAG, Part I</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI proxy middlewares are a hack]]></title>
            <link>https://medium.com/lastmile-ai/ai-proxy-middlewares-are-a-hack-e98a21d26880?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/e98a21d26880</guid>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[developer]]></category>
            <dc:creator><![CDATA[Sarmad Qadri]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 20:22:44 GMT</pubDate>
            <atom:updated>2023-12-21T18:59:43.406Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The current architecture for monitoring, model routing and prompt management is not enough.</em></p><p>Now that I have your attention with the headline, let me take a moment to explain what’s going on. We are starting to see a new concept emerge in the LLM Ops workflow — AI proxy middleware.</p><p>AI proxies are services that stand between your application, and the model inference provider, like OpenAI, Hugging Face, etc. They are responsible for consolidating some important and necessary steps in the generative AI developer workflow:</p><ul><li>Call different models (LLaMA, GPT*, Mixtral) with a single API — usually reusing the openai.chat.completions API with a custom baseUrl.</li><li>Monitor usage, latency, cost, etc.</li><li>Caching &amp; throttling of inference requests</li><li>API key management for inference providers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AVkk_ekrbRMzYWQdrIzHZg.png" /><figcaption>AI proxy middleware sits between your application and model inference providers</figcaption></figure><p>We believe this is, unfortunately, the wrong architecture for solving the right problems. Instead, we propose — and demonstrate — that the capabilities these middlewares provide can be handled more gracefully by frameworks and protocols that avoid middleware altogether.</p><h3>The problem with AI proxy middlewares</h3><p>The motivating problems that AI proxies are trying to solve are valid and important:</p><ol><li><strong>Separation of concerns: </strong>Decouple model-specific logic from application code. <br>It is important to enable applications to invoke different models with the same consistent API surface, removing the need to wrangle model-specific APIs. Developers can invoke different<em> </em>models in the same application flow while using the same consistent API (e.g. using GPT-4 to handle a complicated prompt, and LLaMA or Mixtral for a simpler one).</li><li><strong>Monitoring</strong> generative AI usage, latency, and cost for your application.</li><li><strong>Caching</strong> inference requests using semantic caching, and managing throttling.</li><li>Managing API keys for different inference providers</li></ol><p>Putting all of this in a proxy middleware introduces a new and avoidable set of problems:</p><ul><li><strong>Monolithic design</strong> — monitoring, observability and caching are already well-known concepts in the existing software development workflow, with dedicated systems for each of them. They don’t need to be subsumed by a proxy service.</li><li><strong>Security risk</strong>: Because of this unnecessary service layer between the application and the model provider, we now have an additional security risk, and need to solve for it by encrypting requests/user-specific data.</li><li><strong>Latency and lack of debuggability: </strong>The proxy causes 2 hops to the LLM provider, which can lead to performance degradations. Additionally, if something goes wrong, there’s limited visibility for debugging.</li><li><strong>Local vs. remote</strong>: This service layer doesn’t work for local models, which is something that is becoming increasingly important as models get smaller and more efficient.</li></ul><p>Most importantly, many of these proxy middlewares are closed managed services provided by third party providers. This becomes a critical external dependency with no failover strategy.</p><h3>A better alternative — Frameworks over Proxies</h3><p>We believe the AI proxy layer can be replaced by an open source AI framework and storage format that can still provide a uniform API, and connect to the relevant services to handle monitoring, caching and key management separately.</p><p>To that effect, we have created <a href="https://github.com/lastmile-ai/aiconfig"><strong>AIConfig</strong></a> — a config-driven framework that manages prompts, models and inference settings as JSON-serializable configs.</p><p>These configs are the generative AI artifact for an application, and can be version controlled, evaluated, monitored, and edited in a notebook-like playground. In other words, they embed directly into the developer workflow <em>as it is today</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oD3H8IrZMOw9KkIamF3i6w.png" /></figure><p>There are other AI frameworks out there, but the 2 key observations of AIConfig are:</p><ol><li>prompts, models and inference settings should be saved as config, not code</li><li>a common storage format that is model-agnostic and multi-modal allows switching between different models straightforwardly</li></ol><h4>Comparison with proxy middleware</h4><p>Let’s revisit the things that the proxy middleware enables and do the same thing without a man-in-the-middle. More specifically, let’s do it in a way that doesn’t require a monolithic architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AC1XxjmE2jIIYRwclZ4VrQ.png" /><figcaption>Breaking down a monolithic service into its constituent parts allows you to use existing service providers for inference, monitoring, caching, KMS, etc.</figcaption></figure><p>✅ <strong>Separation of concerns </strong>— Decoupling model-specific logic from application code.</p><p>By design, AIConfig allows you to store and iterate on prompts separately from application code. It provides a uniform API surface across any model and for any modality.</p><p>For example, this aiconfig creates a trip planner app using <a href="https://blog.google/technology/ai/google-gemini-ai/">Gemini</a> and GPT-4:</p><pre>{<br>  &quot;name&quot;: &quot;NYC Trip Planner&quot;,<br>  &quot;description&quot;: &quot;Intrepid explorer with ChatGPT and AIConfig&quot;,<br>  &quot;schema_version&quot;: &quot;latest&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;models&quot;: {<br>      &quot;gemini-pro&quot;: {<br>        &quot;model&quot;: &quot;gemini-pro&quot;<br>      },<br>      &quot;gpt-4&quot;: {<br>        &quot;model&quot;: &quot;gpt-4&quot;,<br>        &quot;max_tokens&quot;: 3000,<br>        &quot;system_prompt&quot;: &quot;You are an expert travel coordinator with exquisite taste.&quot;<br>      }<br>    },<br>    &quot;default_model&quot;: &quot;gemini-pro&quot;<br>  },<br>  &quot;prompts&quot;: [<br>    {<br>      &quot;name&quot;: &quot;get_activities&quot;,<br>      &quot;input&quot;: &quot;Tell me 10 fun attractions to do in NYC.&quot;,<br>      &quot;metadata&quot;: {<br>        &quot;model&quot;: &quot;gemini-pro&quot;<br>      }<br>    },<br>    {<br>      &quot;name&quot;: &quot;gen_itinerary&quot;,<br>      &quot;input&quot;: &quot;Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.&quot;,<br>      &quot;metadata&quot;: {<br>        &quot;model&quot;: &quot;gpt-4&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;order_by&quot;: &quot;geographic location&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>You can invoke either model with the same API:</p><pre>import asyncio<br>from aiconfig import AIConfigRuntime, InferenceOptions<br><br>async def main():<br>  # Load the aiconfig<br>  config = AIConfigRuntime.load(&#39;travel.aiconfig.json&#39;)<br><br>  # Run a Google Gemini prompt (with streaming)<br>  inference_options = InferenceOptions(stream=True)<br>  await config.run(&quot;get_activities&quot;, options=inference_options)<br><br>  # Run a GPT-4 prompt (same API!)<br>  await config.run(&quot;gen_itinerary&quot;, options=inference_options)<br><br>asyncio.run(main())</pre><p>✅ <strong>Monitoring</strong></p><p>The framework provides callback handlers to register usage tracking.</p><p>The main philosophy is that monitoring for generative AI isn’t particularly different from monitoring other services. Monitoring for generative AI should simply hook into wherever you do monitoring for the rest of your application (datadog, cloudwatch, prometheus, etc.)</p><pre>from aiconfig import CallbackManager, CallbackEvent<br>import pprint<br><br>async def custom_callback(event: CallbackEvent) -&gt; None:<br>  &quot;&quot;&quot;<br>    This is a custom callback that prints the event to stdout.<br><br>    Args:<br>        event (CallbackEvent): The event that triggered the callback.<br>  &quot;&quot;&quot;<br>  print(f&quot;Event triggered: {event.name}&quot;)<br>  pprint.pprint(event, width = 150)<br><br>callback_manager = CallbackManager([custom_callback])<br>config.set_callback_manager(callback_manager)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zPk4kDopDPlCg1lRWHlwiA.png" /><figcaption>Existing monitoring services can be hooked into frameworks easily</figcaption></figure><p>✅ <strong>Caching</strong></p><p>There are excellent solutions out there already, like <a href="https://github.com/zilliztech/GPTCache">GPTCache</a>, for semantic caching. Having a framework instead of a proxy allows for a straightforward integration with the best tools for the best job.</p><p>✅ <strong>API Key Management</strong></p><p>We don’t think API key management is a new problem — there are already good KMS services for this, and the same ones can be reused for managing inference endpoint keys.</p><h3>Framework Extensions</h3><p>In addition to the above, there are other parts of the generative AI workflow that are critical for building production applications. Frameworks enable using existing tools to reason over them:</p><h4>Evaluation</h4><p>Having a dedicated config artifact allows developers to define evals for it, and trigger those eval runs as part of CI/CD every time the config changes.</p><p>You can read more about generative AI evaluation <a href="https://github.com/lastmile-ai/aiconfig/blob/main/python/src/aiconfig/eval/README.md">here</a>.</p><h4>Local Experimentation</h4><p>A framework can collapse experimentation and productionization of generative AI into a single workflow. For example, an aiconfig can be opened in a notebook-like playground for visual editing and rapid prototyping, and also used in application code.</p><h4>Governance and version control</h4><p>As a version controlled artifact, aiconfig can be used for reproducibility and provenance of the generative AI bits of your application.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FX_Z-M2ZcpjA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DX_Z-M2ZcpjA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FX_Z-M2ZcpjA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7d44f1ef6b7c2f5edcac2dde62050aa/href">https://medium.com/media/b7d44f1ef6b7c2f5edcac2dde62050aa/href</a></iframe><blockquote>Learn more here: <a href="https://github.com/lastmile-ai/aiconfig">https://github.com/lastmile-ai/aiconfig</a></blockquote><h3>So why are there so many proxy middlewares?</h3><p>As stated above, the proxy middlewares are an attempt to solve legitimate problems with the current generative AI workflow. They are also a convenient way to inject into the critical path for generative AI development. In a competitive market, it is a way to add a sticky dependency.</p><p>However, if we were just evaluating the design from the perspective of the developer, then the capabilities these proxies provide would be broken down into extensible frameworks that provide hooks to external services.</p><h3>Protocols over Frameworks</h3><blockquote>The underlying point in all of this is that we need a standard interaction model that defines how applications work <em>across </em>model providers, how we evaluate generative AI components, how we monitor, etc. (And we haven’t even touched agentic interactions in this blog).</blockquote><p>Standardizing this interaction model as a protocol will eventually make the overall developer workflow more streamlined, and lead to a more open developer ecosystem for generative AI. Our early thinking on this is inspired by SMTP for email, and <a href="https://microsoft.github.io/language-server-protocol/">LSP (Language Server Protocol)</a> for IDEs.</p><p>There is a need for a Generative AI Protocol that defines:</p><ul><li>Standardized storage format for prompts, models and inference settings</li><li>Standardized API for running inference, including routing</li><li>Evaluating generative AI components</li><li>Monitoring and callbacks</li><li>Caching</li><li>Experimentation and UX</li></ul><p><a href="https://github.com/lastmile-ai/aiconfig">AIConfig</a> is a step in this direction, and we’d love to collaborate with the community to take it forward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e98a21d26880" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/ai-proxy-middlewares-are-a-hack-e98a21d26880">AI proxy middlewares are a hack</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-sourcing AIConfig]]></title>
            <link>https://medium.com/lastmile-ai/open-sourcing-aiconfig-6fbab53a59f7?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/6fbab53a59f7</guid>
            <category><![CDATA[sdk]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sarmad Qadri]]></dc:creator>
            <pubDate>Wed, 29 Nov 2023 18:33:05 GMT</pubDate>
            <atom:updated>2023-11-29T21:44:46.408Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A config-driven, source-control-friendly AI application development framework</em></p><p><a href="https://github.com/lastmile-ai/aiconfig">AIConfig</a> is a framework that makes it easy to build generative AI applications for production. It manages generative AI prompts, models and model parameters as JSON-serializable configs that can be version controlled, evaluated, monitored and opened in a notebook playground for rapid prototyping.</p><p>It allows you to store and iterate on generative AI behavior <em>separately from your application code, </em>offering a streamlined AI development workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6yJYiX_5miJcCTF5kGRIg.png" /><figcaption>AIConfig simplifies application code and enables new ways of building with generative AI</figcaption></figure><h3>Wait, but why?</h3><p>Today, prompts, model parameters and model-specific logic are all jumbled in with application code. This tight coupling causes several problems:</p><ul><li><em>Increased complexity</em> — having to wrangle different model providers in core application code.</li><li><em>Difficult to experiment and iterate</em> on prompts or switch from one model to another.</li><li><em>Hard to version and evaluate</em> prompt/model performance.</li></ul><p>AIConfig helps unwind this complexity by capturing the generative AI bits of your application as an AI artifact, which you can:</p><ul><li><strong>Version control</strong> for confidence in production</li><li><strong>Evaluate and monitor</strong></li><li><strong>Experiment and productionize</strong> in the same workflow — open the AIConfig in a playground, switch models, try different parameters, and when done, save the result back as an aiconfig</li><li><strong>Simplify application code </strong>— simply call config.run to invoke a prompt, no matter which model or modality.</li></ul><blockquote>AIConfig is open source, and we welcome contributions: <br><a href="https://github.com/lastmile-ai/aiconfig">https://github.com/lastmile-ai/aiconfig</a></blockquote><h3>Show, don’t tell</h3><p>Let’s quickly demonstrate what AIConfig looks like, and how to use it with your application. We’ll return to this example later, and you can find the code for it here: <a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Getting-Started">https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Getting-Started</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FX_Z-M2ZcpjA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DX_Z-M2ZcpjA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FX_Z-M2ZcpjA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7d44f1ef6b7c2f5edcac2dde62050aa/href">https://medium.com/media/b7d44f1ef6b7c2f5edcac2dde62050aa/href</a></iframe><p>Let’s use ChatGPT to plan a trip to New York.</p><pre>{<br>  &quot;name&quot;: &quot;NYC Trip Planner&quot;,<br>  &quot;description&quot;: &quot;Intrepid explorer with ChatGPT and AIConfig&quot;,<br>  &quot;schema_version&quot;: &quot;latest&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;models&quot;: {<br>      &quot;gpt-3.5-turbo&quot;: {<br>        &quot;model&quot;: &quot;gpt-3.5-turbo&quot;,<br>        &quot;top_p&quot;: 1,<br>        &quot;temperature&quot;: 1<br>      },<br>      &quot;gpt-4&quot;: {<br>        &quot;model&quot;: &quot;gpt-4&quot;,<br>        &quot;max_tokens&quot;: 3000,<br>        &quot;system_prompt&quot;: &quot;You are an expert travel coordinator with exquisite taste.&quot;<br>      }<br>    },<br>    &quot;default_model&quot;: &quot;gpt-3.5-turbo&quot;<br>  },<br>  &quot;prompts&quot;: [<br>    {<br>      &quot;name&quot;: &quot;get_activities&quot;,<br>      &quot;input&quot;: &quot;Tell me 10 fun attractions to do in NYC.&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;gen_itinerary&quot;,<br>      &quot;input&quot;: &quot;Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.&quot;,<br>      &quot;metadata&quot;: {<br>        &quot;model&quot;: &quot;gpt-4&quot;,<br>        &quot;parameters&quot;: {<br>          &quot;order_by&quot;: &quot;geographic location&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>This AIConfig <a href="https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/Getting-Started/travel.aiconfig.json">travel.aiconfig.json</a> contains a prompt chain to get a list of NYC travel activities from an LLM (GPT-3.5), and then generate an itinerary based on user preferences (using GPT-4).</p><p>It’s ok if you don’t understand every part of the AIConfig. You can use a notebook editor, called an AI Workbook, to create it visually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwhfIkvByi1Qr_T68_TR8A.png" /><figcaption>AI Workbook — a notebook playground for AIConfig</figcaption></figure><p>Try it yourself by cloning this workbook:</p><p><a href="https://lastmileai.dev/workbooks/clooqs3p200kkpe53u6n2rhr9">Planning a trip to New Yo... | LastMile AI</a></p><p>Finally, let’s use this AIConfig in our application. We only show Python here, but AIConfig ships with both Node.js and Python SDKs.</p><pre>pip3 install python-aiconfig</pre><pre>import asyncio<br>from aiconfig import AIConfigRuntime, InferenceOptions<br><br>async def main():<br>  # Load the aiconfig<br>  config = AIConfigRuntime.load(&#39;travel.aiconfig.json&#39;)<br><br>  # Run a single prompt (with streaming)<br>  inference_options = InferenceOptions(stream=True)<br>  await config.run(&quot;get_activities&quot;, options=inference_options)<br><br>asyncio.run(main())</pre><p>That’s it! Just reference the prompt you want to run, and use the same config.run API no matter what model provider is being used underneath.</p><p>We’ll return to this example in a moment to demonstrate how to pass data into a config, and chaining prompts and models together.</p><h3>Who should use AIConfig</h3><p>Unlike traditional predictive ML development undertaken largely by ML researchers, generative AI applications are being developed collaboratively by software engineers and product teams.</p><p>Separating prompt management from application development leads to a few powerful consequences:</p><ol><li><strong>Separation of concerns</strong>: You can iterate on prompts and models separately from application code — and different people could be responsible for them, making the overall development more collaborative.</li><li><strong>Notebook editor for prompts</strong>: Having prompts and models in one place allows a notebook-like editor environment to iterate on the aiconfig. This greatly increases the velocity of prototyping and iteration.</li><li><strong>Governance</strong>: As a source-controlled artifact, aiconfig can be used for reproducibility and provenance of the generative AI bits of your application.</li></ol><blockquote>We’ve specifically designed AIConfig for <strong>software developers</strong> and teams building production applications of generative AI.</blockquote><h3>Under the covers</h3><p>AIConfig is <strong>multi-modal</strong> and <strong>model-agnostic</strong>. This enables powerful interop between different models and modalities, including chaining them together (see <a href="https://aiconfig.lastmileai.dev/docs/overview/define-prompt-chain">prompt chaining</a>). For example, a LLaMA2 (text-to-text) prompt can be connected to a DALL-E prompt (text-to-image) to build complex AI applications, all backed by the same aiconfig serialization format.</p><p>AIConfig has 3 core pillars:</p><ol><li><a href="https://aiconfig.lastmileai.dev/docs/overview/ai-config-format">aiconfig file format</a>: a standardized JSON format to store generative AI model settings, prompt inputs and outputs, and flexible multi-purpose metadata.</li><li><a href="https://github.com/lastmile-ai/aiconfig#aiconfig-sdk">AIConfig SDK</a>: Python and Node SDKs to use aiconfig in your application code, and extension points to customize behavior.</li><li><a href="https://lastmileai.dev/workbooks/clm7b9yez00mdqw70majklrmx">AI Workbook editor</a>: A notebook-like playground to edit aiconfig files visually, run prompts, tweak models and model settings, and chain things together.</li></ol><p>Extensibility is a core design principle for the framework, so anyone — including you — can <a href="https://aiconfig.lastmileai.dev/docs/extensibility">create </a><a href="https://aiconfig.lastmileai.dev/docs/extensibility">ModelParser extensions</a> which customize which models to run, how to serialize/deserialize model-specific data into/from an aiconfig, and connect to your custom model deployments.</p><h3>Simplifying your application code with AIConfig</h3><p>Let’s continue from the above example, but this time, instead of just getting a list of activities, let’s ask GPT-4 to create an itinerary for us, ordered by some criteria.</p><p>In our travel.aiconfig.json, we have a gen_itinerary prompt</p><pre>{<br>  &quot;name&quot;: &quot;gen_itinerary&quot;,<br>  &quot;input&quot;: &quot;Generate an itinerary ordered by {{order_by}} for these activities: {{get_activities.output}}.&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;model&quot;: &quot;gpt-4&quot;,<br>      &quot;parameters&quot;: {<br>        &quot;order_by&quot;: &quot;geographic location&quot;<br>      }<br>  }<br>}</pre><p>Observe the following:</p><ol><li>The prompt depends on the output of the get_activities prompt.</li><li>It also depends on an order_by parameter</li><li>It uses GPT-4, whereas the get_activities prompt it depends on uses GPT-3.5.</li></ol><p>Effectively, this is a prompt chain between gen_itinerary and get_activities prompts, <em>as well as</em> as a model chain between gpt-3.5 and gpt-4.</p><p>Let’s run this with AIConfig:</p><pre>import asyncio<br>from aiconfig import AIConfigRuntime, InferenceOptions<br><br>async def main():<br>  # Load the aiconfig<br>  config = AIConfigRuntime.load(&#39;travel.aiconfig.json&#39;)<br><br>  # Run a single prompt (with streaming)<br>  inference_options = InferenceOptions(stream=True)<br>  await config.run(<br>    &quot;gen_itinerary&quot;,<br>    params={&quot;order_by&quot;: &quot;duration&quot;},<br>    options=inference_options,<br>    run_with_dependencies=True)<br><br>asyncio.run(main())</pre><p>Notice how simple the syntax is to perform a fairly complex task — running 2 different prompts across 2 different models and chaining one’s output as part of the input of another — all with a single config.run call! We even passed data into the prompt with the params syntax.</p><p>Finally, you can use AIConfig to monitor your generative AI usage in production. We have defined callback handlers to allow you to connect to your monitoring &amp; observability platform:</p><pre>from aiconfig import CallbackManager, CallbackEvent<br>import pprint<br><br>async def custom_callback(event: CallbackEvent) -&gt; None:<br>  &quot;&quot;&quot;<br>    This is a custom callback that prints the event to stdout.<br><br>    Args:<br>        event (CallbackEvent): The event that triggered the callback.<br>  &quot;&quot;&quot;<br>  print(f&quot;Event triggered: {event.name}&quot;)<br>  pprint.pprint(event, width = 150)<br><br>callback_manager = CallbackManager([custom_callback])<br>config.set_callback_manager(callback_manager)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rErLgZplXoqvWNxB8jvlUg.png" /><figcaption>Sample callback events generated when running the get_activities prompt</figcaption></figure><h3>Why we decided to open source</h3><p>Over the last several months, we’ve been working with several enterprises who have voiced the same issue — they would love to build with generative AI, but they need confidence in production and a developer workflow that is integrated into their existing software engineering workflow.</p><blockquote>We believe that we need a more methodical way of building generative AI applications, that inspires confidence in production, makes it easy to iterate, evaluate performance, and collaborate as a team.</blockquote><p>For any such standard to succeed, it needs to be open source, and needs the developer community’s input. This is our first open source release, and we’d love the community’s feedback on the project, input on the direction, and also contributions to help drive this forward.</p><p>Specifically, you can help with:</p><ol><li><strong>Extension Contributions</strong>: We always welcome code contributions to the library. In particular, <em>model-parsers</em>, <em>extensions</em>, and <em>cookbooks</em> are areas that can always use contributions.</li><li><strong>Cookbooks</strong>: Showcase innovative ways and use cases to use AIConfig in your code through a cookbook.</li></ol><h3>Cookbooks &amp; Guides</h3><p>There is a lot you can do with AIConfig. We have several tutorials to help you get started:</p><ul><li><a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Wizard-GPT">CLI Chatbot</a></li><li><a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/RAG-with-AIConfig">RAG with AIConfig</a></li><li><a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Basic-Prompt-Routing">Prompt routing</a></li><li><a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Function-Calling-OpenAI">OpenAI function calling</a></li><li><a href="https://github.com/lastmile-ai/aiconfig/tree/main/cookbooks/Chain-of-Verification">Chain of thought</a></li></ul><h3>What’s next for AIConfig</h3><p>The travel.aiconfig.json showed how to use OpenAI models, but AIConfig already supports LLaMA2, Google PaLM and Hugging Face text generation. We’ll continue to add additional model support.</p><p>In addition, there’s quite a lot that we are excited to build in the coming weeks.</p><ul><li><em>Evaluation interfaces</em>: allow aiconfig artifacts to be evaluated with user-defined eval functions.</li><li><em>Local editor</em> for aiconfig, so you can iterate on aiconfigs in a locally hosted playground.</li><li><em>OpenAI Assistants API</em> support</li><li><em>Multi-modal model</em> support (such as GPT4-V, DALL-E, Whisper and Hugging Face models)</li></ul><p>Get started at <a href="https://github.com/lastmile-ai/aiconfig">https://github.com/lastmile-ai/aiconfig</a>, and happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fbab53a59f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/open-sourcing-aiconfig-6fbab53a59f7">Open-sourcing AIConfig</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing LastMile AI’s $10MM Seed Round from Gradient]]></title>
            <link>https://medium.com/lastmile-ai/announcing-lastmile-ais-10mm-seed-round-from-gradient-8542222f37e4?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/8542222f37e4</guid>
            <category><![CDATA[announcements]]></category>
            <category><![CDATA[fundraising]]></category>
            <dc:creator><![CDATA[Sarmad Qadri]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 14:21:12 GMT</pubDate>
            <atom:updated>2023-09-15T18:23:38.912Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9Rft-En4pw_G26tHQxCCQ.png" /></figure><p>We are excited to announce our $10MM raise led by <a href="https://www.gradient.com/">Gradient</a>, Google’s AI-focused venture fund, with participation from Jerry Yang, Guillermo Rauch, Exceptional Capital, FirstHand Alliance, 10x Founders and others.</p><p><a href="https://techcrunch.com/2023/09/14/lastmile-ai-closes-10m-seed-round-to-operationalize-ai-models/">LastMile AI closes $10M seed round to &#39;operationalize&#39; AI models | TechCrunch</a></p><p>We are a team of technologists and friends from Meta/Facebook AI. We built many of the core developer tools that thousands of ML engineers and data scientists rely on everyday within Meta — including <a href="https://www.youtube.com/watch?v=f3UfVX4_PD4">Bento</a>, a custom-built Jupyter notebook platform; <a href="https://www.facebook.com/watch/?v=1090799804698974">Mastercook</a>, a collaborative experimentation platform for Ads ranking; and <a href="https://ai.meta.com/blog/meta-ai-ecosystem-management-metrics/">ML Hub</a>, Facebook’s model and feature management platform. We’ve seen firsthand the multiplicative role that good developer tools play in making ML teams more productive, resulting in better products that are easier to iterate.</p><p>With recent advances in foundation models and generative AI, we are witnessing the start of a profound revolution in computing that will fundamentally change our relationship with technology. This revolution is being driven by software engineers and product teams who are using AI as a new part of their software toolkit. Yet ML developer tooling is still mostly geared towards researchers and core ML practitioners.</p><p>To drive the last mile adoption of AI across every industry, we need a new class of AI developer tools built for software engineers, not just ML research scientists.</p><p>Our team has lived and breathed developer tools for the better part of a decade, and we are incredibly excited to make AI more accessible to more teams, businesses and engineers. Our vision is for LastMile to be the integrated development environment for generative AI, allowing you to experiment with, personalize and deploy foundation models for enterprise applications and workflows. It shouldn’t matter what models you use, how many steps are in your AI workflow, or what application you’re using them for — there should be a single, cohesive developer environment for working with AI, and we intend to build it.</p><h3>The next big thing</h3><p>AI has been ‘the next big thing’ for many years, but its promise has largely remained unfulfilled. Until now, leveraging AI required deep expertise and investment, from data preparation and feature engineering to training, evaluation and inference. An organization needed to build sophisticated ML research and infrastructure engineering teams just to get started. So it is no surprise that AI remained the domain of only the most sophisticated tech companies, and remained ‘the next big thing’ for everyone else.</p><p>Like many others, we believe the proliferation of advanced foundation models — especially large language models, but also audio, image and video models — is a game changer for the mass adoption of AI. LLMs and other generative AI models are general-purpose enough that any organization with a software engineering team can start incorporating AI in their applications and workflows. Chatbots really showcased the capabilities of these models, but their value extends far beyond. Already we have seen our design partners use LLMs in production for NLP, classification, intent recognition, data annotation and information retrieval.</p><h3>The AI Development Workflow</h3><p>AI application development today is scattered and disjointed across a mish-mash of tools, libraries and frameworks. We think it needs to be more structured and streamlined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jQxlGcL3AUt6D0iX" /></figure><p>Any developer loop has 4 key phases:</p><ul><li><strong>Inner Dev Loop</strong>: Rapid prototyping, experimenting and iteration</li><li><strong>Outer Dev Loop</strong>: Offline workflows and results evaluation</li><li><strong>Deployment: </strong>Getting things in production</li><li><strong>Monitoring</strong>: Observability, logging and feedback loop</li></ul><p>For AI application development, these phases map to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ITkjZLKSVOXUcaJHdVh4Q.png" /></figure><p>Simply put, some combination of these steps is required to leverage AI in your application. Today each of these steps requires stitching together a number of different tools and providers, and needs nuanced understanding of every step, which increases the barrier to entry.</p><p>Our goal with LastMile is to provide a single developer platform that encompasses the entire lifecycle of AI application development.</p><p>To that effect, we’re taking 3 opinionated design choices:</p><ol><li><em>Model-agnostic. </em><br>An AI developer platform should enable experimenting with a variety/combination of model providers. We don’t think there will be a single dominant model provider that serves every purpose for everyone.</li><li><em>End-to-end.</em><br>A cohesive developer experience should encompass the entire AI application development lifecycle. If it requires putting together different solutions for different steps, then it’s already too complicated.</li><li><em>Built for software engineers and product teams.</em><br>For mass adoption of AI, we need a developer platform for engineers and application developers, not just researchers.</li></ol><p>Our first product, AI Workbooks, is a notebook interface for experimenting with generative AI models. It simplifies the inner dev loop dramatically. <a href="https://lastmileai.dev/">Give it a try today</a>.</p><h3>Why Gradient</h3><p><a href="https://gradient.com/">Gradient</a> is Google’s AI-focused fund. From our very first conversation with <a href="https://www.linkedin.com/in/moosani/">Asif</a> and <a href="https://twitter.com/darian314">Darian</a>, we felt a mind-meld both about the transformative potential of generative AI, and about how lastmile could help turbocharge that transformation across industries. We’re incredibly excited to be part of the Gradient family.</p><h3>Collaborations and Design Partnerships</h3><p>We’re scaling up our design partnerships as we continue to build out the LastMile developer platform. If you are evaluating generative AI to solve specific business challenges, we would love to work with you.</p><p>You can read more about our <a href="https://lastmile-ai.notion.site/LastMile-AI-Design-Partner-Program-bc90f4d926974849aa6b12221592dd1c">Extra Mile Design Partner program</a>. Please reach out to us if you’d like to participate.</p><h3>We’re Expanding Our Team</h3><p>Our journey is not even 1% finished. A comprehensive AI developer platform is such an undertaking that many people think we’re being too ambitious. We take that as a point of pride, because trying to do hard and innovative things gives our work meaning. Sharing that ambition and vision with an incredible team gives us purpose.</p><p><a href="https://www.notion.so/lastmile-ai/We-re-hiring-lastmile-AI-6f74facb4d8d4a7facf6ed9b99b15f6c?pvs=4">We’re hiring</a>, so if our vision appeals to you, please consider applying.</p><p>Let’s get started!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8542222f37e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/announcing-lastmile-ais-10mm-seed-round-from-gradient-8542222f37e4">Announcing LastMile AI’s $10MM Seed Round from Gradient</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Llama 2 Locally in 7 Lines! (Apple Silicon Mac)]]></title>
            <link>https://medium.com/lastmile-ai/run-llama-2-locally-in-7-lines-apple-silicon-mac-c3f46143f327?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/c3f46143f327</guid>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Suyog Sonwalkar]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 15:51:31 GMT</pubDate>
            <atom:updated>2023-07-20T15:50:45.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*81Zzsz8opkq8eBUbpRHlng.png" /><figcaption>Super fast way of getting Llama 2 running locally on your Mac</figcaption></figure><p>TLDR:</p><pre>xcode-select --install # Make sure git &amp; clang are installed<br>git clone https://github.com/ggerganov/llama.cpp.git<br>cd llama.cpp<br>curl -L https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q4_K_M.bin --output ./models/llama-2-7b-chat.ggmlv3.q4_K_M.bin <br>LLAMA_METAL=1 make<br>./main -m ./models/llama-2-7b-chat.ggmlv3.q4_K_M.bin -n 1024 -ngl 1 -p &quot;Give me a list of things to do in NYC&quot;</pre><p>NOTE: The 7B model weights are about 4GB in size, please ensure you have enough space on your machine.</p><h3>What is this doing?</h3><p>This is using the amazing <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> project by Georgi Gerganov to run Llama 2. It downloads a 4-bit optimized set of weights for Llama 7B Chat by TheBloke via their huggingface repo <a href="https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML">here</a>, puts it into the models directory in llama.cpp, then builds llama.cpp with Apple’s Metal optimizations.</p><p>This allows you to run Llama 2 locally with minimal work. The 7B weights should work on machines with 8GB of RAM (but better if you have 16GB). Larger models like 13B or 70B will require significantly more RAM.</p><p>Note that Llama 2 non-chat weights are also available here: <a href="https://huggingface.co/TheBloke/Llama-2-7B-GGML">https://huggingface.co/TheBloke/Llama-2-7B-GGML</a>, however if you want a simpler chat interface, the chat weights are preferred.</p><h3>Performance</h3><p>On an M2 Max MacBook Pro, I was able to get 35–40 tokens per second using the LLAMA_METAL build flag. Your performance may vary depending on Apple Silicon chip.</p><h3>Thanks and acknowledgements</h3><p>It’s amazing that we’re able to get state of the art large language models running locally in such a short amount of time after their release. This wouldn’t be possible without the Llama 2 weights being open sourced by <a href="https://www.meta.com/">Meta</a>, the <a href="https://github.com/ggerganov/llama.cpp">llama.cpp project</a> and TheBloke’s huggingface optimized <a href="https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML">Llama 2 model weights</a>.</p><h3>LastMile AI</h3><p>If you want to learn more about AI tools and large language models like Llama, checkout <a href="https://lastmileai.dev/"><em>LastMile AI</em></a>. You can reach us on:</p><ul><li><a href="https://twitter.com/lastmileai_dev">Twitter</a></li><li><a href="http://discord.gg/xBhNKTetGx">Discord</a></li><li><a href="/service/mailto:support@lastmileai.dev?subject=Llama%20on%20Mac%20locally">Email</a></li></ul><p>We would also appreciate your feedback on our initial product offering, available at<strong> </strong><a href="https://lastmileai.dev/"><strong>lastmileai.dev</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3f46143f327" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/run-llama-2-locally-in-7-lines-apple-silicon-mac-c3f46143f327">Run Llama 2 Locally in 7 Lines! (Apple Silicon Mac)</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating the Performance of Retrieval-Augmented LLM Systems]]></title>
            <link>https://medium.com/lastmile-ai/evaluating-the-performance-of-retrieval-augmented-llm-systems-d95122feb0dd?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/d95122feb0dd</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Felix Fang]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 18:25:51 GMT</pubDate>
            <atom:updated>2023-07-04T04:36:27.024Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (LLMs) that enable AI chatbots like ChatGPT continue to gain popularity as more use cases arise for generative AI. In particular, <a href="https://arxiv.org/pdf/2005.11401.pdf">Retrieval-Augmented Generation</a> (RAG) systems proposed in 2021, and popularized by tools such as <a href="https://python.langchain.com/docs/get_started/introduction.html">langchain</a>, empower many practical applications, such as question-answering with a local knowledge base.</p><p>Evaluating performance and quality of these systems is crucial to assess their capabilities and limitations. Understanding how reliable these systems are is on the top of mind for researchers, developers and consumers alike.</p><p>In this blog post, we explore the various ways to evaluate a Retrieval-Augmented LLM system.</p><h3>Retrieval-Augmented Large Language Models</h3><p>This is the typical set of steps to perform a question answering task based on a local knowledge base:</p><ol><li><strong>Build a vectorstore</strong>: We generate embedding vectors for each documents from the local knowledge base, and store the documents in a vector database indexed by the corresponding embedding vectors;</li><li><strong>Search for context</strong>: We use the same method as previous to embed the input question and find the most relevant documents from the vectorstore;</li><li><strong>Feed the LLM</strong>: We combine the input question with the relevant documents as context and feed them to the LLM to get an answer pertaining to your local knowledge base.</li></ol><p>Voila! This QA system architecture works for almost any local knowledge base, ranging from personal study notes, internal documents, company financial statements, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*wgLlYNRu49Zy2n55" /><figcaption>Diagram of a Typical RAG+LLM System (Image from <a href="https://blog.langchain.dev/retrieval/">https://blog.langchain.dev/retrieval/</a>)</figcaption></figure><p>The aforementioned QA application, empowered by a Retrieval-Augmented LLM system, consists of two components:</p><ol><li><strong>Embedding-based context retrieval</strong> given a question/query, and;</li><li><strong>LLM</strong> that generates a natural language response with the question augmented with relevant context.</li></ol><p>Let’s take a look at how to evaluate each of these components in the rest of the blog post. We start with a quick guide on the concept of embeddings, but if you are familiar with embeddings, feel free to skip to the following section.</p><h3>Embedding 101</h3><p>In the context of Natural Language Processing (NLP), <em>embeddings</em> are numerical representations of words in vector form, enabling the model to interpret their meaning. These vectors consist of multiple dimensions where each dimension represents different aspects of the word. The number of dimensions is predetermined and can differ depending on the embedding model used.</p><p>You can see below how words are translated to a vector where each number represents the score for a particular dimension (ex. living being, feline).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*nsA7x-pnln6WhGwb" /></figure><p>The vector representation is useful because of the concept of distance between vectors which can help determine closeness or similarity. While it’s hard to visualize a vector space with 7 dimensions from the example above, you can calculate the distance between these vectors using various distance measures like Euclidean and cosine distances. The smaller the distance between embeddings, the closer the corresponding words likely are in meaning.</p><p>One of the challenges with vector representation is having too many dimensions. When there are too many dimensions, computational complexity increases significantly. In addition, high dimensionality can also result in problems of overfitting where a model becomes too specialized to the training data and performs poorly on unseen data.</p><p><em>Dimension Reduction</em> is a process to help reduce the number of dimensions in embeddings to overcome issues with high dimensionality. Another benefit of dimensionality reduction is the ability to visualize your embeddings. For instance, see the process below converting a 7-dimension embedding to a 2-dimension embedding and how much easier it is visualize the distances between the embeddings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ewrv-raDbEF5HFbh" /></figure><h3>1/ Evaluation of Embedding-based Context Retrieval</h3><p>Ideally, semantically similar entities should be closer to one another in the embedding space. One of the issues with embeddings as mentioned above is that the vector representation often has hundreds of dimensions, making it hard to visually grasp if semantically similar entities are close to each other when represented as embeddings.</p><p><em>Analyzing the Embedding Space</em></p><p><strong>Dimension reduction</strong> is one process of evaluating the quality of the embedding model by reducing the dimensions of the embeddings, making it easier to visualize and analyze. Dimension reduction techniques such as <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">PCA</a> (linear), <a href="https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding">t-SNE</a> (non-linear), <a href="https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Uniform_manifold_approximation_and_projection">UMAP</a> (similar to t-SNE, better at capturing global structure) help reduce n-dimensional embeddings to 2d or 3d embeddings while preserving certain properties. The lowered dimensionality of embeddings makes it easy for visual exploration, clustering, and analysis of proximity and separation patterns which all help with understanding the quality of the embedding model.</p><p><strong>Pairwise similarity distribution</strong> is another qualitative analysis tool for evaluating the embedding model. Pairwise similarity measures the degree of similarity or relatedness between pairs of embeddings. A good embedding model should capture semantic relationships, ensuring that similar entities have higher similarity scores. By analyzing the pairwise similarity distribution, we can assess whether the embeddings exhibit the desired semantic proximity. A well-performing embedding model will exhibit a higher density of similar embeddings and a lower density of dissimilar embeddings.</p><p><em>Evaluating Embedding Retrieval</em></p><p><strong>Precision and Recall </strong>are popular evaluation metrics for information retrieval like search. Assuming you have ground truth data, precision and recall are excellent at understanding how well the retrieval process is working. Read more about precision and recall in this <a href="https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54">blog post</a>.</p><ul><li>Precision measures the accuracy of the retrieved embeddings, specifically the proportion of relevant items among the retrieved embeddings. Precision@k is the technique used for embedding retrieval where k represents the number of retrieved items.<br>- <strong>Precision@k</strong> = (# of retrieved items @k that are relevant) / (# of retrieved items @k)</li><li>Recall measures the completeness of retrieved results, essentially the proportion of relevant items that are successfully retrieved from the entire set of relevant items. Recall@k is the technique used for embedding retrieval where k represents the number of retrieved items.<br>- <strong>Recall@k</strong> = (# of retrieved items @k that are relevant) / (total # of relevant items)</li></ul><p><strong>Recommendation:</strong></p><ol><li>Use techniques like pairwise similarity distribution and dimension reduction to evaluate the quality of the embedding model being used.</li><li>If you have ground truth data, calculate precision and recall of the embedding retrieval to get a quantifiable score for accuracy.</li></ol><h3>2/ Evaluation of Large Language Models</h3><p>To evaluate the performance of the output, we need to have an idea of the expected output. With the ground truth output (aka reference) and the actual output (aka candidate) from the LLM, we can assess performance through an approach that follows <em>scoring_function(reference, candidate)</em>. The scoring function can be based on:</p><ol><li><em>Exact match</em>. The candidate has to equal the reference. This is helpful for question answering tasks.</li><li><em>Fuzzy match</em>. The candidate needs to be semantically similar to the reference but not necessarily exact. This is applicable for summarization tasks.</li></ol><p><em>Standard Evaluation Measures</em></p><p>Implementing fuzzy match scoring functions for LLM outputs is ambiguous and challenging. There is a <a href="https://towardsdatascience.com/evaluation-metrics-assessing-the-quality-of-nlg-outputs-39749a115ff3">blog post</a> as well as a comprehensive <a href="https://arxiv.org/pdf/2008.12009.pdf">survey</a> on Natural Language Generation (NLG) evaluation metrics, of which here are the commonly used ones:</p><ol><li><strong>BLEU (n-gram based, used for translation):</strong> Measures the precision of the candidate translation by counting the number of matching n-grams between reference translation and candidate translation and penalizes for excessive generation.</li><li><strong>ROUGE (n-gram based, used for summarization): </strong>Measures the overlap and completeness (recall) between the n-gram sequences of the reference summary and the candidate summary.</li><li><strong>BERTScore (embedding-based): </strong>Assess the similarity between candidate text and reference text through the average cosine similarity scores given the BERT embeddings (<a href="https://arxiv.org/pdf/1904.09675.pdf">paper</a>, <a href="https://towardsdatascience.com/bertscore-evaluating-text-generation-with-bert-beb7b3431300">blog post</a>, <a href="https://github.com/Tiiiger/bert_score">GitHub link</a>, <a href="https://huggingface.co/spaces/evaluate-metric/bertscore">HuggingFace demo</a>).</li><li><strong>BLEURT (learned using BERT):</strong> Predicts human ratings of text quality based on a set of reference and candidate text pairs. BLEURT leverages BERT’s contextual representations to compute similarity scores and provides an evaluation metric that aligns with human judgments of text quality (<a href="https://arxiv.org/pdf/2004.04696.pdf">paper</a>, <a href="https://ai.googleblog.com/2020/05/evaluating-natural-language-generation.html">blog post</a>, <a href="https://github.com/google-research/bleurt">GitHub link</a>).</li></ol><p>With regards to <strong>BLEURT</strong> in particular, it is pretrained on metrics such as <strong>BLEU</strong>, <strong>ROUGE</strong>, and <strong>BERTScore</strong> using regression losses, subsequently fine-tuned on human ratings. Thus, <strong>BLEURT</strong> potentially also captures all the other three metrics. Note that BLEURT scores are not calibrated (see <a href="https://github.com/google-research/bleurt/issues/1#issuecomment-627615821">BLEURT score distribution</a> for more details).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*b6eAlJpHL36OZg5o" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*kDJVQSLXaVr6fR_d" /></figure><p><em>Other Evaluation Measures</em></p><ul><li><strong>GPT-4. </strong>Ideally, we would have humans interpret if the output quality is good. Human raters, however, are extremely resource-intensive and not practical at scale. GPT-4 has been used as a fairly good proxy to human-raters. You may want to consider prompt engineering techniques such as <a href="https://www.promptingguide.ai/techniques/fewshot">few shot prompting</a>, <a href="https://www.promptingguide.ai/techniques/cot">chain-of-thought</a>, and <a href="https://www.promptingguide.ai/techniques/consistency">self-consistency</a> to generate more reliable evaluation results from GPT-4.</li><li><strong>Reward Models from RLHF. </strong>Reinforcement learning from human feedback (RLHF) is a technique that learns from a “reward model” that is trained based on human feedback. We can use the reward model, e.g. <a href="https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2">reward-model-deberta-v3-large-v2</a>, to either directly score the output from an LLM or fine-tune the reward model for your specific applications before scoring.</li><li><strong>Word Perplexity. </strong>If you have access to the probabilities of each word output (softmax output), e.g. local LLMs, then you can also compute word perplexity (see <a href="https://towardsdatascience.com/perplexity-in-language-models-87a196019a94">blog post</a> for explanation).</li></ul><p><strong>Recommendation:</strong></p><ol><li>Compute <strong>BLEURT</strong> as a standard evaluation measure for the LLM being used.</li><li>Use a top performing LLM like <strong>GPT-4</strong> to evaluate/score the quality of the outputs (so meta!)</li></ol><h3>Where do we see these metrics used?</h3><p>Here are some references of where we see the aforementioned evaluation metrics for LLMs are used. You may want to look into them to learn more about the details of how to apply these evaluation metrics to your specific LLM applications.</p><p><em>Academic Research Papers and Technical Reports</em></p><ul><li><a href="https://arxiv.org/pdf/2305.10403.pdf">PaLM 2 Technical Report</a> (ROUGE-2, BLEURT)</li><li><a href="https://arxiv.org/pdf/2305.14314.pdf">QLoRA: Efficient Finetuning of Quantized LLMs</a> (GPT-4)</li><li><a href="https://arxiv.org/pdf/2305.18098.pdf">BigTrans: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages</a> (GPT-4)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*u1k10aEZ9fa1v0Sc" /><figcaption>Table 7 from the QLoRA paper</figcaption></figure><p><em>Evaluation Harness (Tools to Facilitate Evaluation of NLP models)</em></p><ul><li><a href="https://github.com/google/BIG-bench/blob/main/docs/doc.md#available-metrics">BIG-bench</a> (BLEU, ROUGE, BLEURT)</li><li><a href="https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md">EleutherAI/lm-evaluation-harness</a> (BLEU, ROUGE, BLEURT; contains a subset of BIG-bench tasks). Note that <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard">Open LLM Leaderboard</a> adopts a subset of this harness</li><li><a href="https://huggingface.co/evaluate-metric">HuggingFace Evaluate</a> (BLEU, ROUGE, BERTScore, BLEURT)</li><li><a href="https://github.com/lm-sys/FastChat/tree/main/fastchat/eval">fastchat/eval</a> (GPT-4; see <a href="https://lmsys.org/blog/2023-03-30-vicuna/">blog post</a> from Vicuna)</li><li><a href="https://github.com/declare-lab/instruct-eval">InstructEval</a> (ChatGPT)<br>- Paper: <a href="https://arxiv.org/pdf/2306.04757.pdf">INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models</a><br>- “… We adopt an automatic approach by leveraging ChatGPT to judge the quality of the generated answers.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*ngb92PnYYLqUE5r-" /><figcaption>Example of GPT-4 Evaluation on Alpaca-13B vs Vicuna-13B</figcaption></figure><h3>Summary: Recommendation for Evaluation Metrics</h3><p><em>Embedding-based context retrieval</em>: we recommend <strong>visualization with dimension reduction</strong> and <strong>pairwise similarity distribution</strong> for qualitative analysis of the embeddings, and <strong>precision@k</strong> and <strong>recall@k</strong> for quantitative evaluation of the retrieval system.</p><p><em>Large Language Models</em>: when ground truth data is available, we recommend <strong>BLEURT</strong> as the primary metric across all LLMs, and <strong>BLEU</strong> and <strong>ROUGE</strong> scores as supplementary metrics. For any applications where there are human ratings available, you may want to consider fine-tune <strong>BLEURT</strong> for those applications.</p><p>For cases where ground truth is not available, we recommend using <strong>GPT-4</strong> as a proxy to an expert human rater, customized with prompt engineering techniques. Leveraging a <strong>reward model</strong> intended for RLHF to compute scores may be worth investigating.</p><h3>LastMile AI</h3><p>We would love to hear how you’re thinking about evaluation metrics. You can reach us on:</p><ul><li><a href="https://twitter.com/lastmileai_dev">Twitter</a></li><li><a href="http://discord.gg/xBhNKTetGx">Discord</a></li><li><a href="http://support@lastmileai.dev/">Email</a></li></ul><p>We are building a generative AI workshop at <a href="http://lastmileai.dev/">lastmileai.dev</a> to allow experimenting with many different types of foundation models, including OpenAI’s ChatGPT, Google’s PaLM2 and others. Evaluating which is good for your use cases is important to us, and to you. Visit us at lastmileai.dev learn more! Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d95122feb0dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/evaluating-the-performance-of-retrieval-augmented-llm-systems-d95122feb0dd">Evaluating the Performance of Retrieval-Augmented LLM Systems</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT-4 as your company’s data engineer]]></title>
            <link>https://medium.com/lastmile-ai/gpt-4-as-your-companys-data-engineer-de0d3887a315?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/de0d3887a315</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Suyog Sonwalkar]]></dc:creator>
            <pubDate>Fri, 26 May 2023 19:30:09 GMT</pubDate>
            <atom:updated>2023-05-27T13:50:17.294Z</atom:updated>
            <content:encoded><![CDATA[<p>Large language models like GPT-4 can already be used to write SQL queries based on natural language questions. For simple prompts with limited knowledge about your database structure, GPT-4 will make reasonable assumptions and still generate SQL.</p><p>However, we can get much better results if we can provide an LLM with information about our database.</p><p>In this post, we’ll start by exploring some relatively straightforward prompt engineering to get significantly better results from GPT-4 as well as discuss some more ideal scenarios that could be enabled via chaining prompts or closer integration of the LLM into the database itself.</p><blockquote>We can get much better results if we can provide an LLM with information about our database.</blockquote><h3>Background</h3><p>For simple prompts like asking for the number of users created in the last week, GPT-4 will make some assumptions about your table name, columns &amp; come up with a response similar to below:</p><pre>Write some SQL that can query the number of users created in the last week from my database</pre><figure><img alt="Prompt: Write some SQL that can query the number of users created in the last week from my database. Response: SELECT COUNT(*) AS UserCount FROM users WHERE created_at &gt;= NOW() — INTERVAL 1 WEEK;" src="https://cdn-images-1.medium.com/max/1024/1*gB4GIkS3crcazmiDp9skJg.png" /></figure><p>We can also reduce GPT-4’s verbosity by specifically asking it to only return SQL.</p><pre>Write some SQL that can query the number of users created in the last week from my database. Do not explain your work, only return the SQL and make assumptions on table names and column names yourself.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Do-klWaLpBMaqvCaDjA0BQ.png" /></figure><h3>Issues</h3><p>As we can already see with the examples above, there are a number of issues trying to directly use this SQL with our database.</p><p>First, GPT-4 doesn’t know the schema of our database, including our table names, column names, column descriptions, and relationships between tables. This prevents us from directly copying the generated SQL into our query tool.</p><p>Second, GPT-4 doesn’t know if we have limitations on our SQL dialect based on the underlying database. Postgres, MySQL, Oracle, and others can have slight differences that become more nuanced as queries become more complex. Even in the basic answers above, Postgres would have a syntax error around the interval clause.</p><p>Third, in large organizations, the industry has moved towards data pipeline solutions like <a href="https://airflow.apache.org/">Airflow</a> and Data warehouses like <a href="https://aws.amazon.com/redshift/">Redshift</a> which can integrate with multiple data sources and run long running jobs to answer more complex queries.</p><p>Thankfully, we can resolve the first two with some prompt engineering and a way of passing our database structure to GPT-4. For the third, we at <a href="https://lastmileai.dev/">LastMile AI</a> want to start a conversation about how to develop industry standards to really leverage the power of LLMs across the entire data stack.</p><h3>Prompt Engineering as a temporary solution</h3><p>We can fix some of the issues by simple prompt engineering. Informing the LLM of the database we’re using as well as just copying the table structure from a client.</p><p>In the following example, I use postgres to retreive table information with a \d public.&quot;User&quot; statement and pass the information to GPT-4 along with the question I want to answer.</p><pre>Given a postgres table schema like below:<br><br>```<br>\d public.&quot;User&quot;;<br>                                    Table &quot;public.User&quot;<br>    Column     |              Type              | Collation | Nullable |      Default      <br>---------------+--------------------------------+-----------+----------+-------------------<br> id            | text                           |           | not null | <br> email         | text                           |           |          | <br> name          | text                           |           |          | <br> attributes    | jsonb                          |           |          | <br> createdAt     | timestamp(3) without time zone |           | not null | CURRENT_TIMESTAMP<br> updatedAt     | timestamp(3) without time zone |           | not null | CURRENT_TIMESTAMP<br> emailVerified | timestamp(3) without time zone |           |          | <br> image         | text                           |           |          | <br>```<br><br>I want to find the number of users who joined in the past week. Give me some SQL that will work against this table in my database. Only give me the SQL, do not explain your work. Make sure to quote column names.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9q4YvjB62rLjoBE4B_YnA.png" /></figure><p>This produces an exact SQL query that I can copy and paste into my psql client to get the result I’m looking for.</p><p>We can take this further and also provide another table’s information in order to perform simple relational queries as well.</p><pre>I also have a Posts table like this:<br>```<br>\d public.”Post”<br>                                           Table &quot;public.Post”<br>        Column         |              Type              | Collation | Nullable |         Default          <br>-----------------------+--------------------------------+-----------+----------+--------------------------<br> id                    | text                           |           | not null | <br> createdAt             | timestamp(3) without time zone |           | not null | CURRENT_TIMESTAMP<br> updatedAt             | timestamp(3) without time zone |           | not null | <br> creatorId             | text                           |           | not null | <br> type                  | “PostType”                     |           | not null | &#39;INFERENCE’::”PostType”<br> content               | text                           |           | not null | <br>Indexes:<br>    “Post_pkey&quot; PRIMARY KEY, btree (id)<br>Foreign-key constraints:<br>    “Post_creatorId_fkey&quot; FOREIGN KEY (&quot;creatorId&quot;) REFERENCES &quot;User&quot;(id) ON UPDATE CASCADE ON DELETE RESTRICT<br>```<br><br>I now want to find how many trials the past week&#39;s new users created, give me the SQL for that. Only return the SQL, do not explain your work. Make sure to quote the column names.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqxCvD4-wSF0IxVlPIny9w.png" /></figure><p>Something not done here would be to pass column descriptions to explain what information columns contain. In the future, it may be beneficial to provide optimized column descriptions for an LLM. Essentially, in addition to providing human readable descriptions, it might be valuable to have token optimized descriptions specifically for LLM understanding.</p><p>This relatively simple solution using prompt engineering can get you more reasonable responses, however they are not always perfect. First the queries may not be optimized. They may also have slight issues like column names not being quoted properly.</p><p>A larger issue is that you might not be able to pass your entire database schema to the LLM due to prompt token limits. In these cases, you may only want to pass the information for the exact tables you’re looking to get data from with your question.</p><h3>Immediate Value</h3><p>We can already see above that with some basic prompt engineering and passing database information to GPT-4, we can get workable code directly from the LLM. We can ask business questions in natural language about our data and use the queries generated to get the answers or metrics we care about.</p><h3>Ideal solutions</h3><p>What would an ideal solution look like for our examples above and for more complicated queries?</p><p>First, it would be ideal to have longer prompt lengths to be able to pass larger schemas to the LLM. With Claude <a href="https://www.anthropic.com/index/100k-context-windows">looking at 100k context limits</a>, this may already be coming soon — although output performance and speed may still be limiting factors for extremely large contexts.</p><p>Maybe the LLM would be able to chain together information about all your tables (\d in Postgres to get the list of relations), then narrow down on detail when you pose your question without having to manually input the schemas. This would require some form of chaining multiple prompts and responses together which ideally would not be in a chat interface.</p><p>In addition, what if a user didn’t have to manually put in any information on their schema and a query tool like <a href="https://www.pgadmin.org/">PgAdmin</a> (or similar for other databases) did this automatically &amp; integrated a prompt UI?</p><p>Ideally the solution would also need to take into account how your schemas are changing over time. A question that is using an out of date schema would potentially give incorrect SQL and could have downstream implications.</p><p>Tools like <a href="https://github.com/yoheinakajima/babyagi">BabyAGI</a> and <a href="https://github.com/Significant-Gravitas/Auto-GPT">AutoGPT</a> are attempting to use LLMs as part of automated agents — would we be comfortable with the LLM taking an action like actually executing SQL without human intervention?</p><p>Finally, in larger organizations, what is an ideal solution deal with data access permissions based on who is performing the queries? And how would a solution integrate with tools like <a href="https://aws.amazon.com/redshift/">Redshift</a>?</p><h3>Working towards better solutions</h3><p>We are just getting started at <a href="https://lastmileai.dev/"><em>LastMile AI</em></a><em>, </em>and would love to hear from you, especially if you’re working towards better solutions integrating data and LLMs. You can reach us here:</p><ul><li><a href="https://twitter.com/lastmileai_dev">Twitter</a></li><li><a href="http://discord.gg/xBhNKTetGx">Discord</a></li><li><a href="/service/mailto:support@lastmileai.dev?subject=GPT%20as%20data%20engineer%20feedback">Email</a></li></ul><p>We would also appreciate your feedback on our initial product offering, available at<strong> </strong><a href="https://lastmileai.dev/"><strong>lastmileai.dev</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de0d3887a315" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/gpt-4-as-your-companys-data-engineer-de0d3887a315">GPT-4 as your company’s data engineer</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using ChatGPT Plugins with LLaMA]]></title>
            <link>https://medium.com/lastmile-ai/using-openais-retrieval-plugin-with-llama-d2e0b6732f14?source=rss----518566bd75e9---4</link>
            <guid isPermaLink="false">https://medium.com/p/d2e0b6732f14</guid>
            <category><![CDATA[protocol]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Sarmad Qadri]]></dc:creator>
            <pubDate>Sun, 26 Mar 2023 04:51:52 GMT</pubDate>
            <atom:updated>2023-03-26T14:37:09.003Z</atom:updated>
            <content:encoded><![CDATA[<p>OpenAI just released initial support for <a href="https://openai.com/blog/chatgpt-plugins">plugins to ChatGPT</a>, allowing the language model to act as agents and interact with the outside world using APIs. Here we show a proof of concept using OpenAI’s <a href="https://github.com/openai/chatgpt-retrieval-plugin">chatgpt-retrieval-plugin</a> with Meta’s <a href="https://arxiv.org/abs/2302.13971">LLaMA</a> language model.</p><p>This is more than just a guide. It is a call-to-action to build an open protocol for foundation model plugins allowing us to share plugins across LLMs, and govern their interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*haTsLMsms7-0RkTUdxDPQQ.png" /><figcaption>LLaMA answering a question about the LLaMA paper with the chatgpt-retrieval-plugin. So Meta!</figcaption></figure><h3>Background</h3><p>OpenAI’s <a href="https://platform.openai.com/docs/plugins/introduction">documentation on plugins</a> explains that plugins are able to enhance ChatGPT’s capabilities by specifying a manifest &amp; an openapi specification.</p><p>There are few details available about how the plugins are wired to ChatGPT, but OpenAI open-sourced the <a href="https://github.com/openai/chatgpt-retrieval-plugin">chatgpt-retrieval-plugin</a> for semantic search and retrieving custom data for additional context.</p><p>In this guide we will take that retrieval plugin, and add a script that integrates it with LLaMA 7B running on your local machine.</p><blockquote>The code that glues the plugin to LLaMA is available in this repo (we welcome contributions):</blockquote><blockquote><a href="https://github.com/lastmile-ai/llama-retrieval-plugin">lastmile-ai/llama-retrieval-plugin</a></blockquote><h3>Limitations</h3><p>This approach successfully adds external context to LLaMA, albeit with gaps compared to OpenAI’s plugin approach:</p><ul><li><strong>Limitations in the underlying model. </strong>LLaMA is far from ChatGPT in many ways. It requires significant additional fine-tuning (such as <a href="https://github.com/tatsu-lab/stanford_alpaca">Alpaca</a>).</li><li><strong>Not generalizable to other plugins. </strong>The <a href="https://platform.openai.com/docs/plugins/introduction/plugin-flow">OpenAI documentation</a> suggests ChatGPT can read a plugin’s API schema, and dynamically construct the right API calls that satisfy the user’s request. By contrast, things didn’t go well when we tried to ask LLaMA to construct a cURL request given an OpenAPI schema. One solution to this would be fine-tuning a model specifically for OpenAPI schemas.</li></ul><h3>Demo</h3><p>We first set up our data store and upload two PDFs to it — the LLaMA<a href="https://arxiv.org/pdf/2302.13971.pdf"> paper</a> and the <a href="https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf">Conda cheatsheet</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMDL744qr-BM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMDL744qr-BM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMDL744qr-BM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b65a23a82d7cc668c730be899d3680a/href">https://medium.com/media/4b65a23a82d7cc668c730be899d3680a/href</a></iframe><p>Then we can query this data, with the relevant embeddings pulled in to the prompt as additional context.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FkkD-IFbRt34%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DkkD-IFbRt34&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FkkD-IFbRt34%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ffd7228461c9a7a20b76be09b1be3e6/href">https://medium.com/media/1ffd7228461c9a7a20b76be09b1be3e6/href</a></iframe><h3>Step-by-step guide</h3><h4>Step 0: Clone the llama-retrieval-plugin repo</h4><p><a href="https://github.com/lastmile-ai/llama-retrieval-plugin">GitHub - lastmile-ai/llama-retrieval-plugin: LLaMa retrieval plugin script using OpenAI&#39;s retrieval plugin</a></p><h4><strong>Step 1: </strong>Set up the data store</h4><p>This step is almost identical to setting up the OpenAI retrieval plugin, but simplified through the use of conda and using pinecone as the vector DB. Following the <a href="https://github.com/lastmile-ai/llama-retrieval-plugin/blob/main/README.md#quickstart">quickstart in the repo</a>:</p><p><em>Set up the environment:</em></p><pre>conda env create -f environment.yml<br>conda activate llama-retrieval-plugin<br>poetry install</pre><p><em>Define the environment variables:</em></p><pre># In production use-cases, make sure to set up the bearer token properly<br>export BEARER_TOKEN=test1234<br>export OPENAI_API_KEY=my_openai_api_key<br><br># We used pinecone for our vector database, but you can use a different one<br>export DATASTORE=pinecone<br>export PINECONE_API_KEY=my_pinecone_api_key<br>export PINECONE_ENVIRONMENT=us-east1-gcp<br>export PINECONE_INDEX=my_pinecone_index_name</pre><p><em>Start the server:</em></p><pre>poetry run start</pre><h4>Step 2: Upload files to the data store</h4><p>For this step, we used the Swagger UI available locally at <a href="http://localhost:8000/docs">http://localhost:8000/docs</a></p><p><em>Authorize:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sOF2jXbQ8NVrBelxhR519A.png" /><figcaption>Authorize using the value of the BEARER_TOKEN you specified in Step 1</figcaption></figure><p><em>Upsert File:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HHgQgAEjW-BPMgXdsXH6HA.png" /><figcaption>Specify any PDF file that you would like to get chunked into embeddings</figcaption></figure><p><em>Query the data store to test:</em></p><p>Take the id returned by the upsert, and construct a query in the Swagger UI to see what embeddings will be returned given a prompt:</p><pre>{<br>  &quot;queries&quot;: [<br>    {<br>      &quot;query&quot;: &quot;What is the title of the LLaMA paper?&quot;,<br>      &quot;filter&quot;: {<br>        &quot;document_id&quot;: &quot;f443884b-d137-421e-aac2-9809113ad53d&quot;<br>      },<br>      &quot;top_k&quot;: 3<br>    }<br>  ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fdb0R-DM9h9avFOxqbR8iw.png" /><figcaption>The query API lets you test out the vector store, and configure the filters</figcaption></figure><h4>Step 3: Set up LLaMA</h4><p>Our repo links to llama.cpp as a submodule, which is what we used for getting LLaMA 7B running locally.</p><p><em>Follow the llama.cpp readme to get set up</em></p><p><a href="https://github.com/ggerganov/llama.cpp/blob/19726169b379bebc96189673a19b89ab1d307659/README.md">llama.cpp/README.md at 19726169b379bebc96189673a19b89ab1d307659 · ggerganov/llama.cpp</a></p><h4>Step 4: Use LLaMA to query your custom data</h4><p>Open a new Terminal, and navigate to the <em>llama-retrieval-plugin</em> repo.</p><p><em>Activate the Conda environment (from Step 1):</em></p><pre>conda activate llama-retrieval-plugin</pre><p><em>Define the environment variables:</em></p><pre># Make sure the BEARER_TOKEN is set to the same value as in Step 1<br>export BEARER_TOKEN=test1234<br># Set the URL to the query endpoint that you tested in Step 2<br>export DATASTORE_QUERY_URL=http://0.0.0.0:8000/query<br># Set to the directory where you have LLaMA set up -- such as the root of the llama.cpp repo<br>export LLAMA_WORKING_DIRECTORY=./llama.cpp</pre><p><em>Run the </em><strong><em>llama_with_retrieval</em></strong><em> script with the desired prompt:</em></p><pre>python3 llama_with_retrieval.py &quot;What is the title of the LLaMA paper?&quot; </pre><p>This script takes the prompt, calls the <em>query </em>endpoint to extract the most relevant embeddings from the data store, and then constructs a prompt to pass to LLaMA that contains these embeddings.</p><p>You can read the code here: <a href="https://github.com/lastmile-ai/llama-retrieval-plugin/blob/main/llama_with_retrieval.py">llama-retrieval-plugin/llama_with_retrieval.py</a></p><h4>Step 5: Tweak and experiment</h4><p>You can modify the <em>llama_with_retrieval </em>script to experiment with different settings that may yield better performance:</p><ul><li>Change the token limit (e.g. reduce it to give more room for the model response).</li><li>Change the prompt template and observe model behavior.</li><li>Change the LLaMA model parameters by modifying the command line. <strong>Note: </strong>You can also specify a custom LLaMA command line by setting the LLAMA_CMD environment variable.</li></ul><blockquote>You can also use <a href="https://lastmileai.dev/">lastmileai.dev</a> to track your various experiments as you tweak and tune models. For example, <a href="https://colab.research.google.com/drive/1sFHDD-lR7_7WMEj-GgzmIBEDd81FFI9_?usp=sharing">here’s a notebook</a> saving some trials using Stable Diffusion.</blockquote><h3>Protocols over Platforms</h3><p>We hope this exercise shows the need for standardizing interactions between foundation models and plugins/extensions. We should be able to use a plugin designed for OpenAI models with another large language model, and vice versa. This is only possible with a Foundation Model Plugin Protocol standard.</p><p>We are in the early stages of a revolution in computing, powered by the advent of state-of-the-art foundation models. We have an opportunity to define the behaviors that govern our interactions with these models, and return to the rich legacy of open protocols of the early internet instead of closed platforms of the modern era.</p><h4>Foundation Model Plugin Protocol</h4><p>The <em>lastmile ai</em> team is exploring what it takes to define a plugin protocol, and spur its adoption. We believe the protocol should be:</p><ul><li><strong>model-agnostic</strong> — support GPTx, LLaMA, Bard, and any other foundation model.</li><li><strong>modal-agnostic </strong>— support different types of inputs and outputs, instead of just text.</li></ul><p>Our early thinking on this is inspired by SMTP for email, and <a href="https://microsoft.github.io/language-server-protocol/">LSP (Language Server Protocol)</a> for IDEs. We will be sharing what we have in this space in the coming days, and <strong>would love to collaborate with you</strong>.</p><h3>Call to action</h3><p>We are just getting started at <em>lastmile ai, </em>and would love to hear from you, especially if you share our vision for an open and interoperable future. You can reach us here:</p><ul><li><a href="https://twitter.com/lastmileai_dev">Twitter</a></li><li><a href="http://discord.gg/xBhNKTetGx">Discord</a></li><li><a href="/service/mailto:sarmad@lastmileai.dev?subject=llama-retrieval-plugin%20followup&amp;body=Hello%20folks%20%40%20lastmile%20ai">Email</a></li></ul><p>We would also appreciate your feedback on our initial product offering, available at<strong> </strong><a href="https://lastmileai.dev/"><strong>lastmileai.dev</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2e0b6732f14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lastmile-ai/using-openais-retrieval-plugin-with-llama-d2e0b6732f14">Using ChatGPT Plugins with LLaMA</a> was originally published in <a href="https://medium.com/lastmile-ai">lastmile ai — blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>