<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Antonio Velasco on Medium]]></title>
        <description><![CDATA[Stories by Antonio Velasco on Medium]]></description>
        <link>https://medium.com/@antoniovelasco1991?source=rss-bf1d72aa1e06------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*HtMR95LmYTri0DNv</url>
            <title>Stories by Antonio Velasco on Medium</title>
            <link>https://medium.com/@antoniovelasco1991?source=rss-bf1d72aa1e06------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:16:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@antoniovelasco1991/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM Utilities — Document Query]]></title>
            <link>https://medium.com/@antoniovelasco1991/llm-utilities-document-query-10e515bc142f?source=rss-bf1d72aa1e06------2</link>
            <guid isPermaLink="false">https://medium.com/p/10e515bc142f</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[chromadb]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Antonio Velasco]]></dc:creator>
            <pubDate>Thu, 17 Oct 2024 15:47:08 GMT</pubDate>
            <atom:updated>2024-10-30T10:19:00.967Z</atom:updated>
            <content:encoded><![CDATA[<h3>LLM Utilities — Document Query Index</h3><p><a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><p>LLM utility tools are a set of simple techniques and applications powered by an LLM (Large Language Model) built on the back of Streamlit for demo purposes.</p><h4>In this article</h4><ul><li>Document Query</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcKti4C1MN22wFOoFpyMYw.gif" /></figure><h4>Previous updates</h4><ul><li><a href="https://medium.com/p/88523921f4d0/edit">Agent Chatbot</a></li><li><a href="https://medium.com/@antoniovelasco1991/llm-utilities-text-summarizer-53c6cd1f04e3">Text Summarizer</a></li></ul><h4>Future updates</h4><ul><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><p>I started developing some of these as learning exercises, for my own personal use, or as demo examples of what can be done for the industry. They enhance quality of life and/or improve efficiency in common day-to-day office tasks, or address industry shortcomings and use cases.</p><p>My intention is to clean them up and write about each one as a showcase and to share the repository as a how-to example. Today, an indexation over long documents that allows the LLM to retrieve relevant information from them and answer user queries accordingly.</p><h3>Tool description</h3><h4><strong>🤔 Document Query</strong></h4><p>The Document Query accepts a drag-and-drop PDF or a pdf URL. It uses Azure’s Document Intelligence to extract all text and split it into pages that are indexed using ChromaDB. The user can then ask any question about the content of the document. The query will be matched against the indexed pages, and the one containing the answer will be brought to an LLM as context. The user will receive the answer and a display of the page for fact-checking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i7t31ihtWRSlx29Xc-4f4w.png" /></figure><p>The indexing works by using our LLM to create a vector for each page of the document. This is called embedding. Creating embeddings is fast and resource efficient. By using them to find the relevant context instead of feeding the whole document to the LLM, we avoid expending too much or running into model’s limits.</p><p><strong>Embeddings </strong>are a way of representing words or chunks of text into vectors, elements with shared common contexts in the corpus will be close together in the vector space. This vectors are stored in a ChromaDB database where they’ll be matched against the user query embedding when prompted and only the relevant page will be brought forward.</p><p><a href="https://docs.trychroma.com/"><strong>Chroma DB</strong></a> is an open-source vector store used for storing and retrieving vector embeddings. Its main use is to save embeddings along with metadata to be used later by large language models.</p><p>Finally, the relevant page and the user query are sent together to a LLM that will answer to the user. This is a Retrieval Augmented Generation solution that allows a LLM to draw knowledge from a given long document with ease and efficient use of resources.</p><h4>Result</h4><p>The demo accepts either a URL pointing to a pdf or a drag-and-drop pdf or doc file. The user may then write a Query on the box and click on “Ask Document”. After a bit of processing, an answer that has had drawn knowledge and information from the document will appear. Below, the context page used by the LLM will be displayed. Allowing for fact check or tasks of content location on long documents.</p><h3>How to use</h3><p>Find the repository here: <a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><h3>Installation</h3><ul><li>Git clone in your local machine.</li><li>Create a ‘.env’ file with:</li></ul><blockquote>SECRET_OPENAI_API_KEY=YOUR_KEY_HERE</blockquote><blockquote>SECRET_FORM_KEY=YOUR_DOCUMENT_INTELLLIGENCE_KEY_HERE</blockquote><blockquote>SECRET_FORM_ENDPOINT=DOCUMENT_INT_ENDPOINT_HERE</blockquote><ul><li>Create a virtual environment (pipenv files available) and install the libraries in ‘requirements.txt’. In the terminal, run the command:</li></ul><blockquote>pip install -r requirements.txt</blockquote><ul><li>In the terminal run the command (either):</li></ul><blockquote>run streamlit run code/Home.py</blockquote><blockquote>pipenv run streamlit run code/Home.py</blockquote><p>That’s it! A html like app will open in your browser where you can navigate to the chat tool on the side and play with it.</p><h4>Future updates</h4><ul><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10e515bc142f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Utilities — Text Summarizer]]></title>
            <link>https://medium.com/@antoniovelasco1991/llm-utilities-text-summarizer-53c6cd1f04e3?source=rss-bf1d72aa1e06------2</link>
            <guid isPermaLink="false">https://medium.com/p/53c6cd1f04e3</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[llmchain]]></category>
            <dc:creator><![CDATA[Antonio Velasco]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 09:15:07 GMT</pubDate>
            <atom:updated>2024-10-30T10:18:27.278Z</atom:updated>
            <content:encoded><![CDATA[<h3>LLM Utilities — Text Summarizer Chain</h3><p><a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><p>LLM utility tools are a set of simple techniques and applications powered by an LLM (Large Language Model) built on the back of Streamlit for demo purposes.</p><h4>In this article</h4><ul><li>Summary Tool</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXdMf5VZmYQO_bFuYfh5cg.gif" /></figure><h4>Previous updates</h4><ul><li><a href="https://medium.com/p/88523921f4d0/edit">Agent Chatbot</a></li></ul><h4>Future updates</h4><ul><li><em>Document Query</em> — (To be documented)</li><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><p>I started developing some of these as learning exercises, for my own personal use, or as demo examples of what can be done for the industry. They enhance quality of life and/or improve efficiency in common day-to-day office tasks, or address industry shortcomings and use cases.</p><p>As I said in <a href="https://medium.com/p/88523921f4d0/edit">the first post</a>, I intend to clean them up and write about each one as a showcase and to share the repository as a how-to example. This week, a simple llm chain that summarizes texts into handy bullet points.</p><h3>Tool description</h3><h4>📃 Document Summarizer</h4><p>The Document Summarizer accepts a drag-and-drop PDF or a URL. It extracts the text from the source and then sends it to a LLM chain that uses Map-Reduce to synthetise the content down to 3 bullet points per number of paragraphs instructed.</p><p>A <strong>chain </strong>is<strong> </strong>a sequence of operations or transformations applied to the input data to generate the desired output. Every step output is the input of the next. In our case, the chain applies a Map Reduce strategy to produce a Summary.</p><p>When summarizing text with a LLM, there are different strategies that can be employed to effectively condense information. Each with its own advantages and drawbacks. One of the simplest and more cost-effective is map-reduce.</p><p><strong>Map-Reduce </strong>involves breaking the long text into smaller sections that fit within the LLM’s context. We then ask the LLM to summarize each section, and finally, we create a summary of these summaries. This should give us a good overall summary of the whole document.</p><p>One great thing about this approach is that since each section’s summary is independent, we can use parallel processing to speed things up.</p><h4>Result</h4><p>The demo accepts either a URL or a drag-and-drop file for summarizing PDFs. When clicking on “Summarize File,” a tree of bullet points with the main ideas of the document will appear. Selecting the number of paragraphs will change the number of paragraphs produced, providing a better understanding of the text.</p><h3>How to use</h3><p>Find the repository here: <a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><h3>Installation</h3><ul><li>Git clone in your local machine.</li><li>Create a ‘.env’ file with:</li></ul><blockquote>SECRET_OPENAI_API_KEY=[YOUR_KEY_HERE]</blockquote><ul><li>Create a virtual environment (pipenv files available) and install the libraries in ‘requirements.txt’. In the terminal, run the command:</li></ul><blockquote>pip install -r requirements.txt</blockquote><ul><li>In the terminal run the command (either):</li></ul><blockquote>run streamlit run code/Home.py</blockquote><blockquote>pipenv run streamlit run code/Home.py</blockquote><p>That’s it! A html like app will open in your browser where you can navigate to the chat tool on the side and play with it.</p><h4>Future updates</h4><ul><li><em>Document Query</em> — (To be documented)</li><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53c6cd1f04e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Utilities — Agent Chatbot]]></title>
            <link>https://medium.com/@antoniovelasco1991/llm-utilities-agent-chatbot-88523921f4d0?source=rss-bf1d72aa1e06------2</link>
            <guid isPermaLink="false">https://medium.com/p/88523921f4d0</guid>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Antonio Velasco]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 12:17:13 GMT</pubDate>
            <atom:updated>2024-10-30T10:17:53.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>LLM Utilities — Chatbot Agent with tools</h3><p><a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><p>LLM utility tools are a set of simple techniques and applications powered by an LLM (Large Language Model) built on the back of Streamlit for demo purposes.</p><h4>In this article</h4><ul><li>Agent Chatbot</li></ul><h4>Future updates</h4><ul><li><em>Summary Tool</em> — (To be updated &amp; documented)</li><li><em>Document Query</em> — (To be documented)</li><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><p>I started developing some of these as learning exercises, for my own personal use, or as demo examples of what can be done for the industry. They enhance quality of life and/or improve efficiency in common day-to-day office tasks, or address industry shortcomings and use cases.</p><p>I intend to clean them up and write a post about each one as a showcase and to share the repository as a how-to example. The first one is a simple agent that provides the same chat experience but enhanced to have access to the web.</p><h3>Tool description</h3><h4>🗨️ Chat agent</h4><p>The chat agent is a simple bot interface agent between the LLM and you with access to a toolset.</p><p>An Agent is a train of thought powered by an LLM that decides on what actions need to be taken to solve the prompted task. The decisions are usually to select a given tool available, to write relevant code or text to employ that tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Eh1uHZaFzC2oNFxUIzTow.png" /></figure><p>This first tool is an Agent that has access to the DuckDuckGO Search Engine, so when prompted it can decide if it needs to use it, what will be its search, read the results and answer in the chat with an accurate answer. This solves the LLMs lack of awareness of real world data outside their training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ka1qxFJjP0p2LWN0mFqD9Q.gif" /></figure><h3>How to use</h3><p>Find the repository here: <a href="https://github.com/Antonio-Velasco/llm_utilities/tree/main">Antonio-Velasco/llm_utilities (github.com)</a></p><h3>Installation</h3><ul><li>Git clone in your local machine.</li><li>Create a ‘.env’ file with:</li></ul><blockquote>SECRET_OPENAI_API_KEY=[YOUR_KEY_HERE]</blockquote><ul><li>Create a virtual environment (pipenv files available) and install the libraries in ‘requirements.txt’. In the terminal, run the command:</li></ul><blockquote>pip install -r requirements.txt</blockquote><ul><li>In the terminal run the command (either):</li></ul><blockquote>run streamlit run code/Home.py</blockquote><blockquote>pipenv run streamlit run code/Home.py</blockquote><p>That’s it! A html like app will open in your browser where you can navigate to the chat tool on the side and play with it.</p><h4>Future updates</h4><ul><li><em>Summary Tool</em> — (To be updated &amp; documented)</li><li><em>Document Query</em> — (To be documented)</li><li><em>Unstructured Data Extraction</em> — (To be updated &amp; documented)</li><li><em>Table Agent</em> — (To be updated &amp; documented)</li><li><em>Text Composer with memory</em> — (WIP)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88523921f4d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt evaluation]]></title>
            <link>https://medium.com/@antoniovelasco1991/prompt-evaluation-3b5b635b549d?source=rss-bf1d72aa1e06------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b5b635b549d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[ll]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Antonio Velasco]]></dc:creator>
            <pubDate>Wed, 09 Aug 2023 11:16:32 GMT</pubDate>
            <atom:updated>2023-08-09T11:16:32.588Z</atom:updated>
            <content:encoded><![CDATA[<h3>Prompt evaluation Example with Python</h3><p>Everybody is talking about Large Language Models (LLMs) and prompt engineering. They are, well, truly large language models able to generalize and that can be applied to multiple tasks they were not trained on.</p><p>The way we instruct them how to approach new scenarios is via prompts. <a href="https://temir.org/teaching/big-data-and-language-technologies-ss22/materials/701-prompt-engineering.pdf">Prompt engineering</a>¹ involves designing robust instructions that effectively steer LLMs towards a desire output. It has become a new useful skill to learn.</p><p>An important part of prompt engineering is prompt evaluation. The enormous power of some of these models makes them an excellent out-of-the-box solution. Small tuning of hyperparameters and a clear prompt will work wonders on most use cases. However, when we implement them in the industry, we need to evaluate their performance.</p><h4>Why?</h4><p>Firstly, evaluating prompt output statistically tames the probabilistic nature of LLMs into a somewhat predictable input-output box that can be integrated into your solutions without wreaking havoc at random. Secondly, prompt drifting. In most cases, these models are not owned and trained by the developers but rather APIs that interact with massively large models that are continuously under development and being retrained. As they change, our desired output might do so. Finally, developing LLM-based solutions should be done in a test-driven way, prompts included.</p><h4>Practical example</h4><p>To ilustrate, let’s run a quick example. Using OpenAi’s basic GPT3.5-turbo and the <a href="https://huggingface.co/datasets/squad">squad</a>² dataset from datasets.</p><pre>from datasets import load_dataset<br><br>dataset = load_dataset(&quot;squad&quot;)</pre><p>This dataset consists of a context, a question about it, and the correct specific answer. It is a perfect dataset to test an LLM whose goal would be to mine out specific values or ideas.</p><p>To the question:</p><p>To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?</p><p>and context (cropped here for simplicity):</p><p>[...] is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple [...]</p><p>the desired answer would be:</p><p>Saint Bernadette Soubirous</p><p>GPT3.5, out of the box is able to perform the task:</p><pre>response = openai.ChatCompletion.create(<br>    model=&quot;gpt-3.5-turbo&quot;,<br>    messages=[<br>      {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: context},<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query},<br>    ]<br>  )<br>print(response.get(&quot;choices&quot;)[0][&quot;message&quot;][&quot;content&quot;])</pre><p>The Virgin Mary allegedly appeared to Saint Bernadette Soubirous in 1858 in Lourdes, France.</p><p>Which is technically correct but verbose.</p><h3>Define tests</h3><pre>from difflib import SequenceMatcher<br>from promptimize import evals<br><br>def sequence_matcher(a, b):<br>    return SequenceMatcher(None, a, b).ratio()<br><br>def jaccard_similarity(a, b):<br>    intersection = len(set(a).intersection(set(b)))<br>    union = len(set(a).union(set(b)))<br>    return intersection / union</pre><p><a href="https://pypi.org/project/promptimize/">Promptimize</a>³ is one example of many open-source tools being developed at the moment to help with prompt evaluation. It has some basic evaluation functions defined that we can use.</p><p><strong>percentage_of_words</strong> meassures from 0 to 1 the percentage of words from a list that are present in the given response. This one is our basic test, the desired answer should be included in the output.</p><pre>evals.percentage_of_words(answer,exact_answer[0])<br>1.0</pre><p><strong>SequenceMatcher</strong> is a class in the difflib module of Python that provides a way to compare pairs of sequences of any type, as long as the sequence elements are hashable.</p><p>The idea is to find the longest contiguous matching subsequence that contains no “junk” elements (these “junk” elements are ones that are uninteresting in some sense, such as blank lines or whitespace). The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence.</p><pre>sequence_matcher(answer,exact_answer[0])<br>0.4406779661016949</pre><p>The <strong>Jaccard similarity</strong> coefficient, also known as the Jaccard index, is a statistic used for gauging the similarity and diversity of sample sets. In our case, two strings. It measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets. In other words, it is calculated as:</p><pre>Jaccard Similarity = (number of observations in both sets) / (number in either set) or J(A, B) = |A∩B| / |A∪B|</pre><p>The Jaccard similarity ranges from 0 to 1. The closer to 1, the more similar the two strings. If two strings are identical, their Jaccard Similarity Index will be 1. Conversely, if they have nothing in common then their similarity will be 0.</p><pre>jaccard_similarity(answer,exact_answer[0])<br>0.4666666666666667</pre><p>Sequence matcher and Jaccard similarity are the tests that allow us to measure how close the output is to the desired answer. A 1 in these would mean that they are identical, but their range between 0 and 1 allow us to see progression.</p><p>Out of the box, we have a model that is able to find the correct answer but is too verbose. Let’s try to improve it while testing statistically.</p><h3>Testing</h3><p>Let’s select a few random entries</p><pre>import numpy as np<br><br># set the seed for reproducibility<br>np.random.seed(42)<br><br># randomly select 20 rows from the dataframe<br>corpus = df.sample(n=20)<br><br># we are not interested in the placement of the answer, so making it the straight entry<br>corpus[&#39;answers&#39;] = corpus[&#39;answers&#39;].apply(lambda x: x[&#39;text&#39;][0])<br>corpus.head(3)</pre><h4>Version 1 — Out of the box performance</h4><p>Let’s make a call to our model for every sample in our corpus df</p><pre>VERSION = &quot;ver001&quot;<br><br>corpus[f&#39;{VERSION}_llm_answer&#39;] = corpus.apply(lambda row:<br>    openai.ChatCompletion.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[<br>        {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: row[&#39;context&#39;]},<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: row [&#39;question&#39;]},<br>        ]<br>    ).get(&quot;choices&quot;)[0][&quot;message&quot;][&quot;content&quot;],<br>    axis = 1<br>    )</pre><p>After applying our tests, we have the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pUM07KyMJgLHTxf_.png" /></figure><p>In this table, the rows are the 3 different tests and each column represent one sample of the dataset. That is context, question, desired answer and model answer. The number is the performance of comparing desired answer and model answer with the proper row criteria.</p><p>As we have seen before, the model excels at identifying the correct answer but is way too verbose. The percentage of desired words is high but the similarity of the strings is low overall.</p><h3>Version 2 — Specific Prompt</h3><p>Let’s see how it responds with a proper prompt and defined hyperparameters. The new expanded system message includes instructions besides the context.</p><pre>config = {<br>    &quot;temperature&quot;: 0,<br>    &quot;max_tokens&quot;: 16,<br>    &quot;top_p&quot;: 1,<br>}<br>corpus_v2[f&#39;{VERSION}_llm_answer&#39;] = corpus_v2.apply(lambda row:<br>  openai.ChatCompletion.create(<br>      model=&quot;gpt-3.5-turbo&quot;,<br>      messages=[<br>      {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: <br>        f&quot;&quot;&quot;<br>        You are an API given the following context: /n<br>        ###<br>        {row[&#39;context&#39;]}<br>        ###<br>        Will reply as concisely as possible to the user question.<br>        &quot;&quot;&quot;<br>      },<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: row [&#39;question&#39;]},<br>      ],<br>    temperature=config[&quot;temperature&quot;],<br>    max_tokens=config[&quot;max_tokens&quot;],<br>    top_p=config[&quot;top_p&quot;],<br>  ).get(&quot;choices&quot;)[0][&quot;message&quot;][&quot;content&quot;],<br>  axis = 1<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cYpj9ylKQ7XmV9mC.png" /></figure><p>Quite poor still. Almost no improvement overall and even worse, in sample 18 it failed to rethrieve the correct data. Let’s see why</p><pre>print(f&#39;To the question: {corpus_v2.iloc[-3][&quot;question&quot;]}&#39;)<br>print(f&#39;with the desired answer: {corpus_v2.iloc[-3][&quot;answers&quot;]}&#39;)<br>print(f&#39;Version 1 answered: {corpus_v2.iloc[-3][&quot;ver001_llm_answer&quot;]}&#39;)<br>print(f&#39;Version 2 answered: {corpus_v2.iloc[-3][&quot;ver002_llm_answer&quot;]}&#39;)</pre><p>To the question: When did Houston begin to regain its dependence on the oil industry?</p><p>with the desired answer: 2000s</p><p>Version 1 answered: Houston began to regain its dependence on the oil industry in the 2000s, specifically after the increase in oil prices during that time.</p><p>Version 2 answered: Houston began to regain its dependence on the oil industry since the increase of oil prices</p><p>However, If we compare with the previous version, we can appreciate some timid progression towards concise answers:</p><pre># Select columns with ver001 and ver002<br>ver001_cols = corpus.columns[-3:]<br>ver002_cols = corpus_v2.columns[-3:]<br><br># Calculate the difference between ver001 and ver002 columns<br>diff_df = corpus_v2[ver002_cols].values - corpus_v2[ver001_cols].values</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KWrCoEv2aWphunB5.png" /></figure><p>From an exploration, it looks like our instruction of being concise is not producing the desired effect. Maybe the model understands its short phrase answers as concise already. We are likely to have failed communicating to it that we want just the exact piece of data.</p><h4>Version 3 — Improved Prompt</h4><p>Let’s try this time rewording the last phrase in our prompt from:</p><p>Will reply as concisely as possible to the user question.</p><p>to</p><p>Will reply concisely, using as few words as possible, to the user question.</p><pre>corpus_v3[f&#39;{VERSION}_llm_answer&#39;] = corpus_v3.apply(lambda row:<br>  openai.ChatCompletion.create(<br>      model=&quot;gpt-3.5-turbo&quot;,<br>      messages=[<br>      {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: <br>        f&quot;&quot;&quot;<br>        You are an API given the following context: /n<br>        ###<br>        {row[&#39;context&#39;]}<br>        ###<br>        Will reply concisely, using as few words as possible, to the user question.<br>        &quot;&quot;&quot;<br>      },<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: row [&#39;question&#39;]},<br>      ],<br>    temperature=config[&quot;temperature&quot;],<br>    max_tokens=config[&quot;max_tokens&quot;],<br>    top_p=config[&quot;top_p&quot;],<br>  ).get(&quot;choices&quot;)[0][&quot;message&quot;][&quot;content&quot;],<br>  axis = 1<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ldFnIciVSDC51zs4.png" /></figure><p>Much better! Now the model is refraining itself from giving verbose answers.</p><pre>print(f&#39;To the question: {corpus_v3.iloc[0][&quot;question&quot;]}&#39;)<br>print(f&#39;with the desired answer: {corpus_v3.iloc[0][&quot;answers&quot;]}&#39;)<br>print(f&#39;Version 1 answered: {corpus_v3.iloc[0][&quot;ver001_llm_answer&quot;]}&#39;)<br>print(f&#39;Version 3 answered: {corpus_v3.iloc[0][&quot;ver003_llm_answer&quot;]}&#39;)</pre><p>To the question: What year was the Banská Akadémia founded?</p><p>with the desired answer: 1735</p><p>Version 1 answered: The Banská Akadémia in Banská Štiavnica, Slovakia was founded in 1735.</p><p>Version 3 answered: 1735</p><p>With a simple change in the prompt, we have greatly improved our out-of-the-box solution, and we have measurable metrics to account for it.</p><p>Now we can iteratively improve our prompt with a real comprehension of the impact of our changes in the prompt wording.</p><h3>Conclusion</h3><p>Although LLMs’ outputs are probabilistic in nature, their related solutions and prompts should be developed in a test-driven way. Developing efforts will profit from tests providing metrics that can be measured and accounted for.</p><p>As we have seen in a brief practical example, setting tests that define how the output of our LLM should behave, we can follow a test-driven development. In this way, development is easier as we understand better the shortcomings, we can assure the net improvement of a given change, and we can set a quality threshold to meet.</p><h3>Repository</h3><p><a href="https://github.com/Antonio-Velasco/promt_evaluation">Antonio-Velasco/promt_evaluation (github.com)</a></p><h3>References</h3><p><a href="https://temir.org/teaching/big-data-and-language-technologies-ss22/materials/701-prompt-engineering.pdf">1</a> Introduction to Prompt Engineering — Temir.</p><p><a href="https://huggingface.co/datasets/squad">2</a> Rajpurkar, P., Zhang, J., Lopyrev, K., &amp; Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, arXiv:1606.05250.</p><p><a href="https://pypi.org/project/promptimize/">3</a> promptimize. (2021). Promptimize. PyPI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b5b635b549d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>