<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Georgian on Medium]]></title>
        <description><![CDATA[Stories by Georgian on Medium]]></description>
        <link>https://medium.com/@georgian-io?source=rss-d3d81cdc0d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FJwbo1EyZGjafbGs212LKw.jpeg</url>
            <title>Stories by Georgian on Medium</title>
            <link>https://medium.com/@georgian-io?source=rss-d3d81cdc0d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:19:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@georgian-io/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Measurably Correct: Strategies for LLM Evaluation.]]></title>
            <link>https://medium.com/georgian-impact-blog/measurably-correct-strategies-for-llm-evaluation-447313621ab1?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/447313621ab1</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 16:26:50 GMT</pubDate>
            <atom:updated>2024-12-09T21:48:26.489Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESvVxD8ezdHJbxHAxQIBsw.jpeg" /></figure><p>By <a href="https://www.linkedin.com/in/benrwilde/">Ben Wilde</a>, <a href="https://www.linkedin.com/in/angelineyasodhara/">Angeline Yasodhara</a> and <a href="https://www.linkedin.com/in/rodrigo-ceballos-lentini/">Rodrigo Ceballos Lentini</a></p><p>How do you know an answer to a question is ‘correct’? For anyone building an application that uses a Large Language Model (LLM) that is much more than just a philosophical question. There is a real need to be able to assess whether or not the generated output of an LLM-application is correct (or not). As LLMs are incorporated into more and more software applications, it may be useful to consider how to incorporate LLM testing into our engineering processes.</p><p>Model evaluation (or simply evaluation or ‘evals’) is the process of determining whether the <em>output</em> from an LLM is correct for a given <em>input. </em>Many readers may have already done an evaluation of a large language model by simply testing some prompts in ChatGPT and reviewing the results. Some might even have clicked on the thumbs up / down and in doing so will have provided OpenAI feedback for their model evaluation process.</p><p>For those coming from a more traditional software background, a useful analogy for evaluations is to think of evaluations as unit and integration tests for LLM applications.<strong> </strong>Given the non-deterministic nature and rapidly changing capabilities of LLMs, we think it is important to take a ‘test driven’ approach when building LLM-based software applications.</p><p><strong>Try, then try again.</strong></p><p>Evaluations start when developers begin evaluating the output of their first few prompts to see if they can get a reasonable answer from the model to which they have access. As they make changes to their prompts, developers continue to evaluate the results. Teams are doing an evaluation when determining whether or not a particular LLM (e.g. Llama3 vs. GPT-4 vs. Mistral) meets the particular requirements of the project.</p><p>Evaluations are also typically conducted to check for changes in output between variations of the same model (e.g. gpt-4–1106-preview vs. gpt-4–0613), changes to prompts and even changes to available context for RAG applications (e.g. adding new documents to an index). Teams also usually evaluate a model after fine tuning or after quantization.</p><p>That’s a lot of evaluation that needs doing and so development teams try to find ways to reduce the amount of manual evaluation required in each case.</p><p><strong>Towards (more) scalable LLM evaluations.</strong></p><p>The proposed approach outlined in Figure 1 below is based on our work with software companies. In this approach, we typically see teams make use of manual evaluation at the start of the LLM selection and prompt development process and at the end. In the middle, teams tend to apply more scalable techniques that enable the ability scale out testing across more data, edge cases, different LLMs, different fine tuning strategies, different versions of the same LLM etc. without requiring extensive manual evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WDJZcEvzrte-BLr4g79yKg.jpeg" /><figcaption><strong>Figure 1: Scaling LLM Evaluation</strong></figcaption></figure><p>To illustrate this approach, consider an example scenario of extracting medical conditions and interventions mentioned in a short section of text. The first step is to try a simple prompt such as “extract conditions and interventions from the following text: …” Example input text and the desired output from the LLM is shown in Figure 2. Once we have adjusted our prompt as necessary to get the desired output, we may want to try it again on an additional sample or two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBZpxgkiah2d0RpGi_FAOQ.jpeg" /><figcaption><strong>Figure 2: Example Input Text and Desired Output (</strong><a href="https://huggingface.co/datasets/owkin/medical_knowledge_from_extracts"><strong>source</strong></a><strong>)</strong></figcaption></figure><p>Next we may consider how to curate a dataset we can use to evaluate model performance. Fortunately for our medical text example there is a <a href="https://huggingface.co/datasets/owkin/medical_knowledge_from_extracts">public dataset</a> available from HuggingFace that provides input and output pairs. The plan will be to use this across each of the LLMs that we evaluate to see which models appear able to provide the desired output given the input text.</p><p>After we have the prompt and dataset available for the evaluations, the next step is to decide on the metrics for evaluating the performance of one LLM vs. another. For example, we might choose to use the ‘correctness’ of the entities extracted by the prompt from the text as our metric. In this scenario we are using a relatively simple way to measure success for each LLM, therefore we may not need to take the final step of manually reviewing the top performing LLMs from the automated stage. In other cases, we may want to create a small set of additional prompt / answer pairs that are more complicated and that take into account edge cases or specific known requirements for a particular use case. However, for scenarios where there is no dataset available, and there is no direct evaluation method to test correctness, e.g. in cases like chatbot scenario or summarization, scaling the evaluation may be more difficult and manual evaluation techniques may be required.</p><p><strong>Sourcing a (good) dataset.</strong></p><p>When selecting the evaluation dataset, it may be prudent to include inputs that are representative of production data, unusual or edge cases, and high-value examples. The dataset should include examples of desired outputs or reference answers, although limited evaluation is still possible without these inputs.</p><p>There are several options for sourcing data including open-source datasets as in our earlier example, historical data that you have access to (and permission to use!), manual creation of new data, and data generated using LLMs. Indeed, one can use LLMs to generate data by expanding an existing dataset by creating new examples, such as generating synonyms for words as well as more sophisticated approaches such as ‘correct by construction’ and ‘resource overboard’.</p><p><strong>Generating a dataset using LLMs</strong></p><p>An alternative to sourcing existing datasets for LLM evaluation is to utilize LLMs themselves to generate evaluation data. Two approaches to generating such data are what we refer to as , ‘Correct by Construction’ and ‘Resource Overboard’ (also known as ‘Knowledge Distillation’).</p><p>The idea behind ‘Correct by Construction’ is that a data set can be designed in such a way that its correctness is ensured. For example, creating a LLM application that is able to generate API calls in Python from a natural language prompt. The input to the LLM would be an instruction asking the LLM to generate the API calls and the output would be code that can then be executed to confirm that the calls are valid. We note that it may be difficult to know if the right API call is being made given a particular prompt because it is possible that a correct call to the wrong API could be generated or the wrong parameters could be generated for the right API endpoint (e.g. a call to a weather API that returns the results in English when German was requested in the prompt).</p><p>One approach to test that the right API calls are being generated is to work backwards. In this scenario, we can start from the right API calls and ask the LLM to generate natural language descriptions of those API calls. Then, when we want to evaluate the model, we can use the LLM-generated descriptions as inputs (prompts) and check to make sure that the API calls (the LLM output) are the same as the ones that we used to create the descriptions in the first place. This approach enables us to confirm that the model is (or is not) generating the appropriate API calls for each possible user request.</p><p>Another example of this type of approach is to evaluate factual question answering via Retrieval Augmented Generation (RAG). Taking chunks of a document that we will be including in our knowledge base, we can ask an LLM to generate relevant questions for each chunk of text. Then we can ask those questions of the RAG application and confirm that the correct chunks of text are returned. This approach allows us to extend this ‘correct by construction’ approach outside of LLM evaluation to check the accuracy of the retrieval stage of the RAG application.</p><p>A second approach to generating data is what we will call “resource overboard” where we can use a more capable LLM, one that is also typically more expensive, to generate desirable outputs for the evaluation dataset. That dataset can be used to test smaller, less capable models to find good candidates for a particular use case. This approach may be useful for chatbots, text summarization and analysis where a model such as GPT 4o or Llama3 405B could be used to generate evaluation data to test suitability of smaller models such as GPT 4o mini, Gemma or Llama3 8B. Note, this approach is similar to the ‘teacher / student’ model where a ‘teacher’ LLM is used to generate training data that is then used to finetune a ‘student’ LLM.</p><p><strong>Measure what matters</strong></p><p>With the evaluation dataset in hand, it is time to think about how to quantify the quality of the responses back from the LLM-application. Here teams have a number of strategies available including:</p><ol><li>Traditional Natural Language Processing (NLP) metrics such as Rouge and BLEU.</li><li>Property Based Metrics (e.g. number of tokens used)</li><li>LLM-as-Judge, an emerging technique that uses a language model to grade output.</li></ol><p><strong>Traditional NLP Metrics</strong></p><p>Examples include ROUGE, BLEU, and Levenshtein distance measure, all of which help us measure the accuracy of generated texts compared to reference texts, but which may not capture semantic similarity.</p><p>ROUGE measures how many words from the reference text are in the generated texts. By contrast, BLEU measures how many words from the generated texts are in the reference text. Another more traditional NLP measure is ‘Levenshtein distance’ which measures the minimum number of single character edits including insertions, deletions, or substitutions, required to change one word in the generated text into a word in the reference text. In each case, reference text, which we can measure against in order to calculate the metrics and may not be available, is needed.</p><p>Each of these measurements are measuring exact similarity. That is, they’re checking the words, the tokens and any character changes. They’re not actually checking whether two sentences, which are only slightly different, are actually saying the same thing. To achieve that we need to look at semantic similarity and that is what the BERT-score can provide us. It measures what we call the “cosine similarity of two pieces of text. It is worth noting that using semantic similarly as a measure of quality has limitations. For example, if you have two sentences, one that says “The stock price goes up” and the other that says “The stock price goes down” then they are both semantically very similar but the meaning is the opposite.</p><p><strong>Property-Based Metrics</strong></p><p>The idea with property-based metrics is that they are simple to measure, giving a quick way to test if the LLM is producing the expected ‘shape’ of output.</p><p>Examples include measuring the total length (in terms of tokens or words) of the output, giving an indication of whether the LLM is rambling (too long) or has truncated the answer. Average sentence length could also be measured to make sure the LLM is generating output that matches writing style expectations.</p><p>Vocabulary size is another property-based metric where the total number of unique words in a text are measured. For example, if a piece of text contains 100 unique words it has a vocabulary size of 100 words. A larger vocabulary size generally indicates a richer or more varied use of words, but it doesn’t provide insight into how these words are distributed throughout the text.</p><p>Lexical diversity can also be used to measure the variety of word usage within a text, reflecting the balance between unique and repeated words.It considers both vocabulary size and word repetition, so two texts could have the same vocabulary size but different lexical diversity depending on how often each word is repeated.</p><p>Finally, where the output is structured such as JSON, testing for valid JSON, inspecting the number of values etc. might provide indications of correctness of the output quickly.</p><p><strong>LLM as Judge: LLMs Evaluating LLMs</strong></p><p>Similar to the ‘Resource Overboard’ approach to generating evaluation data, the concept of LLM-as-Judge uses a more capable (and more expensive) LLM to evaluate the output of another typically smaller and cheaper LLM. This approach allows for the evaluation of open-ended responses because it doesn’t necessarily require reference answers ahead of time, especially when using a more capable model. For example, some models (e.g. GPT-4) can operate without reference answers / examples while others (e.g. GPT-3.5) which may be cheaper and less performance can perform similar evaluations if provided examples [<a href="https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG">ref</a>].</p><p>The basic approach is that there is the LLM-application that needs to be evaluated and then a second evaluator LLM. The second LLM takes in three things: the user prompt, the response from the LLM being evaluated, and an evaluation prompt. The evaluator LLM then generates an evaluation score.</p><p>This approach can approximate human evaluation with LLMs as long as we have the right prompt. In our own work, we have seen it used for measuring creativity, helpfulness, readability and groundedness (testing that the information in the answer is based on a given context / the user input or not).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2V4yzcR4SaNG9MX1bwfXw.jpeg" /><figcaption>Figure 3: LLM as evaluator.</figcaption></figure><p><strong>Implementing Evaluations.</strong></p><p>In our view, it is unlikely that there will be ‘one metric’ that makes sense for any particular use case and teams will likely end up using a mix of metrics. For example, for measuring summarization performance of an LLM, a team might use both property based metrics (length of output text is less than input text) as well as the popular ROUGE. For Retrieval Augmented Generation use cases, the test might first check the LLM output length to observe if the LLM has ‘rambled’ before applying a LLM-as-Judge metric such as RAGAS.</p><p>In each case, the benefit is the use of an easy to calculate heuristic as a ‘sanity check’ before applying the more computationally expensive metric.</p><p>In terms of implementing simple property-based metrics such as word or token counts, those metrics can just be calculated in Python directly. For traditional NLP evaluation metrics HuggingFace’s Evaluate library [<a href="https://huggingface.co/docs/evaluate/en/index">link</a>] provides support for traditional metrics such as ROUGE and BLEU, as well as for getting access to pre-trained models such as those models that <a href="https://huggingface.co/spaces/evaluate-measurement/toxicity">detecti toxicity</a>.</p><p>For “LLM-as-Judge’’ evaluations, there are a number of open source options to consider including: RAGAS [<a href="https://github.com/explodinggradients/ragas">link</a>], Deepeval [<a href="https://github.com/confident-ai/deepeval">link</a>] (which includes RAGAS and 13 other metrics), Uptrain [<a href="https://github.com/uptrain-ai/uptrain">link</a>]. Some of these evaluation frameworks, such as <a href="https://github.com/promptfoo/promptfoo">Promptfoo</a>, span both LLM-as-Judge and more deterministic metrics, including whether or not the LLM output contains certain words [<a href="https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic">ref</a>].</p><p>There are also a growing number of platforms emerging to tackle the challenge of end-to-end LLM evaluations including <a href="https://www.langchain.com/langsmith">Langsmith</a>, <a href="https://langfuse.com/">Langfuse</a>, <a href="https://www.rungalileo.io/">Galileo</a> and <a href="https://deepchecks.com/">Deep Checks</a>.</p><p>For example, Galileo’s Evaluation Intelligence Platform for AI. Galileo’ platform is intended to help teams evaluate model performance, detect hallucinations, optimize prompts with the goal of improving production reliability [<a href="https://docs.rungalileo.io/galileo?__hstc=153249639.fda370877aafe458524e1b1ebfb6d2b5.1729470143350.1729470143350.1729470143350.1&amp;__hssc=153249639.3.1729470143350&amp;__hsfp=3990537976">ref</a>]. Modules include experimentation tools, real-time monitoring, and AI protection. More information is available at <a href="http://galileo.ai/">galileo.ai</a>.</p><p>CleanLab is another platform that provides developers an API as part of its CleanLab Studio that wraps LLM interactions with its “Trusted Language Model” (TLM) [<a href="https://help.cleanlab.ai/tlm/tutorials/tlm/">ref</a>] in order to provide a confidence score for any LLM response [<a href="https://help.cleanlab.ai/tlm/tutorials/tlm_custom_model/">ref</a>], which can be used to automate screening of erroneous responses.</p><p>For developers using LLM-application development frameworks such as LangChain, LlamaIndex, DSPy and Haystack, there are a range of options for incorporating evaluations including. These options include providing built-in evaluation capabilities, the ability to write custom evaluations, and in most cases integrations with evaluation frameworks such as DeepEval or other third party evaluation frameworks [<a href="https://docs.llamaindex.ai/en/stable/examples/evaluation/Deepeval/">ref</a>] [<a href="https://haystack.deepset.ai/integrations/deepeval">ref</a>] [<a href="https://www.promptfoo.dev/docs/configuration/testing-llm-chains/">ref</a>].</p><p>In our view the market is early and teams should stay close to developments over the near term including evaluating multiple platforms to see which one fits a particular use case.</p><p><strong>LLM-as-Judge considerations. </strong>While the LLM-As-Judge approach shows promise, it is an active area of research and development. Potential issues include bias in the judge model due to lack of knowledge on the part of the Judging LLM. For example if the LLM acting as judge has not been sufficiently exposed to domain specific-knowledge it might incorrectly penalize a fine-tuned model’s outputs. Further, the judge LLM might have its own inherent biases or preferences based on its training data, making it less sensitive to the strengths (and weaknesses) of the specialized model.</p><p><strong>Conclusion</strong></p><p>As teams move from a purely experimental approach to an approach where LLM-applications are being moved towards production deployments, we think it is important to take an evaluation / test driven approach to LLM application development. Once in production, evaluations should continue in a continuous fashion. With each prompt or model change, LLM performance should be reevaluated and user adoption and user feedback should be closely monitored to improve the system. As with many aspects of Generative AI right now, given the pace of change in the field at the time of writing, we think teams should take a ‘crawl / walk / run’ approach. This approach helps teams quickly learn, stay close to new developments and switch tools and techniques for evaluations as necessary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=447313621ab1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/measurably-correct-strategies-for-llm-evaluation-447313621ab1">Measurably Correct: Strategies for LLM Evaluation.</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2023: The Year of LLMs]]></title>
            <link>https://medium.com/georgian-impact-blog/2023-the-year-of-llms-3d73bec1e511?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d73bec1e511</guid>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[inference-engine]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 17:35:46 GMT</pubDate>
            <atom:updated>2024-04-05T17:35:46.370Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/benjaminye/">Benjamin Ye</a>, <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a>, <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a> &amp; <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a></p><figure><img alt="An abstract, futuristic and etheral representation of a large language model using a vibrant color palette to further explore the theme of advanced technological networks, illustrating their complexity and elegance." src="https://cdn-images-1.medium.com/max/1024/1*7gKfp7WXa1dBMYKElf3W_A.png" /><figcaption>This image was generated using DALL-E 3.</figcaption></figure><h3>Introduction</h3><p>In 2023, the AI industry and the field of language models in particular, progressed significantly. Following the release of ChatGPT in late 2022, we saw the release of larger, more capable models like GPT-4, Bard and Claude. Tucked away from the spotlight there exists a thriving ecosystem of smaller, open-source models.</p><p>We believe that these smaller models are not necessarily in competition with commercial models; rather, the two complement each other. Whereas commercial models do well with generalized tasks, smaller models excel in specialized tasks. Moreover, given their open-sourced nature, smaller models can be fine-tuned to suit any downstream task.</p><p>The potential of using smaller open-source models to perform domain-specific tasks led us to investigate methods for fine-tuning and serving them in production. In this blog post, we will go over our high-level learnings and also revisit our earlier works related to open-source models.</p><h4>Our Research in 2023</h4><p>In 2023, we delved into the capabilities of popular open-source LLMs. In our experiments, we tested model performance uplift after fine-tuning, as well as latency and cost when these models are put into an inference server. Experiment results are documented for <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">Flan-T5</a>, <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d">Falcon</a>, <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-redpajama-340ae8ea96ec">RedPajama</a>, and <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-llama-2-cdf21d540ce3">Llama 2</a>.</p><p>We also ran a comparison study of the three 7B-parameter models that were considered industry-leading at the time of publication: <a href="https://medium.com/georgian-impact-blog/experimenting-with-fine-tuning-llama2-mistral-and-zephyr-25760895d8ce">Llama 2, Mistral 7B, and Zephyr</a>.</p><p>Lastly, we investigated the <a href="https://medium.com/georgian-impact-blog/exploring-delivery-methods-for-llms-to-customers-e6eeed5769e3">impact of the choice of GPU and inference optimization</a> on the performance (namely throughput and latency) of the inference endpoint.</p><p>In this blog post, we have condensed the main lessons learned from our experiments into two sections: fine-tuning and inference.</p><h3>Fine-tuning</h3><figure><img alt="LLM development timeline. The models below the arrow are closed source while those above the arrow are open source." src="https://cdn-images-1.medium.com/max/1024/1*q5mcBlwgO8WJc4davvsjNg.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2311.16989">Chen et. al</a></figcaption></figure><p>Looking forward to 2024, some speculate that the performance of <a href="https://x.com/ClementDelangue/status/1729158744762626310">OS-LLMs will reach parity with closed-source, commercial alternatives</a>. Indeed, <a href="https://arxiv.org/abs/2311.16989">there is some indication that OS-LLMs are catching up</a> in general performance capabilities as shown by benchmarks such as MT-Bench. When trained in specialized domains (such as coding), open-source models appear to be capable of surpassing their closed-source counterparts, as measured by benchmarks such as HumanEval. These developments are expected to assist in developing enterprise applications where users seek models that are:</p><ul><li>Specialized to complete downstream tasks as opposed to generalized chat;</li><li>Hosted on custom infrastructure where enterprises have control of own data; and</li><li>Cheaper and faster to train and host compared to commercial APIs</li></ul><h4>Observations</h4><p>We have fine-tuned leading OS-LLMs in the 7B- to 30B-parameter space for summarization and classification using QLoRA technique outlined by this<a href="https://arxiv.org/abs/2305.14314"> paper</a>. Our results are discussed below:</p><p><em>Summarization Fine-Tuning Results</em></p><p>We used the <a href="https://aclanthology.org/D19-5409/">Samsum</a> dataset to evaluate dialogue summarization capabilities and observed that open-source models perform fairly well out-of-the-box, but tend to underperform closed-source models such as GPT3.5 and Jurassic J2 x. After fine-tuning, the OS-LLMs in the 7B to 13B parameter spaces performed better than the untuned baselines of tested open and closed-source models.</p><figure><img alt="Text Summarization Fine-Tuning Results." src="https://cdn-images-1.medium.com/max/938/1*asyiBab8TRntQBxCCEEzOA.png" /><figcaption><em>Text Summarization Fine-Tuning Results. Source: Georgian’s LLM Finetuning Hub</em></figcaption></figure><p><em>Text Classification Fine-Tuning Results</em></p><p>For text classification, we used the <a href="http://qwone.com/~jason/20Newsgroups/">20 Newsgroups</a> dataset. We fine-tuned OS-LLMs using different training set sizes (For illustration please see the results table below). We observed that LLMs perform better than BERT in a low-data regime, potentially demonstrating that LLMs can generalize their pre-trained knowledge to new domains. Among the OS-LLMs we tested, Llama-2–13B performed better than its peers. Llama-2–13B also appears to be more sample-efficient. When we used the entire <a href="http://qwone.com/~jason/20Newsgroups/">20 Newsgroups</a> training set, Llama-2–13B performed similarly to a fine-tuned GPT-3.5. This performance is in comparison to other OS-LLMs which achieved accuracies in the 72–75% range, in-line with GPT-3.5 after it was fine-tuned with 2.5% of the training set.</p><figure><img alt="Text Summarization Fine-Tuning Results." src="https://cdn-images-1.medium.com/max/846/1*3qOdzuhHqrHGwrSsyLDDpQ.png" /><figcaption>Text Classification Fine-Tuning Results. Source: Georgian’s LLM Finetuning Hub</figcaption></figure><h4>Performance</h4><h4>Performance</h4><p>OS-LLMs demonstrated a significant uplift in performance after fine-tuning for a particular downstream task. In most cases, fine-tuned OS-LLMs performed better than untuned commercial LLMs — which we view as remarkable given that the size of an OS-LLM is smaller.</p><p>We also tested different methodologies and hyperparameter settings in our experiments and found the following:</p><p><em>Target Modules</em></p><p>By default, Hugging Face’s <a href="https://huggingface.co/docs/peft/index">PEFT</a> library only generates low-rank adaptors for the attention layers (<a href="https://github.com/huggingface/peft/blob/b4faffea8ae031e5bd69a76b55418b3650c04c80/src/peft/utils/other.py#L481">query and value, in particular</a>). This default setting can be overridden by supplying desired target_modules in PEFT’s <a href="https://huggingface.co/docs/peft/en/package_reference/lora#peft.LoraConfig">LoRAConfig</a> object.</p><p>When testing the Zephyr model, we ran one set of experiments with the default target_modules= [“q_proj”, “v_proj”].</p><p>For another set of experiments with Zephyr, we used target_modules=[“q_proj”,”k_proj”,”v_proj”,”o_proj”,”gate_proj”,”up_proj”,”down_proj”,”lm_head”].</p><p>All transformer modules — and low-data regimes in particular — that we tested performed better after tuning. (See the table below.)</p><p>It appears that with full module tuning, Zephyr is able to converge faster. With only 2.5% of training samples, it is able to compete with attention-tuned Zephyr models that have seen 25% of samples. This performance uplift, however, comes with a caveat: we saw a ~5x increase in trainable parameters as a result of tuning all modules in the Zephyr model. This trade-off, however, seems worthwhile if compute and training time is not a concern, especially if one does not have a rich set of labeled data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*JX8P47mXLLe7YnJUPAd3tA.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><p><em>NEFTune</em></p><p>As part of our Zephyr experiments, we also tested whether <a href="https://arxiv.org/abs/2310.05914">NEFTune</a> was capable of increasing performance of an already fine-tuned model. NEFTune appears to improve model performance for those models that are fine-tuned using smaller amounts of data, as seen through the model’s rapid convergence at 2.5% and 5% sample fraction. As training data grows, however, models tuned with NEFTune alone seem to exhibit slower convergence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*TK_Zo-PvjUvGcyPsEI9vJw.png" /></figure><p><em>Prefix Tuning</em></p><p>We also tested <a href="https://huggingface.co/docs/peft/task_guides/seq2seq-prefix-tuning">prefix tuning</a>, and the results did not appear to be promising. For defined downstream tasks, QLoRA were able to achieve better and predictable improvements in performance.</p><h4>Takeaways</h4><p>Based on the results from our experiments, fine-tuning OS-LLMs with LoRA and QLoRA may be a useful method for adapting models to complete specialized downstream tasks. We have run experiments in both summarization and classification tasks and saw OS-LLM performance improvement in both areas following fine-tuning. In addition, OS-LLMs can be trained and served in custom environments without passing data on to a third party.</p><p>Finally, some of the most popular OS-LLMs such as Llama 2, Mistral, and Zephyr may be smaller in terms of parameter counts, but as a result, they may be faster and cheaper to train and serve. In sum, fine-tuning OS-LLMs may be a viable alternative for researchers who are currently relying on commercial APIs to complete specialized tasks.</p><h4>Fine-Tuning Results Table</h4><p>We have distilled our results into the following table as a starting point for researchers seeking to configure custom fine-tuning jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a1srjIQKPFJ9fhgeL1Rupw.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><h3>Inference Speeds</h3><p>A key challenge in deploying LLMs is achieving fast inference speeds. In our view, addressing inference involves answering several critical questions:</p><ul><li>What factors contribute to the slow performance of LLMs in production environments?</li><li>How can LLMs be effectively integrated into complex ML applications with multiple components?</li><li>What strategies can be employed to accelerate LLM inference?</li></ul><p>In this section, we will summarize our findings and insights on inference speeds.</p><h4>Observations</h4><p>Over the course of our experiments on inference speed we have observed that the size of the model, the choice of deployment platform and hardware appear to significantly influence performance. Typically, benchmark metrics improve with smaller model sizes, more powerful GPUs (such as using Nvidia A100 instead of Nvidia A10), and specialized inference servers, rather than custom-built web servers.</p><p>In our <a href="https://medium.com/georgian-impact-blog/exploring-delivery-methods-for-llms-to-customers-e6eeed5769e3">Inference blog post</a>, we discussed the potential capabilities of the new inference engine, vLLM. While servers like Ray and Triton offer better structure, organization and more efficient hardware utilization, some bottlenecks in the LLM inference may remain, such as memory management during attention computation. The memory management bottleneck can be effectively addressed by the PagedAttention mechanism.</p><p>One of our past experiments involved deploying a custom Llama 2 classification model using FastAPI, Ray, Text Generation Inference (TGI) and vLLM to investigate the server’s peak load capacity with varying model sizes and hardware configurations.</p><p>The server’s peak load capacity can be defined as the maximum number of requests that can be processed in one second. Although there’s no guarantee that the server will consistently process that many requests every second without the appropriate infrastructure, our benchmark results provided interesting insights into the relationship between hardware, serving platform and model size.</p><p>See the below charts demonstrating performance for two models: the Llama-2–7B classification model and the Llama-2–13B classification model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b7BKnmQaj6dKQiSwctaT9A.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><p>What conclusions have we drawn from our experiments?</p><ul><li>FastAPI and Ray (represented by the yellow and red plots) demonstrated significantly lower performance compared to TGI and vLLM (depicted by the white and green plots).</li><li>The peak number of requests for vLLM can be up to five times higher than for Ray. Latency decreases drastically with Nvidia A100 compared to Nvidia A10. Latency decreases from 15 seconds to less than five seconds per request for Llama-2–7B.</li></ul><p>The most noticeable difference in the plots lies between the performance of FastAPI/Ray and vLLM/TGI. This variance is attributed to the methods used to generate output tokens during inference. For FastAPI and Ray, we used the built-in generation method from Hugging Face’s <a href="https://huggingface.co/docs/transformers/en/index">transformers library</a>; for vLLM, we used a custom engine using the PagedAttention technique. We think that PagedAttention can more efficiently manage the key and value memory used by the attention mechanism, thereby significantly boosting performance.</p><h4>Performance</h4><p>The performance of LLM inference can be characterized by latency and throughput. Latency refers to the maximum time taken to process 90% of requests. Although we can send a certain number of requests per second, not all of them will be processed simultaneously. This limitation leads to the consideration of throughput — or the number of requests the server can process in one second.</p><p>To obtain these metrics, we used the benchmarking tool Vegeta. (For more information on Vegeta please see <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">our</a> <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d">previous</a> <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-redpajama-340ae8ea96ec">blog</a> <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-llama-2-cdf21d540ce3">posts</a> on the tool.) As previewed above, given that vLLM is the best performer when looking at inference latency and throughput, we used existing integrations of different serving platforms with vLLM. Some platforms (e.g.Ray) use vLLM directly as a library, while others (e.g. TGI) implement specific components like the custom CUDA kernels developed by vLLM in their version of PagedAttention.</p><p>Ultimately we discovered that the generation of tokens, i.e., text generation, occurs similarly regardless of the method of employment of vLLM provided that one employs PagedAttention to address memory bottlenecks. The table below presents our benchmark results for serving the Llama-2–7B classification model on various servers. Over 10 minutes, we sent a fixed number of requests (as indicated in the RPS column) to the server every second. The latency remains consistently under one second across all cases, and it’s noteworthy that the throughput value doesn’t vary significantly from the RPS value, indicating that the number of processed requests per second closely matches the number sent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFvuq6SI5wDJU1Oy9enucg.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><p>Moreover, we observed that the size of the model may influence the number of requests the server can process per second, but does not appear to influence the latency itself.</p><p>Throughput can be improved by strengthening the infrastructure around the LLM. For example, deploying the model on Amazon Sagemaker, which comes with more optimizations including several managed service capabilities like autoscaling, improves the model’s throughput. In a previous <a href="https://medium.com/georgian-impact-blog/exploring-delivery-methods-for-llms-to-customers-e6eeed5769e3">inference-dedicated blog post</a> we saw that it’s possible to send requests to the server over an extended period (one hour in our case) without causing the server to crash.</p><h4>Key Takeaways</h4><p>What features does each LLM server provider cover?</p><p>The following table outlines the most commonly observed features of different inference servers.</p><figure><img alt="Source: Georgian’s LLM Finetuning Hub" src="https://cdn-images-1.medium.com/max/1024/1*kibDwDU3BQiPNKWlukQxQQ.png" /></figure><p>vLLM is primarily used as a library offering an asynchronous LLM engine enhanced by PagedAttention for greater throughput. The <a href="https://blog.vllm.ai/2023/06/20/vllm.html">vLLM team</a> has also developed a FastAPI server utilizing this engine. Beyond these features, however, we think that vLLM offers limited functionality for more complex scenarios. vLLM shares this limitation with TGI.</p><p>Triton and Ray, on the other hand, may be more suitable for building complex ML applications. They share several core features in model management and deployment. Both platforms support concurrent model execution, allowing multiple models to run simultaneously for improved efficiency. They also utilize dynamic batching techniques to group inference requests and thereby enhance throughput. Triton and Ray are both framework agnostic and offer GRPC protocol support for model inference.</p><p>One of Ray’s distinguishing features is its effective resource allocation, which includes fractional resource allocation. This feature may be particularly useful when working with a limited number of GPUs/CPUs, as it allows precise allocation based on the actual needs of the models for inference. Furthermore, Ray performs well when serving multiple applications, making it a good option for scenarios where models or business logic are logically segmented. With Ray, business logic can be divided into distinct applications for organization and management.</p><p>When to use what?</p><p>For quick deployment of LLMs, one of the best methods, in our opinion, is a TGI server, which can be easily integrated into applications using a designated inference client from the Hugging Face Hub library.</p><p>If a company’s application consists of multiple services — for example, a speech recognition and synthesis system with LLMs being one of many components — then Ray paired with the vLLM engine may be a viable option. This combination can offer the advantages of model composition support provided by Ray and the strong serving throughput afforded by vLLM.</p><p>The Hugging Face LLM Inference container with Text Generation support deployed on Amazon Sagemaker may assist servers in handling a larger number of requests, thereby preventing servers from crashing under consistent load. Companies may also consider deploying applications via Ray Clusters onto cloud platforms such as AWS and GCP and manage the cluster via Kubernetes.</p><p>Cost</p><p>Most of our cost calculations were conducted for classification models where we were generating a small number of tokens. In general, the cost varies from $0.0002 — $0.0006 to process 1K tokens. The below table outlines our estimates for several LLMs deployed on the Amazon Sagemaker endpoint, which can be rented at an additional $2.03 per hour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*1ttHjs1Lbf_HefjRdC2_hw.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><p>Or, it is 63 times more expensive to classify 1M sentences using GPT-4 (as an example of commercial LLMs) rather than a custom LLM run on any of the aforementioned servers, according to our experiments. See the chart below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*8Pkb_tHg1xFIhT14bFutPQ.png" /><figcaption>Source: Georgian’s LLM Finetuning Hub</figcaption></figure><h3>Conclusion</h3><p>This post concludes our series of investigations into open-source LLMs. We hope that our findings can help accelerate the timeline and effectiveness of LLM experiments. Code used for the fine-tuning and inference benchmarks referenced herein can be found in Georgian’s <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub repo</a>.</p><p>Key takeaways related to fine-tuning include:</p><ul><li><strong>Fine-tuning with LoRA produces strong results. </strong>Smaller open-source models might not be as eloquent and knowledgeable out-of-the-box as their commercial counterparts. but, after fine-tuning for specific tasks, open-source models can adapt well with very limited training samples. With high-quality training data, the results may match GPT-3.5 and even GPT-4.</li><li><strong>Fine-tuned LLMs are more sample-efficient than BERT.</strong> Across our classification experiments, we observed that newer LLMs may adapt much faster than classic BERT models. While it took BERT 5,332 samples to reach 72% classification accuracy, it only took 1,066 samples for Llama-2–13B to reach the same level of performance.</li><li><strong>Hyperparameters matter. </strong>We have experimented with hyperparameters for fine-tuning, and we find that tuning all modules makes convergence faster. In addition, NEFTune can make LoRA tuning even more sample efficient.</li></ul><p>Key takeaways related to servers include:</p><ul><li><strong>Consider vLLM. </strong>We learned from our experiments that vLLM seems to have a throughput and overall inference speed advantage over other frameworks such as Hugging Face and FasterTransformers.</li><li><strong>Consider Ray (or Triton) for building complex ML applications.</strong> Ray may be a good option for building complex ML applications for users who have a system consisting of many services and components of which an LLM is only one part. With its clean and elegant method of declaring different services and small apps within a system and its smart allocation of resources, Ray may save users a significant amount of time in developing and deploying the final product.</li><li><strong>Experiment with LLM deployment. </strong>Modern tools and platforms reduce the complexity involved with leveraging LLMs in applications, with relatively little investment.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d73bec1e511" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/2023-the-year-of-llms-3d73bec1e511">2023: The Year of LLMs</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring delivery methods for LLMs to customers]]></title>
            <link>https://medium.com/georgian-impact-blog/exploring-delivery-methods-for-llms-to-customers-e6eeed5769e3?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6eeed5769e3</guid>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[llama-2]]></category>
            <category><![CDATA[redpajama]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 19:33:55 GMT</pubDate>
            <atom:updated>2024-02-13T19:53:33.164Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a> &amp; <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a></p><figure><img alt="An abstract image of a landscape of neural networks with connecting glowing nodes, visualizing the concept of ‘inference’." src="https://cdn-images-1.medium.com/max/1024/1*Z3rklOm2gPXlF-Q8vgchMw.png" /><figcaption>This image was generated using DALL-E 3.</figcaption></figure><p>Since the release of ChatGPT, companies have started to explore the potential of GenAI, especially Large Language Models (LLMs), to enhance their business processes, improve product quality, attract more clients and gain a competitive edge in the market.</p><p>We believe that the aim of many trained models is to become accessible for other people to use. Typically, if you are using closed-source LLMs with existing APIs, deployment concerns are minimal. However, with fine-tuned open-source LLMs, the challenge becomes selecting the appropriate way of delivering the LLM to customers.</p><p>If you have been following our series of blog posts about LLM fine-tuning, you might have noticed our efforts to deploy these models and assess the cost of inference.</p><p>In this blog post, we continue to delve into the world of open source tools that facilitate the deployment of machine learning models, including LLMs. Our aim is to provide a potential estimate of the costs involved in deploying your own LLMs using specific tools by comparing different options and explaining the benchmarks that guided our analysis.</p><p>When deploying machine learning models, we believe it is important to consider the capabilities of the developed application. In the case of LLMs, we think these questions are important to ask:</p><ul><li>How much will it cost to process 1,000 input tokens during a certain amount of time?</li><li>Will the users experience a long waiting time? How much time will it take to process one query and generate an answer?</li><li>How many requests per second can the server handle?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4soUchUGVeIXNT1W" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>In our opinion, factors such as (i) the choice of inference server, (ii) the hardware being used for inference (instance type), and (iii) model size matter. We’ll explore how in this blog.</p><h3>Tools and resources</h3><h3>Inference services</h3><p>If you have successfully fine-tuned your open-source model — what’s next? How do you make it accessible to the public?</p><p>The first option is to develop a server where you load the model, define the endpoint and process incoming requests. This process can be achieved using web frameworks such as FastAPI or Flask. However, in our view, there are significant limitations to this approach; these general-purpose web servers are not inherently designed for AI inference tasks. Features that boost inference such as GPU acceleration, dynamic batching or multi-model inference are not readily available by default.</p><p>Therefore, creating such a server on your own with all these features would require considerable human resources and time, particularly if you are at the beginning of your journey into the world of open-source LLMs and their deployment. For this reason, many companies have developed their own inference servers that support the features needed for hosting LLMs and can be relatively easy to use. We will compare efficiency, ease-of-use, how the server handles big workloads and how much support there is for various types of LLMs. Additionally, we will try to estimate the price in case of deployment of the text-generation model that was fine-tuned for a classification task.</p><p>During our experiments, we looked through various services that provide LLM inference capabilities. We looked at the following ones:</p><ul><li><a href="https://github.com/huggingface/text-generation-inference">Text Generation Inference (TGI)</a>: A toolkit developed by Hugging Face for deploying and serving LLMs. TGI enables high-performance text generation for the most popular open-source LLMs, including Llama-2, Falcon, StarCoder, BLOOM, GPT-NeoX and others.</li><li><a href="https://vllm.readthedocs.io/en/latest/">vLLM</a>: A fast and user-friendly library for LLM inference and serving. It can be easily used with Hugging Face model and provides continuous batching of incoming requests, optimized CUDA kernels and high throughput.</li><li><a href="https://vllm.readthedocs.io/en/latest/serving/deploying_with_triton.html">Triton server with vLLM backend</a>: An open source inference serving software that streamlines AI inferencing. With Triton, any AI model from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL and more, can be deployed.</li></ul><p>The Triton backend for vLLM is designed to run supported models on a vLLM engine. When using this backend, all requests are placed on the vLLM AsyncEngine as soon as they are received. Inflight batching and PagedAttention is handled by the vLLM engine.</p><ul><li><a href="https://github.com/ray-project/ray/blob/master/doc/source/serve/doc_code/vllm_example.py">Ray with vLLM support</a>: An open source framework that is helpful for scaling and serving Python and machine learning applications. Among numerous libraries dedicated for handling different parts of the ML lifecycle, Ray also has a library solely dedicated to serving, which is called Ray Serve. The advantage of RayServe is that it is optimized for serving LLMs, and what may be especially beneficial is that it is suited for model compositions and many models serving.</li></ul><p>In terms of vLLM support, it also uses vLLM AsyncEngine for processing incoming requests.</p><p>In our view, what sets TGI and vLLM apart is that, in addition to the features supported by most inference servers, they employ a unique technique called <a href="https://arxiv.org/abs/2309.06180">PagedAttention</a>, which allows efficient management of the attention key and value memory. Specifically, TGI’s implementation of PagedAttention utilizes the custom CUDA kernels developed by <a href="https://github.com/vllm-project/vllm">vLLM</a>.</p><p>Therefore, PagedAttention, in our view, is one of the strongest options to consider for the fast inference for LLMs. Besides using pure vLLM for the inference, additional infrastructure may be needed around models in order to create a more robust system. Many inference servers, such as Triton and Ray, for example, have added integrations with vLLM.</p><h3>Models</h3><p>In this section, we will look at the following models: LLama2 and RedPajama.</p><h4><a href="https://georgian-io.medium.com/the-practical-guide-to-llms-llama-2-cdf21d540ce3">LLaMA-2</a></h4><p>LLaMA-2 is a family of LLMs released by Meta AI. It is available in three variants, each with 7, 13 and 70 billion parameters. While its architecture mirrors that of LLaMA-1, the foundational training for LLaMA-2 utilized 40% more data. Specifically, these LLaMA-2 base models were trained on a 2 trillion-token dataset and refined to exclude websites known for disclosing personal information. Additionally, Meta AI aimed to prioritize more reliable and trustworthy sources in the dataset.</p><h4><a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-redpajama-340ae8ea96ec">RedPajama</a></h4><p><a href="https://together.ai/blog/redpajama-models-v1">RedPajama-INCITE </a>combines Together.ai’s RedPajama dataset and EleutherAI’s Pythia model architecture to form an open source LLM. In our view, one interesting aspect of the RedPajama models family is that there is a 3B parameter size model, which is unusual in our experience. <a href="https://together.ai/blog/redpajama-models-v1">Together.ai</a> reasons that this model size may allow for wider adoption due to smaller hardware requirements and easier experimentation. Together.ai introduces and uses the RedPajama dataset, which is a <a href="https://www.together.ai/blog/redpajama">1.2T token </a>open-source replication of the LLaMa dataset. That is, they follow the same steps in terms of pre-processing and filtering, use the same data sources and extract roughly the same number of tokens from each.</p><p>During this benchmark, we wanted to see how model size influences inference performance. We examined various versions of these models, including:</p><ul><li>LLaMA-2–7B</li><li>LLaMA-2–13B</li><li>RedPajama-3B</li><li>RedPajama-7B</li></ul><p>We selected models fine-tuned using the LoRA method as we previously described in our series of blog posts about LLMs finetuning. Specifically, we were benchmarking models fine-tuned for the classification task on <a href="http://rungalileo/20_Newsgroups_Fixed">this dataset</a>.</p><h3>Hardware</h3><p>We selected AWS as our primary platform for running instances, opting for the g5.4xlarge instance that comes with the Nvidia A10 GPU with 24GB of memory. However, due to a shortage of GPUs and the particular challenge of getting A100 GPUs on AWS, we also used Google Cloud. There, we successfully obtained instances equipped with the Nvidia A100 GPU, with 40GB of memory for a more robust comparison.</p><h3>Benchmark</h3><h3>Latency and Throughput</h3><p>When deploying any machine learning model, we believe it is important to consider the number of users a server can accommodate. If the server isn’t sufficiently optimized, it may struggle to handle consistent batches of incoming requests over extended periods. Thus, in our tests, we sought to simulate “attacks” on the server, mimicking real-world production scenarios. By simulating “attacks,” we could evaluate how effectively each server handles substantial workloads.</p><p>For this purpose, we conducted our benchmark tests using a tool named <a href="https://github.com/tsenart/vegeta">Vegeta</a>. Vegeta is a versatile HTTP load-testing tool designed to target HTTP services with a steady stream of requests. With Vegeta, we were able to set the test’s <strong>duration </strong>(how many seconds a test runs), <strong>rate</strong> (number of requests sent each second) and dynamically provide different inputs using a JSON file. During a test, Vegeta would attack a specific server with a predetermined number of requests every second for the set duration, subsequently providing comprehensive metrics. Specifically, we focused on:</p><ul><li><strong>Latency (90%):</strong> The time it takes to get a response for most (~90%) of the requests. It means that nine out of 10 responses will be faster than this time.</li><li><strong>Throughput</strong>: The number of requests a server can process in one second.</li></ul><p>One of the challenges was finding the optimal duration and rate with which we were sending requests. In the optimal case, we were trying to send a certain number of requests for at least 10 minutes. When the server was crashing under such load, we reduced the duration value and, if necessary, the rate with which we sent the requests. We ran each test three times and found the average value for latency and throughput in order to make the benchmarking process more fair.</p><h3>Benchmark results</h3><p>For each of our requests, we picked random sentences from the Hugging Face dataset, trimming it to the length that would be equal to ~100 tokens and sending it to the server.</p><h4>Text Generation Inference and vLLM</h4><p>In general, TGI and vLLM implement the same range of features as the PagedAttention mechanism; therefore, their performance is pretty similar. It’s noteworthy that in our benchmark tests, the TGI service handled twice as many requests as the vLLM. For instance, when utilizing the same Nvidia A100 GPU, the TGI enabled us to process 40 requests per second for the Llama-7B model, whereas with vLLM, we could only manage 20 requests per second.</p><p>In the table shown below, we tried different combinations of GPUs and servers for deployment. The throughput and RPS (requests per second) value are similar, meaning that the overall time it takes for a request to be processed appears to be relatively small.</p><p><strong>Benchmark results for Text Generation Inference</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*vHj-Gou1MZhoCCCYnkPNyw.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p><strong>Benchmark results for vLLM</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*EsMGUL5LhYt1jWIwsH_BaQ.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p><strong>Ray with vLLM support</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*vHj-Gou1MZhoCCCYnkPNyw.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p><strong>Triton Inference Server with vLLM backend</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*vHj-Gou1MZhoCCCYnkPNyw.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>The performance for Triton Inference Server and Ray is comparable to the plain TGI and vLLM, where the server was able to handle 10 requests per second during 10 minutes.</p><p>This performance leads us to our opinion that using just the inference platform on its own is not enough to provide an ideal user experience. As noted during our previous benchmarks, we were not able to keep the server alive for more than 10 minutes while sending from 10 to 60 requests depending on the model size and hardware. The reason could be the lack of sufficient resource allocation, effective load balancing and an optimized runtime environment tailored for serving LLMs.</p><p>There can be different ways of building a more robust system. For example, in terms of Ray, <a href="https://docs.ray.io/en/latest/cluster/getting-started.html">Ray Clusters</a> may be used to run Ray applications on multiple nodes. Ray also provides native cluster deployment support on AWS, GCP and Kubernetes.</p><p>In our experiments, we deployed some of the models on Amazon SageMaker endpoint using Hugging Face Deep Learning Container (DLC) for inference, which is powered by TGI. This way we could use the optimizations that come with TGI while also working to improve the performance with SageMaker’s managed service capabilities, such as autoscaling.</p><h3>Hugging Face LLM Inference Container for Amazon SageMaker</h3><p>We performed benchmarking using the popular load testing tool <a href="https://locust.io/">Locust</a>. We adjusted the settings to create 10 users at a time, who would all send requests continuously without any breaks. With this setup, each user started sending requests. If the server and network were fast enough, we could get more than 10 requests per second, since each user might make more than one request in that time.</p><p>As a result, we were able to successfully send more than 10 requests per second to the server for the duration of one hour for LLama-2–7B on the AWS instance powered by Nvidia A10 GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gbfcd2Tbj6ZPvZVn" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><h3>Cost</h3><h3>Cost calculation</h3><p>In our previous blog posts, we tried to calculate the cost for inference based on the peak RPS, but this time, we decided to adjust our formula and link it to a throughput value (in this context, it is a number or processed tokens per second).</p><p>If you look at the explanations of the formula written below, you can see that the throughput value depends on the total number of requests the users sent (for the classification task, the model was generating approximately 6 tokens for the input of roughly 100 tokens):</p><p><em>Number of tokens processed = input tokens + output tokens = 100 + 6 = 106</em></p><p><em>Throughput (tokens / s) = Total number of requests * Number of tokens processed / Duration</em></p><p><em>Time to process 1K tokens (min) = 1K / Throughput (tokens / s) / 60</em></p><p><em>Cost to process 1K tokens per hour = Time to process 1K tokens / 60 * Instance cost (per hour)</em></p><p>The process of identifying the specific number of requests mirrored the approach we used during the benchmarking. In this setup, we had 10 users actively sending requests for an hour, with Locust adjusting the frequency of these requests based on the server’s response capabilities.</p><p>By using such calculations, we aimed to imitate the way the inference cost was calculated in <a href="https://www.philschmid.de/sagemaker-llama-benchmark">this blog post</a> written by the Hugging Face team.</p><p>Per the table below, the total number of requests for RedPajama 3B is bigger than for LLama-2–7B and RedPajama 7B, which means that deploying a model trained on fewer parameters may be cheaper ($0.0003 for RedPajama 3B compared to $0.0006 for RedPajama 7B).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*ntpnrE1Yhkbcfx4xdw-FKg.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>We also compared the cost of classifying one million sentences (assuming each contains approximately 100 tokens) using our fine-tuned models with those of GPT-3.5 Turbo and GPT-4.</p><p>As shown in the bar chart presented below, the price is more than three times lower compared to GPT-3.5 Turbo and 63 times lower when compared to GPT-4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2duphxMIOVXoaSII" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><h3>Conclusion</h3><p>In conclusion, here are the takeaways of our views based on our experiments.</p><ul><li>Both TGI and vLLM demonstrate similar performance levels, exhibiting noticeable effectiveness where latency 90% is typically below one second in most scenarios.</li><li>We believe that Nvidia A100 is better than Nvidia A10 as it can serve <strong><em>twice</em></strong> as many requests at the same time when compared to the A10.</li><li>It appears that the smaller the LLM, the higher the number of requests it can serve. RedPajama 3B achieves 30 requests/second on TGI with Nvidia A10 GPU, compared to RedPajama 7B’s 20 requests/second with the same configuration.</li><li>Standalone inference platform appears to fall short in managing heavy user traffic, as seen from earlier load tests, such as TGI and vLLM. We were unable to process incoming requests beyond 10 minutes under moderate loads (likely due to insufficient resource allocation and load management).</li></ul><p>In the end, we deployed some of our models on Amazon SageMaker endpoint with Hugging Face DLC and observed stable performance and ability of the server to handle a large amount of requests.</p><p>Regarding the cost, our calculations demonstrate that using fine-tuned models for classifying one million sentences, each roughly containing 100 tokens, is more cost-effective. Specifically, it costs over three times less than using GPT-3.5 Turbo and is 63 times more affordable than employing GPT-4 for the same task.</p><p>The codebase for serving options is available in our <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM-Fine Tuning-Hub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6eeed5769e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/exploring-delivery-methods-for-llms-to-customers-e6eeed5769e3">Exploring delivery methods for LLMs to customers</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimenting with fine-tuning: Llama 2, Mistral and Zephyr]]></title>
            <link>https://medium.com/georgian-impact-blog/experimenting-with-fine-tuning-llama2-mistral-and-zephyr-25760895d8ce?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/25760895d8ce</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[mistral]]></category>
            <category><![CDATA[zephyr]]></category>
            <category><![CDATA[llama-2]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 21:43:10 GMT</pubDate>
            <atom:updated>2024-01-16T21:40:24.033Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/benjaminye/">Benjamin Ye</a> &amp; <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a></p><figure><img alt="A cartoonish cover art depicting three llamas engaging in a playful battle in a field." src="https://cdn-images-1.medium.com/max/1024/1*E0Hpz_ZIu6RloIyLA1Lu6g.png" /><figcaption>This image was generated using DALL-E 3.</figcaption></figure><p>The world of LLMs evolved quickly in 2023. In July, Meta made big news in the LLM world by releasing its open-access <a href="https://ai.meta.com/llama/">Llama 2</a> model. Three months later, Mistral AI released its <a href="https://mistral.ai/news/announcing-mistral-7b/">7B model</a>, claiming to be leaner and smarter than models with greater parameter counts, and purported to beat Llama2–13B on several benchmarks. In November, Hugging Face’s H4 team produced a fine-tuned version of Mistral, <a href="https://huggingface.co/collections/HuggingFaceH4/zephyr-7b-6538c6d6d5ddd1cbb1744a66">Zephyr 7B</a>, with reported <a href="https://arxiv.org/abs/2310.16944">MT-Bench</a> result that even surpasses that of the Llama2–70B, which up to that point had been <a href="https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard">considered by some to be one of the largest and best-performing models to date</a>.</p><p>Each of these models improved upon one another with refinements such as <a href="https://arxiv.org/abs/2310.06825">tweaking the self-attention mechanism</a> and <a href="https://arxiv.org/abs/2310.16944">distilling knowledge from a larger LLM</a>. Moreover, these smaller, open-access models allow researchers and hobbyists alike to run them on their own infrastructure — even locally with a consumer-grade GPU. These models can also be fine-tuned to enable them to perform specialized downstream tasks — with many anecdotal reports of <a href="https://ragntune.com/blog/gpt3.5-vs-llama2-finetuning">comparable</a> <a href="https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder">results</a> to closed-source OpenAI models at a <a href="https://news.ycombinator.com/item?id=37484135">fraction of the cost and inference time</a>.</p><p>With so many different models, it is hard to choose which one to start our experiment with. To help with the decision, we have done a systematic benchmarking of different models under different tasks. In this blog, we’ll cover:</p><ul><li><strong>Model highlights</strong>: What sets these models apart from another?</li><li><strong>Out-of-the-box benchmarks</strong>: How well do these models do on their own?</li><li><strong>Fine-tuning benchmarks</strong>: How well do these models adapt to downstream tasks?</li><li><strong>Access our experiments in our GitHub repository</strong>: How do I fine-tune with my own data?</li></ul><p><strong>Model Highlights</strong></p><p><em>Llama 2</em></p><p><a href="https://arxiv.org/abs/2307.09288">Llama 2</a>, released by Meta, is an update with improved performance to its predecessor, Llama 1. The model uses a standard transformer architecture with these key features:</p><p><strong>Model varieties</strong>: The base model is offered in a range of parameter sizes — <a href="https://huggingface.co/meta-llama/Llama-2-7b-hf">7B</a>, <a href="https://huggingface.co/meta-llama/Llama-2-13b-hf">13B</a> and <a href="https://huggingface.co/meta-llama/Llama-2-70b-hf">70B</a> — as well as a <a href="https://huggingface.co/meta-llama/Llama-2-70b-chat-hf">version</a> fine-tuned and aligned via supervised fine-tuning and reinforcement learning through human feedback (RLHF).</p><p><strong>Pre-training corpus</strong>: The model is <a href="https://ai.meta.com/resources/models-and-libraries/llama/">pre-trained on a corpus with 2 trillion tokens, 40% more than Llama 1</a>.</p><p><strong>Context length</strong>: the model supports a context length of<a href="https://ai.meta.com/resources/models-and-libraries/llama/"> 4,096 tokens, doubling from that of Llama 1</a>.</p><p>We have written an overview and evaluation of Llama 2 in our previous blog post. To learn more, you can access it <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-llama-2-cdf21d540ce3">here</a>.</p><p><em>Mistral-7B</em></p><p><a href="https://arxiv.org/abs/2310.06825">Mistral-7B</a> was released by Mistral AI. The model’s key differentiating factor is the use of Sliding Window Attention (SWA), proposed by Beltagy et al. in their <a href="https://arxiv.org/abs/2004.05150">Longformer paper</a>. According to the paper, the use of SWA allows the model to be more efficient in both memory and compute. In contrast to the standard Global Attention mechanism, where the number of computational operations increases quadratically in relation to the length of the input, the SWA method has a linear growth in operations as the input length increases. Another benefit of using this attention mechanism is that the model can retain more information from the past, thereby handling longer sequences more effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*ZDjqnG5haID-7LZWrrrMxw.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2310.06825">Jiang et al.</a></figcaption></figure><p>As a result, the model showcased impressive benchmark results in its <a href="https://arxiv.org/abs/2310.06825">paper</a>: beating Llama-2–7B in all benchmarks and the larger Llama-2–13B models in the majority of benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*HPDPhcALvuefjifPToMycg.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2310.06825">Jiang et al.</a></figcaption></figure><p><em>Zephyr-7B</em></p><p><a href="https://arxiv.org/abs/2310.16944">Zephyr-7B</a> was released by Hugging Face’s H4 team. The model is a fine-tuned version of Mistral-7B. The key contribution of the model is the demonstration of the effectiveness of using larger teacher models to generate synthetic datasets upon which the model fine-tunes on.</p><p>For its fine-tuning step, Zephyr used <a href="https://huggingface.co/datasets/stingning/ultrachat">Ultrachat</a> dataset generated from ChatGPT and further aligned using GPT-4-generated <a href="https://huggingface.co/datasets/openbmb/UltraFeedback">UltraFeedback</a> via <a href="https://arxiv.org/abs/2305.18290">Direct Preference Optimization</a> (DPO).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*yewuynB_1zAzeADP-GJsow.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2310.16944">Tunstall et al.</a></figcaption></figure><p>This combination allowed Zephyr to essentially distill knowledge from the teacher models, indirectly learning human preferences the larger models were trained on via RLHF.</p><p>From its <a href="https://arxiv.org/abs/2310.16944">paper</a>, Zephyr’s team reported SOTA MT-Bench results in the 7B model space, even beating most of the open-access models that are 10x bigger. The model also achieved comparable results versus closed-source commercial models like GPT-3.5, demonstrating the effectiveness of its fine-tuning technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*y-GH1kULLtVeyfhiZ1Iqrg.png" /><figcaption>Source: <a href="https://arxiv.org/abs/2310.16944">Tunstall et al.</a></figcaption></figure><p><strong>Out-of-the-Box Benchmarks</strong></p><p>To analyze how these models compare, we evaluated the performance through summarization tasks. We used the Samsum dataset — the corpus consists of 16,369 pairs of dialogues and their accompanying human-labeled gold-standard summaries. To evaluate model performance, we generated summaries on Samsum’s test set and evaluated results using standard <a href="https://aclanthology.org/W04-1013/">ROUGE</a> metrics. In particular, we used ROUGE-1 and ROUGE-2 which measures unigram and bigram overlaps, respectively, between the generated output and the gold label.</p><p><em>Environment Setup</em></p><p><strong>Instance Type: </strong>AWS EC2 g5.2xlarge</p><p><strong>GPU: </strong>1x A10G GPU (24GB VRAM)</p><p><strong>Model Weight Quantization: </strong>4-bit (NF4)</p><p><strong>Calculation Precision: </strong>16-bit (bfloat16)</p><p><em>Summarization Experiment Setup</em></p><p><strong>Dataset: </strong><a href="https://arxiv.org/abs/1911.12237">Samsum</a></p><p><strong>Prompt Template:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*kqh-CC-z2w3zI9bbfFWiKQ.png" /></figure><p><em>Summarization Experiment Results (Zero-Shot &amp; Few-Shot)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*jU2LurkseBZeo--D1eGNIA.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>As we can see from the results, they match our expectations as the newest Zephyr model performed better than both Llama 2 and Mistral in zero-shot generation. However, we did not see as meaningful of a performance uplift from Zephyr with few-shot learning compared to Mistral. Remarkably, all of the 7B open-access models performed in line with the closed-source GPT-3.5-Turbo model offered by OpenAI.</p><p>We were surprised to see the performance of the 13B Llama 2 model lagging behind the rest. However, we noted that ROUGE metrics do not account for nuance such as synonyms.</p><p><strong>Fine-Tuning Benchmarks</strong></p><p>To gauge how well these models adapt to the style of the gold label summaries after fine-tuning, we used the following fine-tuning setup:</p><p><em>Summarization Fine-Tuning Setup</em></p><p><strong>Fine-Tuning Method</strong>: <a href="https://arxiv.org/abs/2305.14314">QLoRA</a></p><p><strong>Learning Rate</strong>: 2e-4</p><p><strong>Dropout</strong>: 0.1</p><p><strong>Rank</strong>: 64</p><p><strong>Alpha</strong>: 16</p><p><strong>Epochs</strong>: 1</p><p><strong>Target Modules</strong>: Attention Modules Only</p><p><em>Summarization Fine-Tuning Result</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*EwatFlsutydhX5Mne22UKA.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>We saw a marked increase in performance after QLoRA fine-tuning, significantly surpassing the summarization performance. Mistral-7B performed better than Llama 2 and Zephyr after fine-tuning. The open-access models continue to hold their own after fine-tuning.</p><p>We noted the drastic improvement of ROUGE metrics from Llama2–13B after fine-tuning, affirming our earlier hypothesis of artificially low scores due to non-conformance of the syntactic style and word choice.</p><p>To further evaluate the effectiveness of QLoRA fine-tuning, we devised another experiment — this time with classification as the downstream task. We used the <a href="https://archive.ics.uci.edu/dataset/113/twenty+newsgroups">newsgroup dataset</a>: 18,828 messages with 20-class labels separated into a train and test set. We conducted an ablation study to see how quickly each model learns the classification task. We fed the model progressively more training examples and evaluated how well the model performed using accuracy metrics at each sample fraction.</p><p><em>Classification Fine-Tuning Setup</em></p><p><strong>Fine-Tuning Method</strong>: <a href="https://arxiv.org/abs/2305.14314">QLoRA</a></p><p><strong>Learning Rate</strong>: 2e-4</p><p><strong>Dropout</strong>: 0.1</p><p><strong>Rank</strong>: 8</p><p><strong>Alpha</strong>: 16</p><p><strong>Epochs</strong>: 5</p><p><strong>Target Modules</strong>: Attention Modules Only</p><p><em>Classification Experiment Setup</em></p><p><strong>Dataset: </strong><a href="https://archive.ics.uci.edu/dataset/113/twenty+newsgroups">newsgroup</a></p><p><strong>Prompt Template:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*ZkmXoVoKrO9uStCk6e6Dig.png" /></figure><p><em>Classification Fine-Tuning Results</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*zm5HyVZIOrgrQBlHtFcyjw.png" /><figcaption>Source: <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Finetuning Hub</a></figcaption></figure><p>We believe our experiment shows that Llama-2–13B is the most sample-efficient model among models we tested; it was able to adapt quicker than the smaller 7B models. Among 7B models, Llama-2–7B seems to adapt very quickly when there’s a low amount (2.5%-5.0%) of training examples. At higher sample fractions, Mistral and Zephyr quickly catch up, performing similarly to Llama2–7B. At 100%, all the 7B models appeared to perform similarly, and unsurprisingly, the 13B version of Llama 2 appeared to perform better than models with lower parameter counts.</p><p>We noted the rather inconsistent performance of GPT-3.5-Turbo across different sample fractions. However, we are unable to draw conclusions as we don’t have any visibility into OpenAI’s internal training method.</p><p><strong>Access our experiments in our GitHub repository</strong></p><p>All of our experiments and scripts to fine-tune these LLMs are available on our <a href="https://github.com/georgian-io/LLM-Finetuning-Hub/tree/main">LLM-Tuning-Hub</a>. The repository is organized by each model we tested. In each folder, the results from our benchmarks as well as the code that does the fine-tuning and inference can be viewed. These scripts can be used as a starting point for custom experimentation.</p><p><strong>Conclusion</strong></p><p>In this blog post, we believe that we have demonstrated that:</p><ol><li><strong>Smaller open-access models may be able to compete with commercial offerings out-of-the-box:</strong> With the refinement of training and alignment techniques, we believe there is an increasing number of smaller models that show promising results when compared to closed-source commercial LLMs.</li><li><strong>Fine-tuning can substantially enhance performance</strong>: Across all models, fine-tuning with techniques like QLoRA appeared to significantly improve performance and domain adaptation. In the case of summarization and classification tasks, we believe all models showed a marked improvement in performance metrics after fine-tuning, indicating that these models can adapt well to specific styles and requirements of a given task.</li><li><strong>Smaller models are viable choices for domain-specific tasks after fine-tuning</strong>: For practitioners who seek a balance between performance and resource constraints, our view is that fine-tuning smaller models like Mistral-7B, Zephyr-7B, or Llama-2–7B with specific datasets can yield results that are on par with, and sometimes even surpass, untuned commercial models. This approach also appears to result in faster inference times and reduced computational costs.</li></ol><p>Thanks to open source resources, it is easier to experiment with open-source LLMs. Developers can host LLMs on their own, or tweak the behavior of the models using their own data.</p><p>When it comes to choosing which model to use for fine-tuning, we suggest trying out smaller models first for simpler natural language processing tasks such as summarization — something a lot of the models already do well out-of-the-box. For more complex tasks or when you have limited training samples, bigger models may be a helpful first step.</p><p>Lastly, if you’d like to know more about what we’ve learned about LoRA parameter settings — levers that can be used to potentially get more performance out of the models — we will share more tips in an upcoming blog post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25760895d8ce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/experimenting-with-fine-tuning-llama2-mistral-and-zephyr-25760895d8ce">Experimenting with fine-tuning: Llama 2, Mistral and Zephyr</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Practical Guide to LLMs: Llama 2]]></title>
            <link>https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-llama-2-cdf21d540ce3?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdf21d540ce3</guid>
            <category><![CDATA[llama-2]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 17:21:18 GMT</pubDate>
            <atom:updated>2023-10-13T17:21:18.460Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a>, <a href="https://www.linkedin.com/in/royalsequeira/">Royal Sequeira</a>, <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a> &amp; <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*hAxeDS_U-T3va6QQ1ge8hg.png" /></figure><p>Continuing our assessment of Large Language Models (LLMs) through the lens of <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">our Evaluation Framework</a>, we turn our attention to Llama 2.</p><p>In this post, we aim to give you:</p><ul><li>A brief overview of Llama 2.</li><li>Different ways to fine-tune Llama 2 on custom datasets.</li><li>A comparison with BERT, DistilBERT and other LLMs on classification and summarization tasks.</li><li>Analysis of how much data and time you need to finetune your own Llama 2, and the cost.</li><li>A look at how to productize Llama 2, and the associated costs.</li></ul><p>We are open-sourcing all of the scripts and our insights in our<a href="https://github.com/georgian-io/llm-tuning-hub"> LLM Github repository</a>.</p><h3>What is Llama 2?</h3><p>Llama 2 is Meta’s latest open-source large language model, released in July 2023, and, according to Meta, can be used for commercial purposes. Llama 2 comes in three different versions: 7B, 13B, and 70B.</p><p>According to <a href="https://arxiv.org/abs/2307.09288">Llama 2: Open Foundation and Fine-Tuned Chat Models</a>, Llama 2 was trained on a mix of publicly available datasets. The paper states that any source containing personal information was removed from the datasets. These datasets equate to <a href="https://arxiv.org/abs/2307.09288">two trillion tokens</a> — 40% more data than its predecessor Llama 1. Llama 2 supports a context length of 4096, twice the length of its predecessor.</p><p>The training process described is very similar to Llama 1, with Llama 2 also using standard Transformer architecture. According to the <a href="https://arxiv.org/abs/2307.09288">paper</a>, the changes from Llama 1 include increased context length and a new form attention mechanism (<a href="https://arxiv.org/pdf/2305.13245v1.pdf">grouped-query attention</a>). Llama 2-Chat, the model’s instruction counterpart, was trained on publicly available instruction datasets with over 1M human annotations.</p><p>The same <a href="https://arxiv.org/abs/2307.09288">paper</a> states that Llama 2 was evaluated on popular benchmarks for both pre-training and fine-tuning abilities. The tasks used to benchmark against pre-trained models included, among others, common-sense reasoning, mathematical abilities and general knowledge. <a href="https://arxiv.org/abs/2307.09288">Meta found</a> that Llama 2 outperforms the other open-source models they tested (Falcon, MPT, and Llama1) across all benchmarks. When pitted against closed-source models, <a href="https://arxiv.org/abs/2307.09288">Llama 2 performs</a> similarly to GPT-3.5 in certain tasks such as language understanding and math word problems, but lags behind in coding tasks. In contrast, Meta found that <a href="https://arxiv.org/abs/2307.09288">Llama 2 matches or surpasses</a> PaLM-540B in nearly all benchmark tasks, yet there remain substantial performance gaps between Llama 2, PaLM-2-L, and GPT-4.</p><h3>Evaluating Llama 2</h3><p>For a holistic evaluation, we assess the 7B and 13B versions of Llama 2 across the four pillars of our Evaluation Framework: <strong><em>Performance</em></strong>, <strong><em>Time to Train</em></strong>, <strong><em>Costs</em></strong> and <strong><em>Inference</em></strong>.</p><h3>Llama 2 Performance</h3><p>Let’s move on to the first pillar of our evaluation framework: <strong><em>Performance</em></strong>.</p><p>We evaluated the performance of Llama 2 across the tasks of classification and summarization. All experiments were conducted on an <a href="https://aws.amazon.com/ec2/instance-types/g5/">AWS EC2 instance: g5.2xlarge</a> that comes with 1 NVIDIA A10G GPU. It has 24GB memory, and costs US$1.212 / hour.</p><p>We perform supervised <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">fine-tuning via QLoRA</a> on Llama 2 for both classification and summarization tasks. Here is an example of the prompt for fine-tuning for the task of classifying categories of news documents, which is similar to the prompts used in <a href="https://medium.com/georgian-impact-blog">our previous blog posts on benchmarking LLMs</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOpQKMUfRnAuf15fnWFsjQ.png" /></figure><p>Because the task is classification-based, we track accuracy as the main metric. We compare Llama 2 against other popular open-source models, such as Flan-T5-Large, Falcon and RedPajama.</p><p>The table below shows the performance of different LLMs with sample efficiency. The last row of the table contains the performance when the entire dataset is used. In these results, Llama 2–13B’s accuracy appears to be higher than all other models we have tested to date. We can also see that the first row of the table — corresponding to the lowest fraction of training samples — shows similar results. Llama 2–13B appears to achieve the optimal performance in a low data situation across the models we have tested so far.</p><p>Llama 2–7B, the smallest version of Llama, appears to achieve lower accuracy results than the other models across different sample fractions. These accuracy results indicate the impact of the extra parameters contained in Llama 2–13B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yjAK0Ns8g1oytk7TOOUXA.png" /></figure><p>Now let’s turn to the summarization task. For this task, we use the Samsum dataset where the goal is to summarize chat conversations. We create three prompts corresponding to the three settings: (i) Zero-shot, (ii) Few-shot, and (iii) Fine-Tuning with QLoRA. You can see example prompts below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pamfBaTOUZptwQnvoESIvg.png" /></figure><p>We observe a few trends in the tables below. First, Llama 2–7B’s performance appears to be higher than Llama 2–13B in a zero-shot and few-shot setting. We see that Llama 2–7B’s ROUGE-1 and ROUGE-2 scores are both higher than Llama 2–13B for zero- and few-shot. However, after fine-tuning with QLoRA, Llama 2–13B’s scores appear higher by a small margin. In our opinion, these results indicate that Llama 2–7B could be a strong candidate to consider for summarization and Q&amp;A tasks as it delivers these results despite being smaller than Llama 2–13B.</p><p><strong>Llama-2 7B Summarization Performance:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_tms04bDfM1sOSscOYenA.png" /></figure><p><strong>Llama-2 13B Summarization Performance:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NVc-mWsfasKr66lcpNmgtQ.png" /></figure><p>Next, we compare Llama 2–7B and Llama 2–13B with other models. Both versions of Llama 2 achieve competitive results, with Llama 2–13B appearing to achieve the highest results of the models we tested. Looking at these results, Llama 2 and Falcon appear to be good candidates to consider for summarization tasks. In our view, the 7B versions of both Llama 2 and Falcon can deliver good performance at potentially lower latencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uotN3Hv6ZRzfo06S_dL71A.png" /></figure><h3>Llama 2 Time and Cost to Train</h3><p>Next we move to the 2nd and 3rd pillars of our evaluation framework: <strong><em>Time</em></strong> and <strong><em>Cost</em></strong> to Train.</p><p><strong>Llama-2 7B Time and Cost to Train Stats:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*0C6PgEBEdK0-YREtlhpWxA.png" /></figure><p><strong>Llama-2 13B Time and Cost to Train Stats:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*CkaqkA5LFlU8qvLg2Qwclg.png" /></figure><p>As mentioned earlier, all experiments were conducted on an AWS EC2 instance: g5.2xlarge that costs US$1.212 / hour. We can see that the training costs are <em>just a few dollars</em>.</p><p>Llama 2–13B’s fine-tuning takes longer than Llama 2–7B owing to its relatively larger model size. As a result, gradient updates take more time, which leads to higher training costs.</p><h3>Llama 2 Inference</h3><p>The fourth pillar of our evaluation framework is <strong><em>Inference</em></strong>. With Llama 2, <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">in addition to our previous set up</a>, we also deployed using a new method — an open-source unified compute framework Ray. We also tested all three methods on an instance equipped with an Nvidia A100 40GB GPU in addition to an Nvidia A10 23B GPU. We tried this method because we wanted to check how much the inference changes with more compute power.</p><p>As in <a href="https://medium.com/georgian-impact-blog">our previous assessments</a>, we use a load testing tool Vegeta and with the aim of finding the maximum number of requests that the server is able to handle (RPS). We also measured throughput and latency. We used a set of sample sentences, each comprising approximately 100 tokens, to generate the requests. For each request during the load testing, we randomly picked a sentence from this sample set. This method aims to maintain the consistency of our test outcomes. Through these tests, we identified the RPS capacities for each model and service for classification and summarization.</p><p>The Nvidia A100 40GB GPU on GCP cost US$3.81 an hour. The Nvidia A10G 24GB GPU on AWS g5.4xlarge cost US$1.624 an hour.</p><p>We previously mentioned our exploration of new frameworks for model serving. One of these is Ray, a tool that offers a scalable library called <a href="https://docs.ray.io/en/latest/serve/index.html">Ray Serve</a>, specifically designed for constructing online inference APIs. For our testing, we kept it simple, avoiding advanced optimizations like dynamic batching or multi-node/multi-GPU serving. Another tool we experimented with is vLLm, a swift and user-friendly library tailored for LLM inference and serving. Deploying vLLm was straightforward; it required just a single command and a model stored in a HuggingFace repository. For FastApi, we used two workers to serve the model. We found that using two workers prevents “Out of Memory” errors and introduces parallelism to the request handling process. As with our previous assessments, we merged the base LLama models with Lora layers for TGI in order to complete the <a href="https://github.com/huggingface/text-generation-inference#:~:text=Other%20architectures%20are%20supported%20on%20a%20best%20effort%20basis%20using%3A">requirement</a> for serving.</p><p><strong>Llama-2- 7B Classification</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWlCpoxYNu_NqcynuWU1rA.png" /></figure><p>For the classification task, TGI and vLLm outperformed all other deployment methods that we tested. However, Ray handled more requests than FastApi for a fraction of the cost. As for the vLLm, it can easily handle even more requests than TGI and provides higher throughput (87.06 compared to TGI’s 54.41 for Nvidia A100).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*esNRVJi9OM7iNpmj4ZGj7g.png" /></figure><p>We have provided the results for TGI at different RPS in the graph below to illustrate the difference in performance when working with different instances. You can see the difference in latency when using A100 GPU in the graph. We also saw higher throughput (54.41 A100 compared to 19.81 A10) while the cost for the 1K tokens was not much higher for the A100 (US$0.00007) than when using A10 instance (US$0.00003).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*Pimh0JMv0QDKLLlXuPsu8w.png" /></figure><p><strong>Llama-2–13B Classification</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5X_HZ_K4EOY6fst5cqDD5w.png" /></figure><p>The results follow a similar pattern using the LLama-2–13B model to Llama-2–7B for the classification task. Looking at our results in the table above, TGI and vLLm again appear to outperform the other deployment methods, with vLLm handling more requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2dC5-bWq8MgrOMhoDHOcw.png" /></figure><p><strong>Llama-2–7B Summarization</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z1Hn68wUGWuaYGs3OJ1GrA.png" /></figure><p>For the summarization task, although RPS is similar across all deployment options, we observed that TGI on an A100 appeared to result in higher RPS (205), with a latency of 0.75 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RfRpmmmzlV1_APz6_WVICQ.png" /></figure><p><strong>Llama-2–13B Summarization</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbhdDVxuJDLlPyeoOaTNRg.png" /></figure><p>Looking at the summarization results for Llama-2–13B, we observe that vLLm appears to achieve the highest RPS for both Nvidia A10 and A100. Of the other approaches, Ray appears to outperform FastApi (55 RPS compared to FastApi 10 RPS on Nvidia A10) with lower costs (US$0.00006 on both instances).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OALLdSHnNsmyRImU7HsYzw.png" /></figure><h3>Conclusion</h3><p>Based on our analysis, Llama 2 could be important for companies leveraging LLMs owing to its strong performance in low data situations and low costs to train<strong>.</strong></p><p>We evaluated different methods to leverage Llama 2–7b and Llama-13B, i.e. in-context learning and fine-tuning with QLoRA, and found fine-tuning to outperform other methods across both tasks of classification and summarization.</p><p>We discussed how Llama 2–7B and Llama 2–13B appear to outperform language models such as the BERT family, and other LLMs. We observed that Llama 2–13B fine-tuned with QLoRA appears to achieve stronger results than the other models tested across both tasks, even at different sample efficiencies, based on our assessments to date. For summarization tasks, Llama 2–7B performs better than Llama 2–13B in zero-shot and few-shot settings, making Llama 2–7B an option to consider for building out-of-the-box Q&amp;A applications.</p><p>Fine-tuning both versions of Llama 2 takes a reasonable amount of time, and the associated costs to train are low. Llama 2–13B takes longer to fine-tune when compared to Llama 2–7B, owing to the differences in their model sizes.</p><p>For inference, we tested four deployment methods on two instances. Although a more powerful instance often leads to greater throughput and reduced latency, the key appears to be selecting an efficient deployment platform. Based on our results, we believe that is vLLm and Text Generation Inference. We believe Ray may be a good fit as an inference server for LLMs if we leverage additional optimizations. In our examination of various compute capabilities, we noted that using a more robust instance, specifically the Nvidia A100, results in enhanced performance and a higher throughput and peak RPS. While this setup comes at a slightly higher cost, it may be a justifiable investment for specific scenarios where achieving optimal throughput/RPS is crucial.</p><p>If you are looking for more detail on our work, our <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">GitHub repository</a> contains insights about hyperparameter optimization while fine-tuning Llama 2–7B and Llama 2–13B.</p><p>This blog is the fourth in a series of posts exploring LLMs that we intend to make available alongside the <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">GitHub repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdf21d540ce3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-llama-2-cdf21d540ce3">The Practical Guide to LLMs: Llama 2</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Practical Guide to LLMs: RedPajama]]></title>
            <link>https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-redpajama-340ae8ea96ec?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/340ae8ea96ec</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[redpajama]]></category>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 20:41:00 GMT</pubDate>
            <atom:updated>2023-10-10T14:35:20.360Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a>, <a href="https://www.linkedin.com/in/akashsara/">Akash Saravanan</a>, <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a> &amp; <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*LEaF1IvhRMvrlI-QG5C4DQ.png" /></figure><p>Continuing our assessment of Large Language Models (LLMs) through the lens of <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">our Evaluation Framework</a>, we turn our attention to RedPajama.</p><p>In this post, we aim to give you:</p><ul><li>A brief overview of RedPajama.</li><li>Different ways to fine-tune RedPajama on custom datasets.</li><li>A comparison with BERT, DistilBERT and other LLMs on classification and summarization tasks.</li><li>Analysis of how much data and time you need to finetune your own RedPajama, and the cost.</li><li>A look at how to productize RedPajama, and the associated costs.</li></ul><p>We are open-sourcing all of the scripts and our insights in our<a href="https://github.com/georgian-io/llm-tuning-hub"> LLM Github repository</a>.</p><h3>What is RedPajama?</h3><p><a href="https://together.ai/blog/redpajama-models-v1">RedPajama-INCITE </a>combines Together.ai’s RedPajama dataset and EleutherAI’s Pythia model architecture to form an open source LLM. In our view, one interesting aspect is that there is a 3B parameter size model, which is unusual in our experience. <a href="https://together.ai/blog/redpajama-models-v1">Together.ai</a> reasons that this model size will allow for wider adoption due to smaller hardware requirements and easier experimentation.</p><p>Together.ai introduces and uses the RedPajama dataset, which is a 1.2T token open-source replication of the LLaMa dataset. That is, they follow the same steps in terms of pre-processing and filtering, use the same data sources and extract roughly the same number of tokens from each.<a href="https://together.ai/blog/redpajama-models-v1"> But there’s no guarantee that this is the exact data that LLaMa used</a>. All the datasets that have been used are fairly standard and public datasets. RedPajama is primarily an English language dataset but the authors note that the Wikipedia dataset does contain texts in 20 different languages. The specific breakdown of the dataset is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*4a420jXKcVfNrrI9TlcVJQ.png" /></figure><p>RedPajama comes in two sizes; 3B and 7B. Each model has 3 variations — base, instruction fine-tuned, and chat. Both 3B and 7B versions follow the Pythia model architecture. The Pythia model is based on GPT-3 (hence it’s decoder-only) with the following changes:</p><ol><li>It uses fully dense attention layers instead of alternating sparse and dense attention layers. <a href="https://arxiv.org/pdf/2304.01373.pdf">Subsequent research to GPT-3 found that this worked better</a>.</li><li>FlashAttention (Dao et al., 2022): An exact-attention algorithm that is IO-aware and uses fewer memory accesses. <a href="https://arxiv.org/abs/2205.14135">FlashAttention was shown to increase speed by up to 3x compared to standard attention</a>.</li><li>Rotary Positional Embeddings (RoPE) (Su et al., 2021): RoPE attempts to unify absolute and relative positional embeddings.</li><li>Parallelized Attention (Wang &amp; Komatsuzaki 2021): The attention and feedforward layers are organized in parallel instead of sequentially. This arrangement is an optimization done to speed up the training process. Specifically, some matrix multiplications can be combined into a single operation.</li><li><a href="https://arxiv.org/pdf/2304.01373.pdf">It uses untied embedding matrices.</a> That is, the embedding and reverse-embedding matrices are not the same. This matrix set up is intended to increase interpretability.</li></ol><h3>Evaluating RedPajama</h3><p>For a holistic evaluation, we assess both 3B and 7B versions of RedPajama across the four pillars of our Evaluation Framework: <strong><em>Performance</em></strong>, <strong><em>Time to train</em></strong>, <strong><em>Costs</em></strong> and <strong><em>Inference</em></strong>.</p><h3>RedPajama Performance</h3><p>Let’s move on to the first pillar of our evaluation framework: <strong><em>Performance</em></strong>.</p><p>We evaluated the performance of RedPajama across the tasks of classification and summarization. All experiments were conducted on an AWS EC2 instance: g5.2xlarge that comes with 1 NVIDIA A10 GPU. It has 24GB memory, and costs US$1.212 / hour.</p><p>We use two distinct methods with RedPajama for the tasks of classification and summarization: (i) In-Context Learning, and (ii) Fine-Tuning via QLoRA. If you are unfamiliar with these two methods, <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d">our previous blog</a> explains them with example prompts.</p><p>Here is an example of the prompts for zero-shot prompting, few-shot prompting and fine-tuning for the task of classifying categories of news documents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*jWg1z5YNujN66eFW-3oTpg.png" /></figure><p>Now let’s look at some numbers. Because we are solving a classification task, we track accuracy as the main metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*39VZBrHSoyIS3ufBPwP6aw.png" /></figure><p>With accuracy scores of 0%, we saw that the model struggles to predict correctly when used in a zero-shot setting. In the few-shot setting, similar to <a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d">our findings about Falcon-7B</a>, the model returns an error because the prompt becomes too long after including several examples. Fine-tuning, on the other hand, improves the accuracy over the zero-shot setting, with 72.34% accuracy for 3B and 75.52% for 7B, showing the merits of tuning RedPajama on proprietary datasets.</p><p>Comparing RedPajama alongside Flan-T5-Large, Falcon-7B and other popular language models, we notice the following accuracies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*zs3Jssrdr-YAQ-iuT0JpjA.png" /></figure><p>Both versions of RedPajama are competitive when compared to other language models. The 7B version comes close to Falcon-7B’s performance. Furthermore, when trained on fewer samples, RedPajama performs at a similar level to other LLMs. The ablation study on sample complexity (see table below) suggests that RedPajama is also a strong candidate to consider in low data situations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*_tqDN7ZsVqspodtKlUtClg.png" /></figure><p>Now let’s turn to the summarization task. For this task, we use the Samsum dataset where the goal is to summarize chat conversations. Similar to classification, we create three prompts corresponding to the three settings: (i) Zero-shot, (ii) Few-shot, and (iii) Fine-Tuning with QLoRA. You can see example prompts below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*NhtTWY2L3GoGUNLxadw-RA.png" /></figure><p>RedPajama does a better job at summarizing dialogues than classifying news documents in zero-shot and few-shot settings. However, our results indicate that fine-tuning is still more effective as it helps RedPajama learn the summarization style specific to the dataset as opposed to creating a generic summary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*LP7O7h090V-sHUwWZ8T4dA.png" /></figure><p>When compared with other LLMs we have tested, we observe that RedPajama achieves competitive metrics, although Falcon-7B leads the table (see below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*evkGaAeNT7q1Do4bqYfqNQ.png" /></figure><h3>RedPajama Time and Cost to Train</h3><p>Next we move to the 2nd and 3rd pillars of our evaluation framework: <strong><em>Time</em></strong> and <strong><em>Cost</em></strong> to Train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*e7t_epXbN5AWPvMcF598jw.png" /></figure><p>As mentioned earlier, all experiments were conducted on an AWS EC2 instance: g5.2xlarge that costs USUS$1.212 / hour. We can see that the training costs are <em>just a few dollars</em>.</p><h3>RedPajama Inference</h3><p>Let’s move on to the 4th pillar of our evaluation framework: <strong><em>Inference</em></strong>.</p><p>As with previous assessments in this series, we used a load testing tool called Vegeta to test how effectively the system handles a large number of requests. We selected the HuggingFace Text Generation Inference Server and FastAPI as our deployment options. We aimed to determine the maximum requests per second (RPS) each model can handle, as well as the throughput, latency and cost per 1,000 tokens. We built a collection of example sentences each consisting of ~100 tokens in order to produce the requests. Then we randomly selected one of these sentences for each request during the load testing experiment. This approach attempts to keep our testing results consistent. Through experimentation, we determined the typical ranges of RPS that each model and service could handle for each task.</p><p>To evaluate RedPajama models through FastAPI, we decided to run the service with a single worker. Under this approach, the GPU memory could be allocated for just one model instance at any given time. By doing so, we successfully averted the occurrence of “Out of Memory” errors, as the memory demands of RedPajama models surpassed the available GPU memory when multiple instances were executed concurrently. Since the TGI <a href="https://github.com/huggingface/text-generation-inference#:~:text=Other%20architectures%20are%20supported%20on%20a%20best%20effort%20basis%20using%3A">requires</a> a standalone model, we have to merge the base model with the LoRA layers.</p><p>All load testing experiments have been performed on an AWS g5.4xlarge instance that costs US$1.624 per hour.</p><h4>Classification</h4><p>For the classification task we tested the FastAPI service for RPS ranging from 1 to 4 with a step size of 1. Then, we evaluated Text Generation Inference (TGI) for RPS ranging from 10 to 150, using a step size of 15. We then calculated the average throughput and latency for the maximum possible RPS. The tables and plots demonstrate significant differences in the response speed and load capacity when deploying the RedPajama model through FastAPI versus TGI.</p><p>RedPajama-3B + LoRA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*DSZ6HpVLenpBhnSyDlKvAQ.png" /></figure><p>The inference cost for FastAPI (US$0,001 / 1K tokens) is higher than for TGI (US$0,00003 / 1K tokens). Moreover, looking at the latency value we can say that it would cost US$0.0006 to get responses on 135 requests in 1.44 seconds using TGI, which is not possible to achieve with FastAPI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*Bn_wVlKFgQ35Vba_cj9hAA.png" /></figure><p>RedPajama-7B + LoRA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*U0CbBwH0sNumUDcZ3gWujA.png" /></figure><p>RedPajama-7B performs similarly to RedPajama-3B, but with a slightly lower RPS for text-generation; this result is expected due to the larger model size.</p><p>Taking into account the latency value, it costs US$0.001 to get responses on 125 requests with the RedPajama-7B model deployed on TGI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*4W92Q0iYd36Jz9OZtuXjCw.png" /></figure><h4>Summarization</h4><p>For the summarization task we were testing the models for the RPS in range from 10 to 200 with a step equal to 15. You can see our results in the tables below.</p><p>RedPajama-3B + LoRA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*YqrRW8wjqMMfc-TgRcA7Tw.png" /></figure><p>Even though the maximum RPS is quite similar for FastAPI and Text Generation using<strong> </strong>RedPajama-3B, the throughput and latency differ, which influences price per number of responses. According to our results, it will cost US$0.0003 to get responses for 195 requests using TGI and US$0.01 for 160 requests using FastAPI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*-Re0J8EERgMHYhYknGQ5uQ.png" /></figure><p>RedPajama-7B + LoRA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*7nFxlHTf0D7QdJhzmg96fQ.png" /></figure><p>As we saw with the classification task, the load testing performance for summarization of RedPajama-7B is similar to RedPajama-3B. The maximum RPS for FastAPI stays the same and for TGI it is lower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*pU8Y0uo0qP68xMJt7MVLMw.png" /></figure><p>Our load testing experiments highlight a key finding: the difference in model size does not have a big impact on inference. Instead, the factor influencing performance is the choice of deployment platform. In our experience, using an option like TGI has been consistently more effective than FastAPI. For instance, in the scenario of RedPajama, we observed a latency improvement of up to 12 times for both classification and summarization tasks.</p><h3>Conclusion</h3><p>To summarize our RedPajama assessment:</p><ul><li>We discussed and evaluated different methods to leverage RedPajama; both 3B and 7B versions.</li><li>We assessed how RedPajama can compete with similar language models, such as Falcon-7B, Flan-T5-Large and the BERT family.</li><li>We shared the time and cost it took to train RedPajama 3B and 7B in our tests.</li><li>Finally, we showed the different ways you could productize the LLM and the associated costs.</li></ul><p>If you are looking for more detail, <a href="https://github.com/georgian-io/llm-tuning-hub">our GitHub repository</a> contains insights about hyperparameter optimization while fine-tuning RedPajama, and which search space yields better results.</p><p>This blog is the third in a series of posts exploring LLMs that we intend to make available alongside the <a href="https://github.com/georgian-io/llm-tuning-hub">Github repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340ae8ea96ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-redpajama-340ae8ea96ec">The Practical Guide to LLMs: RedPajama</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Practical Guide to LLMs: Falcon]]></title>
            <link>https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2d43ecf6d2d</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[falcons]]></category>
            <category><![CDATA[summarization]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Thu, 31 Aug 2023 20:38:57 GMT</pubDate>
            <atom:updated>2023-10-10T14:33:45.562Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a>, <a href="https://www.linkedin.com/in/angelineyasodhara/">Angeline Yasodhara</a>, <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a> &amp; <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*8PVAMledv1OdXVGS_6ybZg.png" /></figure><p>Continuing our assessment of Large Language Models (LLMs) through the lens of <a href="https://georgian-io.medium.com/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">Georgian’s Evaluation Framework</a>, we turn our attention to Hugging Face’s <a href="https://huggingface.co/blog/falcon">Falcon</a>.</p><p><a href="https://georgian-io.medium.com/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">Our first blog</a> lays out the Evaluation Framework we are using in our assessments in detail. At a high level, we are testing against four pillars that are common considerations when building verticalized LLM solutions: <strong><em>Performance</em></strong>, <strong><em>Time to Train</em></strong>, <strong><em>Costs</em></strong> and <strong><em>Inference</em></strong>.</p><p>With around 4.4 million downloads last month,¹ <a href="https://huggingface.co/tiiuae/falcon-7b">Falcon has received a lot of attention from the research community</a> as the model emphasizes the importance of high-quality training data. Before researchers could use Falcon, foundational models were primarily trained on <strong><em>smaller</em></strong> high-quality curated datasets and internet data. The internet data used to train these other foundational models was filtered for quality (either by the link’s popularity or with a classifier trained on curated datasets). However, this lack of <a href="https://huggingface.co/blog/falcon">high-quality data</a> for training caused them to <em>hallucinate</em>, i.e., generate inaccurate responses.</p><p>In an effort to curb hallucinations, researchers at the Technology Innovation Institute in Abu Dhabi created Falcon, which is not only open-source but comes with the Apache 2.0 License, meaning it can be used by businesses for <a href="https://huggingface.co/blog/falcon">commercial purposes</a>.</p><p>In this post, we aim to provide:</p><ul><li>A brief explanation of Falcon.</li><li>Different ways in which Falcon may be fine-tuned on custom datasets.</li><li>A comparison of our tests of BERT, DistilBERT and Flan-T5-Large across the tasks of classification and summarization.</li><li>Analysis of how much data and time may be required to fine-tune a Falcon model, and the potential cost.</li><li>A look at some possible ways to productize Falcon, and the potential associated costs.</li></ul><p>We are open-sourcing all of the scripts and our insights in our <a href="https://github.com/georgian-io/LLM-Finetuning-Hub">LLM Fine Tuning Github repository</a>.</p><h3>What is Falcon?</h3><p>Falcon is a causal decoder-only model, i.e, given a sequence of words, it predicts the <a href="https://huggingface.co/blog/falcon">most-likely next word.</a></p><p>Falcon comes in two sizes — 7 billion parameters (called Falcon-7B) and 40 billion parameters (called Falcon 40B). Each of the two sizes has two versions: (i) <em>base</em>, which has been pre-trained on large corpuses of text and can be fine-tuned on downstream tasks, and (ii) <em>instruct</em>, which has already been fine-tuned on instructions, making it, in our view, favorable for out-of-the-box chatbot and Q&amp;A applications.</p><p>As of the time of writing, Falcon’s research paper is not out. However, the model uses two ideas that make it optimized for inference:</p><p>(i) <em>FlashAttention</em> (Dao et al., 2022): an exact-attention algorithm that is IO-aware and uses fewer memory accesses. The exact-attention algorithm was shown to give a speedup of up to 3x compared to standard attention.</p><p>(ii) <em>Multiquery</em> (Shazeer et al., 2019): an improvement on multi-head attention layers, where the keys and values are shared across all of the different attention “heads”. This process reduces memory and was shown to increase decoding speed.</p><p>What distinguishes Falcon from other LLMs is the dataset it was trained on. The following table gives an overview of the different sources that contributed to <a href="https://arxiv.org/abs/2306.01116">Falcon’s dataset</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*UXISdfLN9pA3NDjgLTk5sw.png" /></figure><p>At a high-level, <a href="https://arxiv.org/abs/2306.01116">the dataset has 800B tokens from RefinedWeb</a> (from May 2013 to September 2022), and does not contain duplicated and toxic content. This dataset is also<a href="https://huggingface.co/datasets/tiiuae/falcon-refinedweb"> available for commercial use</a> and is hosted on Hugging Face.</p><p>For the purposes of this blog post and our experiments, we chose Falcon-7B.</p><h3>Falcon-7B Performance</h3><p>We evaluated the performance of Falcon-7B across the tasks of classification and summarization. All experiments were conducted on an AWS EC2 instance: g5.2xlarge that comes with 1 NVIDIA A10G GPU. It has 24GB memory, and costs US $1.212/hour.</p><p>We use two distinct methods with Falcon for the tasks of classification and summarization: (i) In-Context Learning, and (ii) Fine-tuning via QLoRA.</p><h4>Method I: In-Context Learning</h4><p>In this method, we take Falcon as is and use it to solve our task. A popular name for this method is <em>prompting</em>, where the model generates text based on instructions provided by the user.</p><p>It is important to note that this method <em>does not</em> involve any tuning of model weights/parameters. For the task of classifying categories of news documents, here is an example of what the prompt looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*GBLc8d9IqnFaE5THS7beTQ.png" /></figure><p>In the prompt above, we provide instructions about the task to Falcon, the list of possible classes and the example sentence, and then ask it to predict the class. In the data science community, this approach is called <em>zero-shot prompting</em>. The LLM has to predict the class without prior experience about the particular domain.</p><p>However, zero-shot prompting is challenging, so researchers employ <em>few-shot prompting</em> to make things a bit easier for the model. Using this method, we provide a few examples of news documents and their corresponding categories to guide the model. Here is what the modified prompt looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*o17EM3k9L1QC5XGSx40giw.png" /></figure><p>Now that we are familiar with In-Context Learning, let’s move on to fine-tuning with QLoRA.</p><h4>Method II: Fine-tuning with QLoRA</h4><p>QLoRA is the <em>quantized version</em> of LoRA. Given the size of Falcon-7B, it is not possible to fit the entire model on 1 GPU without configuration. To make this happen, the model is loaded in a 4-bit or 8-bit configuration as opposed to the usual 32 bits. This approach allows the model to occupy less memory and allows for fine-tuning on a commercial grade GPU. <a href="https://georgian-io.medium.com/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">Our first blog in this series evaluating Flan-T5-Large</a> contains more detail on LoRa.</p><p>We <em>fine-tune</em> Falcon-7B through the mechanism of Causal Language Modeling. In simple words, given some text, the model learns how words are associated with each other and which words follow other words. To make this happen, we take the dataset containing news documents and their corresponding categories and create instructions out of them.</p><p>Here is an example of what an instruction looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*glyxixvVCQCqXPqXYDLTXA.png" /></figure><p>Once the model has learned the associations between news documents (sentence) and their corresponding categories (label), we ask it to predict the category for a new news document via:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*VTiN2NuPyvJTo2IweZI9zg.png" /></figure><p>Now let’s look at some numbers. Given we are solving a classification task, we track accuracy as the main metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*HpYAtoOaTXeTgn6dUIpedw.png" /></figure><p>These results show that the model struggles to make accurate predictions when used in a zero-shot setting. In the few-shot setting, the model errors out because the prompt becomes too long after including several examples. Fine-tuning, on the other hand, improves the accuracy over the zero-shot setting by more than 75%, demonstrating, in our view, the merits of tuning LLMs on proprietary datasets.</p><p>Comparing Falcon-7B alongside Flan-T5-Large and other popular language models, we observe the following accuracy results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*J2yMxIC-Dieaz7GIlfLdWQ.png" /></figure><p>In our tests, Falcon-7B outperforms the other models, achieving 76.37% accuracy. Furthermore, when trained on fewer samples, Falcon-7B maintains its lead in performance compared to other models. At roughly 5,332 samples, other models catch up to Falcon-7B’s performance, indicating Falcon-7B’s effectiveness in low-data regimes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*WrGfSx4E78yBg12v1UcYow.png" /></figure><p>Now let’s turn to the summarization task. For this task, we use the Samsum dataset where the goal is to summarize chat conversations. Similar to classification, we create three prompts corresponding to the three settings: (i) Zero-shot, (ii) Few-shot, and (iii) Fine-tuning with QLoRA. Below is a snapshot of what the prompts look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*biTQ1cul0QWrVOG13TWf-g.png" /></figure><p>Based on our tests, Falcon-7B does a better job at summarizing dialogues than classifying news documents in zero-shot and few-shot settings. But fine-tuning is still more effective than zero- and few-shot as it helps Falcon-7B learn the summarization style specific to the dataset as opposed to creating a generic summary. We track the ROUGE-1 and ROUGE-2 scores that tell us how close the generated text is to the actual summary. ROUGE-1 computes the presence of similar individual words across both the generated text and actual summary. ROUGE-2 computes the presence of similar <em>word-pairs</em> across both texts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*ID4CajzEBOScAM_7XEwnNQ.png" /></figure><p>In these cases, Falcon-7B outperforms Flan-T5-Large and Flan-T5-Base versions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*MDkJSd-WNxrMghF6Ijq2Bw.png" /></figure><p>Across both classification and summarization tasks, we see the merits of fine-tuning Falcon-7B on target datasets as opposed to using it out-of-the-box.</p><h3>Falcon-7B Time and Cost to Train</h3><p>Next we move to the second and third pillars of our evaluation framework: <strong><em>Time</em></strong> and <strong><em>Cost</em></strong> to Train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*ttdyYsCqEMrZXaqOduK-UA.png" /></figure><p>As mentioned earlier, all experiments were conducted on an AWS EC2 instance: g5.2xlarge that costs US$1.212/hour, which is low in our experience.</p><h3>Falcon-7B Inference</h3><p>With performance, and time &amp; cost to train out of the way, let’s move on to the 4th pillar of our evaluation framework: <strong><em>Inference</em></strong>.</p><p>With Inference, we used the same approach for deployment and cost estimation for the Flan model.</p><p>Following the same process we used to test Flan-T5-Large, we are using the load testing tool, Vegeta, on Falcon. We created a script that sent varying numbers of requests (ranging from 5 to 185) in three sets, with a three-second interval to give the server time to recover. Afterward, we examined the results, excluding instances where a “too many requests” error occurred. We calculated the average throughput and latency (90%) for the maximum possible requests per second (RPS) and used this data to calculate the cost. Again, following the same process we used to test Flan-T5-Large, all of the load testing experiments have been executed on a g5.4xlarge instance.</p><p>For the summarization task, we varied the RPS from five to 180. Ninety percent of all requests had a response time equal to or less than 1.82 seconds for 145 RPS (which is the maximum number of requests the server was able to handle).</p><p>To get this number of responses in one second, taking the throughput value of 53.8, (or to get 145 responses in ~1.82 seconds) costs US$0.0008.</p><p>Falcon-7B + LoRA for summarization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*yI9-FEi55nl9B0WsPBBWQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*_mSRDud9rSzZ_1zF" /></figure><p>The performance of the classification model during Inference is similar to the summarization model. The maximum RPS that the Text Generation Inference (TGI) was able to handle equals 125.</p><p>Taking into account the latency value, it will cost US$0.001 to get responses for 125 requests in 2.7 seconds.</p><p>Falcon-7B + LoRA for classification:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*jt3zWwyp3W7prO66v9qVjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*N_JQtUlgiR4JhXzojqfg7A.png" /></figure><h3>Conclusion</h3><p>Thanks to its strong performance in low data situations and low costs to train<strong>,</strong> Falcon could be important for companies leveraging LLMs.</p><p>We evaluated different methods to leverage Falcon-7B, i.e. in-context learning and fine-tuning with QLoRA, and found fine-tuning to outperform other methods across both tasks of classification and summarization.</p><p>We discussed how Falcon-7B can compete with similar language models such as Flan-T5-Large and the BERT family, and found Falcon-7B to perform better than Flan-T5-Large and BERT family in our tests. Furthermore, Falcon-7B performs particularly well when the amount of available labeled data is limited, in our experience.</p><p>Fine-tuning Falcon-7B takes a reasonable amount of time, and the associated costs to train are low. Finally, we benchmarked Falcon-7B’s throughput via Hugging Face’s text-generation pipeline. We found that Falcon -7B performs better in terms of speed for summarization than for classification, although the difference in speed between the two tasks isn’t large. As a result, the Text Generation Inference can be a fitting solution to deploy Falcon-7B for both classification and summarization tasks.</p><p>If you are looking for more detail on our work, our <a href="https://github.com/georgian-io/llm-tuning-hub">GitHub repository</a> contains insights about hyperparameter optimization while fine-tuning Falcon-7B.</p><p>This blog is the second in a series of posts exploring LLMs that we intend to make available alongside the <a href="https://github.com/georgian-io/llm-tuning-hub">GitHub repository</a>.</p><p>[1]: As of August 31, 2023</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2d43ecf6d2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-falcon-d2d43ecf6d2d">The Practical Guide to LLMs: Falcon</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons learned from our GenAI boot camp and hackathon]]></title>
            <link>https://medium.com/georgian-impact-blog/lessons-learned-from-our-genai-boot-camp-and-hackathon-e71357f06c02?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/e71357f06c02</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[bootcamp]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 18:30:28 GMT</pubDate>
            <atom:updated>2023-08-17T09:06:11.841Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/azin-asgarian/?originalSubdomain=ca">Azin Asgarian</a>, <a href="https://www.linkedin.com/in/chelsealefaivre/">Chelsea Lefaivre</a> &amp; <a href="https://www.linkedin.com/in/davidjtingle/">David Tingle</a></p><p>With generative AI (GenAI) development moving quickly, the Georgian team held a GenAI boot camp for our portfolio companies (our customers) to help them kickstart their own GenAI journey. In this blog, we aim to share our takeaways from running the boot camp.</p><h3>The Genesis of our GenAI Boot Camp</h3><p>At Georgian, our 30+ person R&amp;D team provides support to our companies as they look to adopt emerging technologies like GenAI and build new and differentiated products or product features. The team runs hackathons (which typically run for two to three weeks) and more in-depth engagements that span several months.</p><p>In our typical hackathon model, we partner with a single company to address a specific issue the team is tackling. From our experience participating in the Vector Institute’s event Prompt Engineering Lab, we recognized the value of a group-oriented model delivered. The approach encouraged collective learning and cross-pollination of ideas. Since GenAI is evolving so quickly and much of the knowledge, concepts and code could be relevant across different companies’ use cases, we decided to compress the timeline and work with many of our customers at the same time in our very first GenAI boot camp.</p><p>In this boot camp, we aimed to provide education and support for 22 teams from 21 companies across our portfolio and companies in <a href="https://georgian.io/georgian-colab/">CoLab</a> — our pre-investment program for early-stage startups — with the goal of educating our companies on GenAI and supporting them as they took a GenAI use case from ideation into implementation in a matter of days.</p><h3>Benefits of a GenAI Boot Camp</h3><p>By adapting our typical offerings to this new format, we were able to accomplish our two goals:</p><ol><li><strong>Educating companies on GenAI: </strong>The boot camp was designed to boost the participants’ understanding of GenAI and offer them a clear perspective into the potential possibilities and limitations within this domain.</li><li><strong>Scaling our hackathon model to help 21 companies simultaneously:</strong> The boot camp set out to provide guidance on how to identify, structure and prioritize diverse opportunities aligned with each company’s unique business needs. The format allowed teams to implement their proofs of concept (POC) within just a few days, enabling them to promptly witness the results of their efforts. By scaling our usual 1:1 hackathon model this way, we were able to be more efficient and productive.</li></ol><h3>Planning a Successful GenAI Boot Camp</h3><h3>The Organizing Committee</h3><p>Given the tight timeline, we realized that the success of the boot camp would stem from having representatives from various parts of our business. Our core team was the same cross-functional team that organizes <a href="https://georgian-io.typeform.com/to/thI9lxiL?typeform-source=growth.georgian.io#hubspot_utk=xxxxx&amp;hubspot_page_name=xxxxx&amp;hubspot_page_url=xxxxx">Transferred Learnings</a>, a community for learning about Applied AI. This team includes technical expertise, marketing acumen and community management — all important functions for initiatives of this nature. This core group was augmented with 11 other team members from across Georgian.</p><p>In addition to our internal team, Deval Pandya, David Emerson and others at the Vector Institute helped us with planning. Having gone through a similar experience, they were able to share knowledge and practical insights, which helped in the execution of the boot camp.</p><h3>Boot Camp Structure</h3><p>By the end of the boot camp, we wanted the participants to have a solid foundation in GenAI and be equipped with the skills to apply the technology to their own use cases. To that end, we structure the boot camp across three phases:</p><ol><li>Use case identification and evaluation</li><li>Tutorials to educate and access to tools and demos</li><li>Hackathon for building prototypes</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pt7xhpPTI6QBUzRU" /></figure><p>In the initial phase, members from our Product and AI teams worked with each team, to clarify their use case and product strategy.</p><p>Since our participants had different levels of experience with Large Language Models (LLMs), the next phase provided educational presentations. Georgian practitioners and others in the GenAI ecosystem — including the <a href="https://vectorinstitute.ai/generative-ai-discussions/">Vector Institute</a>, <a href="https://cloud.google.com/ai/generative-ai">Google</a> and <a href="https://www.private-ai.com/">Private AI</a> — acted as our tutors.</p><p>During the tutorial sessions, we covered introductory materials on LLMs. We complemented these tutorials with demo notebooks and code assets. In addition, machine learning practitioners were on hand during open office hours to answer queries and address any challenges the teams encountered.</p><p>In the final phase, we provided one-on-one technical support to help teams create rapid prototypes for their use cases. By giving personalized support for their specific technical needs we aimed to accelerate the prototyping process.</p><h3>Inviting Participants</h3><p>The teams were divided in two categories depending on their GenAI maturity:</p><ol><li><strong>Education</strong>: If teams were earlier in the GenAI journey, they had the option to attend the educational portion without committing to a hackathon project.</li><li><strong>Hackathon</strong>: If teams had the time, resources and GenAI experience to take on a hackathon project, they would be given extra support from the technical team.</li></ol><p>For teams participating in the hackathon portion, teams generally included different backgrounds and perspectives to help with ideation and implementation of feasible GenAI solutions. Typically, teams included a blend of technical team members (who were familiar with Python and machine learning concepts) and at least one product manager (who had an understanding of the business’s internal and customer processes).</p><h3>Use Case Identification</h3><p>While our boot camp use cases were mainly focused on natural language processing (NLP), for GenAI broadly, you can differentiate between text, speech and image (computer vision) applications.</p><p>In our case, we prompted our teams to think about different applications, including:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*3GvNzt4rZlmzjOhb" /></figure><ol><li><strong>Asset Generation</strong>: GenAI is used to create new assets such as code, images, audio and text. This application encompasses a wide range of sub-applications and use cases, and it forms the core of many emerging companies in this space.</li><li><strong>Conversational UI</strong>: GenAI enables conversational interfaces, including chatbots. Conversational UIs serve as important interfaces for complex products, knowledge bases and processes, providing a flexible and user-friendly method of interaction.</li><li><strong>Knowledge Retrieval</strong>: GenAI models, paired with <a href="https://venturebeat.com/ai/how-vector-databases-can-revolutionize-our-relationship-with-generative-ai/">vector databases</a>, are effective for knowledge retrieval. These models can understand and surface relevant information from various data sources like documentation, FAQs or help desk materials. While there is overlap with conversational UI, knowledge retrieval is distinct in its focus on surfacing information.</li><li><strong>Text Utilities</strong>: Language models excel at understanding inputs, and text utilities leverage this ability. Tasks like sentiment scoring, extracting meta-information, translation and summarization fall into this category. Text utilities enable data analysis and insights by utilizing the capabilities of language models.</li><li><strong>Task Execution</strong>: GenAI models can be used to perform specific tasks within software environments. This application is an emerging area where models can execute activities that would typically be done through other means. For example, in analytics, these models can query and analyze datasets based on natural language prompts, freeing analysts from manual tasks to do higher value work. Additionally, models can interact with other software systems, executing specific actions and tasks based on predefined parameters. While still in its early stages, task execution presents an interesting area to explore for leveraging the capabilities of GenAI models in software environments.</li></ol><h3>A Framework to Evaluate GenAI Projects</h3><p>Once the teams had identified potential projects, we used the evaluation framework below to prioritize:</p><ul><li><strong>Desirability</strong>: Does your proposed project solve a high-priority pain point or enable an opportunity for your customers? Assess market desire and the overall impact the project will have.</li><li><strong>Viability</strong>: Consider economic value creation for the business. Does the project offer more upside than costs, considering factors such as margins and deployment expenses?</li><li><strong>Feasibility</strong>: Is the project technically feasible to build? Consider maintenance and support over the long-term along with market implementation, including sales potential.</li></ul><h3>Support for Participants</h3><p>The Georgian R&amp;D team offered technical support for teams in two formats: drop-in office hours and 1:1 bookable time slots.</p><p>Because the development period was fairly short — only two dedicated days — these support touch points were important for teams to make progress on their prototypes.</p><p>Most of the support requests fell into these three areas:</p><ol><li><strong>Ideation and problem scoping</strong>: In several cases, the original use cases were too broad to accomplish in the timeline, and teams needed help to break it down into smaller tasks/scope it into the boot camp timeline.</li><li><strong>Setup and tooling</strong>: Another common request was how to use specific tools, like LangChain and PandaAI, or how to set up their environment.</li><li><strong>Prompt engineering and customizing LLMs</strong>: The third common support request was helping to customize LLMs or to tweak their prompts. Often this type of request involved taking one large task that the LLM did passably and creating multiple smaller tasks that the LLM could handle better.</li></ol><h3>Trends We Noticed</h3><p>Based on our previous hackathons and workshops, we came into this engagement with a pretty good idea of what to expect. Here are some trends we noticed that surprised us!</p><p><strong>Use Case Ideal Types vs. Combinations<br></strong>Many teams combined use cases from the list above in interesting ways (e.g., a chatbot leveraging a vector database).</p><p><strong>Analytics Is a Recurring Theme<br></strong>One combination we saw several times was related to analytics: specifically, using a combination of a natural language interface, a vector database to store embeddings for underlying data and a prompt chain that generated and then executed analytics queries to explore or analyze that data using language.</p><p><strong>Chat Interfaces Come to the Fore<br></strong>Many teams leveraged the power of natural language interfaces to build better experiences for users. However, chat functionality alone doesn’t make a product successful. The most successful applications of chat, in our experience, are typically linked to a specific purpose or goal for the conversation, one that matters for the user. Helping the user solve key challenges or unlocking new capabilities is essential for finding product-market fit for your conversational interface.</p><p><strong>Internal and External Use Case Balance<br></strong>Many teams scoped and built customer-facing POCs, but a number of teams also built internal-facing functionality that automated or augmented key processes for colleagues in their organization.</p><p><strong>Teams Built More (And Faster) Than We Anticipated<br></strong>Going into the boot camp, our objective was to make technical development / building a POC as easy as possible. With that said, we expected that many participants would focus on learning and brainstorming rather than coding. We were surprised to see the vast majority of teams submit at least one POC for the competition at the end of the boot camp, which shows how fast development can happen in the era of GenAI and how accessible the technology is.</p><p><strong>Edge Cases Matter<br></strong>Looking across the teams’ experience, it would appear that building a GenAI use case that gets 80% of the way towards viable functionality is relatively straightforward, but solving the edge cases and failure modes in the last 20% takes a lot of effort. Many teams spent more time than expected resolving these types of issues.</p><p><strong>Continued Progress<br></strong>After the boot camp ended, we were happy to see teams continuing to work on their projects. Some teams even shared that they were planning to run similar hackathons with their own teams to continue their learning and development.</p><h3>Recognition</h3><p>We recognized three teams for their projects:</p><ul><li><strong>The Impactful Innovator Award:</strong> Awarded to the team that has developed a feature that has the potential for positive, real-world impact and is closest to production readiness. This award underscores the important intersection of practicality and large-scale impact.</li><li><strong>The Pioneering Prodigy Award</strong>: Awarded to the team whose AI features are novel, innovative and “out of the box”. This award emphasizes the creative and forward-thinking aspects of the hackathon.</li><li><strong>The Ethical Engineer Award</strong>: Awarded in recognition of the team that incorporates responsible AI practices into their project, such as fairness, privacy or transparency considerations. This award reflects the importance of ethical considerations in AI development.</li></ul><h3>Lessons Learned</h3><p>By bringing in experienced practitioners from across the various Georgian teams and our network, we believe that we were able to deliver a smooth experience for our participants. All the teams boosted their adoption and understanding of a rapidly evolving technology and produced prototypes that, in our view, could potentially evolve their own products or business practices.</p><p>Specifically, our teams appreciated having pre-made examples, demos and notebooks to ground their learning. Access to LLM platforms like OpenAI GPT-4, Google and Hugging Face models like Falcon 40B also helped teams accelerate their projects. They noted that easy file access, tutorial recordings they could watch and share asynchronously and consistent communication helped make their boot camp experience a good one.</p><p>We also had a few learnings of our own! In the future, we plan to offer a standalone session dedicated to environment setup before the boot camp to get people set up early. We also plan on developing a more robust progress tracking system so that we can track company stage and maturity to tailor the support we offer.</p><h3>Special Thanks</h3><p>Thanks to our guest speakers <a href="https://ca.linkedin.com/in/david-emerson-1b9b2225">David Emerson</a> from the Vector Institute, <a href="https://ca.linkedin.com/in/erik-saarenvirta">Erik Saarenvirta</a> from Google, <a href="https://ca.linkedin.com/in/mikeyoung6">Michael Young</a> from Private AI. Their help and support with office hours for the participants, access and guidance on using their tools was appreciated.</p><p>Also thanks to the Georgian team for supporting this project — especially to our Georgian presenters <a href="https://ca.linkedin.com/in/akashsara">Akash Saravanan</a>, <a href="https://ca.linkedin.com/in/alex-manea-cso">Alex Manea</a>, <a href="https://ca.linkedin.com/in/angelineyasodhara">Angeline Yasodhara</a>, <a href="https://www.linkedin.com/in/davidjtingle">David Tingle</a>, <a href="https://www.linkedin.com/in/eli-scott-609698159/">Eli Scott</a>, <a href="https://www.linkedin.com/in/rodrigo-ceballos-lentini">Rodrigo Ceballos Lentini</a>, <a href="https://ca.linkedin.com/in/rohit-saha-ai">Rohit Saha</a>, <a href="https://ca.linkedin.com/in/royalsequeira">Royal Sequiera</a>.</p><p><strong>Disclosures. </strong>The material and information presented here is for discussion and general informational purposes only and is not intended to be, and should not be construed as, legal, business, tax, investment advice or other professional advice. The material and information do not constitute a recommendation, offer, solicitation or invitation for the sale of any securities, financial instruments, investments or other services, including any securities of any investment fund or other entity managed or advised, directly or indirectly, by Georgian or any of its affiliates. Past performance is not an indication of future performance and may not be repeated. Any forward-looking statements, including estimations, expectations, projections, forecasts or predictions, such as anticipated outcomes, proceeds or performance (“<strong>Projection</strong>s”), are for illustration purposes only and should not be relied on and do not reflect any actual outcomes, proceeds or performance, which will be materially higher or lower than the Projections. There can be no assurance that the Projections will be attained. The information and materials herein are as of August 4, 2023 unless otherwise indicated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e71357f06c02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/lessons-learned-from-our-genai-boot-camp-and-hackathon-e71357f06c02">Lessons learned from our GenAI boot camp and hackathon</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Practical Guide to LLMs: Flan-T5]]></title>
            <link>https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d26cc5f14c0</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[flan-t5]]></category>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Fri, 04 Aug 2023 17:50:15 GMT</pubDate>
            <atom:updated>2023-10-10T14:34:49.890Z</atom:updated>
            <content:encoded><![CDATA[<p>By: <a href="https://www.linkedin.com/in/rohit-saha-ai/">Rohit Saha</a>, <a href="https://www.linkedin.com/in/maria-ponomarenko-71b465179/">Mariia Ponomarenko</a> &amp; <a href="https://www.linkedin.com/in/kyryl-truskovskyi-275b7967/">Kyryl Truskovskyi</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*69lhoorDo-piv2sCnZtX7g.png" /></figure><p>If the last 6 months of AI research felt like a decade to you, you are not alone! With a new Large Language Model (LLM) released every other week, it has been challenging for the research community to keep up with the current pace of innovation in AI. While there has been a flurry of blog posts, tweets and code snippets on social media showcasing the power of LLMs and how to set up chat applications using them, we have seen few efforts that stress-test them for real-life business use-cases.</p><p>Through a series of blog posts, we aim to bring you insights that we have learned via experimentation with a suite of LLMs. Today, we will be focusing on one of the most popular open-source LLMs: <a href="https://huggingface.co/docs/transformers/model_doc/flan-t5">Flan-T5 </a>.</p><p>In this post, we aim to give you:</p><ul><li>A brief explanation of Flan-T5.</li><li>A comparison with the popular <a href="https://huggingface.co/docs/transformers/model_doc/bert">Bert</a> and <a href="https://huggingface.co/docs/transformers/model_doc/distilbert">Distilbert</a> models across classification and summarization tasks.</li><li>Analysis of how much data and time you need to train your own Flan-T5, and the cost.</li><li>A look at some of the leading ways in our experience to productize Flan-T5, and the associated costs.</li></ul><p>What’s more, we are open-sourcing all of the scripts in the form of an <a href="https://github.com/georgian-io/llm-tuning-hub">LLM Github repository</a>. This blog is the first in a series of posts exploring LLMs that we intend to make available alongside the Github repository.</p><h3>What is Flan-T5?</h3><p>Flan-T5 is an open-source LLM that’s available for commercial usage. Published by Google researchers, Flan-T5 is an encoder-decoder model pre-trained on a variety of language tasks. The model has been trained on supervised and unsupervised datasets with the goal of learning mappings between sequences of text, i.e., text-to-text.</p><p>In our view, what sets Flan-T5 apart from other models is that its training is based on prompting. In other words, the model has knowledge of performing specific tasks such as summarization, classification and translation to name a few. For instance, if you were to feed this blog post into Flan-T5 and tell it to “summarize this article”, it would know that it needs to generate a shorter version of this article. If you download the model from Hugging Face, you can start using it right away for general purpose applications.</p><h3>Evaluating Flan-T5</h3><p>However, to build verticalized solutions on your proprietary data with this model, you would first have to consider: <strong><em>Performance</em></strong>, <strong><em>Time to train</em></strong>, <strong><em>Costs</em></strong> and <strong><em>Inference</em></strong>.</p><p>If you have already trained and deployed language models, such as Bert, you may already be familiar with this evaluation framework. In other words, Flan-T5 is very similar to Bert but is a larger and more powerful model.</p><h3>Flan-T5-Large Performance</h3><p>Let’s move on to the first pillar of our evaluation framework: <strong><em>Performance</em></strong>.</p><p>We compare the performance of fine-tuning Flan-T5-Large on two tasks: Classification and Summarization. Flan-T5 comes in various sizes; for our experiments, we chose Flan-T5-Large, which has 780M parameters. Furthermore, all experiments were conducted on an AWS EC2 instance: g5.2xlarge that comes with 1 NVIDIA A10G GPU. It has 24GB memory, and costs $1.212 / hour.</p><p>A quick aside: Fine-tuning the full Flan-T5 model comes with significant hardware and overfitting challenges. Imagine fitting such a huge model in 1 GPU! A nightmare, right? To circumvent this, we will be performing Parameter Efficient Fine-Tuning (PeFT) via <a href="https://huggingface.co/docs/peft/conceptual_guides/lora">LoRA</a>. In simple words, LoRA is a technique to fine-tune massive models (LLMs) without actually fine-tuning the base model. It works by adding some extra layers throughout the base model, and <strong><em>tuning only those layers</em></strong>. In the case of Flan-T5-Large, LoRA adds layers that contain only 4.7M parameters. You can think of these layers as experts that are focused on learning patterns present in your dataset, while leveraging the generalist knowledge already available in Flan-T5-Large’s 780M parameters.</p><p>Back to the task. For classification, we used the open-source <a href="https://www.kaggle.com/datasets/crawford/20-newsgroups">News Group dataset</a>. This dataset contains news documents that are labeled into their corresponding categories such as: sports, politics, finance, etc. We compare Flan-T5-Large’s performance against Bert (110M) and Distilbert (66M), and observe the following accuracy scores:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*_gFePsBkqDPdmeh3N8piVA.png" /></figure><p>At first glance, it appears that Flan-T5-Large achieves parity alongside Distilbert and Bert when the entire dataset is used for fine-tuning. However, often businesses don’t have as much labeled data to train these models as the 10,664 in this dataset. Next, we performed an ablation study to see what happens to performance when we decrease the amount of training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*GS-nl8SwCUnEpIxbhK-XfQ.png" /></figure><p>We can see that Flan-T5-Large does a significantly better job when compared to Distilbert and Bert on a sample size as low as ~250! As we steadily increase the number of samples, Distilbert and Bert finally catch-up to Flan-T5-Large, making Flan-T5-Large, in our opinion, a good candidate to consider in low-data situations.</p><p>In case you are wondering whether the training conditions were different causing one model to outperform the other, we fine-tuned each model for exactly five epochs, i.e., five full passes over the training dataset, and evaluated it on a consistent test set.</p><h3>Flan-T5 Time and Cost to Train</h3><p>Next we move to the 2nd and 3rd pillars of our evaluation framework: <strong><em>Time</em></strong> and <strong><em>Cost</em></strong> to Train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*lKLryFQS13hjfCyOJJU0Og.png" /></figure><p>As mentioned earlier, all experiments were conducted on an AWS EC2 instance: g5.2xlarge that costs $1.212 / hour. Training such a huge model on roughly 10,000 samples cost less than a cup of coffee, which is surprisingly low in our experience.</p><p>At this time, we will evaluate Flan-T5-Large on a summarization task for which we used the <a href="https://huggingface.co/datasets/samsum">Samsum dataset</a>. We compare the performance of Flan-T5-Large + LoRA (4.7M parameters) against full fine-tuning of Flan-T5-Base, i.e., tuning the whole 250M parameters. Full fine-tuning Flan-T5-Large without LoRA is challenging so we chose Flan-T5-Base for this comparison. The metric being computed is called <a href="https://medium.com/nlplanet/two-minutes-nlp-learn-the-rouge-metric-by-examples-f179cc285499">ROUGE1</a> which is commonly used to evaluate models for the task of summarization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*gm750mUVcKZOixbHcUTvqA.png" /></figure><p>The above training took 2 hours 47 minutes over 5 epochs, costing ~$3.6.</p><h3>Flan T-5 Inference</h3><p>With performance, and time &amp; cost to train out of the way, let’s move on to the 4th pillar of our evaluation framework: <strong><em>Inference</em></strong>.</p><p>We have observed that companies can be worried about using their own LLMs because of the cost of maintaining them in production. So far, we have not seen any discussions on estimating costs for productizing LLMs. As a result, we will calculate them based on assumptions that are commonplace in ML production systems. Of course, inference costs are dependent on your infrastructure, usage and existing engineering system.</p><p>To keep things simple, we have created a framework to estimate prices. We have added some fundamental assumptions for a specific example. Using this framework, you can:</p><ol><li>Evaluate costs of different models, and find which one is suitable for your use.</li><li>Estimate cost for your specific task by changing the input and load format in provided boilerplate code.</li></ol><p>There are many web servers for deploying LLMs such as <a href="https://github.com/huggingface/text-generation-inference">text-generation-inference</a>, <a href="https://github.com/bentoml/OpenLLM">OpenLLM</a> and <a href="https://github.com/mosecorg/mosec">mosec</a>. Eventually, we hope to compare them, but for today, we will use a basic setup with FastAPI and Triton inference server without any optimization under the hood.</p><p>All benchmarks were conducted on a g5.4xlarge AWS instance costing $1.624 (on-demand price as of June 2023). For stress-testing purposes, we used a load-testing tool called Vegeta and loaded the web servers with an increasing number of requests per second (RPS) until latency started to degrade significantly, or we started getting timeout errors. We conducted experiments for each RPS value multiple times (3–6 times) and calculated the average latency and throughput.</p><p>It is worth mentioning that when we perform a load test with a tool like Vegeta and set the request rate to n requests per second, it means that the tool attempts to simulate an average of n requests per second over the duration of the test. It doesn’t guarantee that exactly n requests will be served and completed within each second.</p><p>Inference costs are derived from:</p><ul><li>Total tokens server can process in 1 hour = (rps * average number of tokens (input + output) * 60 seconds * 60 minutes)</li><li>Price per hour = from AWS</li><li>Inference cost = Price per hour / (Total tokens in 1 hour / 1000)</li></ul><p>This is a specific calculation, but using it makes it easy to compare other LLMs, including closed LLM APIs (such as GPT-4 and Writer Parmila). In future posts, we intend to add more optimization, with the aim of reducing these numbers. Remember, these numbers are only references.</p><p>Flan-T5-Large + LoRA for summarization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*s4I5--zAxeAb5xR_1YXx1g.png" /></figure><p>Flan-T5-Large + LoRA for classification:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*LyY-8MzE8W7xammyWtodcQ.png" /></figure><h4>FastAPI</h4><p>For the summarization task, we varied the RPS from 5 to 30, and examined the system’s responsiveness across different load levels. We discovered that 90% of all requests had a response time equal to or less than 18.27 seconds (for 30 RPS). The plot also shows that as RPS increases, the 90th percentile latency rises gradually, signaling potential performance limitations. We found out that 35 requests per second is a critical threshold where the system fails.</p><p>The throughput value was reported as 1.5. This value represents the average number of requests successfully completed per second during the load test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*2Sz5J_w-2mCvu8vk" /></figure><p>Next, we performed the same load testing experiments for the classification task. Here the maximum number of requests the system can cope with is much higher — 185 requests per second. Respectively 90% of all requests had a response time equal to or less than 28.01 seconds. However, as with the summarization task, the throughput is much lower, at 5.84, meaning that approximately 5.84 requests were processed and received a response per second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*5hchVCk17GNh9vY6" /></figure><h4>Hugging Face Text Generation Inference</h4><p>Text Generation Inference (TGI) server developed by Hugging Face allows faster text generation by using advanced techniques like Tensor Parallelism and dynamic batching with popular open-source Language Model Libraries (LLMs) such as StarCoder, BLOOM, GPT-NeoX, Llama, and T5.</p><p>This time for the summarization task we varied the RPS value from 5 to 120. 90% of all requests had a response time equal to or less than 2.03 seconds (for 120 RPS).</p><p>The throughput value was reported as 45.5, which is much higher than the value we were able to get using FastAPI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*T-RdAtgOSIQDKfLk" /></figure><p>As for the classification task, the maximum RPS that TGI was able to handle equals to 145. The throughput value is 78.5 which is much higher than the value we got during load testing of FastAPI. Moreover, the Latency (90%) is also lower and equals to 1.5 s per request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/0*J_drC7hA4UeNa-cH" /></figure><h3>Conclusion</h3><p>This brings us to the end of our Flan-T5-Large assessment. To summarize:</p><ul><li>We showcased how an LLM, in this case Flan-T5-Large, can compete with the BERT family in a low-data regime.</li><li>We shared the time and cost it takes to train a Flan-T5-Large.</li><li>Finally, we showed the different ways you can productize the LLM and the associated costs.</li></ul><p>If you are looking for more detail, our <a href="https://github.com/georgian-io/llm-tuning-hub">GitHub repository</a> contains insights about hyperparameter optimization while fine-tuning Flan-T5, and which search space yields better results.</p><p>This blog is the first in a series of posts exploring LLMs that we intend to make available alongside the Github repository.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d26cc5f14c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/the-practical-guide-to-llms-flan-t5-6d26cc5f14c0">The Practical Guide to LLMs: Flan-T5</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Generative AI: 4 ways to customize your models]]></title>
            <link>https://medium.com/georgian-impact-blog/getting-started-with-generative-ai-4-ways-to-customize-your-models-f6871b44cb6a?source=rss-d3d81cdc0d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6871b44cb6a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <dc:creator><![CDATA[Georgian]]></dc:creator>
            <pubDate>Wed, 05 Apr 2023 21:38:00 GMT</pubDate>
            <atom:updated>2023-04-14T21:09:12.909Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mJ1UBHxwCO2SvcxXzl3udw.jpeg" /></figure><p>By: <a href="https://www.linkedin.com/in/azin-asgarian/?originalSubdomain=ca">Azin Asgarian</a></p><p>Generative AI models such as <a href="https://openai.com/research/gpt-4">GPT-4</a>, <a href="https://blog.google/technology/ai/bard-google-ai-search-updates/">Bard</a> and <a href="https://ai.facebook.com/blog/large-language-model-llama-meta-ai/">LLaMA</a> have become increasingly popular due to their remarkable capabilities in various tasks, including text generation, summarization and categorization. While big tech companies have the resources to train and maintain these powerful models, smaller businesses and organizations may need to explore cost-effective ways to adapt and customize existing models to address their specific requirements.</p><p>The process of selecting the right model and customizing it can be a complex endeavor, particularly for those companies new to this field. You need to think about factors like the intended application, available computer power, task difficulty, dataset size and quality, scalability and the necessary expertise for successful implementation, among others, when considering the appropriate model and customization approach.</p><p>In this blog post, we present our insights from working with growth-stage software companies and experimenting with generative AI models and technologies. Specifically, we’ll discuss and compare various types of generative AI models and outline four techniques for tailoring these models to an organization’s unique needs. From fine-tuning to prompt engineering, prompt optimization and reinforcement learning from human feedback (RLHF), this guide aims to lay out some considerations for determining the most suitable approach for adopting the GenAI models to specific use cases.</p><h3>What is Generative AI?</h3><p>Generative AI has taken the world by storm, with models like <a href="https://openai.com/research/gpt-4">GPT-4</a>, <a href="https://openai.com/blog/chatgpt">ChatGPT</a> and <a href="https://stablediffusionweb.com/">Stable Diffusion</a> transforming human-machine interactions. These models empower users to generate inventive visual and textual content, engage in life-like conversations and more. [<a href="https://www.nvidia.com/en-us/glossary/data-science/generative-ai/?ncid=so-link-326637#cid=dl31_so-link_en-us">1</a>]</p><p>What sets generative AI models apart from previous deep learning models is their complexity, size and extensive training on massive datasets. As a result, they exhibit advanced capabilities, such as generating new content from prompts, engaging in logical reasoning, solving mathematical problems and passing human-like tests, which were previously unattainable with the previous generation of AI models.</p><p>In contrast to traditional deep learning models that rely on supervised or semi-supervised methods for specific tasks, generative AI models employ unsupervised techniques, making them suitable for a wide array of downstream applications. This versatility <a href="https://arxiv.org/abs/2108.07258">has cemented their status</a> as foundational models in the field.</p><h3>What are the different types of Generative AI models?</h3><p>Currently, there are four main types of generative AI models:</p><ol><li><strong>Generative Adversarial Networks (GANs): </strong><a href="https://arxiv.org/abs/1406.2661">Introduced in 2014</a>, GANs consist of two neural networks: a generator that creates new examples and a discriminator that distinguishes between real and generated content. As the generator improves content quality, the discriminator becomes better at identifying generated content, resulting in a continuous cycle of improvement. GANs are known for producing high-quality samples quickly but often lack sample diversity, making them better suited for domain-specific data generation.</li><li><strong>Variational Autoencoders (VAEs): </strong><a href="https://arxiv.org/abs/1312.6114">VAEs</a> consist of two neural networks, the encoder and decoder. The encoder compresses input data into a dense representation, while the decoder reconstructs the original input data. This process allows for the generation of novel data by sampling new latent representations that are mapped through the decoder. Although VAEs generate outputs faster than diffusion models, they often lack detail in comparison.</li><li><strong>Diffusion Models: </strong>Diffusion models, also known as <a href="https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf">denoising diffusion probabilistic models (DDPMs)</a>, have become the new state-of-the-art in image generation, outperforming previous approaches like GANs. These models involve a two-step process: forward diffusion, which adds random noise to training data, and reverse diffusion, which reconstructs the data samples by reversing the noise. By starting the reverse de-noising process from random noise, novel data can be generated. Although diffusion models are considered foundational models due to their high-quality outputs and flexibility, their reverse sampling process can be slow and time-consuming.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJCr17jFXANbkq81XCMxaQ.png" /><figcaption>Photo Credit: <a href="https://www.nvidia.com/en-us/glossary/data-science/generative-ai/?ncid=so-link-326637#cid=dl31_so-link_en-us">Nvidia</a></figcaption></figure><p>Diffusion models are closely related to GANs, and have largely replaced GANs in many applications. Moreover, they share similarities with <a href="https://towardsdatascience.com/denoising-autoencoders-explained-dbb82467fc2">de-noising autoencoders</a>, to the extent that <a href="https://sander.ai/2022/01/31/diffusion.html">some researchers</a> have even contended that diffusion models are a specific type of autoencoders. The performance of diffusion models in various applications, such as image synthesis, video generation and molecule design, has captured the attention of the machine learning community, turning it into a burgeoning research field.</p><p>For a comprehensive literature review on diffusion models, please refer to <a href="https://arxiv.org/pdf/2209.00796.pdf">this survey paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*rbh-_ktNtfaK1G9atQqNSw.png" /><figcaption>Credit: <em>Diffusion Models: </em><a href="https://arxiv.org/pdf/2209.00796.pdf"><em>A Comprehensive Survey of Methods and Applications</em></a></figcaption></figure><p><strong>4. Transformer-based models:</strong> Transformers, which were introduced in the <a href="https://research.google/pubs/pub46201/">groundbreaking 2017 paper “Attention is All You Need,”</a> are a class of deep learning models that have transformed natural language processing.</p><p>These models, also called Large Language Models (LLMs), are distinct in that they process sequential input data in a non-sequential manner and rely on self-attention and positional encoding mechanisms. Self-attention determines the importance of different parts of the input, while positional encoding captures the order of input words.</p><p>Transformer networks consist of multiple layers, such as self-attention, feed-forward and normalization layers, and are excellent at deciphering and predicting tokenized data, making them highly effective for text-based generative AI applications. For more info about the attention mechanism and transformer architecture, you can read this <a href="https://jaketae.github.io/study/transformer/">article</a> or watch this <a href="https://www.youtube.com/watch?v=iDulhoQ2pro">video</a>.</p><p>In the last few years, dozens of models belonging to the transformer family have emerged, each with a quirky name that may not be self-explanatory.</p><p>For a comprehensive but straightforward overview of these models, check out <a href="https://arxiv.org/pdf/2302.07730.pdf">this survey paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gFFMveyoVqjtG-Hoc0zRw.png" /><figcaption>Photo Credit: <a href="https://arxiv.org/pdf/2302.07730.pdf">Transformer Models: An introduction and catalog</a></figcaption></figure><p>To evaluate and compare LLMs, benchmarks such as <a href="https://crfm.stanford.edu/helm/latest/?">HELM</a> from Stanford University are available (36 models, 57 metrics, and 42 scenarios).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I06xiijJTIXyQktV5_yAAw.png" /><figcaption>Photo credit: <a href="https://crfm.stanford.edu/helm/latest/?groups=1">HELM</a></figcaption></figure><p>And survey papers such as <a href="https://arxiv.org/pdf/2303.18223.pdf">this paper</a> from University of China:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*au8ZXUh31W9KzhgCstLdRQ.png" /></figure><h3>Four ways to customize pre-trained models</h3><p>As alluded to earlier, established players have access to the data and intellectual property needed for training large-scale foundation models.</p><p>In contrast, numerous small to medium-sized startups typically do not possess the resources (such as expertise, computational power, and funding) to build GPT-like models from scratch.</p><p>To address this, we present four alternative methods for customizing pre-trained models, which may help startups to achieve differentiation and deliver a personalized experience to their customers without the burden of training massive models from the ground up [<a href="https://bootcamp.uxdesign.cc/3-ways-to-tailor-foundation-language-models-like-gpt-for-your-business-e68530a763bd">1</a>, <a href="https://towardsdatascience.com/specialized-llms-chatgpt-lamda-galactica-codex-sparrow-and-more-ccccdd9f666f">2</a>, <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">3</a>, <a href="https://research.ibm.com/blog/what-is-ai-prompt-tuning">4</a>]:</p><ol><li><strong>Fine-tuning</strong></li></ol><p>Fine-tuning is a popular approach that involves updating some of a pre-trained model’s parameters with additional labeled data to adapt a general-purpose model for a specific task. This technique enables the fine-tuned model to retain knowledge from its pre-training process while becoming more proficient in a specialized domain. However, fine-tuning carries the risk of overfitting, where the model becomes excessively specialized in one area, losing its ability to perform well in other tasks.</p><p>A recent variation of fine-tuning, particularly for LLMs, is called <a href="https://arxiv.org/pdf/2109.01652.pdf#:~:text=Instruction%20tuning%20finetunes%20a%20pretrained,were%20seen%20during%20instruction%20tuning.">instruction tuning</a>. Instruction tuning refines a pre-trained language model on a mixture of tasks formulated as instructions. As shown in <a href="https://arxiv.org/pdf/2109.01652.pdf#:~:text=Instruction%20tuning%20finetunes%20a%20pretrained,were%20seen%20during%20instruction%20tuning.">this paper</a> by Google, this method can enhance the model’s performance on specific tasks that require intricate instructions.</p><p>There are <a href="https://www.youtube.com/watch?v=nnylYEh4bpI">several common fine-tuning strategies</a>, including:</p><p><strong>a. Updating the embedding layer(s):</strong> This strategy focuses on refining the layers responsible for converting input data into meaningful representations. For T5-large, this strategy involves updating approximately 32 million parameters.</p><p><strong>b. Updating the language modeling head:</strong> This approach targets the components of the model responsible for generating output predictions. In the case of T5-large, this strategy would involve updating approximately 32 million parameters.</p><p><strong>c. Updating all parameters of the model:</strong> This comprehensive strategy involves updating the entire model, including both the aforementioned components and other layers. For T5-large, this strategy means refining approximately 770 million parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*ItBW7EaNgOP3VOVH4IMUWg.png" /><figcaption>Photo Credit: <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">Google</a></figcaption></figure><p><strong>2. Prompt engineering</strong></p><p>Prompt engineering offers an alternative to fine-tuning large language models for specific tasks. This approach emerged with <a href="https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html">GPT-3</a>, which convincingly demonstrated that a frozen model could be conditioned to perform various tasks through “in-context” learning.</p><p>With prompt engineering, users condition the model for a particular task by hand-crafting a text prompt that includes a description (zero-shot) or examples of the task (few-shot). For instance, to prime a model for sentiment analysis, one could use the prompt, “Is the following movie review positive or negative?” followed by the input sequence, “This movie was amazing!” Researchers found that specific prompt engineering methods, such as <a href="https://www.allabtai.com/prompt-engineering-tips-zero-one-and-few-shot-prompting/#:~:text=Few%2Dshot%20prompting%20is%20a,examples%20of%20previously%20seen%20objects.">few-shot prompting</a> or <a href="https://arxiv.org/pdf/2201.11903.pdf">chain-of-thought prompting</a>, can significantly enhance output quality without the need for fine-tuning.</p><p>This approach simplifies the process of serving a model for multiple downstream tasks by allowing a single frozen pre-trained language model to be shared across tasks. Unlike fine-tuning, it eliminates the need to store and serve separate tuned models for each downstream task.</p><p>However, based on these studies and research papers, there may be some drawbacks. Text prompts require manual effort to design, and their performance often falls short compared to fine-tuned models. <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">For example</a>, a frozen GPT-3 model with 175 billion parameters scores 5 points lower on the SuperGLUE benchmark than a fine-tuned T5 model with 800 times fewer parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*i9n0MlFawxICoIkRItDvEg.png" /><figcaption>Photo Credit: <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">Google</a></figcaption></figure><p><strong>3. Prompt-tuning or optimization</strong></p><p>The research community soon realized that by moving beyond prompt engineering (manually crafted prompts) and treating prompts as tunable parameters, they could optimize LLM performance for specific tasks.</p><p>Prompt-tuning serves as a compromise between fine-tuning and prompt engineering, as it only tunes the prompt parameters for a specific task without retraining the model or updating its other parameters. This method conserves resources compared to fine-tuning, delivers higher-quality outputs than prompt engineering and reduces the need for manual labor in crafting prompts.</p><p>For instance, in the case of T5-large, prompt-tuning involves adjusting approximately 50,000 parameters, as opposed to the 32 to 770 million parameters that would be updated during fine-tuning [<a href="https://www.youtube.com/watch?v=nnylYEh4bpI">1</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*V_b-tur8NcTAFz3fInIDXA.png" /><figcaption>Photo Credit: <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">Google</a></figcaption></figure><p>In particular, the automatically-induced prompts can be further separated into two groups: 1) discrete or hard prompts, where the prompt is an actual text string and 2) continuous or soft prompts, where the prompt is instead described directly in the embedding space of the underlying LLM [<a href="https://dl.acm.org/doi/pdf/10.1145/3560815">1</a>].</p><p>We will briefly explain both hard or soft prompts below:</p><p><strong>Discrete or hard prompts</strong></p><p>Discrete prompts, also known as hard prompts, involve automatically searching for templates in a discrete space, usually corresponding to natural language phrases.</p><p>Various methods have been proposed for discrete prompts [<a href="https://dl.acm.org/doi/pdf/10.1145/3560815">1</a>] including:</p><p><strong>a. Prompt Mining:</strong> Automatically finds templates using a large text corpus, searching for frequent middle words or dependency paths between inputs and outputs.<br><strong>b. Gradient-based Search: </strong>Iteratively searches for short sequences that trigger pre-trained language models to generate desired predictions by applying gradient-based search over tokens.<br><strong>c. Prompt Generation: </strong>Treats prompt creation as a text generation task and uses pre-trained models like T5 or reinforcement learning techniques to generate prompts.</p><p><strong>Continuous or soft prompts</strong></p><p>Similar to hand-crafted prompts, soft prompts are combined with input text. However, instead of using existing vocabulary tokens, soft prompts consist of learnable vectors that can be optimized end-to-end across a training dataset. While soft prompts are not immediately interpretable, they intuitively extract evidence on how to perform a task from labeled datasets.</p><p>This approach serves the same purpose as manually written text prompts but without the constraints of discrete language. It eliminates the need for manual prompt creation and allows the soft prompt to efficiently process and condense information from extensive datasets containing thousands or even millions of examples. In contrast, engineered prompts typically face limitations due to model input length constraints, accommodating fewer than 50 examples.</p><p><a href="https://arxiv.org/abs/2104.08691">In a recent paper</a>, Google researchers demonstrated that it is possible to create a soft prompt for a specific task by initially configuring the prompt as a fixed-length sequence of vectors (for example, 20 tokens long). These vectors are appended to the beginning of each embedded input, and the combined sequence is then fed into the model. The model’s prediction is compared to the target in order to calculate the loss, and the error is back-propagated to compute gradients. However, only the learnable vectors are updated, while the core model remains unchanged.</p><p>Since this discovery, numerous soft prompt optimization techniques have been developed, including:</p><p><strong>a. Prefix Tuning: </strong>Prepends a sequence of continuous task-specific vectors to the input while keeping the language model parameters frozen. This method is more sensitive to different initialization in low-data settings than discrete prompts.<br><strong>b. Tuning Initialized with Discrete Prompts: </strong>Initializes the search for a continuous prompt using a prompt created or discovered via discrete prompt search methods. This approach can provide a better starting point for the search process.<br><strong>c. Hard-Soft Prompt Hybrid Tuning:</strong> Inserts tunable embeddings into a hard prompt template, combining learnable and fixed elements. One example is P-tuning, which represents prompt embeddings as the output of a BiLSTM and introduces task-related anchor tokens within the template. Another example is Prompt Tuning with Rules (PTR), which uses manually crafted sub-templates and logic rules to compose a complete template, inserting virtual tokens with tunable embeddings for enhanced representation ability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*SinvgH5VbKL8ztwsBthsnA.png" /><figcaption><strong>Left</strong>: With model tuning, incoming data are routed to task-specific models. <strong>Right</strong>: With prompt tuning, examples and prompts from different tasks can flow through a single frozen model in large batches, better utilizing serving resources. Photo Credit: <a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">Google</a>.</figcaption></figure><p>Prompt-tuning, which originated with large language models, has since been extended to other domains such as audio and video due to its practical efficiency. Prompts can take the form of text snippets, speech streams or pixel blocks in still images or videos. For more information, refer to this <a href="https://dl.acm.org/doi/pdf/10.1145/3560815">comprehensive survey paper</a>.</p><p><strong>4. Reinforcement learning from human feedback (RLHF)</strong></p><p>Reinforcement Learning from Human Feedback (RLHF) is a method used to improve language models by aligning them with human values that are not easily quantifiable (e.g., funny, helpful). They can help models avoid erroneous answers, reducing bias and hallucinations.</p><p>The process consists of <a href="https://huggingface.co/blog/rlhf">three main steps</a>:</p><p><strong>a. Pre-training a Language Model (LM):</strong> This step involves using an existing language model as a starting point. This pre-trained model has already been trained on a large corpus of text data and can generate text based on given prompts. Examples include smaller versions of GPT-3 or DeepMind’s Gopher. The choice of the initial model depends on the specific application and is not yet standardized.</p><p><strong>b. Gathering Data and Training a Reward Model (RM):</strong> In this step, human feedback is collected to create a dataset that assigns a scalar reward to the text based on human preferences. Prompts are passed through the initial language model to generate new text, which is then ranked by human annotators. Various ranking methods can be used to convert these human rankings into a scalar reward signal for training. The reward model aims to represent human preferences numerically, and it is used to guide the fine-tuning of the language model in the next step.</p><p><strong>c. Fine-tuning the Language Model with Reinforcement Learning (RL):</strong> The final step involves optimizing the initial language model using reinforcement learning. The objective is to update the model parameters to maximize the reward metrics obtained from the reward model. The fine-tuning process involves formulating the task as a Reinforcement Learning (RL) problem, defining the policy, action space, observation space and reward function, and then using an RL algorithm (e.g., Proximal Policy Optimization (PPO) or Advantage Actor-Critic (A2C)) to optimize the model based on these components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*luOL5f21_ckJtXg7ZbWh7A.png" /><figcaption>Photo Credit: <a href="https://huggingface.co/blog/rlhf">Hugging Face</a></figcaption></figure><p>Overall, RLHF is an exciting and complex research area that aims to improve language models by directly optimizing them using human feedback. The method has been successfully implemented in models like GPT-4 and ChatGPT but requires further exploration to optimize the design space and understand its full potential. For more technical details please check these blog posts (<a href="https://huggingface.co/blog/rlhf">1</a> and <a href="https://huggingface.co/blog/trl-peft">2</a>) from Hugging Face.</p><h3>Comparison between the four methods</h3><p>Here we briefly describe the differences between these four methods.</p><p>We do this by looking into four different dimensions and examining how each method ranks for each dimension:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_GN6PH7EdKYerliMX3HGKw.png" /><figcaption>Photo Credit: <a href="https://georgian.io/resources/">Georgian</a></figcaption></figure><ol><li><strong>Data Volume:</strong> When it comes to data volume, in our view, prompt engineering stands out as the most efficient method.<br>Prompt engineering can be executed with just a task description (zero examples) or a task description accompanied by as many examples as the model allows. Due to model input size limitations, most models can accommodate fewer than 100 examples. If there are more examples than the model’s input can handle, prompt optimization methods should be considered, and soft prompts can be utilized for higher performance if explainability is not a concern. Prompt optimization methods demand significantly less data compared to fine-tuning, as they update far fewer parameters. Depending on the size of your model, you may need a few thousand to a few hundred thousand examples to go with fine-tuning or RLHF. The choice between fine-tuning or RLHF will depend on the nature of the objective. In our view, it would be efficient to opt for fine-tuning if the objective is easily quantifiable, such as improving performance for a specific task, or choose RLHF if the objective is not easily quantifiable, like making a chat assistant more ethical and harmless.</li><li><strong>Training Costs:</strong> There are no training costs for prompt engineering, and the costs remain low if you want to experiment with various prompts and examples to optimize performance. Prompt optimization methods also have relatively low costs, as they involve tuning a small number of parameters. In comparison, fine-tuning can incur higher costs, as it involves updating some or all of the model’s parameters.<br>RLHF has relatively high costs, as not only does it require tuning some or all of the model’s parameters, but it also necessitates paying human annotators to create high-quality data. It’s crucial to note that data quality is vital for achieving exceptional performance and should not be compromised.</li><li><strong>Technical Difficulty: </strong>In our view, among the four approaches, prompt engineering and fine-tuning are the easiest to implement, as numerous platforms and tools support execution with just a few lines of code. In contrast, prompt optimization methods are more recent, and there is a lack of sufficient tools or platforms to assist the user. The same applies to RLHF. While there have been significant advancements, such as the emergence of platforms that simplify collecting human feedback (e.g., <a href="https://www.surge.ai/">Surge.AI</a>) and support from key players like Hugging Face (<a href="https://huggingface.co/blog/trl-peft">code example</a>), we believe that this area is still relatively new and requires time to mature.</li><li><strong>Performance Gain:</strong> Although prompt engineering methods can improve performance, particularly when providing the model with a few examples, they may still lag behind other approaches. <br>Prompt optimization methods have proven to be more effective than prompt engineering methods, but they only achieve parity with fine-tuning for models with 10 billion or more parameters [<a href="https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html">1</a>]. Both fine-tuning and RLHF require a substantial amount of data, but they can lead to the most significant performance gains. The choice between fine-tuning and RLHF ultimately depends on the specific goal and whether it is quantifiable or not.</li></ol><h3>Takeaways</h3><p>In this post, we discussed various types of generative AI models and four approaches to customize them for a specific use case. We believe that it is important to have a good understanding of what is available, choose the relevant models for the use case, and further tailor the models using appropriate techniques and relevant data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6871b44cb6a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/georgian-impact-blog/getting-started-with-generative-ai-4-ways-to-customize-your-models-f6871b44cb6a">Getting started with Generative AI: 4 ways to customize your models</a> was originally published in <a href="https://medium.com/georgian-impact-blog">Georgian Impact Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>