<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gokul on Medium]]></title>
        <description><![CDATA[Stories by Gokul on Medium]]></description>
        <link>https://medium.com/@gokulrajar?source=rss-5f2e53709304------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YHm543qvJZQxLrvFfxybIg.jpeg</url>
            <title>Stories by Gokul on Medium</title>
            <link>https://medium.com/@gokulrajar?source=rss-5f2e53709304------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:50:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gokulrajar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Game-Changer for Language Models: Self-Play Fine-Tuning Takes AI to New Heights!]]></title>
            <link>https://medium.com/@gokulrajar/the-game-changer-for-language-models-self-play-fine-tuning-takes-ai-to-new-heights-a0d54d475059?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0d54d475059</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Wed, 10 Jan 2024 17:00:01 GMT</pubDate>
            <atom:updated>2024-01-10T17:00:01.095Z</atom:updated>
            <content:encoded><![CDATA[<h4>SPIN: An Easy Way to Make Language Models Better Without More Human Help.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*Qq8FpeSicVDXixCz.jpg" /></figure><p><a href="https://arxiv.org/abs/2401.01335">SPIN</a> works by letting the language model practice with itself, like playing a game against its earlier versions. This helps the model learn better without needing extra data from humans. The paper shows that SPIN can make language models more accurate and efficient.</p><h4><strong>Challenges with Current Fine-Tuning Methods</strong></h4><p>There are mainly two stages involved in aligning a Large Language Model (LLM) to act as a more responsible AI.</p><ol><li>Supervised Fine-Tuning : This is when a Large Language Model (LLM) is trained again with a smaller set of specific examples to get better at certain tasks. But, doing this training over and over with the same examples doesn’t really help the model improve much after a certain point.</li><li>RLHF: In this method, the LLM is trained by rewarding it when it does things correctly. However, finding the right way to reward the model is hard and requires a lot of data that humans curated which more costly. It’s a method that needs a lot of effort and resources to work well.</li></ol><p>Both of these methods have their challenges. SFT can reach a limit in how much it helps, and RL fine-tuning needs a lot of preparation and resources.</p><p>The paper introduces a new method called <strong>Self-Play Fine-Tuning (SPIN)</strong>. This method lets the LLM create its own training data by essentially practicing on its own. This new approach could overcome the limitations of the other two methods by allowing the LLM to keep improving without needing <strong>large amounts of new data</strong> or <strong>extensive help from humans</strong>.</p><h4>SPIN — A Novel Approach</h4><p>Self-Play Fine-Tuning (SPIN) is a new way to make large language models better. It works by having the model talk to itself. The model creates its own training data from these self-conversations and learns to tell the difference between its own responses and those from humans. This helps the model get better at matching the kind of language used in real data. SPIN improves LLMs without needing new human-made data or expert help. The goal is for the model to reach a point where it works perfectly with the type of language it’s being trained on. Tests have shown that this method really helps improve how well LLMs perform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B5iKRc1ZzytGP1ovaIAj4Q.png" /></figure><h4>Steps to implement SPIN approach.</h4><p><strong>1.Select a Supervised Fine-Tuned (SFT) LLM</strong>: Start with an LLM that has been fine-tuned with a specific dataset.</p><p><strong>2. Generate New Synthetic Responses</strong>: Use this SFT model to create new, synthetic responses for the dataset. For instance, generate about 50,000 samples<strong>(</strong><a href="https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized"><strong>Refer Here</strong></a><strong>).</strong></p><p><strong>3. Create a Pairwise Dataset</strong>: Form a new dataset comprising pairs of original (accepted) responses and the new synthetic (rejected) responses.</p><p><strong>4. Train the LLM on the Pairwise Dataset</strong>: Teach the LLM to better predict real responses over the synthetic ones, using a specific kind of error measurement called logistic loss.</p><p><strong>5. Evaluate the Refined LLM</strong>: Finally, test the improved LLM on various benchmarks to see how well it performs after this self-play fine-tuning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wnm1kkXidndaR4J8_b8a-A.png" /></figure><p>the steps 1 to 3 of the Self-Play Fine-Tuning (SPIN) method are repeated like this:</p><ol><li>Select the Latest Model: Choose the most recently improved language model.</li><li>Generate New Responses: Make this model create new responses. Now, these responses can include ones from the current version of the model and the version just before it.</li><li>Create a New Comparison Dataset: Put together a dataset that has both these new responses and the original human responses.</li></ol><p>By doing these steps over and over, the model gets better at making responses that are more like those a human would give. Each time it goes through this process, it learns to be more human-like in its responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*XGjOvR1cek4EKxZJAInRmQ.png" /></figure><p>This approach allows the LLM to refine its understanding and generation of language by learning to distinguish between real human responses and its own generated ones, thereby enhancing its overall performance.</p><p><strong>Models</strong> : <a href="https://huggingface.co/UCLA-AGI">https://huggingface.co/UCLA-AGI</a></p><h4>Conclusion</h4><p>In summary, Self-Play Fine-Tuning (SPIN) is a really cool and effective way to make large language models (LLMs) much smarter. It’s like teaching the model to have conversations with itself and learn from them. This way, the model gets better at understanding and responding like a human would. It’s like a self-improvement journey for the model, and it’s pretty impressive how much it can learn and grow through this process. It’s exciting to see such smart technology getting even smarter!</p><p>Reference : <a href="https://arxiv.org/abs/2401.01335">https://arxiv.org/abs/2401.01335</a></p><p>Follow us for more update😀.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0d54d475059" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention Mechanism Math Illustration : Transformers Series - Part 1]]></title>
            <link>https://medium.com/@gokulrajar/attention-mechanism-math-illustration-transformers-series-part-1-37c24ac9d2f2?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/37c24ac9d2f2</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Sun, 07 Jan 2024 18:24:23 GMT</pubDate>
            <atom:updated>2026-04-01T16:14:29.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>Attention Mechanism Illustration : Transformers Series - Part 1.</h3><h4>In this article, I will provide a step-by-step explanation of how the attention mechanism works and why such mathematical calculations are used.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kf871smtQKXAf3dSYOlVPA.png" /><figcaption>Image by author</figcaption></figure><h3>Transformers</h3><p>Transformers is a type of neural network architecture that has become particularly popular for handling sequential data, such as text or time series data. The <a href="https://arxiv.org/abs/1706.03762">Transformer architecture</a> was introduced in June 2017. The focus of the original research was on translation tasks.</p><h3>Attention Mechanism</h3><p>The main idea behind self-attention mechanism is quite simple: it’s like each part of the input (like each word in a sentence) has a pair of “magic glasses” that allows it to look at every other part and understand how important they are to it. much like how we are focus on particular aspects of a scene or conversation.</p><h4><strong>Scenario:</strong></h4><p>Imagine you are reading a recipe for making a cake. The recipe has various steps and ingredients listed. Your goal is to understand and remember the steps to make the cake.</p><p><strong>Without Attention Mechanism: </strong>You read the entire recipe from start to finish in one go. Then, you try to recall and execute the steps from memory. If the recipe is long or complex, you might forget or mix up the steps.</p><p><strong>With Attention Mechanism: </strong>As you read each step, you focus specifically on that part of the recipe, understanding and remembering it in detail.When you’re actually performing a step, your attention goes back to the specific part of the recipe relevant to that step.</p><p>For example, when mixing ingredients, you focus on the section listing quantities and mixing techniques. This way, you efficiently recall and apply the information relevant to each step, leading to better execution.</p><h3><strong>Attention Equation</strong></h3><p>The formula behind the attention mechanism basically a matrix multiplication and scaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*HBqw4CR6aa_Sowa1Z2gn5Q.gif" /><figcaption>Image by author</figcaption></figure><blockquote>look’s confusing right ? let’s deep dive into that</blockquote><h3><strong>Attention Architecture Breakdown</strong></h3><p>The following terms used in the diagram was</p><p><em>n — number of token(Assume number of words in a sentence)</em></p><p><strong>For example:</strong> “The deer has run away” = 5 tokens. for simplicity we’ll take word as a token but in reality may be the word is splitted into 2–3 tokens.</p><p><em>d — dimension of those tokens(length of the embeddings)</em></p><p><strong>For example:</strong> embedding for “I love transformers”</p><p>“I”: [-0.2, 0.5, -0.1, 0.8]</p><p>“love”: [-0.4, 0.7, 0.2, -0.5]</p><p>“transformers”: [0.6, -0.3, 0.7, -0.2]</p><p>the dimension of the embedding is 4.</p><p>Note: here I ignore the softmax and <em>sqrt(dk)</em> scaling term in the attention equation for visual simplicity, but I’ll discuss those later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7sy0KlRgyh3n0M0SH-6xqQ.gif" /><figcaption>Image by author</figcaption></figure><p>There are three main inputs to the attention mechanism: queries (Q), keys (K), and values (V). These inputs can all be described in terms of a “information retrieval” system of sorts. The general idea is that we submit a <em>query</em> for each token in the input sequence. We then take those queries and match them against a series of <em>keys</em> that describe <em>values</em> that we want to know something about. The similarity of a given query to the keys determines how much information from each value to retrieve for that particular query</p><p>For now, don’t worry too much about where Q, K, and V come from, we’ll cover this in upcoming blogs. However, for now, This is typically done using three different linear transformations (i.e., multiplying each feature vector by three different weight matrices). These matrices are learned during the training process. So the attention operation is an effective way of spreading information between tokens of the same sequence. This is known as <em>self attention</em>.</p><h4>Attention Matrix</h4><p>The output of the first matrix multiplication, where we take the similarity of each query to each of the keys, is known as the <em>attention matrix</em>. The attention matrix depicts how much each token in the sequence is paying attention to each of the keys (hence the <em>n</em> x <em>n</em> shape). If we zoom in to our earlier depiction of the attention operation we see the matrix multiplication to produce the attention matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tU8I08MkUqsg2UgP0lKCKg.gif" /><figcaption>Image by author</figcaption></figure><p>Let’s deep dive into the illustrative visualization of the Attention matrix. let’s take an example of Q, K, V vectors</p><p><strong>Q = [[0.2, 0.5, -0.1, 0.8], [-0.4, 0.7, 0.1, -0.7], [0.6, -0.3, 0.3, -0.2]]</strong></p><p><strong>K = [[0.8, 0.2, -0.9, 0.4], [-0.3, 0.1, 0.2, -0.5], [0.6, -0.8, 0.5, -0.7]]</strong></p><p><strong>V = [[-0.1, 0.7, 0.9, -0.2], [-0.6, 0.3, -0.5, 0.5], [-0.2, 0.4, 0.2, -0.6]]</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jx-Tfw_I0AJvWgKukGnYfw.gif" /><figcaption>Image by author</figcaption></figure><p>here we get a<strong> Attention Matrix(Q.K^t).</strong></p><p>But what’s the intuition behind why this matrix multiplication between the queries and keys works?</p><p>Before answering the above, let’s quickly review what the dot product between two vectors represents. For two vectors whose direction is similar the dot product will be large, while for two vectors pointing in different directions the dot product will be small.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWL4zmJl1yNoHtquvFTiJg.gif" /><figcaption>Image by author</figcaption></figure><p>Let us now think of our queries as a series of vectors, one for each token in the input matrix. We take the dot product of each query vector with each key (q1-d1, q1-d2, q1-d3, q1-d4)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfOgdrsFIyId5qLuflt5bw.gif" /><figcaption>Image by author</figcaption></figure><p>In the above illustration, we take the dot product for a single query with each key and are left with four scalar values. For keys close to the same direction as the query, the values will be large, while for keys differing in direction from the query vector the values would be low. These values are then passed through a softmax function to scale the values to a probability distribution that adds up to 1, and also sharpens the distribution so the higher values are even higher and lower values even lower. These values are what make up the <em>attention matrix,</em> and are a representation of how much each query matches a given key. Since typically the queries and keys are derived from the same sequence of input tokens, the attention values indicate how much each token in the sequence is “paying attention” to other tokens in the sequence.</p><p>Let’s continue the calculation which involves scaling the attention matrix <em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9mqRp6pRFiNNbhtFkvh5A.gif" /><figcaption>Image by author</figcaption></figure><h4>Softmax Function</h4><p>The softmax is statistical function for converting a vector of numerical values into a probability distribution. The values outputted by the softmax function are in the range (0, 1) and sum up to 1, making them ideal for representing probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_FnYsi15GfFDfSeJCr39w.gif" /><figcaption>Image by author</figcaption></figure><p>Let’s apply the softmax in the scaled attention matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Glirw8U7FJDmnlRiMVfHMw.gif" /></figure><h4>Routing Information</h4><p>Once we have the attention values, we must organize the information in the value matrix V. Now that we know how we want to guide information, we need to go ahead and guide the actual information stored in the values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PlXZb0VCZkBYxxjACm4kg.gif" /><figcaption>Image by author</figcaption></figure><p>To guide the information, we perform another dot product. This time, it’s between the softmax attention matrix and the value matrix. By doing this, we essentially “index” into the information for each value in the matrix based on the attention described in the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4414eiYfTJ2Ia_EQKcHXpg.gif" /><figcaption>Image by author</figcaption></figure><p>At the end, we obtain the output from the attention mechanism. This output is then passed through a feed-forward network and a normalization layer to stabilize the matrix. We will cover this in detail in upcoming articles.</p><p>This article features animations and a blog by community member <a href="https://www.linkedin.com/in/gokul-raja-1541b8226/"><strong>Gokul</strong></a>.</p><p>If you find this article useful, please follow and give it applause. This will help us create quality content and continue this series.</p><p>Reference</p><ol><li><a href="https://jalammar.github.io/illustrated-transformer/">https://jalammar.github.io/illustrated-transformer/</a></li><li><a href="https://arxiv.org/abs/1706.03762">https://arxiv.org/abs/1706.03762</a></li><li><a href="https://youtu.be/QCJQG4DuHT0?si=8QUsi4Vq0xFAFbQE">https://youtu.be/QCJQG4DuHT0?si=8QUsi4Vq0xFAFbQE</a></li><li><a href="https://storrs.io/">https://storrs.io/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37c24ac9d2f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeciLM-7B : The Fastest and Most Accurate 7 Billion-Parameter LLM to Date]]></title>
            <link>https://medium.com/@gokulrajar/decilm-7b-the-fastest-and-most-accurate-7-billion-parameter-llm-to-date-2c2acb75c234?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c2acb75c234</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[deci]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Tue, 12 Dec 2023 17:10:14 GMT</pubDate>
            <atom:updated>2024-10-23T08:09:09.430Z</atom:updated>
            <content:encoded><![CDATA[<h3>DeciLM-7B : The Fastest and Most Accurate 7 Billion-Parameter LLM to Date</h3><h4>Attention AI enthusiasts! Introducing the tabletop DeciLM-7B, the fastest and most accurate model ever, with an official fine-tuning guide.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LJEDzhbyeGU8L0ei.jpg" /></figure><p>Today <strong>deci</strong> have introduced a their <strong>DeciLM-7B</strong> parameter model which outperforms all 7B parameters models currently available and became a fastest and most proficient 7-billion-parameter base LLM <strong>available today</strong>, redefining the benchmarks for speed and accuracy.</p><h4>Breakthrough</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m59e4U-Zs0NVcSNs.png" /></figure><ol><li>Great Accuracy: DeciLM-7B achieved an impressive average score of 61.55 on the Open LLM Leaderboard. It outperformed competitors in its class, like Mistral 7B, showing that it gives more reliable and precise responses, especially in applications like customer service bots and data analysis.</li><li>Faster Performance: In a PyTorch benchmark, DeciLM-7B demonstrated improved speed. It performed 1.83 times faster than Mistral 7B and 2.39 times faster than Llama 2 7B when handling sequences of 2048 tokens for both input and output.</li><li>Speed Boost with Infery-LLM: When combined with Infery-LLM, the world’s fastest inference engine, DeciLM-7B showcased even faster speeds. It achieved speeds 4.4 times greater than Mistral 7B with vLLM. This makes it suitable for sectors that require high-speed and high-volume interactions, such as telecommunications, online retail, and cloud services.</li><li>Innovative Architecture: DeciLM-7B was created using the Neural Architecture Search-powered engine called AutoNAC. It uses something called variable Grouped Query Attention, which is a breakthrough in finding the right balance between accuracy and speed.</li><li>Instruction-Tuned Variant: Another version of DeciLM-7B, called DeciLM-7B-instruct, was fine-tuned using LoRA on the SlimOrca dataset. This version achieved an even higher average score of 63.19 on the Open LLM Leaderboard, showing its effectiveness without relying on some optimization techniques.</li></ol><figure><img alt="Competing with 5 to 10 times bigger models." src="https://cdn-images-1.medium.com/max/1024/1*uZZU22mOYdUnHJbw5LE0RA.png" /><figcaption>Competing with 5 to 10 times bigger models.</figcaption></figure><h4>Benchmarks</h4><p>DeciLM-7B is a top-performing model in a competition for large language models. It’s better than other models with similar or even double its size, like Llama-2–13B. DeciLM-7B does really well in different tests and has an average score of 61.55, beating the previous best model, Mistral 7B, which scored 60.97.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*rkQNCQ1DwRH8fOoA.png" /></figure><p><strong>Hugginface Playground : </strong><a href="https://huggingface.co/spaces/Deci/DeciLM-7B-instruct"><strong>DeciLM-7B-Instruct</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f6BQidMA1Yb552mMxbsUkQ.png" /></figure><h4>Enhanced Throughput and Efficiency</h4><p>DeciLM-7B is not only accurate but also sets a new standard in how quickly it processes information, leaving its competitors far behind. Built on NVIDIA A10 GPUs, it achieves an impressive 83% increase in throughput compared to Mistral 7B and an even more remarkable 139% improvement over Llama 2 7B. This showcases its outstanding design and engineering, establishing a new benchmark in the field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HGMaMa3sUGUmd2CK.png" /></figure><h4>The Infery-LLM Edge: Unparalleled Acceleration at High Volumes</h4><p>When DeciLM-7B works together with Infery-LLM optimization and Inference SDK, it performs much better. Compared to Mistral 7B, it’s 4.4 times faster, and compared to Llama-2–7B, it’s 5.8 times faster. This is still true even when Mistral 7B and Llama-2–7B use additional libraries like vLLM for inference and serving.</p><h4>Cost-Effective Deployment with DeciLM-7B</h4><p>Choosing DeciLM-7B with Infery-LLM is not just about getting better performance; it’s also a smart money move. This combo not only makes the model better but also saves a lot of money compared to what other providers charge for similar services. It’s a great choice for businesses that want to create, use, and expand applications with large language models while keeping their computing costs down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YGqasWUm_ywtz5Gj.png" /></figure><h4>Advantages</h4><ol><li>DeciLM-7B Overview: This model, DeciLM-7B, works really well because it uses something called Variable Grouped Query Attention (GQA), which is better than other methods.</li><li>Multi-Query Attention (MQA) and Limitations: MQA is a method that helps the model run faster but has some problems. It makes the model focus on certain things, but this can limit its ability to understand different patterns and relationships in data.</li><li>Grouped Query Attention (GQA): GQA is a better way. It lets the model look at different aspects of information by organizing its attention into groups. This helps the model understand things more accurately.</li><li>DeciLM-7B’s Variable GQA Implementation: DeciLM-7B uses a smart approach. It keeps some things the same for each layer but changes others. This way, some parts of the model can work really fast (like MQA), and others can focus more on accuracy (like GQA).</li><li>NAS Engine (AutoNAC) Behind DeciLM-7B: DeciLM-7B was made using a clever tool called AutoNAC. This tool helps find the best way to build models without needing a lot of computer power. It played a big role in creating other efficient models like DeciCoder 1B, DeciDiffusion 1.0, YOLO-NAS, and YOLO-NAS pose. For DeciLM-7B, AutoNAC was especially important in figuring out the best settings for different parts of the model.</li></ol><h4>Conclusion</h4><p>DeciLM-7B’s outstanding performance, combined with substantial cost savings and a dedication to open-source principles, positions it as a fundamental element in the advancement of applications based on large language models (LLM). As we strive to explore the limits of AI, DeciLM-7B serves as proof of our commitment to innovation and making advancements accessible in the field.</p><ul><li>Dive Deeper: Delve into their <a href="https://bit.ly/decilm-7b-notebook">fine-tuning notebook</a> for a detailed guide on fine-tuning DeciLM-7B.</li><li>Playground : <a href="https://console.deci.ai/infery-llm-demo?utm_campaign=Deci%20Blog&amp;utm_source=blog&amp;utm_medium=Link&amp;utm_content=Introducing%20DeciLM-7B%20%3A%20The%20Fastest%20and%20Most%20Accurate%20Language%20Model%20to%20Date">DeciLM-7B</a></li><li>Get Started: Access and <a href="https://huggingface.co/Deci/DeciLM-7B">download the model</a> seamlessly from the Hugging Face repository.</li></ul><p><strong>Follow us on medium for more update.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c2acb75c234" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy LLM for Free of Cost.]]></title>
            <link>https://medium.com/@gokulrajar/how-to-deploy-llm-for-free-of-cost-6e7947d9b64a?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e7947d9b64a</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Fri, 08 Dec 2023 17:47:21 GMT</pubDate>
            <atom:updated>2024-10-23T08:11:49.981Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide to deploy a large language models on production at zero cost.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*yIJqvC4P_qYsVnBBC04pcg.png" /></figure><p>In the previous article, we saw the production-ready RAG system at zero cost. in that we utilized the hugging face space-free tier for deployment.</p><p><a href="https://gathnex.medium.com/build-your-own-production-ready-retrival-augmented-genration-system-at-zero-cost-b619c26c10c1">Build your own Production ready Retrival Augmented Genration-System at Zero Cost.</a></p><p>We are going to use the same Hugging Face space to deploy our LLM. Hugging Face provides CPU Basic, 2 vCPU with 16GB RAM, and 50GB Storage Free!</p><blockquote>Wait! What? Can we deploy the LLM on this resource? Is this possible?</blockquote><p>The answer was yes! It is possible with some catch. We can run an LLM on the CPU with the help of <a href="https://github.com/marella/ctransformers">Ctransformers</a>. <strong>CTransformer</strong> library is a Python package that provides access to Transformer models implemented in C/C++ using the GGML library.</p><h4>GGML</h4><p><a href="https://github.com/ggerganov/ggml"><strong>GGML</strong></a> is a C library for machine learning. It helps run large language models (LLMs) on regular computer chips (CPUs). It uses a special way of representing data (binary format) to share these models. To make it work well on common hardware, GGML uses a technique called quantization. This technique comes in different levels, like 4-bit, 5-bit, and 8-bit quantization. Each level has its own balance between efficiency and performance.</p><h4>Drawbacks</h4><p>The main drawback here is latency. Since this model will run on CPUs, we won’t receive responses as quickly as a model deployed on a GPU , but the latency is not excessive. On average, it takes around 1 minute to generate 140–150 tokens on huggingface space. Actually, it performed quite well on local system with a 16-core CPU, providing responses is less than 15 sec.</p><h4>Model</h4><p>For demo purposes, we are going to deploy the Zephyr-7B-Beta SOTA model. You can find various sizes of Zephyr GGUF quantized format in <a href="https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/tree/main">Huggingface</a>.</p><blockquote>Don’t get confused. GGUF is updated version of GGML, offering more flexibility, extensibility, and compatibility. It aims to simplify the user experience and accommodate various models. GGML, while a valuable early effort, had limitations that GGUF seeks to overcome.</blockquote><p><strong>Zephyr GGUF </strong>: <a href="https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/tree/main">https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/tree/main</a></p><p>We have selected <a href="https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/tree/main">Zephyr-7b-beta.Q4_K_S.gguf</a>. It’s a 4-bit quantized small model, which is fast and has minimal loss of quality.</p><p><strong>Note</strong> : Response, latency, and accuracy may change according to the size of the models you select.</p><h3><strong>Coding Time</strong></h3><p>Let’s explore deployment in detail. We’ve deployed the Zephyr model in two environments:</p><ol><li>One designed for creating APIs.</li><li>Another serving as a playground, enabling you to interact with the model.</li></ol><h4>1. Deployment LLM for API</h4><p>Deployment structure</p><p>LLM_Deployment_at_zerocost<br>├── Dockerfile<br>├── main.py<br>├── requirements.txt<br>└── zephyr-7b-beta.Q4_K_S.gguf</p><p><strong>Refer to our Hugging Face space files structure to get an idea</strong> : <a href="https://huggingface.co/spaces/gathnex/LLM-deployment-zerocost-api/tree/main">https://huggingface.co/spaces/gathnex/LLM-deployment-zerocost-api/tree/main</a></p><p><strong>requirements.txt</strong></p><pre>python-multipart<br>fastapi<br>pydantic<br>uvicorn<br>requests<br>python-dotenv<br>ctransformers</pre><p><strong>zephyr-7b-beta.Q4_K_S.gguf</strong></p><p>Download zephyr-7b-beta.Q4_K_S.gguf model on <a href="https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/tree/main">huggingface</a>.</p><p><strong>main.py</strong></p><p>The main.py file contains a FastAPI function that returns a Zephyr-7B completion.</p><pre>from ctransformers import AutoModelForCausalLM<br>from fastapi import FastAPI<br>from pydantic import BaseModel<br><br><br>llm = AutoModelForCausalLM.from_pretrained(&quot;zephyr-7b-beta.Q4_K_S.gguf&quot;,<br>model_type=&#39;mistral&#39;,<br>max_new_tokens = 1096,<br>threads = 3,<br>)<br><br>#Pydantic object<br>class validation(BaseModel):<br>    prompt: str<br>#Fast API<br>app = FastAPI()<br><br>@app.post(&quot;/llm_on_cpu&quot;)<br>async def stream(item: validation):<br>    system_prompt = &#39;Below is an instruction that describes a task. Write a response that appropriately completes the request.&#39;<br>    E_INST = &quot;&lt;/s&gt;&quot;<br>    user, assistant = &quot;&lt;|user|&gt;&quot;, &quot;&lt;|assistant|&gt;&quot;<br>    prompt = f&quot;{system_prompt}{E_INST}\n{user}\n{item.prompt}{E_INST}\n{assistant}\n&quot;<br>    return llm(prompt)</pre><p><strong>Dockerfile</strong></p><p>Finally, the Dockerfile was used to containerize our application for deployment.</p><pre>FROM python:3.9<br><br>WORKDIR /code<br><br>COPY ./requirements.txt /code/requirements.txt<br><br>RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt<br><br>COPY ./zephyr-7b-beta.Q4_K_S.gguf /code/zephyr-7b-beta.Q4_K_S.gguf<br>COPY ./main.py /code/main.py<br><br>CMD [&quot;uvicorn&quot;, &quot;main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;7860&quot;]</pre><p>Now, our files are ready for deployment.</p><p><strong>Deployment</strong></p><p>Huggingface space</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*wb46QjzMwe8V-Lu2.png" /></figure><p>Step 1: Create a new <a href="https://huggingface.co/spaces"><strong>Hugging Face space.</strong></a></p><p>Step 2: Enter the space name and choose the license type.</p><p>Step 3: Select the Space SDK-Docker (Blank) option.</p><p>Step 4: Choose ‘Public’ initially to copy our deployment endpoint, and then hit ‘Create Space.</p><p><strong>Refer to our Hugging Face space files structure to get an idea</strong> : <a href="https://huggingface.co/spaces/gathnex/LLM-deployment-zerocost-api/tree/main">https://huggingface.co/spaces/gathnex/LLM-deployment-zerocost-api/tree/main</a></p><p>Step 5: Upload the files we have created above.</p><p>LLM_Deployment_at_zerocost<br>├── Dockerfile<br>├── main.py<br>├── requirements.txt<br>└── zephyr-7b-beta.Q4_K_S.gguf</p><p>Congratulations! you have deployed your LLM.</p><p>If there are no errors in the code, you’ll see that it’s running, and you can check the logs if an error occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*DEGkbpm29ajP1NF-1O0BvQ.png" /></figure><p>Step 6: Now the space created, then click on “Embed this space” and copy the space link (This serves as your API key for production).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/0*oMegIUqgN2a-phI7.png" /></figure><p>Copy the direct URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ijcsyWpF1opC6HvkELFJ5A.png" /></figure><p>Step 7: Now make space visibility private ⚠️ in setting (Note: Your data and credentials will be exposed in a public space).</p><p>Everything is now completed, and we have deployed our LLM successfully. Let’s proceed to test our API.</p><p>This is our Space link : <a href="https://gathnex-llm-deployment-zerocost-api.hf.space/">https://gathnex-llm-deployment-zerocost-api.hf.space</a></p><p>Add /docs to the URL to enable Swagger UI and test our API.</p><p>Fast API swagger : <a href="https://gathnex-llm-deployment-zerocost-api.hf.space/docs">https://gathnex-llm-deployment-zerocost-api.hf.space/docs</a></p><p>Endpoint : <a href="https://gathnex-llm-deployment-zerocost-api.hf.space/llm_on_cpu">https://gathnex-llm-deployment-zerocost-api.hf.space/llm_on_cpu</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXgK7oTYucTbn-mmAlB8Zg.png" /></figure><p>We also deployed Zephyr in the Gradio space for the chat playground.</p><p><strong>2. Chat UI Playground on HF CPU : </strong><a href="https://huggingface.co/spaces/gathnex/LLM_deployment_zerocost_playground">https://huggingface.co/spaces/gathnex/LLM_deployment_zerocost_playground</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VA2MVl4aG3JYLNdV9XVX2g.png" /></figure><h4>Conclusion</h4><p>We have successfully deployed LLM in production with an API. You can use it for small automation tasks, text generation, agent tasks, etc. It’s very useful for those looking to deploy their own customized model without the high costs associated with GPUs. GPU costs are often too high, making it difficult for many people to afford.</p><blockquote>Opportunity is everywhere; stop thinking and start doing with what you have.</blockquote><p><strong>Follow us on medium for more update.</strong></p><p>If you have any questions, feel free to reach out to us on LinkedIn.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e7947d9b64a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own Production ready Retrival Augmented Genration-System at Zero Cost.]]></title>
            <link>https://medium.com/@gokulrajar/build-your-own-production-ready-retrival-augmented-genration-system-at-zero-cost-b619c26c10c1?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/b619c26c10c1</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 16:36:24 GMT</pubDate>
            <atom:updated>2023-11-28T10:01:31.545Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build your own Production ready RAG-System at Zero Cost.</h3><p>In this article, we have implemented a production-ready Advanced RAG System or Vector Search, at zero cost😎. This is the same system we have deployed in our production environment using Azure-based products at $200/month. Now, we are going to demonstrate how to implement it for free of cost. Sounds interesting, right? Let’s deep dive into RAG🚀.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qZVrqO1fuYErqIaY" /></figure><h4><strong>What is RAG?</strong></h4><p>Retrieval-Augmented Generation (RAG) is a popular technique that makes Large Language Model (LLM) generation more effective and accurate by incorporating an external knowledge base.</p><p>we are going build this system and deploy it on production level using the free recourses that are available.</p><h3>Table of Contents:</h3><ol><li><strong>Retrieval Augmented Generation</strong></li><li><strong>Tech stacks</strong></li><li><strong>Running RAG with Your Own Data (with code)<br>3</strong>.1. Storing data in Vector database<br>3.2. Deployment</li></ol><h4><strong>1. Retrieval Augmented Generation.</strong></h4><p>RAG provides a way to optimize the output of an LLM with targeted information without modifying the underlying model itself; that targeted information can be more up-to-date than the LLM as well as specific to a particular organization and industry. That means the generative AI system can provide more contextually appropriate answers to prompts as well as base those answers on extremely current data.</p><p>It’s basically works on two stages:</p><p><strong>Retrieval Stage</strong></p><p>When user send a query, the RAG system first scours its connected database to find relevant information. where the data’s are converted into vector embeddings and stored in higher dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/0*jPospoEMa8GXE8Qj.jpg" /></figure><p>there are many types of methods are used to find relevant information from the database which are Cosine similarity, Dot Product similarity, Euclidean distance, etc. we’ll cover this working in separate article.</p><p><strong>Generation Stage</strong></p><p>Once the relevant data is retrieved, the generative component of the model synthesizes this information, integrating it with what it has learned during its training to create coherent, contextually rich responses.</p><h4><strong>2. Tech Stacks</strong></h4><p>The tech stacks we are going to using for this end-to-end deployment is:</p><p><a href="https://www.pinecone.io/"><strong>Pinecone</strong></a><strong>:</strong> Pinecone vector database is a vector-based database that offers high-performance search and similarity matching. Pinecone offers a basic free version of vector databases, so we plan to utilize that for our vector storage.</p><p><a href="https://platform.openai.com/docs/overview"><strong>OpenAI</strong></a><strong>:</strong> We are going to use the free tier of OpenAI GPT models for completion and embedding models for converting the data into vectors. you can also use any other open source models or api.</p><p>We highly suggest using Cohere’s free APIs (Chat, text generation, embedding, etc.) because they’re way better than OpenAI’s free APIs. Refer to our previous article and get more insights on Cohere’s APIs.</p><p><a href="https://gathnex.medium.com/breaking-chains-coheres-free-llm-api-s-shakes-openai-s-foundation-b91f6156c89d">Breaking Chains: Cohere’s Free LLM API’s Shakes OpenAI’s Foundation.</a></p><p><a href="https://huggingface.co/spaces"><strong>Huggingface Space:</strong></a><strong> </strong>we can build, host, and share your ML apps using Spaces. they are providing 16GB RAM, 2 vCPU cores and 50GB of disk space hardware for Free. we are going use it as deployment space.</p><p><strong>FastAPI and Docker:</strong> <a href="https://fastapi.tiangolo.com/tutorial/">FastAPI</a> is a Python framework for building APIs and <a href="https://www.docker.com/101-tutorial/">Docker</a> is used for containerize our AI Application for deployment in huggingface space.</p><p>Above mentioned tech stacks are open source and closed source free tier versions.</p><h4><strong>3. Running RAG with Your Own Data (with code)</strong></h4><p>There are two phases to the development of this RAG system. The data will be transformed into embeddings and stored in a vector database in phase 1. phase 2 will be deploying the retrieval with completion setup in production.</p><h4>3.1 Embed and index</h4><ul><li>Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data).</li><li>Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ugz4zAMfpWMf9fiP3d1nbg.gif" /><figcaption>Data Storage</figcaption></figure><p><strong>Code</strong></p><pre># Environment Setup<br>!pip install -q pinecone-client openai datasets</pre><p>To create embeddings we must first initialize our connection to OpenAI Embeddings, we sign up for an API key at <a href="https://beta.openai.com/signup">OpenAI</a>.</p><pre>from openai import OpenAI<br>from google.colab import userdata<br>import os, pinecone<br><br>client = OpenAI(api_key= &quot;Openai_api_key&quot;)</pre><p>We can now create embeddings with the OpenAI Ada similarity model like so:</p><pre>res = client.embeddings.create(<br>    input=[<br>        &quot;The Gathnex AI community is a group of individuals who are interested in learning about and using AI. The community is open to anyone who wants to participate, and there are no prerequisites for membership. The community is a great place to learn about AI, share your experiences, and connect with other like-minded individuals.&quot;,<br>    ],<br>    model=&quot;text-embedding-ada-002&quot;<br>)<br>print(res.data[0].embedding)</pre><p><strong>Initializing a Pinecone Index</strong></p><p>Next, we initialize an index to store the vector embeddings. For this we need a Pinecone API key, <a href="https://app.pinecone.io/">sign up for one here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSGVvXBifEKgFeecaDK7Ww.png" /></figure><p>Step 1: Create an index by entering the index name, dimension (assuming the dimension is 1536), and metric (choose a vector search method). We are going to use the starter pod because it’s free. Then, hit the ‘Create Index’ button.</p><p>Step 2: Navigate to the API Key tab and copy the Pinecone API for connection.</p><pre>import pinecone<br><br>index_name = &#39;rag-demo&#39;<br><br># initialize connection to pinecone (get API key at app.pinecone.io)<br>pinecone.init(<br>    api_key =  &quot;Pinecone_api_key&quot;,<br>    environment = &quot;gcp-starter&quot;  # find next to api key in console<br>)<br># check if &#39;openai&#39; index already exists (only create index if not)<br>if index_name not in pinecone.list_indexes():<br>    pinecone.create_index(index_name, dimension=len(res.data[0].embedding))<br># connect to index<br>index = pinecone.Index(index_name)</pre><p>Now we’ll load our sample dataset</p><pre>from datasets import load_dataset<br><br># load the first 150 rows of the indian law dataset<br>data = load_dataset(&#39;nisaar/Lawyer_GPT_India&#39;, split=&#39;train&#39;)<br>data</pre><p>Then we create a vector embedding for each phrase using OpenAI, and upsert the ID, vector embedding, and original text for each phrase to Pinecone.</p><p>Note: We highly recommend using the Cohere embedding model due to some limitations in the OpenAI API during the conversion of text into vectors. Refer to our previous article about Cohere’s free API access for more information.</p><p><a href="https://gathnex.medium.com/breaking-chains-coheres-free-llm-api-s-shakes-openai-s-foundation-b91f6156c89d">Breaking Chains: Cohere’s Free LLM API’s Shakes OpenAI’s Foundation.</a></p><pre>from tqdm.auto import tqdm<br>import time<br><br>count = 0  # we&#39;ll use the count to create unique IDs<br>batch_size = 1  <br>for i in tqdm(range(0, len(data[&#39;answer&#39;]), batch_size)):<br>    # set end position of batch<br>    i_end = min(i+batch_size, len(data[&#39;answer&#39;]))<br>    # get batch of lines and IDs<br>    lines_batch = data[&#39;answer&#39;][i: i+batch_size]<br>    ids_batch = [str(n) for n in range(i, i_end)]<br>    # create embeddings<br>    res = client.embeddings.create(input=lines_batch,model=&quot;text-embedding-ada-002&quot;)<br>    time.sleep(20)<br>    embeds = [res.data[0].embedding]<br>    # prep metadata and upsert batch<br>    meta = [{&#39;text&#39;: line} for line in lines_batch]<br>    to_upsert = zip(ids_batch, embeds, meta)<br>    # upsert to Pinecone<br>    index.upsert(vectors=list(to_upsert))</pre><p>After the data has been successfully uploaded, it’s time for testing.</p><p><strong>Query</strong></p><p>With our data indexed, we’re now ready to move onto performing searches. This follows a similar process to indexing. We start with a text query, that we would like to use to find similar sentences. As before we encode this with OpenAI&#39;s text similarity ada model to create a <em>query vector</em> xq. We then use xq to query the Pinecone index.</p><pre>query = &quot;transportation law&quot;<br>xq = client.embeddings.create(input=query,model=&quot;text-embedding-ada-002&quot;)<br>#Retrieving relevent data from pinecone DB<br>res = index.query([xq.data[0].embedding], top_k=2, include_metadata=True)<br>res<br><br>#Output<br>{&#39;matches&#39;: [{&#39;id&#39;: &#39;78&#39;,<br>              &#39;metadata&#39;: {&#39;text&#39;: &#39;The Motor Vehicles (Amendment) Act, 2019 &#39;<br>                                   &#39;has substantial implications for road &#39;<br>                                   &#39;safety. It increases penalties for traffic &#39;<br>                                   &#39;violations, introduces provisions for &#39;<br>                                   &#39;electronic enforcement, and provides for a &#39;<br>                                   &#39;Motor Vehicle Accident Fund. It also &#39;<br>                                   &#39;mandates the central government to &#39;<br>                                   &#39;regulate taxi aggregators and establishes &#39;<br>                                   &#39;a National Road Safety Board. This Act is &#39;<br>                                   &#39;expected to enhance compliance with &#39;<br>                                   &#39;traffic rules and reduce road accidents.&#39;},<br>              &#39;score&#39;: 0.805287659,<br>              &#39;values&#39;: []},<br>             {&#39;id&#39;: &#39;8&#39;,<br>              &#39;metadata&#39;: {&#39;text&#39;: &#39;The Motor Vehicles (Amendment) Act 2019 &#39;<br>                                   &#39;introduces stricter penalties for traffic &#39;<br>                                   &#39;offenses, improving road safety. It &#39;<br>                                   &#39;establishes a Motor Vehicle Accident Fund &#39;<br>                                   &#39;and mandates insurance. However, it also &#39;<br>                                   &#39;raises concerns about state autonomy, as &#39;<br>                                   &#39;it encroaches upon their legislative &#39;<br>                                   &#39;purview.&#39;},<br>              &#39;score&#39;: 0.800310254,<br>              &#39;values&#39;: []}],<br> &#39;namespace&#39;: &#39;&#39;}</pre><p>It’s clear from this example that the semantic search pipeline is clearly able to identify the meaning between each of our queries. Using these embeddings with Pinecone allows us to return the most semantically similar document from the already indexed Indian law dataset.</p><p>Refer the code : <a href="https://colab.research.google.com/drive/1U9SBwbGLtAhsTwPgWO5rTCiiJir2yYNY?usp=sharing">https://colab.research.google.com/drive/1U9SBwbGLtAhsTwPgWO5rTCiiJir2yYNY?usp=sharing</a></p><h4>3.2 Vector Search with completion and deployment</h4><p>Here is a detailed description of our RAG system.</p><ol><li>We are going to deploy the structured files below in Hugging Face and create an API.</li><li>The initial user request is passed to the Hugging Face space, which contains our container.</li><li>After that, it’s passed through the embedding model and converted into vectors.</li><li>Take the resulting vector embedding and send it as a <a href="https://docs.pinecone.io/docs/query-data">query</a> to Pinecone.</li><li>Get back semantically similar documents, even if they don’t share any keywords with the query.</li><li>Then, it’s passed to GPT-3.5-turbo, which efficiently completes the user query using the retrieved documents. Next, The user gets a response.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*okkdr8yeQfL1LkLEbQA-BA.gif" /><figcaption>RAG Architecture</figcaption></figure><p>The file structure for deployment will be as follows:</p><p><em>Rag_Demo/<br>| — __init__.py<br>| — credentials.env<br>| — Dockerfile<br>| — main.py<br>| — rag_retriver.py<br>| — requirements.txt</em></p><p>Please refer to this GitHub repository for a better understanding of the structure : <a href="https://github.com/gathnexadmin/Production-Ready-RAG-System-at-Zero-Cost">https://github.com/gathnexadmin/Production-Ready-RAG-System-at-Zero-Cost</a></p><p><strong>requirements.txt</strong></p><p>This file contains a list of packages or libraries needed to work on a project, all of which can be installed using the file.</p><pre>openai<br>python-multipart<br>fastapi<br>pydantic<br>uvicorn<br>python-dotenv<br>pinecone-client</pre><p><strong><em>credentials.env</em></strong></p><p>Generally, .env files are used to store sensitive variables or credentials.</p><pre>openai_api = &quot;openai_api_key&quot;<br>Pinecone_api_key = &quot;api key&quot;<br>Pinecone_environment = &quot;gcp-starter&quot;<br>index_name = &quot;index_name&quot;</pre><p><strong><em>rag_retriver.py</em></strong></p><p>The rag_retriever.py file contains GPT completion, vector search, and retriever functionalities from the Pinecone database.</p><pre><br>import pinecone<br>from openai import OpenAI<br>from dotenv import dotenv_values<br><br>#Loading Credentials<br>env_name = &quot;credentials.env&quot;<br>config = dotenv_values(env_name)<br>client = OpenAI(api_key= config[&quot;openai_api&quot;])<br><br><br>#Connection<br>index_name = config[&quot;index_name&quot;]<br># initialize connection to pinecone (get API key at app.pinecone.io)<br>pinecone.init(<br>    api_key =  config[&quot;Pinecone_api_key&quot;],<br>    environment =  config[&quot;Pinecone_environment&quot;]<br>)<br>index = pinecone.Index(index_name)<br><br>#Vector Search<br>def Vector_search(query):<br>  Rag_data = &quot;&quot;<br>  xq = client.embeddings.create(input=query,model=&quot;text-embedding-ada-002&quot;)<br>  res = index.query([xq.data[0].embedding], top_k=2, include_metadata=True)<br>  for match in res[&#39;matches&#39;]:<br>      if match[&#39;score&#39;] &lt; 0.80:<br>        continue <br>      Rag_data += match[&#39;metadata&#39;][&#39;text&#39;]<br>  return Rag_data <br><br>#GPT Completion<br>def GPT_completion_with_vector_search(prompt, rag):<br>    DEFAULT_SYSTEM_PROMPT = &#39;&#39;&#39;You are a helpful, respectful and honest INTP-T AI Assistant named Gathnex AI. You are talking to a human User.<br>    Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.<br>    If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don&#39;t know the answer to a question, please don&#39;t share false information.<br>    You also have access to RAG vectore database access which has Indian Law data. Be careful when giving response, sometime irrelevent Rag content will be there so give response effectivly to user based on the prompt.<br>    You can speak fluently in English.<br>    Note: Sometimes the Context is not relevant to Question, so give Answer according to that sutiation.<br>    &#39;&#39;&#39;<br>    response = client.chat.completions.create(<br>    model=&quot;gpt-3.5-turbo-1106&quot;,<br>    messages=[<br>        {f&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: DEFAULT_SYSTEM_PROMPT},<br>        {f&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: rag +&quot;, Prompt: &quot;+ prompt},<br>    ]<br>    )<br><br>    return response.choices[0].message.content</pre><p><strong>main.py</strong></p><p>The main.py file contains a FastAPI function that returns a RAG completion.</p><pre>from Rag_Retriver import Vector_search, GPT_completion_with_vector_search<br>from fastapi import FastAPI<br>from pydantic import BaseModel<br><br>#Pydantic object<br>class validation(BaseModel):<br>    prompt: str<br>    <br>#Fast API<br>app = FastAPI()<br><br><br>@app.post(&quot;/Gathnex_Rag_System&quot;)<br>async def retrival_augmented_generation(item: validation):<br>    rag = Vector_search(item.prompt)<br>    completion = GPT_completion_with_vector_search(item.prompt, rag)<br>    return completion</pre><p><strong>Dockerfile</strong></p><p>Finally, the Dockerfile was used to containerize our application for deployment.</p><pre>FROM python:3.9<br><br>WORKDIR /code<br><br>COPY ./requirements.txt /code/requirements.txt<br><br>RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt<br><br>COPY ./__init__.py /code/__init__.py<br>COPY ./credentials.env /code/credentials.env<br>COPY ./rag_retriver.py /code/rag_retriver.py<br>COPY ./main.py /code/main.py<br><br>CMD [&quot;uvicorn&quot;, &quot;main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;7860&quot;]</pre><p>Now, our files are ready for deployment.</p><h4><strong>Deployment</strong></h4><p>Huggingface space</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*bKxGGOPGIDRXkUvIXSTdTA.png" /></figure><p>Step 1: Create a new <a href="https://huggingface.co/spaces"><strong>Hugging Face space.</strong></a></p><p>Step 2: Enter the space name and choose the license type.</p><p>Step 3: Select the Space SDK-Docker (Blank) option.</p><p>Step 4: Choose ‘Public’ initially to copy our deployment endpoint, and then hit ‘Create Space.</p><p><strong>Refer to our Hugging Face space files structure to get an idea</strong> : <a href="https://huggingface.co/spaces/gathnex/rag-demo/tree/main">https://huggingface.co/spaces/gathnex/rag-demo/tree/main</a></p><p>Step 5: Upload the files we have created above.</p><p><em>Rag_Demo/<br>| — __init__.py<br>| — credentials.env<br>| — Dockerfile<br>| — main.py<br>| — rag_retriver.py<br>| — requirements.txt</em></p><p>Congratulations! Your production-ready application has been deployed successfully.</p><p>If there are no errors in the code, you’ll see that it’s running, and you can check the logs if an error occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Q7YQV87cWj_r2jqMySqbQ.png" /></figure><p>Step 6: Now the space created, then click on “Embed this space” and copy the space link (This serves as your API key for production).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*5OfHdn2EubB1W9kjTmcCfQ.png" /></figure><p>Copy the direct URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HW9EmFRK1ZZrRVKhanDXuQ.png" /></figure><p>Step 7: Now make space visibility private ⚠️ in setting (Note: Your data and credentials will be exposed in a public space).</p><p>Everything is now completed, and our RAG system has been deployed successfully. Let’s proceed to test our API.</p><p>This is our Space link : <a href="https://gathnex-rag-demo.hf.space/docs">https://gathnex-rag-demo.hf.space</a></p><p>Add /docs to the URL to enable Swagger UI and test our API.</p><p>Fast API swagger : <a href="https://gathnex-rag-demo.hf.space/docs">https://gathnex-rag-demo.hf.space/docs</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cW0xFTqzCEFlFBJVkToQrQ.png" /></figure><p>Endpoint : <a href="https://gathnex-rag-demo.hf.space/Gathnex_Rag_System">https://gathnex-rag-demo.hf.space/Gathnex_Rag_System</a></p><p>We have implemented the RAG system with an API endpoint at the production level, utilizing open-source tools and other free-tier resources. This project can be employed for high school projects, and it is suitable for production use as well, as there is minimal latency.</p><p>Follow us on medium for more update.</p><p><strong>Follow us on LinkedIn</strong> : <a href="https://www.linkedin.com/company/gathnex/">https://www.linkedin.com/company/gathnex/</a></p><p>If you have any questions, feel free to reach out to us on LinkedIn.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b619c26c10c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking Chains: Cohere’s Free LLM API’s Shakes OpenAI’s Foundation.]]></title>
            <link>https://medium.com/@gokulrajar/breaking-chains-coheres-free-llm-api-s-shakes-openai-s-foundation-b91f6156c89d?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/b91f6156c89d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cohere]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 16:43:06 GMT</pubDate>
            <atom:updated>2023-11-24T16:43:06.531Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we are going to test Cohere’s free LLM APIs and assess their potential significant impact on open source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*V33Ey1fIq7m1EYlE.png" /></figure><h3><strong>Cohere</strong></h3><p>Cohere’s platform gives developers and businesses access to NLP, which is powered by large language models. Cohere’s AI is designed to be easy to use, accessible, and private.</p><p>Cohere’s recent breakthroughs in LLM development are remarkable steps toward creating unbiased and responsible AI solutions for consumers. Despite these advancements, LLMs are still in the development phase, and Cohere strives to address potential biases and ensure responsible deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cbq9damgmsJ5VaP9h2hvvQ.png" /></figure><p>Play with coral model <a href="https://coral.cohere.com/"><strong>here</strong></a><strong>.</strong></p><h4><strong>Cohere API’s</strong></h4><p>Cohere provides a variety of powerful APIs that enable developers to integrate cutting-edge language models into their projects. Here are some of the key APIs offered by Cohere:</p><ul><li>Chat</li><li>Generation</li><li>Embeddings</li><li>Reranking</li><li>Classification</li><li>Language Detection</li><li>Summarization</li><li>Tokenize</li><li>Detokenize</li></ul><p><strong>Chat Endpoint</strong></p><p>The chat endpoint allows users to have conversations with a Large Language Model (LLM) from Cohere. Users can send messages as part of a persisted conversation using the conversation_id parameter, or they can pass in their own conversation history using the chat_history parameter.<br>The endpoint features additional parameters such as connectors and documents that enable conversations enriched by external knowledge. We call this &quot;Retrieval Augmented Generation&quot;, or &quot;RAG&quot;.</p><pre>!pip -q install cohere<br>import cohere<br>co = cohere.Client(&#39;&lt;&lt;apiKey&gt;&gt;&#39;)</pre><p>You can find your API key in API <a href="https://docs.cohere.com/reference/chat">Reference page</a>.</p><pre>response = co.chat(<br>  chat_history=[<br>    {&quot;role&quot;: &quot;USER&quot;, &quot;message&quot;: &quot;Who is current CEO of Openai&quot;},<br>    {&quot;role&quot;: &quot;CHATBOT&quot;, &quot;message&quot;: &quot;There has been a lot of turmoil surrounding the leadership of OpenAI in recent months. Former CEO, Sam Altman, was briefly ousted in November 2023, with the board appointing Chief Technology Officer, Mira Murati, as interim CEO. However, just days later, a deal was struck and Altman returned as CEO of the company. There has been speculation about Altman&#39;s dealings with the board, with claims that he had been dishonest in his communication with them, and a desire from Altman to install an entirely new slate of directors.&quot;}<br>  ],<br>  message=&quot;What happened in openai in lastweek, explain clearly&quot;,<br>  # perform web search before answering the question. You can also use your own custom connector.<br>  connectors=[{&quot;id&quot;: &quot;web-search&quot;}] <br>)<br>print(response.text)<br><br>#Output<br>&#39;&#39;&#39;<br>It was quite a turbulent week for OpenAI, with the board initially announcing that Sam Altman would be departing as CEO, and technology chief Mira Murati would be taking on the role of interim CEO. This was met with a large proportion of employees calling for Altman&#39;s return, including Murati herself. After negotiations, it was then announced that Altman would be returning to the company as CEO just 5 days after his initial departure, with a new initial board of directors.<br><br>Here are the new members of the board:<br>- Bret Taylor (Chair)<br>- Larry Summers<br>- Adam D&#39;Angelo<br><br>Is there anything else you&#39;d like to know about the events at OpenAI last week?<br>&#39;&#39;&#39;<br></pre><p><strong>Generate Endpoint</strong></p><p>This endpoint generates realistic text conditioned on a given input.</p><pre>response = co.generate(<br>    model=&#39;command-nightly&#39;,<br>    prompt = &quot;Write an introductory paragraph for a blog post about language models.&quot;,<br>    )<br><br>print(response.generations[0].text)<br><br>#Output<br>&#39;&#39;&#39;<br>Language models have emerged as powerful tools for natural language processing and artificial intelligence. These models are designed to analyze, generate, and understand human language, and have revolutionized various industries and applications.<br>The popularity and usage of language models have skyrocketed, showcasing their effectiveness and potential in numerous domains. As researchers and developers continue to innovate, the future for language models remains bright, enabling us to tackle complex language-based challenges and unlock new opportunities for human-computer interaction.<br>&#39;&#39;&#39;</pre><p><strong>Embedding Endpoint</strong></p><p>This endpoint returns text embeddings. An embedding is a list of floating point numbers that captures semantic information about the text that it represents.</p><p>Embeddings can be used to create text classifiers as well as empower semantic search, etc.</p><pre>response = co.embed(<br>  texts=[&#39;what is large language models&#39;],<br>  model=&#39;embed-english-v3.0&#39;,<br>  input_type=&#39;classification&#39;<br>)<br>print(response.embeddings[0])<br><br>#Output<br>&#39;&#39;&#39;<br>[-0.04324341, -0.04257202, -0.029647827, -0.05340576, -0.06439209, 0.021331787, -0.09307861,......]<br>&#39;&#39;&#39;</pre><p><strong>Reranking Endpoint</strong></p><p>This endpoint takes in a query and a list of texts and produces an ordered array with each text assigned a relevance score.</p><pre>docs = [&#39;Carson City is the capital city of the American state of Nevada.&#39;,<br>&#39;The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.&#39;,<br>&#39;Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.&#39;,<br>&#39;Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states.&#39;]<br><br>response = co.rerank(<br>  model = &#39;rerank-english-v2.0&#39;,<br>  query = &#39;What is the capital of the United States?&#39;,<br>  documents = docs,<br>  top_n = 3,<br>)<br>print(response)<br><br>#Output<br>&#39;&#39;&#39;<br>[RerankResult&lt;document[&#39;text&#39;]: Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.,index: 2, relevance_score: 0.98005307&gt;, RerankResult&lt;document[&#39;text&#39;]: Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states., index: 3, relevance_score: 0.27904198&gt;, RerankResult&lt;document[&#39;text&#39;]: Carson City is the capital city of the American state of Nevada., index: 0, relevance_score: 0.10194652&gt;]<br>&#39;&#39;&#39;<br></pre><p><strong>Classification Endpoint</strong></p><p>This endpoint makes a prediction about which label fits the specified text inputs best. To make a prediction, Classify uses the provided examples of text + label pairs as a reference.</p><pre>from cohere.responses.classify import Example<br>examples=[<br>  Example(&quot;Dermatologists don&#39;t like her!&quot;, &quot;Spam&quot;),<br>  Example(&quot;&#39;Hello, open to this?&#39;&quot;, &quot;Spam&quot;),<br>  Example(&quot;I need help please wire me $1000 right now&quot;, &quot;Spam&quot;),<br>  Example(&quot;Nice to know you ;)&quot;, &quot;Spam&quot;),<br>  Example(&quot;Please help me?&quot;, &quot;Spam&quot;),<br>  Example(&quot;Your parcel will be delivered today&quot;, &quot;Not spam&quot;),<br>  Example(&quot;Review changes to our Terms and Conditions&quot;, &quot;Not spam&quot;),<br>  Example(&quot;Weekly sync notes&quot;, &quot;Not spam&quot;),<br>  Example(&quot;&#39;Re: Follow up from today&#39;s meeting&#39;&quot;, &quot;Not spam&quot;),<br>  Example(&quot;Pre-read for tomorrow&quot;, &quot;Not spam&quot;),<br>]<br>inputs=[<br>  &quot;Confirm your email address&quot;,<br>  &quot;hey i need u to send some $&quot;,<br>]<br>response = co.classify(<br>  inputs=inputs,<br>  examples=examples,<br>)<br>print(response)<br><br>#Output<br>&#39;&#39;&#39;<br>[Classification&lt;prediction: &quot;Not spam&quot;, confidence: 0.5661598, labels: {&#39;Not spam&#39;: LabelPrediction(confidence=0.5661598), &#39;Spam&#39;: LabelPrediction(confidence=0.43384025)}&gt;, Classification&lt;prediction: &quot;Spam&quot;, confidence: 0.9909811, labels: {&#39;Not spam&#39;: LabelPrediction(confidence=0.009018883), &#39;Spam&#39;: LabelPrediction(confidence=0.9909811)}&gt;]<br>&#39;&#39;&#39;</pre><p><strong>Language detection endpoint</strong></p><p>This endpoint identifies which language each of the provided texts is written in.</p><pre>response = co.detect_language(<br>  texts=[&#39;தமிழ், உலகில் உள்ள முதன்மையான மொழிகளில் ஒன்றும் செம்மொழியும் ஆகும்.&#39;, &#39;Hello world&#39;]<br>)<br>print(response.results)<br>#Output<br>&#39;&#39;&#39;<br>[Language&lt;language_code: &quot;ta&quot;, language_name: &quot;Tamil&quot;&gt;, Language&lt;language_code: &quot;en&quot;, language_name: &quot;English&quot;&gt;]<br>&#39;&#39;&#39;<br></pre><p><strong>Summarize Endpoint</strong></p><p>This endpoint generates a summary in English for a given text.</p><pre>text=(<br>  &quot;Ice cream is a sweetened frozen food typically eaten as a snack or dessert. &quot;<br>  &quot;It may be made from milk or cream and is flavoured with a sweetener, &quot;<br>  &quot;either sugar or an alternative, and a spice, such as cocoa or vanilla, &quot;<br>  &quot;or with fruit such as strawberries or peaches. &quot;<br>  &quot;It can also be made by whisking a flavored cream base and liquid nitrogen together. &quot;<br>  &quot;Food coloring is sometimes added, in addition to stabilizers. &quot;<br>  &quot;The mixture is cooled below the freezing point of water and stirred to incorporate air spaces &quot;<br>  &quot;and to prevent detectable ice crystals from forming. The result is a smooth, &quot;<br>  &quot;semi-solid foam that is solid at very low temperatures (below 2 °C or 35 °F). &quot;<br>  &quot;It becomes more malleable as its temperature increases.\n\n&quot;<br>  &quot;The meaning of the name \&quot;ice cream\&quot; varies from one country to another. &quot;<br>  &quot;In some countries, such as the United States, \&quot;ice cream\&quot; applies only to a specific variety, &quot;<br>  &quot;and most governments regulate the commercial use of the various terms according to the &quot;<br>  &quot;relative quantities of the main ingredients, notably the amount of cream. &quot;<br>  &quot;Products that do not meet the criteria to be called ice cream are sometimes labelled &quot;<br>  &quot;\&quot;frozen dairy dessert\&quot; instead. In other countries, such as Italy and Argentina, &quot;<br>  &quot;one word is used fo\r all variants. Analogues made from dairy alternatives, &quot;<br>  &quot;such as goat&#39;s or sheep&#39;s milk, or milk substitutes &quot;<br>  &quot;(e.g., soy, cashew, coconut, almond milk or tofu), are available for those who are &quot;<br>  &quot;lactose intolerant, allergic to dairy protein or vegan.&quot;<br>)<br><br>response = co.summarize(<br>  text=text,<br>)<br>print(response.summary)<br><br>#Output<br>&#39;&#39;&#39;<br>Ice cream is a popular frozen dessert made from dairy products or alternatives, sweeteners, and spices. It is a foam made by cooling and stirring a liquid mixture to incorporate air spaces. While it is solid at very low temperatures, it becomes more malleable as its temperature increases.<br> The name &quot;ice cream&quot; has different definitions depending on the country, with some countries having stricter regulations on commercial use. Ice cream made from dairy alternatives is available for those who are lactose intolerant or vegan.<br>&#39;&#39;&#39;</pre><p><strong>Tokenize Endpoint</strong></p><p>This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.</p><pre>response = co.tokenize(<br>  text=&#39;tokenize me! :D&#39;,<br>  model=&#39;command&#39; <br>)<br>print(response)<br><br>#output<br>cohere.Tokens {<br> tokens: [10002, 2261, 2012, 8, 2792, 43]<br> token_strings: [&#39;token&#39;, &#39;ize&#39;, &#39; me&#39;, &#39;!&#39;, &#39; :&#39;, &#39;D&#39;]<br> meta: {&#39;api_version&#39;: {&#39;version&#39;: &#39;1&#39;}}<br>}</pre><p><strong>Detokenize Endpoint</strong></p><p>This endpoint takes tokens using byte-pair encoding and returns their text representation. To learn more about tokenization and byte pair encoding, see the tokens page.</p><pre>response = co.detokenize(<br>  tokens=[10104, 12221, 1315, 34, 1420, 69],<br>  model=&quot;command&quot;<br>)<br>print(response)<br><br>#Output<br>Anton Mun🟣;🥭^</pre><p><strong>Conclusion</strong></p><p>The model’s performance was great overall. Cohere has primarily focused on reducing the model’s hallucination effectively. It was fast and accurate in providing responses. This stands out in comparison to OpenAI’s APIs, which currently have delays. Cohere not only offers a more affordable solution but also a superior model compared to OpenAI.</p><p>Reference : <a href="https://docs.cohere.com/docs">https://docs.cohere.com/docs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b91f6156c89d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create your own LLM Agent from Scratch: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@gokulrajar/how-to-create-your-own-llm-agent-from-scratch-a-step-by-step-guide-14b763e5b3b8?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/14b763e5b3b8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 13:01:56 GMT</pubDate>
            <atom:updated>2023-12-07T16:32:06.229Z</atom:updated>
            <content:encoded><![CDATA[<p>In this Article , we are going to build a LLM Agent from scratch using Python✅. No Langchain❌, No Llama Index❌, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*anquVtL69_N897vK" /></figure><h3>What are Agents?</h3><p>LLM agents are are programs that use large language models to decide how and when to use tools to complete tasks.</p><p>Why are LLM agents essential for tasks? LLMs are powerful; however, they may not be able to perform certain tasks. Let’s explore the reasons why.</p><p>Large Language Models are powerful neural networks trained on massive amounts of text data. They can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way but not for doing a tasks.</p><p>The core idea of agents is to use a language model to choose a sequence of actions to take. In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order.</p><p>In simpler terms, LLMs are like the engine that powers a car, while LLM agents are the car itself, with additional features that make it useful for real-world tasks.</p><h3>Agents and Tools</h3><p>To use agents, we require three things:</p><ul><li>A base LLM,</li><li>A tool that we will be interacting with,</li><li>An agent to control the interaction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXI0Z99lCDn7UzYSKbzgkw.gif" /></figure><h3>Agent Implementation</h3><pre>!pip install -q openai py_expression_eval google-api-python-client</pre><pre>from openai import OpenAI<br>from googleapiclient.discovery import build<br>from py_expression_eval import Parser<br>import re, time, os</pre><ul><li>Set up a Custom Search Engine and Google API Key, following <a href="https://stackoverflow.com/questions/37083058/programmatically-searching-google-in-python-using-custom-search"><strong><em>these instructions</em></strong></a>.</li><li>Get an API Key and Custom Search Engine ID from the previous step, and set them as environment variables GOOGLE_API_KEY and GOOGLE_CSE_ID respectively.</li></ul><pre>client = OpenAI(api_key=&#39;Openai_api_key&#39;)<br>os.environ[&quot;GOOGLE_CSE_ID&quot;] = &quot;Google CSE ID&quot;<br>os.environ[&quot;GOOGLE_API_KEY&quot;] = &quot;Google API Key&quot;</pre><p><strong>Tools</strong></p><p>Search: Conduct a search engine query using Google’s official custom search engine. You can send 100 requests per day in the free tier.</p><p>Calculator: I use py_expression_eval as a calculator (good balance between being able to run complex math expressions, without many of the risks of trying to pull in a full Python REPL/eval).</p><p>We have used two tools for the demo. You can use whatever tool you want as per your requirements.</p><pre>#Google search engine<br>def search(search_term):<br>    search_result = &quot;&quot;<br>    service = build(&quot;customsearch&quot;, &quot;v1&quot;, developerKey=os.environ.get(&quot;GOOGLE_API_KEY&quot;))<br>    res = service.cse().list(q=search_term, cx=os.environ.get(&quot;GOOGLE_CSE_ID&quot;), num = 10).execute()<br>    for result in res[&#39;items&#39;]:<br>        search_result = search_result + result[&#39;snippet&#39;]<br>    return search_result<br><br>#Calculator<br>parser = Parser()<br>def calculator(str):<br>    return parser.parse(str).evaluate({})</pre><p><strong>Prompt</strong></p><p>First, let’s set up the system prompt.</p><pre>System_prompt = &quot;&quot;&quot;<br>Answer the following questions and obey the following commands as best you can.<br><br>You have access to the following tools:<br><br>Search: Search: useful for when you need to answer questions about current events. You should ask targeted questions.<br>Calculator: Useful for when you need to answer questions about math. Use python code, eg: 2 + 2<br>Response To Human: When you need to respond to the human you are talking to.<br><br>You will receive a message from the human, then you should start a loop and do one of two things<br><br>Option 1: You use a tool to answer the question.<br>For this, you should use the following format:<br>Thought: you should always think about what to do<br>Action: the action to take, should be one of [Search, Calculator]<br>Action Input: &quot;the input to the action, to be sent to the tool&quot;<br><br>After this, the human will respond with an observation, and you will continue.<br><br>Option 2: You respond to the human.<br>For this, you should use the following format:<br>Action: Response To Human<br>Action Input: &quot;your response to the human, summarizing what you did and what you learned&quot;<br><br>Begin!<br>&quot;&quot;&quot;</pre><p>So what the above loop can do ?</p><p>Telling the model that it will be run in the loop. In that loop, the LLM has two options: it can either ‘use a tool,’ giving that tool an input, or it can respond to the human. We give the model a list of the tools and a description of when/how to use each one. The thought-action pattern creates a ‘chain of thought,’ telling the model to think about what it’s doing.</p><p>We used the GPT-4 model for this demo, but you can also use open source models like Llama, Mistral, Zephyr, etc.</p><pre>def Stream_agent(prompt):<br>    messages = [<br>        { &quot;role&quot;: &quot;system&quot;, &quot;content&quot;: System_prompt },<br>        { &quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt },<br>    ]<br>    def extract_action_and_input(text):<br>          action_pattern = r&quot;Action: (.+?)\n&quot;<br>          input_pattern = r&quot;Action Input: \&quot;(.+?)\&quot;&quot;<br>          action = re.findall(action_pattern, text)<br>          action_input = re.findall(input_pattern, text)<br>          return action, action_input<br>    while True:<br>        response = client.chat.completions.create(<br>            model=&quot;gpt-4&quot;,<br>            messages=messages,<br>            temperature=0,<br>            top_p=1,)<br>        response_text = response.choices[0].message.content<br>        print(response_text)<br>        #To prevent the Rate Limit error for free-tier users, we need to decrease the number of requests/minute.<br>        time.sleep(20)<br>        action, action_input = extract_action_and_input(response_text)<br>        if action[-1] == &quot;Search&quot;:<br>            tool = search<br>        elif action[-1] == &quot;Calculator&quot;:<br>            tool = calculator<br>        elif action[-1] == &quot;Response To Human&quot;:<br>            print(f&quot;Response: {action_input[-1]}&quot;)<br>            break<br>        observation = tool(action_input[-1])<br>        print(&quot;Observation: &quot;, observation)<br>        messages.extend([<br>            { &quot;role&quot;: &quot;system&quot;, &quot;content&quot;: response_text },<br>            { &quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Observation: {observation}&quot; },<br>            ])</pre><p>Let’s test our agent.</p><pre>Stream_agent(&quot;who is new openai board members&quot;)</pre><pre>Output<br>Thought: I need to find the current information about the new board members of OpenAI. <br>Action: Search<br>Action Input: &quot;New OpenAI board members&quot;<br>Observation:  7 days ago ... I am returning to OpenAI as CEO. Mira will return to her role as CTO. The new initial board will consist of Bret Taylor (Chair), Larry Summers, ...Nov 21, 2023 ... Altman were jettisoned from the board, whose members now include Bret Taylor, an early Facebook officer and former co-chief executive of ...OpenAI is governed by the board of the OpenAI Nonprofit, currently comprised of Independent Directors Bret Taylor (Chair), Larry Summers, and Adam D&#39;Angelo.Nov 22, 2023 ... Here are the newest members of OpenAI&#39;s board · Bret Taylor, board chair · Larry Summers · Adam D&#39;Angelo.Nov 22, 2023 ... The company will have an initial board consisting of Bret Taylor, the former chair of Twitter&#39;s board before its takeover by Elon Musk; Larry ...Nov 23, 2023 ... Who are OpenAI&#39;s new board members as Sam Altman returns? ... Nov 22 (Reuters) - ChatGPT-maker OpenAI on Tuesday said it reached an agreement for ...Nov 22, 2023 ... https://www.bloomberg.com/news/articles/2023-11-22/sam-altman-to-return-as-openai-ceo-with-a-new-board What are your thoughts?6 days ago ... Who is Bret Taylor? New OpenAI Board Member is Former Salesforce CEO · Bret Taylor is a 43-year-old American computer programmer, entrepreneur ...The new initial board included former Salesforce co-CEO Bret Taylor as chairman. OpenAI also announced that Microsoft would have a non-voting board seat at ...Nov 22, 2023 ... The company that developed ChatGPT has added new members to its board as it welcomes Sam Altman back as CEO · Sam Altman is the value in OpenAI: ...<br>Action: Response To Human<br>Action Input: &quot;The new board members of OpenAI include Bret Taylor, who is serving as the Chair, Larry Summers, and Adam D&#39;Angelo.&quot;<br>Response: The new board members of OpenAI include Bret Taylor, who is serving as the Chair, Larry Summers, and Adam D&#39;Angelo.</pre><p>Resourse : <a href="https://github.com/gathnexadmin/LLM_Agent/tree/main">https://github.com/gathnexadmin/LLM_Agent/tree/main</a></p><p><strong>Conclusion</strong></p><p>This is the basic idea of an LLM agent, which is built based on this paper. The output was really good when compared to Langchain and Llamaindex agents.</p><p><strong>Reference</strong></p><ol><li><a href="https://arxiv.org/abs/2210.03629">https://arxiv.org/abs/2210.03629</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14b763e5b3b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating the LinkedIn posts using Generative AI(LLM). Part-2(Post Automation Implementation)]]></title>
            <link>https://medium.com/@gokulrajar/automating-the-linkedin-posts-using-generative-ai-llm-part-2-post-automation-implementation-e9a910777238?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9a910777238</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[linkedin]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 17:22:33 GMT</pubDate>
            <atom:updated>2023-11-11T09:42:42.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G0-Y3Hmzvk7T5FAf" /><figcaption>Photo by <a href="https://unsplash.com/@alexbemore?utm_source=medium&amp;utm_medium=referral">Alexander Shatov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://gathnex.medium.com/automating-the-linkedin-posts-using-generative-ai-llm-part-1-how-to-create-linkedin-api-e5f77fa46e5f">Automating the LinkedIn posts using Generative AI(LLM). Part-1(How to create LinkedIn API)</a></p><p>In the previous article (Part 1), we discussed how to create a LinkedIn access token. In this blog, we will now explore the implementation of automating LinkedIn posts.</p><p><strong>Features of this tool</strong></p><ul><li>Automated content creation: Generate compelling LinkedIn posts based on your intent and provided links.</li><li>Intelligent posting: Determine the optimal time to post your content for maximum engagement.</li><li>Content crafting: Automatically craft the best title and description for LinkedIn by accessing the link you provided.</li><li>Image capturing: Automatically find and include relevant images to accompany your posts.</li><li>Time-saving efficiency: Streamline your LinkedIn posting process and save valuable time.</li></ul><p><strong>Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYwL9XhfrHHhMDTISP8CpA.jpeg" /></figure><p>The LinkedIn automation posting system using Gen AI utilizes a two-pronged approach to handle different types of requests.</p><p><strong>Normal Query Flow</strong></p><p>For standard queries, the<strong> Intent Identification System</strong> directs the input to the GPT in the upside. GPT excels at generating text, translating languages, and writing various creative content. It analyses the query and produces a comprehensive and informative response tailored to the user’s request.</p><p><strong>Create Content and Post LinkedIn Post with a Link Flow</strong></p><p>When a user intends to create content and post it on LinkedIn along with a link, the <strong>Intent Identification System</strong> takes a different route. It bypasses GPT and directly enters the Intelligent posting System. This path is specifically designed to efficiently process requests involving content creation and LinkedIn posting.</p><p>Intelligent posting System module extracts relevant information from the provided link and organizes it into a structured format. This structured data then feeds into the <strong>Content Creator module</strong>, which crafts a high-quality LinkedIn post based on the extracted information. the <strong>Image</strong> <strong>Capturer module</strong> retrieves an appropriate image from the internet to accompany the post.</p><p>The above-extracted contents and information have been well formatted and organised. Finally, the formatted post is sent to the LinkedIn API for publishing on the user’s LinkedIn profile.</p><p>This streamlined flow ensures that content creation and LinkedIn posting requests are handled efficiently and effectively, saving time and effort for users.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-7hzBzgquUs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-7hzBzgquUs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-7hzBzgquUs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a36757f0d64fe199af60275edcdc1615/href">https://medium.com/media/a36757f0d64fe199af60275edcdc1615/href</a></iframe><p><strong>Code implementation</strong></p><p>Install our <a href="https://github.com/gathnexadmin/Linkedin_Automation_with_Generative_AI">Linkedin_Automation_with_Generative_AI</a> github repo.</p><pre>!pip install -q git+https://github.com/gathnexadmin/Linkedin_Automation_with_Generative_AI.git</pre><p>Import the llm_automation and Linkedin_post modules, which contain all pre-written functions and automation to connect to LinkedIn. Set up the OpenAI API and your LinkedIn access token.</p><p>To know more about how to create LinkedIn access token checkout previous blog : <a href="https://medium.com/@gathnex/automating-the-linkedin-posts-using-generative-ai-llm-part-1-how-to-create-linkedin-api-e5f77fa46e5f">https://medium.com/@gathnex/automating-the-linkedin-posts-using-generative-ai-llm-part-1-how-to-create-linkedin-api-e5f77fa46e5f</a></p><pre>from psi import llm_automation, Linkedin_post<br><br>OPENAI_API_KEY = &quot;Your openai key&quot;<br>access_token = &quot;your linkedin access token&quot;</pre><p><strong>PSI function</strong></p><pre>def Gathnex_AI(prompt):<br>    llm = llm_automation.llm_auto(prompt, OPENAI_API_KEY)<br>    if llm.intent_indentifier() == &quot;#Post&quot;:<br>        url = llm.prompt_link_capturer()<br>        res = Linkedin_post.LinkedinAutomate(access_token, url, OPENAI_API_KEY).main_func()<br>        return llm.posted_or_not(res)<br>    else:<br>        return llm.normal_gpt()</pre><p><strong>Time to test our automation system.</strong></p><p>Test 1.</p><pre>Gathnex_AI(&quot;explain quantum computing&quot;)</pre><pre>#Output<br>Sure! 🤓 Quantum computing is a fascinating field that aims to use the principles of quantum mechanics to perform computations more efficiently than classical computers. 🚀<br><br>In classical computing, data is stored as binary digits called &quot;bits&quot; that can be either 0 or 1. But in quantum computing, information is stored as &quot;qubits,&quot; which can exist in a superposition of both 0 and 1 states simultaneously. This allows quantum computers to perform parallel computations on multiple possible solutions at once.<br><br>Moreover, quantum computers leverage the concept of &quot;entanglement&quot; which allows qubits to be linked together in a way that the measurement of one qubit instantly determines the state of another, regardless of the distance between them.<br><br>These unique properties of qubits and entanglement enable quantum computers to perform certain tasks much faster than classical computers. For example, quantum computers have the potential to solve complex optimization problems, simulate quantum systems, and break certain cryptographic algorithms.<br><br>However, it&#39;s important to note that quantum computing is still in its early stages and faces significant challenges, such as maintaining the delicate quantum states, minimizing errors, and scaling up the number of qubits. 🧪<br><br>Did that help? Let me know if you have more questions! 😊</pre><p>Test 2.</p><pre>Gathnex_AI(&quot;create content about my new medium blog post https://medium.com/@gathnex/new-generative-ai-course-by-deeplearning-ai-daf34e24e9c8 and post it on my linked&quot;)</pre><pre>#Output<br>Great news! 🎉 Your LinkedIn post has been successfully posted! 🚀 Congrats on sharing your thoughts, ideas, or updates with your network.<br>🙌 Keep up the fantastic work! 👏 If you need any assistance with anything else, feel free to ask. 😊</pre><p>Linkedin :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*ucgeeumQpbyKmfbSzPjkWg.png" /></figure><p>Here, we can see the post is successfully posted on LinkedIn when we explain the prompt clearly.</p><p><strong>UI Playground</strong></p><p>Now you can easily use this LinkedIn automation tool with a UI by simply entering the OpenAI and LinkedIn access tokens.</p><p>Huggingface Space UI Link : <a href="https://huggingface.co/spaces/gathnex/Linkedin_Automation">https://huggingface.co/spaces/gathnex/Linkedin_Automation</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeZ8v1Gp0Vr6bHJPxKYssA.png" /></figure><p>We highly recommend cloning the Hugging Face repository and utilizing this tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63Oq2rOCWPSiC5TP8MaNDQ.png" /></figure><h3>Conclusion</h3><p>In this article, we explored how to set up the LinkedIn automation system. If PSI is enhanced with chat history and an intent identification system with multiple specialties, it will yield even better results. Furthermore, we extend an open invitation to individuals who wish to contribute to this community, as their input and contributions are highly valued.</p><p>To get more update in AI follow us on LinkedIn : <a href="https://www.linkedin.com/company/gathnex/">https://www.linkedin.com/company/gathnex/</a></p><p>A special thanks to <a href="https://www.linkedin.com/in/gokul-raja-1541b8226/">Gokul Raja R</a>, who made a significant contribution to this project</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9a910777238" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Generative AI Course by DeepLearning.AI]]></title>
            <link>https://medium.com/@gokulrajar/new-generative-ai-course-by-deeplearning-ai-daf34e24e9c8?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/daf34e24e9c8</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[course]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Wed, 01 Nov 2023 15:44:12 GMT</pubDate>
            <atom:updated>2023-11-01T15:44:12.046Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ITlVadI2sR5OcBRo" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the ever-evolving landscape of artificial intelligence, understanding the nuances of generative AI has become imperative for professionals across various domains. Luckily, there’s a beacon of knowledge in the form of a course: “<a href="https://www.deeplearning.ai/courses/generative-ai-for-everyone/?utm_campaign=genai4e-launch&amp;utm_medium=video&amp;utm_source=youtube&amp;utm_content=teaser"><strong>Generative AI for Everyone</strong></a>” by DeepLearning.AI, guided by the <a href="https://www.linkedin.com/in/andrewyng/">Andrew Ng</a>.</p><p><em>Week 1: Introduction to Generative AI</em></p><p>The course kicks off with a comprehensive introduction to Generative AI. Through a series of engaging videos and insightful readings, participants unravel the fundamentals of generative AI. Andrew Ng’s expertise shines as he explains the intricacies, offering a clear understanding of what generative AI is and exploring its diverse applications.</p><p><em>Week 2: Exploring Generative AI Projects</em></p><p>Week two delves into the heart of generative AI projects. Participants are immersed in a world of possibilities as they explore various software applications. The course doesn’t stop at the basics; it ventures into advanced technologies, teaching participants to move beyond mere prompting. This deeper exploration equips learners to apply generative AI creatively and effectively.</p><p><em>Week 3: Generative AI’s Impact on Business and Society</em></p><p>The final week of the course is a revelation, shedding light on the intersection of generative AI with business and society. Participants gain insights into how generative AI is transforming the business landscape, opening new avenues and strategies. Moreover, the course delves into the societal implications, fostering a holistic understanding of the technology’s impact on our world.</p><p><a href="https://www.deeplearning.ai/courses/generative-ai-for-everyone/?utm_campaign=genai4e-launch&amp;utm_medium=video&amp;utm_source=youtube&amp;utm_content=teaser">Enroll now </a>: <a href="https://www.deeplearning.ai/courses/generative-ai-for-everyone/?utm_campaign=genai4e-launch&amp;utm_medium=video&amp;utm_source=youtube&amp;utm_content=teaser">https://www.deeplearning.ai/courses/generative-ai-for-everyone/?utm_campaign=genai4e-launch&amp;utm_medium=video&amp;utm_source=youtube&amp;utm_content=teaser</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=daf34e24e9c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zephyr 7B  beats ChatGPT. HuggingFace Open challenge to OpenAI.]]></title>
            <link>https://medium.com/@gokulrajar/zephyr-7b-beats-chatgpt-huggingface-open-challenge-to-openai-bba2fceecd9c?source=rss-5f2e53709304------2</link>
            <guid isPermaLink="false">https://medium.com/p/bba2fceecd9c</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[zephyr]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Gokul]]></dc:creator>
            <pubDate>Sat, 28 Oct 2023 18:18:45 GMT</pubDate>
            <atom:updated>2023-10-31T08:26:51.251Z</atom:updated>
            <content:encoded><![CDATA[<h3>Zephyr 7B Beta beats ChatGPT: HuggingFace’s Open challenge to OpenAI.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fXpzzIdpCWOr25HW" /><figcaption>Photo by <a href="https://unsplash.com/@maria_shalabaieva?utm_source=medium&amp;utm_medium=referral">Mariia Shalabaieva</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://huggingface.co/HuggingFaceH4">HuggingFace H4 Team</a> focuses on research and development of aligning language models to be helpful, honest, harmless. They released the <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr 7B alpha</a> model 3 weeks ago. Zephyr is a series of language models that are trained to act as helpful assistants. <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr-7B-α</a> is a fine tuned version of mistrial 7B , which outperforms massive language models like GPT-3.5, Llama-13B-chat, Falcon-40B and many more large language model’s.</p><p>Now the H4 team has released <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-beta">Zephyr 7B Beta</a> models which outperforms all large language models including gpt-3.5-turbo, llama 70b, etc and equally competes with gpt-4 in <a href="https://tatsu-lab.github.io/alpaca_eval/">alpaca benchmarks</a>. So the key fact is that zephyr 7b is 25 times smaller gpt-3.5 model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*lbVW8oPLjSURFlPGU-zEoQ.png" /></figure><h3>About Zephyr 7B β</h3><p>Zephyr-7B-β is the second model in the series, and is a fine-tuned version of <a href="https://huggingface.co/mistralai/Mistral-7B-v0.1">mistralai/Mistral-7B-v0.1</a> that was trained on on a mix of publicly available, synthetic datasets using <a href="https://arxiv.org/abs/2305.18290">Direct Preference Optimization (DPO)</a>.</p><p><strong>Model description</strong></p><ul><li>Model type: A 7B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.</li><li>Language(s) (NLP): Primarily English</li><li>License: MIT</li><li>Finetuned from model: <a href="https://huggingface.co/mistralai/Mistral-7B-v0.1">mistralai/Mistral-7B-v0.1</a></li></ul><p><strong>Performance</strong></p><p>At the time of release, Zephyr-7B-β is the highest ranked 7B chat model on the <a href="https://huggingface.co/spaces/lmsys/mt-bench">MT-Bench</a> and <a href="https://tatsu-lab.github.io/alpaca_eval/">AlpacaEval</a> benchmarks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*H06JhVNC-dEnBfG7xvpMew.png" /></figure><p>In particular, on several categories of MT-Bench, Zephyr-7B-β has strong performance compared to larger open models like Llama2-Chat-70B:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hnb84yMfkvxLLD4w.png" /></figure><p><strong>How to use Zephyr 7B Beta Model on consumer hardware.</strong></p><pre>!pip install -q accelerate bitsandbytes gradio transformers langchain xformers einops</pre><pre>import torch<br>import transformers<br>from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, TextStreamer, gradio</pre><pre>torch.set_default_device(&#39;cuda&#39;)<br><br>model = AutoModelForCausalLM.from_pretrained(&quot;HuggingFaceH4/zephyr-7b-beta&quot;,<br>                                             torch_dtype=&quot;auto&quot;, load_in_4bit=True)<br><br>tokenizer = AutoTokenizer.from_pretrained(&quot;HuggingFaceH4/zephyr-7b-beta&quot;,<br>                                          torch_dtype=&quot;auto&quot;)<br>def stream(user_prompt):<br>    runtimeFlag = &quot;cuda:0&quot;<br>    system_prompt = &#39;You are Gathnex AI, an intelligent assistant dedicated to providing effective solutions. Your responses will include emojis to add a friendly and engaging touch. 😊 Analyze user queries and provide clear and practical answers, incorporating emojis to enhance the user experience. Focus on delivering solutions that are accurate, actionable, and helpful. If additional information is required for a more precise solution, politely ask clarifying questions. Your goal is to assist users by providing effective and reliable solutions to their queries.&#39;<br>    E_INST = &quot;&lt;/s&gt;&quot;<br>    user, assistant = &quot;&lt;|user|&gt;&quot;, &quot;&lt;|assistant|&gt;&quot;<br><br>    prompt = f&quot;{system_prompt}{E_INST}\n{user}\n{user_prompt.strip()}{E_INST}\n{assistant}\n&quot;<br><br>    inputs = tokenizer([prompt], return_tensors=&quot;pt&quot;).to(runtimeFlag)<br><br>    streamer = TextStreamer(tokenizer,  skip_prompt=True, skip_special_tokens=True)<br><br>    _ = model.generate(**inputs, streamer=streamer, max_new_tokens=500)</pre><p>Let’s test the model</p><pre>stream(&quot;write a email about the product delivery deley for 1day, which is 29-10-23 will be the delivery data&quot;)</pre><p>Output:</p><pre><br>Subject: Product Delivery Delay Notification<br><br>Dear Valued Customer,<br><br>We regret to inform you that there has been a delay in the delivery of your recent order. The expected delivery date for your product is now 29-10-23, which is a one-day delay from the original delivery date.<br><br>We apologize for any inconvenience this may cause and assure you that we are doing everything possible to expedite the delivery process. The delay is due to unforeseen circumstances beyond our control, which has affected our logistics and supply chain.<br><br>We understand the importance of timely delivery, and we are working diligently to minimize the impact of this delay. We will keep you updated on the progress of your order and provide you with a new delivery window as soon as possible.<br><br>We appreciate your patience and understanding during this time, and we assure you that we will make every effort to ensure that your order is delivered to you as soon as possible.<br><br>If you have any further questions or concerns, please do not hesitate to contact us. Our customer service team is available 24/7 to assist you.<br><br>Thank you for your continued business, and we apologize once again for any inconvenience caused.<br><br>Best regards,<br><br>[Your Name]<br><br>[Your Company Name]<br><br>📦 Delivery Delay 🚚<br><br>📩 Email Notification 📧<br><br>📅 New Delivery Date: 29-10-23 📅<br><br>📱 Contact Us Anytime 📱<br><br>🤝 We&#39;re Here to Help 🤝</pre><p>Source : <a href="https://colab.research.google.com/drive/1-3cs5fm0zTNSSq69CcXiaVD3Xu54UhVW?usp=sharing">Google colab</a></p><p><strong>Why Zephyr 7B β is outperforming</strong></p><p>Just as with the alpha release, what is interesting about the model is not just the metrics, but how it was trained. Zephyr is a fine-tune with these components:</p><ul><li>Fine-tune of the best small open-source pretrained model out there: Mistral 7B</li><li>Usage of large scale preferences dataset: UltraFeedback</li><li>Drop RL to use Direct Preference Optimization (DPO)</li><li>Overfitting on the preference dataset surprisingly yields better chat results</li></ul><p><strong>Training Steps</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*LAD4FU0FjI93reiqRTsZXg.png" /></figure><ol><li>Distilled Supervised fine-tuning (dSFT): Build a large scale, self-instruct-style dataset (<a href="https://huggingface.co/datasets/stingning/ultrachat">UltraChat</a>) and then do distilled SFT.</li><li>AI Feedback (AIF) collection: 4 different LLMs generate completions and then GPT-4 is used to rank the responses (<a href="https://huggingface.co/datasets/openbmb/UltraFeedback">UltraFeedback</a>).</li><li>Distilled direct preference optimization (dDPO): We do DPO of the dSFT model (from step 1) using the feedback data (from step 2). DPO is an alternative to PPO that removes the need for a reward model. Zephyr beta trains for more DPO epochs (than Zephyr alpha) leading to better chat results!</li></ol><p><strong>Major facts of DPO</strong></p><ul><li>Overfitting with DPO leads to a better chat model according to all benchmarks</li><li>We did ablation experiments to see if SFT and DPO were really needed. Conclusions: DPO with no SFT leads to the model not learning the chat template. SFT + DPO yield the best results.</li><li>The feedback received for Zephyr Alpha was that there was incorrect casing (e.g. “Hi. how are you?”) and some responses were prefaced weirdly (e.g. “I don’t have personal X”), so we did some additional filtering for that.</li></ul><p>Huggingface released the language model alignment handbook as open source <a href="https://github.com/huggingface/alignment-handbook">https://github.com/huggingface/alignment-handbook</a> .</p><p><strong>Demo:</strong></p><p><a href="https://huggingface.co/spaces/HuggingFaceH4/zephyr-chat">Zephyr Chat - a Hugging Face Space by HuggingFaceH4</a></p><p>You can evaluate Zephyr 7B beta with other language models using Chatbot Arena, LMSYS arena: <a href="http://arena.lmsys.org/">http://arena.lmsys.org</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bba2fceecd9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>