<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shreya Sri on Medium]]></title>
        <description><![CDATA[Stories by Shreya Sri on Medium]]></description>
        <link>https://medium.com/@shreyasri.ankala?source=rss-f2e2bdc551f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GmpVJkes6Wj04XeQKYsQfQ.jpeg</url>
            <title>Stories by Shreya Sri on Medium</title>
            <link>https://medium.com/@shreyasri.ankala?source=rss-f2e2bdc551f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:30:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shreyasri.ankala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Recursive Language Models]]></title>
            <link>https://medium.com/ai-simplified-in-plain-english/recursive-language-models-056eea0d4762?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/056eea0d4762</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Thu, 22 Jan 2026 17:29:10 GMT</pubDate>
            <atom:updated>2026-01-28T18:00:12.287Z</atom:updated>
            <content:encoded><![CDATA[<h3>Recursive Language Models Explained Through Classic Algorithms and Real Systems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LLhUeVQmsYbVTS9Y" /><figcaption>Photo by <a href="https://unsplash.com/@kanhaiyasharma?utm_source=medium&amp;utm_medium=referral">Kanhaiya Sharma</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Have you ever tried to ask a language model a complex question by providing it with information from an external source? <br>For example, you wanted to learn Python, and you tried to copy and paste the whole Python documentation. Did it accept the input?</p><p>You would have met with a polite refusal.<br>“Sorry, this input exceeds the maximum context length.”<br>Or maybe it accepted it, but it didn’t remember key details from your input.</p><p>I was in a similar situation when it hit me:<br>LLMs aren’t actually “remembering” everything.<br>Everything they reason about must fit <strong>inside a limited token window</strong>.</p><p>As humans, we encounter massive amounts of information all the time. We read books, research papers, articles from various sources, debug code, and also learn a lot from our surroundings. How do we do it?</p><p>We use recursion.</p><p>And that’s what led to the invention of <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2512.24601">Recursive Language Models</a>.</p><h3><strong>The Recursion We Already Know:</strong></h3><p>Let’s start with the recursion we already know.</p><pre># A basic recursive Python code that adds elements to an array<br>def sum_array(arr):<br>    if len(arr) == 0:<br>        return 0<br>    return arr[0] + sum_array(arr[1:])</pre><p>What happens here?<br> 1. Breaking down the problem: sum the first element and the rest instead of summing the whole array.<br> 2. Call itself: Solve smaller pieces of code using the same function<br> 3. Combine results: Add the first element back in<br> 4. Stop eventually: Base cases help stop recursing when they’re encountered.</p><p>The classic recursion comes with a red flag: <strong>stack space. </strong>Each call adds a frame to memory, and going too deep leads you to a stack overflow.</p><h3><strong>Why Stack Space isn’t a problem?</strong></h3><p>Recursion in RLM occurs at the system level, not in memory.</p><p>In reality:</p><p>Each recursive step is <strong>a separate model call</strong>.<br>Memory for the model itself <strong>does not grow</strong> with recursion depth.<br>Intermediate data is stored externally in files, variables, or databases and can persist between calls.</p><p>Think of it like this:<br><strong>Classic Recursion:</strong></p><pre>Call f(100)<br>    Call f(99)<br>        Call f(98)<br>            ...</pre><p><strong>RLM Recursion:</strong></p><pre>Call model → get result → store result<br>Call model → get result → store result<br>Call model → get result → store result</pre><p>This makes sure that there’s no growing call stack inside the model. Each step is independent, so it never leads to a stack overflow. Instead, there’s a trade-off between compute and latency.</p><h3>How Current LLMs Remember Stuff?</h3><p>When you think a model remembers the previous conversation, this happens in reality:</p><p>It’s more like a <strong>sliding window </strong>over your conversation.<br>However, in contrast, RLMs store information in a structured environment, separate from the model itself. <br>Variable, files, databases, or data lakes can hold the intermediate results indefinitely, and the model then accesses them whenever needed. This is similar to how we humans take notes or highlight sections while reading a book.</p><p>But that doesn’t mean we should ditch LLMs completely; we need to understand how standard LLMs are efficient for small-scale tasks.</p><p>We should acknowledge how large some of the current LLMs’ context windows are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QkshAVi1ZRa3lZFsknuZcw.png" /><figcaption>source: <a href="https://artificialanalysis.ai/models?models=gpt-oss-120b%2Cgpt-oss-20b%2Cgpt-5-1%2Cgpt-5-2%2Cllama-4-maverick%2Cgemini-3-flash-reasoning%2Cgemini-3-pro%2Cclaude-opus-4-5-thinking%2Cclaude-4-5-sonnet-thinking%2Cmistral-large-3%2Cdeepseek-v3-2-reasoning%2Cfalcon-h1r-7b%2Cgrok-4-1-fast-reasoning%2Cgrok-4%2Cnova-2-0-lite-reasoning-medium%2Cnova-2-0-pro-reasoning-medium%2Cminimax-m2-1%2Cnvidia-nemotron-3-nano-30b-a3b-reasoning%2Ckimi-k2-thinking%2Ck-exaone%2Cmimo-v2-flash-reasoning%2Ckat-coder-pro-v1%2Ck2-v2%2Cmi-dm-k-2-5-pro-dec28%2Chyperclova-x-seed-think-32b%2Cglm-4-7%2Cqwen3-235b-a22b-instruct-2507-reasoning#context-window">ArtificialAnalysis.ai</a></figcaption></figure><p>If the input fits comfortably inside the context window, a single call is fast and simple. We don’t have to manage external memory or recursion. Less compute overhead with only one forward pass through the model.<br>Examples:<br> - Summarizing a single article or email<br> - Answering a short question<br> - Generating small pieces of code</p><p>For these tasks, introducing RLM recursion would be overkill; it’s like using recursion to add two numbers.</p><p>RLMs shine when we work with large, dense, or structured inputs where every detail matters:<br>Massive documents where you can’t safely summarize<br>Long conversations or multi-year logs<br>Pairwise comparisons, like checking relationships between all sections of a contract<br>Codebases that need systematic analysis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*9w6UlkBQuoCC1Q9XqFTGIg.png" /><figcaption>source: <a href="https://arxiv.org/pdf/2512.24601">RECURSIVE LANGUAGE MODELS</a></figcaption></figure><p>RLMs approach this by:<br>1. Breaking the task into smaller, manageable subproblems.<br>2. Processing each subproblem separately<br>3. Storing results externally.<br>4. Combining results recursively to build the final answer.</p><p>This way, it doesn’t rely on the model to remember everything in a token at once. This helps trade latency for scalability and reliability. which may be acceptable for high-stakes tasks.</p><h3>We May Think, Why Not Just Use a RAG?</h3><p>A RAG works by getting pieces of data or documents and feeding them into the model. This can be useful when we know what to retrieve, but it has its own limitations.</p><p>Retrieval may <strong>miss information</strong> needed for exhaustive reasoning.<br>RAG is less suitable for multi-pass, hierarchical, or pairwise tasks, where every part may influence the outcome.<br>It doesn’t natively support recursive reasoning over structured subproblems.</p><p>The following is a code snippet to see how RLMs work in action:</p><pre>document = load_large_document() too big to fit in one LLM call<br><br>def count_errors(doc_chunk):<br>    if len(doc_chunk) &lt; 2000:  # fits in context<br>        return model(f&quot;Count &#39;error&#39; in this text:\n{doc_chunk}&quot;)<br><br>    chunks = split_into_chunks(doc_chunk)  # break the document<br>    counts = [count_errors(c) for c in chunks]  #recursive call<br>    return sum(counts)# combine results<br><br>total_errors = count_errors(document)<br>print(&quot;Total errors:&quot;, total_errors)</pre><p>When we compare this to a standard LLM, if the document is too big, the model might fail or forget important context. RLMs help in covering the entire document systematically.</p><blockquote><strong>Knowing if we really need an RLM is like choosing whether to bring an umbrella; sometimes it’s overkill, but when it rains, you’re glad you have it.</strong></blockquote><p>It depends on the following factors:<br> 1. Memory needs: token-limited vs external storage<br> 2. Compute cost: multiple model calls vs one-pass model<br> 3. Task structure: shallow reasoning vs multi-pass, hierarchical reasoning</p><p>So while RLM recursion might feel “heavier,” it’s actually the right tool for the right problem.</p><h3>References</h3><ul><li><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2512.24601">Recursive Language Models</a></li></ul><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLMs and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=056eea0d4762" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-simplified-in-plain-english/recursive-language-models-056eea0d4762">Recursive Language Models</a> was originally published in <a href="https://medium.com/ai-simplified-in-plain-english">AI Simplified in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Will AI Agents Replace Humans?  — An Introduction To AI Agents]]></title>
            <link>https://medium.com/@shreyasri.ankala/will-ai-agents-replace-humans-an-introduction-to-ai-agents-753cb939c3f9?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/753cb939c3f9</guid>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[future]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 17:44:58 GMT</pubDate>
            <atom:updated>2025-02-28T17:44:59.710Z</atom:updated>
            <content:encoded><![CDATA[<h3>Will AI Agents Replace Humans? 🤖 — An Introduction To AI Agents</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nt7QCtt6NZhMnu9c" /></figure><p>AI is no longer just about answering questions — it’s about taking action. We’ve entered an era where AI is not just reactive but <strong>proactive</strong>, capable of reasoning, planning, and <strong>executing complex tasks autonomously</strong>. These AI agents are intelligent digital workers who can function independently, learn from experiences, and adapt to challenges. Unlike LLMs that generate text or RAGs that fetch information, <strong>AI agents interact with their environment, make decisions, and complete tasks with minimal supervision.</strong></p><p>Imagine an AI-powered assistant that not only answers your emails but also schedules meetings, books flights, manages finances and even writes code for you — <strong>without you explicitly telling it to do so each time</strong>. Sounds futuristic? It’s happening right now.</p><p>In this article, we’ll explore what AI agents are, how they work, their different types, and how they compare to LLMs and RAGs. Buckle up; the AI revolution is just getting started! 🚀</p><h3>What Are AI Agents? 🤖</h3><p>An <strong>AI agent</strong> is an intelligent system designed to <strong>observe, analyze, plan, and act</strong> in an environment <strong>without human intervention</strong>. Unlike traditional AI models that require prompts or predefined instructions, AI agents are more like autonomous decision-makers capable of strategizing and executing tasks.</p><h4>Key Features of AI Agents:</h4><p>✔ <strong>Autonomy</strong> — They function without continuous human guidance.</p><p>✔ <strong>Memory &amp; Adaptability</strong> — They can recall past interactions, learn from experiences, and improve decision-making.</p><p>✔ <strong>Tool &amp; API Integration</strong> — They can interact with external APIs, databases, and software applications.</p><p>✔ <strong>Multi-Step Planning</strong> — They don’t just react; they <strong>strategize and execute multi-step processes</strong> to complete tasks.</p><p>✔ <strong>Continuous Learning</strong> — Many use reinforcement learning and data-driven strategies to refine their decision-making.</p><p>Think of AI agents as <strong>digital employees</strong> — some can manage your tasks efficiently, while others might need more training before they get things right! 😅</p><h3>Key Principles That Define AI Agents 🧠</h3><p>AI agents are not just any software program — they are <strong>rational agents</strong> that make decisions based on their perceptions and data to <strong>achieve optimal performance</strong>. Here’s what sets them apart:</p><p>✔ <strong>Perception &amp; Sensory Input</strong> — AI agents collect data from their environment through physical sensors (e.g., cameras, microphones) or software inputs (e.g., APIs, user queries).</p><p>✔ <strong>Decision-Making &amp; Planning</strong> — They analyze information to determine the best possible actions for achieving their goals.</p><p>✔ <strong>Autonomy &amp; Adaptability</strong> — AI agents act independently and refine their strategies based on past experiences.</p><p>✔ <strong>Goal-Oriented Actions</strong> — Every action an AI agent takes is directed toward a specific outcome.</p><p>✔ <strong>Learning &amp; Improvement</strong> — Many AI agents continuously <strong>learn from feedback</strong> and adapt to changing environments.</p><h3>Key Components of AI Agent Architecture 🏗️</h3><p>✔ <strong>Architecture</strong> — The foundational structure (software-based or physical hardware) that supports AI agent operations.</p><p>✔ <strong>Agent Function</strong> — Defines how the AI agent transforms inputs into actions.</p><p>✔ <strong>Agent Program</strong> — The implementation layer where logic, ML models, and algorithms operate.</p><h3>Types of AI Agents ⚙️</h3><p>AI agents come in different forms, each suited for specific use cases and levels of complexity. Let’s explore the main categories:</p><h4>1️⃣ Reactive Agents — Instant Responders 🏃‍♂️</h4><p>These agents respond to inputs based on predefined rules, without storing past experiences. They are useful for simple automation tasks but lack adaptability.</p><p><strong>Example:</strong> <strong>Basic chatbots, spam filters, rule-based automation.</strong></p><h4>2️⃣ Deliberative (Goal-Oriented) Agents — Think Before Acting 🧠</h4><p>These agents analyze a situation, evaluate multiple options, and make decisions based on their goals.</p><p><strong>Example:</strong> <strong>AutoGPT, BabyAGI, Devin AI for software development.</strong></p><h4>3️⃣ Learning Agents — Getting Smarter Over Time 📈</h4><p>These agents use <strong>machine learning</strong> to refine their decision-making based on past interactions. They can adapt and improve over time.</p><p><strong>Example:</strong> <strong>AI-powered financial advisors, healthcare diagnostic systems.</strong></p><h4>4️⃣ Multi-Agent Systems — AI Teams Collaborating 🤝</h4><p>Multiple AI agents working together, communicating, and collaborating to solve complex problems.</p><p><strong>Example:</strong> <strong>Swarm robotics, collaborative AI research assistants.</strong></p><h3>AI Agents vs. LLMs vs. RAGs 🤔</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*dwei1ElFvVah34dsqm6SCw.png" /><figcaption>AI Agents vs LLMs vs RAGs</figcaption></figure><p><strong>LLMs</strong> (like GPT-4) are great at generating text but lack memory and autonomy. <strong>RAGs</strong> improve LLMs by fetching relevant external information. However, <strong>AI agents go further by actively working towards solving problems, planning tasks, and functioning independently.</strong> 🚀</p><h3>How AI Agents Work 🔍</h3><p>AI agents operate in a structured manner, often following these key steps:</p><p>1️⃣ <strong>Perception:</strong> The agent observes its surroundings, gathering information from various sources, such as user inputs, sensors, or APIs.</p><p>2️⃣ <strong>Understanding &amp; Reasoning:</strong> The agent processes this data using algorithms and models, identifying patterns, making inferences, and assessing possible actions.</p><p>3️⃣ <strong>Decision-Making:</strong> The agent evaluates different options and selects the most effective action based on its goals.</p><p>4️⃣ <strong>Action Execution:</strong> The agent carries out the selected task, whether it’s executing a script, sending a message, making a transaction, or interacting with other software.</p><p>5️⃣ <strong>Feedback Loop &amp; Learning:</strong> The agent monitors the results of its actions, refines its decision-making, and improves over time based on new data and experiences.</p><p>🔹 <strong>Example: AI Customer Support Agent</strong> Imagine you contact an AI-powered customer service chatbot. Unlike a basic chatbot, an AI agent can:</p><ul><li><strong>Understand your problem</strong> by analyzing previous conversations.</li><li><strong>Take action autonomously</strong> by processing refunds, updating your account, or escalating issues to a human agent if needed.</li><li><strong>Learn over time</strong> by improving responses based on customer feedback.</li></ul><h3>Real-World Use Cases 🌍</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hzbQHTwHyFsZGX4P" /><figcaption>A depiction of real world use cases</figcaption></figure><p>AI agents are already making waves across industries. Here’s how they’re transforming different sectors:</p><p>✅ <strong>Autonomous Coding Assistants</strong> — AI agents like <strong>Devin AI</strong> can debug, refactor, and develop software independently.</p><p>✅ <strong>AI Research Assistants</strong> — Automating hypothesis generation, testing, and refinement for scientific discovery.</p><p>✅ <strong>AI-Powered Customer Support</strong> — Going beyond answering queries, these agents actively troubleshoot issues.</p><p>✅ <strong>Financial AI Traders</strong> — Analyzing market trends and executing trades autonomously.</p><p>✅ <strong>Healthcare AI Agents</strong> — AI-powered diagnostics, treatment recommendations, and patient monitoring.</p><h3>Challenges of Using AI Agents ⚠️</h3><p>🚨 <strong>Data Privacy Concerns</strong> — AI agents require large datasets, raising security risks.</p><p>🚨 <strong>Ethical Issues</strong> — AI bias and unfair decision-making require regulatory oversight.</p><p>🚨 <strong>Technical Complexities</strong> — Developing AI agents demands specialized ML expertise.</p><p>🚨 <strong>Compute Resources</strong> — Training AI agents is resource-intensive and costly.</p><h3>The Future: Will AI Agents Replace Humans? 🤖</h3><p>AI agents are incredibly powerful, but they are not perfect. They still require <strong>human oversight, ethical considerations, and continuous fine-tuning</strong>. However, their ability to <strong>automate complex workflows, adapt to new challenges, and work with minimal supervision</strong> makes them a game-changer.</p><p>In the near future, we might see:</p><p>✔ AI agents managing entire <strong>business operations</strong> autonomously.</p><p>✔ AI-powered <strong>healthcare diagnostics</strong> providing real-time medical insights.</p><p>✔ Fully autonomous <strong>AI research labs</strong> accelerating discoveries in science and technology.</p><p>One thing is clear — AI agents are not just another tool; they are <strong>an evolution</strong> of AI itself. The question is, are we ready for them? 🚀</p><p>💬 <strong>What do you think? Could an AI agent handle your daily tasks? Let’s discuss!</strong> 🚀</p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=753cb939c3f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT 3.5 vs. 4 vs. 4 Omni: The Ultimate AI Showdown in a Tale of Tech and Use Cases]]></title>
            <link>https://generativeai.pub/gpt-3-5-vs-4-vs-4-omni-the-ultimate-ai-showdown-in-a-tale-of-tech-and-use-cases-762cbbff6e6f?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/762cbbff6e6f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Mon, 20 Jan 2025 07:33:55 GMT</pubDate>
            <atom:updated>2025-01-20T07:33:55.623Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*97qLBtD92mV2wF43" /><figcaption>Photo by <a href="https://unsplash.com/@saycheezestudios?utm_source=medium&amp;utm_medium=referral">Say Cheeze Studios</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Once upon a time in the ever-evolving kingdom of Artificial Intelligence, there lived three extraordinary siblings: GPT-3.5, GPT-4, and GPT-4-Turbo. Each had their unique abilities, strengths, and quirks that made them shine in their own way. This is the story of their adventures and how they changed the world.</p><h4>The Eldest — GPT-3.5</h4><p>GPT-3.5 was the trailblazer, a prodigy who carried forward the legacy of GPT-3. Known for its impressive intellect and quick wit, GPT-3.5 became the go-to wizard for solving riddles, crafting stories, and answering questions. However, it had its limitations. While fast and fairly clever, GPT-3.5 occasionally fumbled when tasks became too nuanced or required deep contextual understanding.</p><p>In the kingdom, GPT-3.5 found itself widely adored. Developers relied on it for coding assistance, and businesses used it for chatbots. But the villagers began to wonder, “Could there be a wiser and more capable sibling?”</p><h4>The Scholar — GPT-4</h4><p>Enter GPT-4, the elder sibling with a thirst for deeper knowledge and precision. GPT-4 was not just an improvement — it was a transformation. It could understand complex emotions, tackle intricate problems, and engage in conversations with a depth and elegance previously unseen. The scholars in the kingdom marveled at its ability to write essays that felt human and its knack for making subtle distinctions in meaning.</p><p>Where GPT-3.5 might stumble over ambiguous questions, GPT-4 could gracefully untangle them. It became the trusted companion for researchers, legal experts, and those working in fields requiring exceptional accuracy. However, being this wise came with a cost — GPT-4 was slower than GPT-3.5, and its services were slightly more expensive. It was a scholar, after all, and scholarly wisdom required patience.</p><h4>The Speedster — GPT-4-Turbo</h4><p>Then came the youngest sibling, GPT-4-Turbo, a sprightly and efficient version of GPT-4. It retained much of the intelligence and nuance of its elder sibling but came with a twist — it was faster and cheaper! Imagine the brilliance of GPT-4, but with the energy of a young athlete ready to sprint through tasks.</p><p>GPT-4-Turbo quickly became a favorite for real-time applications. Businesses loved it for powering chatbots, customer service tools, and even real-time data analysis. Its affordability made advanced AI accessible to more villagers, ensuring that the benefits of AI could spread far and wide.</p><h4>The Tech Behind the Magic</h4><p>The magic of these siblings wasn’t just in their abilities but in the technology that powered them.</p><ul><li><strong>GPT-3.5</strong> relied on foundational transformer architectures, excelling in generating human-like text but limited in depth and nuance.</li><li><strong>GPT-4</strong> introduced a more refined architecture with larger training datasets and better optimization techniques. It handled complex tasks with grace, capturing subtle patterns in data and producing highly contextual outputs.</li><li><strong>GPT-4-Turbo</strong>, on the other hand, was designed for efficiency. Its architecture was optimized for speed without compromising much on quality, making it a practical choice for real-time applications.</li></ul><p>Behind the scenes, their creators at OpenAI continually fine-tuned these siblings, ensuring that they could meet the evolving needs of the kingdom.</p><h4>Real-Time Adventures</h4><p>The three siblings ventured far and wide, finding their place in various domains:</p><ol><li><strong>GPT-3.5</strong>: The storyteller and assistant, perfect for drafting emails, writing scripts, and brainstorming ideas. It was the friend you turned to for creative projects and quick solutions.</li><li><strong>GPT-4</strong>: The intellectual, solving complex coding problems, crafting detailed legal documents, and analyzing research data. It was the sibling you called upon for heavy lifting when accuracy and depth were paramount.</li><li><strong>GPT-4-Turbo</strong>: The multitasking speedster, excelling in powering conversational AI, generating product recommendations, and assisting in fast-paced environments like e-commerce and customer support.</li></ol><p>According to a tweet by Sam Altman, founder of OpenAI, ChatGPT 4 “hallucinates significantly less.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PRtTkyYDPwGx0bkV" /><figcaption><em>Source -Twitter account of Sam Altman</em></figcaption></figure><h3>The Visionary — GPT-4 Omni</h3><p>As the tale of the three siblings unfolded, whispers spread across the kingdom of a fourth sibling on the horizon. This sibling, known as GPT-4 Omni, was unlike any that came before — a visionary who could see, interpret, and understand the world not only through words but through images, graphs, and beyond. Omni was heralded as the “jack-of-all-modalities,” bridging the gap between text and vision to unlock new dimensions of AI capabilities.</p><p>GPT-4 Omni wasn’t just a conversationalist or writer — it was an observer. It could analyze visual data, describe the contents of an image, interpret trends from graphs, and even assist with tasks like medical image analysis. The kingdom marveled at its ability to explain a picture in exquisite detail or combine visual and textual data for profound insights.</p><p>This made Omni a favorite for domains like:</p><ul><li><strong>Healthcare:</strong> Reading and interpreting X-rays or MRIs alongside medical notes.</li><li><strong>Education:</strong> Teaching through interactive visuals and detailed explanations.</li><li><strong>Research and Design:</strong> Assisting in visual-heavy fields like architecture and engineering.</li></ul><p>But as powerful as Omni was, it had to compete with its younger sibling, GPT-4 Turbo, in specific domains. This set the stage for an exciting rivalry.</p><h3>GPT-4 Omni vs. GPT-4 Turbo — A Clash of Strengths</h3><p>Though they were siblings, GPT-4 Omni and GPT-4 Turbo were designed with different strengths, and their rivalry became the talk of the kingdom. Here’s how they compared:</p><p><strong>FeatureGPT-4 OmniGPT-4 TurboPrimary Strength</strong>Multimodal genius — processing both text and visual data for richer insights.Lightning-fast, cost-efficient, and text-focused mastery.<strong>Use Cases</strong>Complex tasks involving visual and textual analysis (e.g., interpreting images, creating captions, or data visualization).High-speed conversational AI for customer support, e-commerce, and text-heavy tasks.<strong>Speed</strong>Slower due to the additional processing required for visual data.Significantly faster for text-only applications.<strong>Cost</strong>Higher, as it integrates advanced multimodal capabilities.More affordable, making it accessible for real-time, large-scale deployments.<strong>Innovation Factor</strong>Pioneering new frontiers in AI through multimodal abilities.Optimized for high performance and scalability in text-based scenarios.</p><h3>Sibling Synergy</h3><p>Despite their differences, the kingdom soon realized that the true power of the siblings lay in their ability to complement each other.</p><ul><li>GPT-4 Turbo could handle the fast-paced, everyday needs of businesses and consumers, ensuring efficiency and affordability.</li><li>GPT-4 Omni stepped in for more intricate tasks, blending vision and language to deliver unparalleled insights.</li></ul><p>For example:</p><ul><li>A <strong>retail platform</strong> might use GPT-4 Turbo for real-time customer support and GPT-4 Omni to analyze product images, identify defects, or generate creative marketing visuals.</li><li>In <strong>healthcare</strong>, Turbo could provide rapid responses to general medical inquiries, while Omni could assist doctors in diagnosing conditions by interpreting both medical text and imaging.</li></ul><h3>Choosing the Right Sibling for the Task</h3><p>With so many brilliant siblings in the GPT family, villagers often found themselves asking, <em>“Which one should I use for my task?”</em> To help them decide, the siblings created a guide that compared their strengths and ideal use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*sgpoKSOcKcYvEr4qkR_05w.png" /><figcaption>When to use which model?</figcaption></figure><h3>Latest GPT-4 Models: Comparison</h3><p>OpenAI continues to refine its GPT models to meet evolving user needs. Since the launch of GPT-4 in March 2023, significant advancements have been made, leading to the development of GPT-4 Turbo and GPT-4 Omni (GPT-4o). Each model offers unique strengths tailored to specific applications, ensuring users have the right tool for their diverse requirements.</p><p>The table below highlights the key differences and strengths of the latest GPT-4 models, helping you determine which one is best suited for your use case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*Zu_M3_4SH_Z3nrHJ1GprSg.png" /><figcaption>key comparisions</figcaption></figure><h3>Pricing Comparison of GPT Models</h3><p>OpenAI’s suite of GPT models offers flexibility in pricing based on input and output token usage, catering to diverse needs ranging from cost-effective general tasks to premium, high-context applications.For all further details, please refer to the <a href="https://openai.com/api/pricing"><strong>Pricing page of OpenAI</strong></a><strong>.</strong> The table below provides a breakdown of the costs associated with each model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*ktgyEno33_enoAwjAd1m1A.png" /><figcaption>Pricing comparision</figcaption></figure><h3>Key Differences</h3><ul><li><strong>Use GPT-3.5</strong> when you need quick, general-purpose assistance at an affordable cost.</li><li><strong>Choose GPT-4</strong> for tasks requiring high accuracy, deeper reasoning, and nuanced understanding.</li><li><strong>Pick GPT-4 Turbo</strong> for real-time, scalable applications where speed and cost-efficiency matter.</li><li><strong>Select GPT-4 Omni</strong> for complex multimodal tasks that require analyzing both visual and textual data.</li></ul><p>This table became a cherished guide for the villagers, helping them harness the full potential of the GPT family. Whether they were writing stories, building chatbots, or analyzing images, there was always a sibling ready to lend a hand.</p><h3>The Evolving Family</h3><p>The tale of GPT-4 Omni and its siblings reminds us that innovation doesn’t end with a single breakthrough — it’s an ongoing journey. Each sibling plays a vital role in advancing the kingdom of AI, pushing boundaries, and creating new opportunities.</p><p>As the family grows, the kingdom eagerly awaits the next chapter in this saga. Will there be a new sibling on the horizon? Only time will tell. For now, the world celebrates the brilliance of GPT-3.5, GPT-4, GPT-4 Turbo, and GPT-4 Omni — a family that has transformed the way we live, work, and dream.</p><h3>Other Such Articles</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*cmNeHDGS7LCai9TT.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*C60VCJpFIvKWrBL2.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=762cbbff6e6f" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/gpt-3-5-vs-4-vs-4-omni-the-ultimate-ai-showdown-in-a-tale-of-tech-and-use-cases-762cbbff6e6f">GPT 3.5 vs. 4 vs. 4 Omni: The Ultimate AI Showdown in a Tale of Tech and Use Cases</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Deployment in the Cloud: A Practical Guide ]]></title>
            <link>https://generativeai.pub/llm-deployment-in-the-cloud-a-practical-guide-435db7fda3b7?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/435db7fda3b7</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Fri, 06 Dec 2024 05:22:33 GMT</pubDate>
            <atom:updated>2024-12-06T15:04:23.508Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*KjwdrxWAUPHa3E2C" /><figcaption>source:<a href="https://giphy.com/">https://giphy.com/</a></figcaption></figure><p>Imagine this: You’ve developed an incredible large language model (LLM) that generates poetry, answers complex questions, or powers chatbots. But there’s one problem — it’s sitting idle on your local machine. How can you make it accessible to millions worldwide? 🌍 The answer lies in deploying your LLM on the cloud. Let’s dive into the process step-by-step, combining practicality with detailed code snippets for you to follow along.</p><h3>Essential Considerations Before Deploying LLMs in the Cloud 🤔</h3><p>Before jumping into the deployment process, it’s essential to address some critical considerations. Deploying LLMs isn’t just about uploading a model; it’s about creating a robust, efficient, and secure system. Here are the questions you should answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*IMHv5Tav0ZxozzWn" /><figcaption>Model management(source:<a href="https://giphy.com/">https://giphy.com/</a>)</figcaption></figure><h4>Model Management 🛠️</h4><ul><li>How will you version and manage multiple models (e.g., base, fine-tuned, adapters) effectively?</li><li>Can you roll back or update models without disrupting the system? Ensuring smooth updates and maintaining consistency is vital for production-grade deployments.</li></ul><h4>Scalability and Performance ⚡</h4><ul><li>Is your infrastructure designed to handle traffic surges without downtime? Cloud-native solutions like autoscaling and load balancing play a critical role here.</li><li>Have you implemented strategies to minimize inference latency? Preloading models and optimizing inference pipelines are key to delivering fast responses.</li></ul><h4>Cost Optimization 💰</h4><ul><li>Are you utilizing techniques like model compression to reduce resource usage? Efficient models can significantly cut down operational costs.</li><li>Is your autoscaling configured to optimize costs during low-traffic periods? Balancing performance and cost is essential for sustainable deployments.</li></ul><h4>Security and Compliance 🔒</h4><ul><li>Are your deployments compliant with data regulations (e.g., GDPR, HIPAA)? Non-compliance can lead to hefty fines and legal issues.</li><li>Have you implemented encryption and access control for sensitive data? Protecting user data and model integrity should be a top priority.</li></ul><h4>Monitoring and Maintenance 🛡️</h4><ul><li>Do you have observability tools in place to monitor performance, utilization, and errors? Proactive monitoring helps in identifying and resolving issues quickly.</li><li>How will you manage and scale your deployments over time? A well-defined maintenance strategy ensures longevity and reliability.</li></ul><h3>Strategies for Deploying LLMs on the Cloud 🌐</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*LQjSDVvcHzPJgbJw" /><figcaption>Strategy(source:<a href="https://giphy.com/">https://giphy.com/</a>)</figcaption></figure><p>Addressing the above questions requires implementing best practices tailored to cloud-based LLM deployments. Let’s explore some strategies:</p><h4>Centralize Model Management 🗂️</h4><p>Use tools like MLflow to track and manage model versions. A centralized model registry allows seamless updates and rollbacks, keeping your deployments organized and compliant.</p><h4>Optimize Scalability with Autoscaling 📈</h4><p>Leverage Kubernetes autoscaling to ensure your infrastructure dynamically adjusts to traffic demands. Preloading foundation models into distributed storage (e.g., S3, NFS) minimizes delays during scaling.</p><h4>Reduce Costs with Model Compression 🧠</h4><p>Techniques like quantization and model distillation can make your models smaller and faster. These optimizations reduce computational costs while maintaining accuracy.</p><h4>Enhance Versatility with LoRA Adapters 🎛️</h4><p>Instead of deploying multiple foundation models, use lightweight, fine-tuned adapters like LoRA. These modules can be dynamically loaded for specific tasks, saving GPU memory and resources.</p><h4>Build Resilient Inference Pipelines 🏗️</h4><ul><li>Implement sharding techniques like pipeline parallelism or tensor parallelism for large models.</li><li>Use a hybrid cloud and edge strategy to bring inference closer to the user, reducing latency and improving performance.</li></ul><h4>Prioritize Observability 🔍</h4><ul><li>Monitor GPU utilization, cache hit rates, and inference latency with tools like Prometheus and Grafana.</li><li>Use inference-specific metrics to fine-tune scaling policies and identify bottlenecks in your system.</li></ul><h3>Best Practices for Cloud-Based LLM Deployments 🌟</h3><ul><li><strong>Pre-Caching Models:</strong> Reduce initialization delays by caching foundation models into local or distributed storage before scaling.</li><li><strong>Latency-Aware Load Balancing:</strong> Distribute traffic intelligently to minimize response times and maximize resource utilization.</li><li><strong>Dynamic Adapter Loading:</strong> Use lightweight, task-specific adapters to avoid deploying multiple resource-heavy models.</li><li><strong>End-to-End Monitoring:</strong> Track hardware, application, and model-specific metrics to ensure optimal performance.</li><li><strong>Compliance-First Approach:</strong> Regularly audit deployments to ensure adherence to data protection regulations.</li></ul><h3>The Setup: Choosing the Right Cloud Provider ☁️</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*L8DF9qn0MET8DlGi" /><figcaption>source:<a href="https://giphy.com/">https://giphy.com/</a></figcaption></figure><p>Before you deploy, you need a home for your model. AWS, Google Cloud Platform (GCP), and Azure are popular choices. Each offers robust infrastructure for hosting AI models, but for simplicity, let’s focus on deploying using <strong>AWS</strong>.</p><h3>Prerequisites ✅</h3><ol><li>An AWS account.</li><li>Basic knowledge of Docker.</li><li>Your trained LLM model (for example, a Hugging Face model).</li><li>AWS CLI installed and configured.</li></ol><h3>Step 1: Preparing Your Model 🧳</h3><p>We’ll use the Hugging Face Transformers library to prepare the model.</p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br>import torch</pre><pre># Load your model and tokenizer<br>model_name = &quot;gpt2&quot;  # Replace with your model<br>model = AutoModelForCausalLM.from_pretrained(model_name)<br>tokenizer = AutoTokenizer.from_pretrained(model_name)</pre><pre># Save the model locally<br>model.save_pretrained(&quot;./model&quot;)<br>tokenizer.save_pretrained(&quot;./model&quot;)</pre><p>This saves your model and tokenizer files locally, ready for deployment.</p><h3>Step 2: Dockerize Your Model 🐳</h3><p>Containerization ensures consistency across environments. Create a Dockerfile to package your model and API server.</p><p><strong>Dockerfile:</strong></p><pre># Base image<br>FROM python:3.9</pre><pre># Set working directory<br>WORKDIR /app</pre><pre># Copy requirements and install dependencies<br>COPY requirements.txt ./<br>RUN pip install --no-cache-dir -r requirements.txt</pre><pre># Copy application code<br>COPY . .</pre><pre># Expose port<br>EXPOSE 8080</pre><pre># Start the server<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><p><strong>requirements.txt:</strong></p><pre>fastapi<br>uvicorn<br>transformers<br>torch</pre><p><strong>app.py:</strong></p><pre>from fastapi import FastAPI<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>import torch<br><br>app = FastAPI()<br># Load model and tokenizer<br>model_name = &quot;./model&quot;  # Path to your saved model<br>model = AutoModelForCausalLM.from_pretrained(model_name)<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br>@app.post(&quot;/generate/&quot;)<br>async def generate(prompt: str):<br>    inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;)<br>    outputs = model.generate(inputs[&quot;input_ids&quot;], max_length=50)<br>    response = tokenizer.decode(outputs[0], skip_special_tokens=True)<br>    return {&quot;response&quot;: response}</pre><p>Build the Docker image:</p><pre>docker build -t llm-api .</pre><h3>Step 3: Push Docker Image to AWS ECR 📦</h3><ol><li><strong>Create an ECR repository:</strong></li></ol><ul><li>aws ecr create-repository --repository-name llm-api</li></ul><p><strong>2. Authenticate Docker to ECR:</strong></p><ul><li>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin &lt;your_account_id&gt;.dkr.ecr.us-east-1.amazonaws.com</li></ul><p><strong>3. Tag and push the image:</strong></p><ul><li>docker tag llm-api:latest &lt;your_account_id&gt;.dkr.ecr.us-east-1.amazonaws.com/llm-api docker push &lt;your_account_id&gt;.dkr.ecr.us-east-1.amazonaws.com/llm-api</li></ul><h3>Step 4: Deploy to AWS ECS 🚀</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/0*3JSgkRCJW4bRgHce" /><figcaption>source:<a href="https://giphy.com/">https://giphy.com/</a></figcaption></figure><ol><li><strong>Create a Task Definition:</strong> Define your container settings using the ECS console or CLI. Example JSON:</li></ol><pre>{     &quot;family&quot;: &quot;llm-api-task&quot;,     <br>&quot;containerDefinitions&quot;: [         <br>{             &quot;name&quot;: &quot;llm-api&quot;,             <br>&quot;image&quot;: &quot;&lt;your_account_id&gt;.dkr.ecr.us-east-1.amazonaws.com/llm-api&quot;,<br>             &quot;memory&quot;: 512,             <br>&quot;cpu&quot;: 256,             <br>&quot;essential&quot;: true,           <br>  &quot;portMappings&quot;: [                 <br>{                     &quot;containerPort&quot;: 8080,                     &quot;hostPort&quot;: 8080                 }             ]         }     ] }</pre><ol><li>Use the CLI to register it:</li></ol><ul><li>aws ecs register-task-definition --cli-input-json file://task-def.json</li></ul><p><strong>2. Create a Cluster and Service:</strong> Launch the task on an ECS cluster. You can create a new cluster</p><ol><li><strong>Launch a Service:</strong> Use the ECS CLI or AWS Management Console to launch your service, ensuring the task you defined runs on the cluster:</li></ol><pre>aws ecs create-service \     --cluster llm-cluster \     --service-name llm-service \     --task-definition llm-api-task \     --desired-count 1 \     --launch-type FARGATE \     --network-configuration &quot;awsvpcConfiguration={subnets=[subnet-12345abc],securityGroups=[sg-12345abc],assignPublicIp=ENABLED}&quot;</pre><ol><li>This command specifies Fargate as the launch type for serverless deployment. Replace subnet-12345abc and sg-12345abc with your actual subnet and security group IDs.</li></ol><h3>Step 5: Configure Load Balancing and Auto Scaling</h3><ol><li><strong>Attach a Load Balancer:</strong> Create an Application Load Balancer (ALB) to distribute traffic and attach it to your ECS service. You can configure this in the AWS Console under the “Load Balancers” section or use the CLI.</li><li><strong>Set Up Auto Scaling:</strong> Enable auto-scaling for your service to adjust the number of running tasks based on traffic:</li></ol><pre>aws application-autoscaling register-scalable-target \     --service-namespace ecs \     --resource-id service/llm-cluster/llm-service \     --scalable-dimension ecs:service:DesiredCount \     --min-capacity 1 \     --max-capacity 10</pre><p>3. Define scaling policies for CPU or memory utilization:</p><pre>aws application-autoscaling put-scaling-policy \     --policy-name cpu-scaling-policy \     --service-namespace ecs \     --resource-id service/llm-cluster/llm-service \     --scalable-dimension ecs:service:DesiredCount \     --policy-type TargetTrackingScaling \     --target-tracking-scaling-policy-configuration file://cpu-policy.json</pre><p>Example cpu-policy.json:</p><pre>{     &quot;TargetValue&quot;: 70.0,     &quot;PredefinedMetricSpecification&quot;: {         &quot;PredefinedMetricType&quot;: &quot;ECSServiceAverageCPUUtilization&quot;     },     &quot;ScaleInCooldown&quot;: 60,     &quot;ScaleOutCooldown&quot;: 60 }</pre><h3>Step 6: Test Your Deployment</h3><p>Once your service is running, access it through the ALB’s DNS name or public IP. Send requests to the /generate/ endpoint using tools like curl or Postman to ensure your model responds correctly:</p><pre>curl -X POST http://&lt;your-alb-dns&gt;/generate/ \<br>     -H &quot;Content-Type: application/json&quot; \<br>     -d &#39;{&quot;prompt&quot;: &quot;Write a poem about the sea&quot;}&#39;</pre><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*V6nacoG5hunSm3_R.gif" /><figcaption>source:<a href="https://giphy.com/">https://giphy.com/</a></figcaption></figure><p>Deploying large language models on the cloud is a journey that requires careful planning, robust architecture, and ongoing monitoring. By addressing key questions, implementing proven strategies, and following this comprehensive step-by-step guide, you’ll unlock the full potential of your models to scale and serve users effectively.</p><p>Whether you’re building an AI-powered application or experimenting with the latest LLMs, cloud deployment bridges the gap between local experimentation and global impact. 🌐</p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=435db7fda3b7" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/llm-deployment-in-the-cloud-a-practical-guide-435db7fda3b7">LLM Deployment in the Cloud: A Practical Guide 🚀</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vision Language Models (VLM): Bridging the Gap Between Vision and Text]]></title>
            <link>https://generativeai.pub/vision-language-models-vlm-bridging-the-gap-between-vision-and-text-9ff2e5a45920?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ff2e5a45920</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Wed, 20 Nov 2024 08:50:38 GMT</pubDate>
            <atom:updated>2024-11-20T08:50:38.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C_sCx1aasbBLjqfK" /><figcaption>Photo by <a href="https://unsplash.com/@dtravisphd?utm_source=medium&amp;utm_medium=referral">David Travis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine you’re at an art gallery. A painting catches your eye, and you’re curious about its story. What if a system could explain not just what the painting depicts but also narrate its artistic significance? Welcome to the world of <strong>Vison Language Models (VLM)</strong>! These models combine the power of computer vision and natural language processing to understand and generate text based on images. Let’s dive into the journey of VLMs, their architecture, and a simple implementation to bring this technology to life.</p><h3><strong>The Origin of VLMs</strong></h3><p>The concept of VLMs emerged from the need to interpret and describe the visual world in human language. Initially, computer vision models like Convolutional Neural Networks (CNNs) focused on recognizing objects in images, while NLP models specialized in text understanding. The idea of merging these capabilities gained traction with advancements in transformer architectures (like BERT and Vision Transformers). The integration of vision and language models opened doors to tasks like <strong>image captioning</strong>, <strong>visual question answering (VQA)</strong>, and <strong>image-text retrieval</strong>.</p><p>Key milestones in this journey:</p><ul><li><strong>ImageNet (2012)</strong>: Revolutionized computer vision with large-scale image classification.</li><li><strong>Transformers (2017)</strong>: Introduced by Vaswani et al., transformers became the backbone of language models.</li><li><strong>CLIP (2021)</strong>: OpenAI’s breakthrough in aligning vision and text using contrastive learning.</li><li><strong>BLIP, Flamingo, and others</strong>: Pushed the boundaries of multimodal learning.</li></ul><h3>What are Vison Language Models (VLMs)?</h3><p><strong>Vison Language Models (VLMs)</strong> are advanced AI systems designed to understand, interpret, and generate meaningful connections between visual data (like images and videos) and textual information (like descriptions and queries). They bridge two of the most prominent domains in artificial intelligence: <strong>computer vision</strong> and <strong>natural language processing (NLP)</strong>.</p><p>To grasp the essence of VLMs, imagine a system that not only “sees” an image but also “understands” and “talks about” it. For instance, given a picture of a sunset, a VLM might say, <em>“A stunning orange sunset over a calm ocean.”</em> This capability is made possible by combining the strengths of vision and language processing models.</p><h3>Why Were VLMs Created?</h3><p>Before the advent of VLMs, AI models specialized in either processing visual data (like identifying objects in an image) or understanding language (like answering questions or generating text). However, many real-world problems require both abilities to work together.</p><p>Examples:</p><ol><li><strong>Image Captioning</strong>: Writing a caption that describes the content of an image.</li><li><strong>Visual Question Answering (VQA)</strong>: Answering a user’s question based on an image.</li><li><strong>Image-Text Retrieval</strong>: Searching for relevant images based on a textual query (or vice versa).</li></ol><p>These challenges highlighted the need for models capable of <strong>multimodal understanding</strong>, combining visual and textual data seamlessly.</p><h3>Why Are VLMs Important?</h3><p>The ability of VLMs to work across modalities makes them incredibly versatile and impactful:</p><ol><li><strong>Accessibility</strong>: Describing images for visually impaired individuals.</li><li><strong>Efficiency</strong>: Automating content moderation on platforms like social media.</li><li><strong>Search Optimization</strong>: Improving search engines to work seamlessly with images and text queries.</li><li><strong>Creativity</strong>: Generating art or storytelling by blending visual and textual inputs.</li></ol><h3>An Analogy: How VLMs See the World</h3><p>Think of a VLM as a <em>travel guide</em> in a foreign land. The guide can both:</p><ul><li>Look at a historical monument (visual understanding).</li><li>Explain its significance and history in your language (textual generation).</li></ul><p>This ability to interpret, relate, and communicate across two domains makes VLMs a powerful tool for understanding the world.</p><p>Would you like to explore more about a specific aspect, such as its use in applications like ChatGPT Vision or DALL-E? Let me know!</p><h3>How Do VLMs Work?</h3><p>At the heart of every VLM lies the ability to <strong>process images and text in a unified framework</strong>. Here’s how it typically works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*OzBw1g0YNcRqgxr96Rmevg.png" /><figcaption>Generic architecture</figcaption></figure><ol><li><strong>Vision Encoder</strong></li></ol><ul><li>This module processes images.</li><li>Modern VLMs use Vision Transformers (ViTs) instead of traditional CNNs. ViTs divide the image into patches and process them similarly to words in text.</li><li>Output: A set of embeddings representing the visual features.</li></ul><p><strong>2. Text Encoder</strong></p><ul><li>This module processes text.</li><li>Popular choices include transformer-based models like BERT or GPT.</li><li>Output: Text embeddings capturing semantic meaning.</li></ul><p><strong>3. Multimodal Fusion</strong></p><ul><li>The heart of VLMs lies in combining visual and textual information.</li><li>Common approaches:</li><li><strong>Cross-attention</strong>: Enables interaction between image and text embeddings.</li><li><strong>Contrastive Learning</strong>: Aligns images and text in a shared latent space (used in CLIP).</li></ul><p>Here’s a simplified flowchart:</p><pre>[Image] --&gt; Vision Encoder --&gt; Image Embeddings  <br>[Text] --&gt; Text Encoder --&gt; Text Embeddings  <br>         Fusion Layer (e.g., Cross-Attention or Contrastive Loss)  <br>                      ↓  <br>          Multimodal Representations</pre><h3><strong>Hands-On: Building a Simple VLM with CLIP</strong></h3><p>Let’s implement a basic Vison Language Model using OpenAI’s CLIP, which is pretrained for multimodal tasks. We’ll use the transformers and torch libraries.</p><ol><li><strong>Setup</strong><br>Install the required libraries:</li></ol><pre>pip install torch torchvision transformers</pre><p><strong>2. Code Implementation</strong></p><pre>import torch<br>from PIL import Image<br>from transformers import CLIPProcessor, CLIPModel<br><br># Load CLIP model and processor<br>model = CLIPModel.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>processor = CLIPProcessor.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br><br># Load an image and text<br>image = Image.open(&quot;example_image.jpg&quot;)  # Replace with your image file<br>text = [&quot;A cute cat sitting on a couch&quot;, &quot;A dog running in a park&quot;]<br><br># Preprocess inputs<br>inputs = processor(text=text, images=image, return_tensors=&quot;pt&quot;, padding=True)<br><br># Get image and text embeddings<br>outputs = model(**inputs)<br>image_embeddings = outputs.image_embeds<br>text_embeddings = outputs.text_embeds<br><br># Calculate similarity<br>similarity = torch.nn.functional.cosine_similarity(image_embeddings, text_embeddings)<br>print(&quot;Similarity Scores:&quot;, similarity)</pre><p><strong>Explanation</strong></p><ul><li>The CLIP model aligns image and text embeddings. Higher similarity scores indicate better matches between the image and the text.</li><li>Try experimenting with different images and captions to see how the model performs.</li></ul><p><strong>Applications of VLMs</strong></p><ul><li><strong>Image Captioning</strong>: Automatically generating descriptive captions for images.</li><li><strong>Visual Question Answering (VQA)</strong>: Answering questions based on image content.</li><li><strong>Content Moderation</strong>: Detecting inappropriate content in images and videos.</li><li><strong>Creative Tasks</strong>: Designing AI art tools that blend visual and textual creativity.</li></ul><p><strong>Conclusion</strong><br>Vison Language Models are a testament to how AI can bridge modalities, bringing us closer to machines that truly understand our world. As the technology evolves, the possibilities for human-AI collaboration will only expand. So, the next time you admire a painting or capture a photo, imagine the stories a VLM could tell!</p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*CNNbwGdGcGMSvsvw.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*tPU4xfn4gZ05gigw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff2e5a45920" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/vision-language-models-vlm-bridging-the-gap-between-vision-and-text-9ff2e5a45920">Vision Language Models (VLM): Bridging the Gap Between Vision and Text</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GraphRAG Explained: Combining Graphs and Generative AI for Better Retrieval]]></title>
            <link>https://generativeai.pub/graphrag-explained-combining-graphs-and-generative-ai-for-better-retrieval-fd324d0a2ddd?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd324d0a2ddd</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 05:53:58 GMT</pubDate>
            <atom:updated>2024-10-28T05:53:58.262Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xF930m_adj0qh_8a" /><figcaption>Photo by <a href="https://unsplash.com/@martz90?utm_source=medium&amp;utm_medium=referral">Martin Martz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine you’re navigating a vast library filled with every piece of information known to humanity. In front of you stands a guide, well-versed in retrieving exactly what you need, no matter the complexity of your question. This is the power of GraphRAG, a sophisticated technique that combines Graph Neural Networks (GNNs) with Retrieval-Augmented Generation (RAG) to provide ultra-precise answers and insights. Our journey here will introduce GraphRAG’s purpose, the mechanics behind its processes, and an implementation guide to help you create your own version.</p><h3>The Why Behind GraphRAG</h3><p>Large Language Models (LLMs) can perform remarkably well, but they face limitations when it comes to handling complex, interconnected data. Standard RAG-based systems improve retrieval by fetching relevant chunks of information from external sources, yet they treat data as independent entries. This is where <strong>GraphRAG</strong> steps in, using <strong>graphs to model relationships</strong> among entities, concepts, and contexts.</p><p>GraphRAG can capture dependencies between nodes (representing data) and edges (representing relationships), making it more suitable for applications that need to interpret relationships, such as scientific literature search, recommendation engines, and knowledge graphs. With GraphRAG, you’re not only retrieving data but also understanding how each piece relates to another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w5QNbgG-TkB3JpaW.png" /></figure><p><em>The GraphRAG Pipeline (Image Source:</em> <a href="https://arxiv.org/pdf/2404.16130"><em>GraphRAG Paper</em></a><em>)</em></p><h3>Building Blocks of GraphRAG</h3><p>To understand how GraphRAG works, let’s break down its core components:</p><ol><li><strong>Graph Neural Networks (GNNs):</strong> GNNs enable understanding by considering each node and the information contained in its neighboring nodes. This way, relationships within the graph can be leveraged to infer deeper insights.</li><li><strong>Retrieval-Augmented Generation (RAG):</strong> RAG models combine retrieval (fetching relevant data) with generation (producing human-like responses). Typically, this involves embedding documents into vectors, storing them in a database, and retrieving relevant information for generation.</li></ol><p>Learn more about RAG here:</p><p><a href="https://medium.com/generative-ai/rag-app-development-from-start-to-finish-2fb9c70ae8e1">RAG App Development: From Start to Finish</a></p><p><a href="https://medium.com/generative-ai/building-local-rag-from-start-to-finish-a61aa5d3a653">Building Local RAG:From Start to Finish</a></p><p><strong>3. The Fusion:</strong> In GraphRAG, graph embeddings represent entities and their relationships. This means the retrieval process is both aware of the data’s content and its structure, which is incredibly powerful.</p><h4><strong>Varieties of GraphRAG</strong></h4><p>GraphRAG enhances the traditional Retrieval-Augmented Generation (RAG) approach by introducing a graph database as a source for context-rich information. By doing so, it empowers language models (LLMs) with structured insights beyond simple text chunks, addressing key issues like lack of context, factual inaccuracies, and language ambiguity. Instead of merely passing textual chunks of documents to the LLM, GraphRAG incorporates structured entity information, including descriptive properties and relationships, fostering a deeper and more accurate understanding of complex subjects. Each record in the vector database can now hold contextually enhanced representations, helping the LLM make sense of specialized domains more effectively. By blending GraphRAG with the traditional RAG approach, we get the best of both worlds — the structured, factually grounded context from graphs combined with the extensive textual richness of large documents.</p><p>GraphRAG can be adapted in various ways, each depending on the nature of questions, the specific domain, and the knowledge graph content:</p><ol><li><strong>Graph as a Content Store</strong><br>In this approach, relevant text chunks are extracted from documents within a knowledge graph and passed to the LLM to answer questions. This requires a knowledge graph rich in textual content and metadata and a connection to a vector database to store and retrieve these chunks effectively.</li><li><strong>Graph as a Subject Matter Expert</strong><br>Here, the knowledge graph provides descriptions of key concepts and entities related to the question, enhancing the LLM’s contextual understanding with “semantic context.” These descriptions include relationships between relevant concepts, making this variety ideal for knowledge graphs built with a comprehensive conceptual model, including ontologies, taxonomies, or detailed entity descriptions. Implementing this approach requires entity linking or another mechanism to identify the relevant concepts.</li><li><strong>Graph as a Database</strong><br>In this variation, GraphRAG uses a graph query to directly retrieve factual information from the knowledge graph in response to natural language questions. This method relies on a knowledge graph rich in factual data and typically involves a natural language-to-graph query tool and entity linking. The retrieved information is then summarized by the LLM for a coherent, contextually accurate response.</li></ol><p>Each of these approaches tailors GraphRAG to specific applications, empowering LLMs with the structured, interconnected knowledge stored in knowledge graphs to enhance accuracy, context, and relevance.</p><h3>Implementing GraphRAG — Setting Up the Tools</h3><p>Now that we have a solid understanding of what GraphRAG offers, let’s dive into the implementation.</p><h4>Step 1: Preparing the Environment</h4><p>Let’s set up the tools you’ll need for this project:</p><ol><li><strong>Python and Essential Libraries</strong>: Make sure you have Python installed. We’ll use PyTorch for GNN models, Hugging Face Transformers for RAG models, and additional libraries for handling graph structures and embeddings.</li><li><strong>Graph Database</strong>: Use Neo4j or NetworkX to create and manage your graph structure.</li></ol><h4>Step 2: Importing Libraries and Preparing Data</h4><p>Let’s start by importing the necessary libraries:</p><pre>import torch<br>import networkx as nx<br>from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from torch_geometric.nn import GCNConv  # for building GNNs</pre><p>For this example, we’ll create a simple graph with documents as nodes and connections based on topic similarity.</p><h4>Step 3: Building a Graph from Documents</h4><p>Suppose you have a dataset with documents on various topics. Each document is a node, and edges represent thematic similarity:</p><pre>documents = [<br>    &quot;Artificial intelligence is transforming industries.&quot;,<br>    &quot;Machine learning improves AI by refining algorithms.&quot;,<br>    &quot;Graph theory plays a key role in network analysis.&quot;,<br>    &quot;Neural networks are a subset of AI technologies.&quot;<br>]<br>vectorizer = TfidfVectorizer()<br>doc_embeddings = vectorizer.fit_transform(documents).toarray()<br>graph = nx.Graph()<br># Adding nodes and edges based on similarity<br>for i, doc in enumerate(doc_embeddings):<br>    graph.add_node(i, text=documents[i], embedding=doc)<br>    for j, other_doc in enumerate(doc_embeddings):<br>        if i != j:<br>            similarity = cosine_similarity([doc], [other_doc])[0][0]<br>            if similarity &gt; 0.5:  # threshold for similarity<br>                graph.add_edge(i, j, weight=similarity)</pre><h4>Step 4: Building the Graph Neural Network (GNN)</h4><p>Now, let’s create a GNN that will understand relationships in the graph. We’ll use a Graph Convolutional Network (GCN) as a foundational structure.</p><pre>class GraphRAGModel(torch.nn.Module):<br>    def __init__(self):<br>        super(GraphRAGModel, self).__init__()<br>        self.conv1 = GCNConv(in_channels=doc_embeddings.shape[1], out_channels=64)<br>        self.conv2 = GCNConv(64, 32)<br>        self.conv3 = GCNConv(32, 16)<br>def forward(self, x, edge_index):<br>        x = self.conv1(x, edge_index).relu()<br>        x = self.conv2(x, edge_index).relu()<br>        x = self.conv3(x, edge_index)<br>        return x<br># Convert NetworkX graph to PyTorch geometric data format<br>import torch_geometric<br>data = torch_geometric.utils.from_networkx(graph)<br>model = GraphRAGModel()<br># Passing document embeddings through the GNN<br>gnn_embeddings = model(data.x, data.edge_index)</pre><h4>Step 5: Integrating RAG Model with GNN</h4><p>Now that we have the graph-enhanced embeddings, we’ll use them in a RAG model to retrieve and generate responses based on context.</p><pre>tokenizer = RagTokenizer.from_pretrained(&quot;facebook/rag-token-nq&quot;)<br>retriever = RagRetriever.from_pretrained(&quot;facebook/rag-token-nq&quot;, index_name=&quot;custom&quot;)<br>model = RagTokenForGeneration.from_pretrained(&quot;facebook/rag-token-nq&quot;, retriever=retriever)<br><br># Use GNN-enhanced embeddings for document retrieval<br>retriever.index.index_embeddings = gnn_embeddings.detach().numpy()<br># Sample query<br>query = &quot;How does AI impact neural network research?&quot;<br>input_ids = tokenizer(query, return_tensors=&quot;pt&quot;).input_ids<br>generated = model.generate(input_ids=input_ids)<br>print(tokenizer.decode(generated[0], skip_special_tokens=True))</pre><h3>Testing and Tuning the Model</h3><p>With GraphRAG set up, it’s time to test it on queries to assess its effectiveness. Notice how it doesn’t just retrieve random documents; it returns information with contextual relevance, thanks to the graph structure and GNN-powered embeddings.</p><h3>Future Directions and Applications</h3><p>Imagine the versatility of GraphRAG in fields like research assistance, complex question answering, or in building recommendation systems. By combining relational understanding from graphs with powerful language generation, GraphRAG opens up a future where machines can answer questions in a truly human-like, contextually aware manner.</p><h3>Final Thoughts on the GraphRAG Journey</h3><p>GraphRAG represents the next evolution in RAG, providing a robust framework that not only retrieves data but also understands the connections within it. Its implementation combines cutting-edge GNNs with the retrieval strength of RAG models, offering a promising direction for applications that rely on deeply interconnected data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vlPsZUbb5VojxQIS.png" /></figure><p><strong><em>⭐️ Follow me on </em></strong><a href="https://www.linkedin.com/in/a-shreya-sri-a83a20205/"><strong><em>LinkedIn</em></strong></a><strong><em> for more such interesting articles⭐️</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FHhEEqN2Nv4Bk5O9.png" /></figure><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*GJvFKDlsvlsZ7NqI.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*cRcUA-Iix6fGjg9w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd324d0a2ddd" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/graphrag-explained-combining-graphs-and-generative-ai-for-better-retrieval-fd324d0a2ddd">GraphRAG Explained: Combining Graphs and Generative AI for Better Retrieval</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Strategic Chain-of-Thought: Elevating AI Reasoning with Smarter Prompts]]></title>
            <link>https://generativeai.pub/strategic-chain-of-thought-elevating-ai-reasoning-with-smarter-prompts-762c8c4c8215?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/762c8c4c8215</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 00:55:10 GMT</pubDate>
            <atom:updated>2024-09-23T00:55:10.955Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tp2eAyXeKr0qe68p" /><figcaption>Photo by <a href="https://unsplash.com/@mikezo?utm_source=medium&amp;utm_medium=referral">Mike Alonzo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine you are a skilled mapmaker, tasked with charting a vast and complex landscape — one filled with winding paths, hidden treasures, and potential hazards. You have a team of explorers, all eager to set out and navigate these unknown terrains. But here’s the catch: they need a solid plan, a clear strategy, before they can even begin. After all, without a thoughtful approach, they might get lost or overlook important discoveries. This is where <strong>Strategic Chain-of-Thought (SCoT)</strong> becomes your secret weapon.</p><p>Much like the explorers needing a plan, AI agents today need more than just brute force processing power — they need strategic thinking. And at the heart of this process is <strong>prompt engineering</strong>. Picture prompt engineering as the bridge between the explorers (the AI agents) and the map of the terrain (the Language Model or LLM). It’s through these carefully designed prompts that the agents know where to start, how to think, and what steps to take to reach their destination.</p><p>In our story, traditional prompt engineering methods were like giving the explorers a map without instructions. They could follow the paths, but often they’d wander, get lost, or find themselves at dead ends. But then came the discovery of <strong>Chain-of-Thought (CoT) prompting</strong> — a powerful new method where explorers were encouraged to think out loud, step by step, as they ventured through the terrain. They didn’t just follow the map blindly anymore; they carefully considered each choice, reasoned through challenges, and ended up making much better decisions along the way.</p><h3>A Strategic Evolution: SCoT</h3><p>As more and more explorers benefited from CoT, something interesting happened. Some of the more experienced ones realized that while thinking step-by-step helped, they needed something more. What if, before they even set out, they had a clear, strategic game plan? What if they could use the knowledge from previous journeys to guide their exploration?</p><p>This is where <strong>Strategic Chain-of-Thought (SCoT)</strong> emerged, an evolution of the Chain-of-Thought method. Instead of simply thinking aloud as they went, the explorers were now equipped with a carefully crafted strategy, a master plan drawn from a <strong>predefined strategic knowledge base</strong> — essentially, a collection of insights from past expeditions.</p><p>Think of this as giving the explorers two things: first, a mental blueprint of how they should approach their task, and second, the ability to think through the journey in stages. <strong>SCoT’s two-stage approach</strong> was born.</p><p>In the first stage, the mapmaker (that’s you) doesn’t just hand over a map. Instead, you work with the explorers to establish a strategy — how they will tackle the journey, what kinds of problems they should anticipate, and what steps they should follow when they encounter forks in the road. This strategy is built using your wealth of knowledge from past expeditions. It’s like saying, “We’ve crossed this forest before, and we know the best way through. Trust the strategy.”</p><p>Once the strategy is clear, the explorers enter the second stage: the actual journey itself. With the strategic plan in hand, they now begin their trek, following the <strong>Chain-of-Thought paths</strong> that guide them step by step, informed by the blueprint they created earlier. This way, they don’t just make good decisions — they make the <em>right</em> decisions based on strategic foresight. It’s no longer just about thinking; it’s about thinking strategically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tf2Kouw2rkh-kAVk.png" /><figcaption>Framework</figcaption></figure><h3>Strategic Chain-of-Thought (SCoT)</h3><p>As our explorers set out on more challenging journeys, the mapmaker knew there was still room for improvement in how they approached problem-solving. This led to the discovery of a new method, now called <strong>Strategic Chain-of-Thought (SCoT)</strong>.</p><p>SCoT wasn’t just another way to guide explorers through the terrain; it was a refined strategy designed to improve the quality of their paths. Much like how a seasoned traveler learns to adapt their route based on changing weather or terrain conditions, SCoT integrates <strong>strategic knowledge</strong> directly into the planning process.</p><p>Here’s how it works: imagine that instead of choosing a single path at the start, the mapmaker first explores <strong>multiple problem-solving strategies</strong>. Each strategy is tested and considered, and the most effective one is chosen as the guiding strategy. It’s like planning the journey by comparing different routes, factoring in past experiences, and deciding which one is most likely to lead the explorers to success.</p><p>Once the right strategy is selected, it doesn’t just stay on paper. This strategic plan <strong>directs the explorers</strong> throughout their entire journey, helping them navigate obstacles and stay focused on the goal. Thanks to this approach, the explorers generate not only higher-quality Chain-of-Thought paths but also more <strong>accurate final answers</strong>. The whole reasoning process becomes more precise and effective, leading them to their destination faster and with fewer mistakes.</p><h3>SCoT’s Efficiency: The Few-Shot Method</h3><p>Now, imagine a particularly difficult journey where the explorers aren’t just navigating based on strategy — they also need to learn from past journeys. This is where SCoT takes things to the next level with the <strong>few-shot method</strong>. Here, strategic knowledge is used to select the most <strong>relevant examples</strong> from past expeditions. These examples serve as demonstrations, showing the explorers how to tackle new challenges by applying lessons from previous journeys.</p><p>What’s remarkable is that SCoT does this <strong>without the need for multiple queries</strong> or external knowledge sources. Just like a mapmaker who has everything they need in their journal of past travels, SCoT eliminates the need to search for additional information. By doing so, it <strong>reduces the computational overhead</strong> — the explorers don’t need to stop and ask for directions at every fork in the road. This makes SCoT a more <strong>practical and resource-efficient</strong> solution, saving time, energy, and resources.</p><h3>The Origins of Chain-of-Thought (CoT)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SGCa-_K3UWr3cu9Z.png" /><figcaption>Components of CoT</figcaption></figure><p>Before we discovered SCoT, there was the foundation of <strong>Chain-of-Thought (CoT)</strong> prompting. As I explored in a previous journey (or article, if you will), CoT was a powerful technique that encouraged <strong>Large Language Models (LLMs)</strong> to think in steps, much like how we do when we solve problems in real life.</p><p>Picture an explorer breaking down a difficult route into smaller segments. Instead of rushing through the forest in one go, they focus on each segment, making sure each step is carefully considered. This method mirrors <strong>human problem-solving</strong>, where large tasks are broken down into more manageable pieces. The result? The explorers are able to move forward with greater precision and care, ensuring that every part of the journey is given the attention it deserves.</p><p>One of the fascinating things about CoT is that even when certain steps aren’t completely accurate, the overall journey can still be successful. This shows how important it is to have a well-structured plan — sometimes, even small mistakes can be corrected along the way if the bigger picture is clear.</p><h3>The Broader Concept: Chain-of-X</h3><p>Over time, the mapmaker realized that CoT was just one part of a much larger process, leading to the broader concept of <strong>“Chain-of-X.”</strong> Here, explorers weren’t just thinking step-by-step — they were learning how to tackle <strong>complex challenges</strong> by breaking them down into smaller, <strong>sub-problems</strong>.</p><p>In these complex journeys, the explorers used new tools. One of these was <strong>Bridging</strong>, where symbolic items like landmarks or markers helped them connect different parts of the map. <strong>Language Templates</strong> were another important tool, offering textual hints or clues that guided their reasoning. Together, these elements helped the explorers stay <strong>coherent and relevant</strong>, ensuring that their problem-solving remained focused and on track.</p><h3>Looking to the Future</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F9RC07OoaAV2kxIf.png" /></figure><p>As explorers venture into even more challenging and uncharted territories, CoT will remain an essential tool in their toolkit. Its <strong>simplicity and transparency</strong> are key to effective problem-solving. However, as journeys grow longer and the landscapes become more intricate, <strong>managing the complexity</strong> of these explorations will require more advanced tools — SCoT being one of the most promising.</p><p>With <strong>multi-inference architectures</strong> (or multi-layered maps) and increasingly sophisticated AI agents, prompt engineering will need to evolve. The mapmaker will need to focus on <strong>data-centric approaches</strong>, where careful oversight ensures that each expedition maintains the highest standards of integrity. After all, even the best map is useless if the explorers can’t follow it properly.</p><p>In this new era of AI-driven exploration, <strong>human oversight</strong> will play a vital role. The mapmaker’s journal might be filled with strategic knowledge, but it’s the wisdom and careful guidance of the mapmaker that will ultimately ensure every journey is a success.</p><h3>Accommodating Complexity</h3><p>As the mapmaker’s expeditions grew in scope, the landscapes themselves became more complex — larger terrains, steeper mountains, and denser forests. With each new challenge, it became clear that handling these vast and intricate routes required more than just a good map. The explorers needed a more <strong>resilient platform</strong> to accommodate the growing functionality of their missions.</p><p>Imagine that the simple tools that once worked for easy day trips were no longer enough. The mapmaker now had to manage not just the path ahead, but also the logistics of how the explorers would communicate, process information, and adjust their plans mid-journey. This meant building an <strong>infrastructure</strong> capable of handling multiple variables at once — like the flow of data, unexpected detours, and the need for <strong>multi-step reasoning</strong>.</p><p>However, as these journeys scaled in size and complexity, the mapmaker realized that merely managing tasks manually wouldn’t be enough. This is where the idea of creating <strong>agentic explorers</strong> came into play. These agents weren’t just explorers following a set path — they were capable of breaking down <strong>complex requests</strong> into smaller, manageable tasks, optimizing how the journey was conducted. The agents could autonomously adjust to changing conditions and re-route themselves if needed, improving both <strong>performance and scalability</strong>.</p><p>But where would this complexity be hosted? It became clear that the explorers needed an <strong>adaptable system</strong> that could support real-time adjustments and seamless integration with the broader landscape they were navigating. This system would allow them to interact with data in real-time, adjusting their path as new information came in. In much the same way, modern AI applications need resilient platforms that support real-time processing, integrating with <strong>AI ecosystems</strong> to handle increasingly complex demands.</p><h3>SCoT &amp; Strategic Knowledge</h3><p>Behind every successful journey, there was one thing that made all the difference: <strong>strategic knowledge</strong>. This wasn’t just any information; it was the kind of knowledge that provided clear guidance — a set of principles that helped the explorers navigate toward the right solutions and avoid pitfalls.</p><p>Strategic knowledge was the <strong>mapmaker’s wisdom</strong>, gained over years of charting unknown territories. It provided a <strong>structured process</strong> for the explorers to follow, making sure they didn’t just stumble through their journey but proceeded with purpose. By using strategic knowledge, the explorers could consistently reach their goals with <strong>stability</strong> and precision.</p><p>Think of it as a <strong>correct and comprehensive approach</strong> to problem-solving. Instead of relying on guesswork or trial and error, strategic knowledge offered a <strong>systematic method</strong> that guided the explorers through each step, ensuring that they arrived at the correct destination. The steps were carefully crafted to be simple enough to follow but detailed enough to ensure that no important detail was overlooked. This way, they avoided ambiguity and kept their course steady.</p><h3>Eliciting and Applying Strategic Knowledge: The Two-Step Process</h3><p>Rather than immediately charging ahead, the mapmaker’s explorers would first pause to <strong>elicit strategic knowledge</strong>. This was the critical first step. Much like deciding on the best route through a challenging landscape, the explorers needed to identify the most <strong>efficient method</strong> for solving the problem. Once they had this method in hand, it became their guiding strategy, a <strong>compass</strong> to follow throughout the journey.</p><p>Next, the explorers would apply this <strong>strategic knowledge</strong> to their specific challenge. In doing so, they didn’t just rush toward the final answer; they followed a clear path that ensured their steps were deliberate and well-thought-out. This two-step process — eliciting strategic knowledge and then applying it — helped the explorers arrive at the correct solution with confidence.</p><h3>SCoT’s Single-Query Efficiency</h3><p>Unlike traditional methods where explorers had to continually stop, check their bearings, and consult additional information, <strong>SCoT</strong> allowed them to proceed efficiently with a <strong>single-query method</strong>. This meant they didn’t need to keep asking for new directions or pulling in outside knowledge sources as they went along. Everything they needed was contained within the initial strategy.</p><p>In this sense, SCoT became a highly efficient solution for explorers, eliminating the need for <strong>multiple queries</strong> and making the journey far smoother and more resource-efficient. However, as the explorers continued to scale their expeditions, the mapmaker realized that there might still be moments when <strong>dynamic demonstrations</strong> — real-time examples from new sources of knowledge — would be needed to guide them through uncharted territory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5gvnXEOORu6Bs-dF.png" /><figcaption>Comparisions</figcaption></figure><h3>The Few-Shot Method</h3><p>To address this, the mapmaker refined the SCoT process by introducing a <strong>few-shot method</strong>. In this approach, the explorers didn’t just rely on one predefined strategy — they also learned from <strong>previous journeys</strong>. By selecting the most relevant demonstrations from past expeditions, the explorers could use these examples to inform their current path.</p><p>This approach worked in two stages:</p><ol><li><strong>Creating a strategy-based demonstration corpus</strong> — a collection of the most successful past journeys, offering insights into how similar challenges were overcome.</li><li><strong>Performing model inference</strong> — where the explorers applied these demonstrations to inform their decisions on the current journey.</li></ol><p>Thanks to this refined method, the explorers were better prepared for the challenges ahead. They didn’t just rely on a single strategy, but combined strategic knowledge with relevant examples from the past, leading to a more <strong>comprehensive and adaptive solution</strong>.</p><h3>Example 1: The Complex Math Problem</h3><p><strong>Regular Prompting: Step-by-Step Linear Approach</strong></p><p>Let’s consider the explorers facing a complex math problem — say, solving an equation that involves multiple variables and nested functions. In a <strong>regular prompting</strong> scenario, the Large Language Model (LLM) is simply instructed to solve the problem using a <strong>step-by-step approach</strong>. This is a basic Chain-of-Thought (CoT) technique where the model proceeds linearly, solving one part of the problem before moving on to the next.</p><p>For example, the regular prompt might look like this:</p><p><strong>Prompt:</strong><br><em>“Solve the equation 2(x + 3)² — 5x = 17. Show the steps.”</em></p><p>The LLM would break the problem down into smaller steps:</p><ol><li>Expand the squared term.</li><li>Simplify the equation.</li><li>Isolate the variable x.</li></ol><p>The model completes the task, but without any advanced strategy. It follows the steps rigidly, and if there’s an ambiguity or tricky calculation, the process might break down or produce errors, as it lacks deeper reasoning or strategic foresight.</p><p><strong>SCoT: Strategic Math Solver</strong></p><p>Now, let’s see how <strong>Strategic Chain-of-Thought (SCoT)</strong> would handle the same problem. Instead of jumping straight into solving the equation, the SCoT approach would first identify the <strong>best problem-solving strategy</strong> by drawing from a <strong>strategic knowledge base</strong>.</p><p>For instance, the SCoT prompt might look like this:</p><p><strong>Prompt:</strong><br><em>“Using strategic knowledge about quadratic equations, solve 2(x + 3)² — 5x = 17. Apply the most efficient method for expanding and solving the equation, and explain the rationale behind each step.”</em></p><p>In this case, before even expanding the equation, the model would analyze the structure of the problem. The strategic knowledge might suggest factoring or completing the square as alternative methods based on the complexity of the equation, optimizing the process. This avoids unnecessary steps or dead-ends that might occur in a regular approach.</p><p>SCoT doesn’t just follow rules — it <strong>strategizes</strong> based on the nature of the task, ensuring a more <strong>efficient and accurate solution</strong>. If any intermediate calculation gets complex, the model can fall back on its predefined strategies to adjust course and avoid errors. This reduces the risk of incorrect results while maintaining <strong>clarity and precision</strong> in reasoning.</p><h3>Example 2: Coding a Sorting Algorithm</h3><p><strong>Regular Prompting: Direct Implementation</strong></p><p>Let’s take a more technical example where an LLM is tasked with implementing a <strong>sorting algorithm</strong> in Python. In <strong>regular prompting</strong>, the model would simply be asked to write the code for a quicksort or mergesort algorithm.</p><p><strong>Prompt:</strong><br><em>“Write Python code for a quicksort algorithm.”</em></p><p>The LLM will likely generate functional code, directly laying out the steps of the algorithm as it has been trained to. This works fine for basic tasks, but it lacks a deeper understanding of which sorting algorithm is best suited for specific types of data. If the prompt changes or the dataset has particular features (e.g., already partially sorted or with repeated values), the regular prompt would not adapt to this nuance.</p><p><strong>SCoT: Strategic Algorithm Selection</strong></p><p>With <strong>SCoT</strong>, the LLM doesn’t simply implement quicksort. It first analyzes the task strategically by drawing on a <strong>knowledge base</strong> that contains information about different sorting methods and their efficiencies for specific types of data.</p><p><strong>Prompt:</strong><br><em>“Using strategic knowledge about sorting algorithms, write Python code for sorting a list of integers. First, identify the most efficient sorting method based on the characteristics of the data, and explain why this method was chosen before implementing it.”</em></p><p>In this scenario, before jumping to code, the model would first decide whether quicksort is the best option. If the list is small, <strong>insertion sort</strong> might be more efficient; if the data is nearly sorted, <strong>bubble sort</strong> could suffice. The model selects a strategy that’s <strong>contextually appropriate</strong> for the problem at hand.</p><p>SCoT’s strategic approach saves computational resources by picking the most efficient path forward, generating <strong>high-quality results</strong> while <strong>minimizing unnecessary calculations</strong>. This is especially important in real-world applications, where not every task is best solved with a one-size-fits-all approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AIwT6MmqYH1429P-.png" /></figure><p><strong><em>⭐️ Please follow me on </em></strong><a href="https://www.linkedin.com/in/a-shreya-sri-a83a20205/"><strong><em>LinkedIn</em></strong></a><strong><em> for more such interesting articles⭐️</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AIwT6MmqYH1429P-.png" /></figure><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Ukj4mTJx7ArCzKKx.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*-Kgm_UlAwzTOg_4V.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=762c8c4c8215" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/strategic-chain-of-thought-elevating-ai-reasoning-with-smarter-prompts-762c8c4c8215">Strategic Chain-of-Thought: Elevating AI Reasoning with Smarter Prompts</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making AI Portable: Compressing Large Language Models]]></title>
            <link>https://generativeai.pub/making-ai-portable-compressing-large-language-models-92f4d464e750?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/92f4d464e750</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 13:17:37 GMT</pubDate>
            <atom:updated>2024-09-05T13:17:37.173Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1FmeDZG2WRln4wqC" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Imagine a small startup buzzing with creativity. The team had a vision: to build an app that could translate languages in real-time, breaking down communication barriers across the globe. The solution seemed obvious — tap into one of the top-tier large language models (LLMs), known for their exceptional performance in tasks like translation, text generation, and more.</p><p>Excited by the possibilities, the team dove into the project. But soon, they encountered an unexpected hurdle. The LLMs, as powerful as they were, came with a significant drawback: they were massive. Running these models demanded not just gigabytes, but dozens or even hundreds of gigabytes of GPU memory. For a startup with limited resources, this was a serious problem. The models were expensive to run, difficult to set up, and nearly impossible to deploy on smaller devices or edge servers without access to robust cloud infrastructure.</p><p>The dream of making the app accessible to everyone, regardless of their device, started to feel like trying to squeeze a giant into a small box. How could they make this work without compromising on the app’s performance?</p><p>The team began exploring solutions and discovered a path forward — <strong>compression techniques</strong>. These innovative methods promised to shrink the giant LLMs down to a manageable size, allowing them to fit into smaller devices and run more efficiently. It felt like finding a secret formula that could bring their dream back to life.</p><p>As they dug deeper, the team learned about various compression techniques. <strong>Pruning</strong>, for instance, involved trimming away parts of the model that were less essential, much like a gardener trimming branches to help a tree grow stronger. <strong>Quantization</strong> was another technique, reducing the precision of the model’s computations to save space and speed things up without sacrificing too much accuracy. And then there was <strong>knowledge distillation</strong>, where a smaller model learned from the larger one, capturing its knowledge in a more compact form.</p><p>Each technique came with its own set of trade-offs. While these methods made the models lighter and faster, they also required careful balancing to ensure that the model’s performance didn’t suffer. But the promise of deploying the app on a wide range of devices, from high-end servers to everyday smartphones, made the challenge worth taking on.</p><p>After implementing these compression techniques, the team found that their once-unwieldy LLM had transformed into a streamlined version of itself. The app was now faster, more efficient, and could run on devices with limited resources. Most importantly, the vision of breaking down language barriers became a reality, reaching people around the world.</p><p>This journey through LLM compression taught the team a valuable lesson: that even the most powerful tools can be refined and optimized, making them accessible to more people. It also highlighted the importance of innovation in solving real-world problems, turning obstacles into opportunities.</p><p>In this article, we’ll explore the same journey they took — delving into the world of LLM compression techniques. We’ll explain how they work, the trade-offs involved, and how they are transforming the future of AI applications.</p><h3>LLM pruning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*1rwDXd_tLlizglSAt2WFgg.png" /></figure><p>After discovering the world of compression techniques, the team was eager to explore more ways to slim down their large language model (LLM). They had already seen how effective these methods could be, but the journey was far from over. The next challenge was to understand the intricate components of their model and figure out which parts were essential — and which could be trimmed without sacrificing too much performance.</p><p>Like many other deep neural networks, LLMs are composed of countless components working together. However, not all of these components contribute equally to the model’s output. Some, as the team soon realized, had little to no effect at all. This was good news because it meant there was potential to make the model even more compact. The question was: how could they do this effectively?</p><p>The answer lay in <strong>pruning</strong>. The idea behind pruning is simple: remove the non-essential parts of the model to make it leaner while keeping its core functionality intact. But as the team learned, there were different ways to approach pruning, each with its own set of benefits and challenges.</p><p><strong>Unstructured pruning</strong> was the first method they encountered. It was a straightforward approach — remove irrelevant parameters, regardless of the model’s structure. In this technique, parameters that fell below a certain threshold were set to zero, effectively eliminating their impact. The result was a sparse model, where zero and non-zero weights were scattered randomly. It was like trimming off random branches from a tree without worrying about the overall shape.</p><p>While unstructured pruning was easy to implement, it wasn’t without its challenges. The random distribution of weights made it tough to optimize the model for hardware, requiring extra computation and processing steps. Additionally, after pruning, the model often needed further retraining to regain its original performance. Despite these hurdles, the team found hope in recent advancements. For example, <strong>SparseGPT</strong> emerged as a powerful technique. Developed by researchers at the Institute of Science and Technology Austria (ISTA), SparseGPT performed one-shot pruning on large transformer models like BLOOM and OPT. The best part? It eliminated the need for retraining altogether.</p><p>Another technique that caught the team’s attention was <strong>LoRAPrune</strong>. This method combined <strong>low-rank adaptation (LoRA)</strong> with pruning, making it a highly efficient way to improve LLM performance on specific tasks. LoRA was already known as a parameter-efficient fine-tuning technique, only updating a small subset of the model’s parameters. By integrating this with pruning, LoRAPrune offered an elegant solution for making the model both compact and effective.</p><p>But the story didn’t end there. The team soon realized that <strong>structured pruning</strong> was another powerful tool in their arsenal. Unlike unstructured pruning, which focused on removing individual parameters, structured pruning targeted larger components — entire neurons, channels, or layers. The advantage was clear: by removing entire parts of the model, the compression process became simpler, and hardware efficiency improved. For example, eliminating an entire layer reduced the model’s computational complexity without creating irregularities in the structure.</p><p>Yet, structured pruning came with its own risks. Removing entire neurons or layers could potentially strip the model of important learned features. It required a deep understanding of the model’s architecture and how each part contributed to overall performance. One misstep, and the model’s accuracy could suffer.</p><p>Among the techniques that stood out was <strong>LLM-Pruner</strong>. This task-agnostic method took a smart approach to structured pruning. Instead of relying heavily on the original training data, LLM-Pruner used gradient information to selectively remove non-critical coupled structures. It was like having a guide who knew exactly which branches of the tree to cut, ensuring that the tree remained strong and functional.</p><p>With these new tools in hand, the team continued their journey. They found that by carefully pruning their model — whether through unstructured methods like SparseGPT or structured approaches like LLM-Pruner — they could achieve the impossible. Their once-giant LLM was now lean, efficient, and ready to power the app they had always dreamed of building.</p><p>The story of pruning taught them a valuable lesson: sometimes, less is more. By understanding which parts of the model were essential and which could be removed, they were able to create a version of the LLM that was not only powerful but also accessible to a wider range of devices and users.</p><h3>LLM Knowledge Distillation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*mpK7HVWg5aHheFKwMNepIQ.png" /></figure><p>After pruning the large language model (LLM) down to a manageable size, the team felt a sense of accomplishment. But they knew the journey wasn’t over yet. While pruning had helped them shed some of the model’s excess weight, they still needed to ensure it remained just as capable as the original, more complex version. That’s when they stumbled upon another powerful tool in the world of LLM compression — <strong>knowledge distillation</strong>.</p><p>Knowledge distillation sounded like something out of a science fiction novel. The concept was simple yet intriguing: a smaller “student” model learns to emulate the behavior of a larger, more complex “teacher” model. By transferring the knowledge from the teacher to the student, the result is a more compact yet highly capable model. It was like teaching an apprentice all the skills of a master craftsman, but in a way that allowed them to work more efficiently and with fewer resources.</p><p>In the realm of LLMs, the team discovered that knowledge distillation techniques came in two main flavors. The first was <strong>standard knowledge distillation</strong>. This method aimed to transfer the general knowledge from the teacher model to the student. For example, they could gather a series of prompts and responses from a powerful model like ChatGPT and use them to train a smaller, open-source LLM. However, the team quickly realized there were challenges with this approach. Capturing the underlying data distributions accurately was no easy task. Moreover, they had to navigate restrictions on training LLMs with data gathered from commercial models, which added another layer of complexity.</p><p>But just as they were beginning to feel the weight of these challenges, they found hope in <strong>MiniLLM</strong>. This technique, developed by researchers at Tsinghua University and Microsoft Research, took a different approach. MiniLLM employed specialized objective and optimization functions specifically designed for LLMs. By doing so, it enhanced the effectiveness of the distillation process, making it easier to train a smaller model that could perform nearly as well as the original.</p><p>While standard knowledge distillation focused on transferring general knowledge, the team also explored another fascinating technique called <strong>emergent ability distillation</strong>. This method was different — it targeted specific abilities that the larger teacher model had learned and transferred them to the smaller student model. Emergent abilities were like hidden treasures within the larger models, capabilities that didn’t exist in smaller ones. For example, the team could gather prompts and responses related to mathematics or reasoning problems from a powerful model like GPT-4 and try to transfer those abilities to a smaller model like Vicuna.</p><p>The beauty of emergent ability distillation was that it was much easier to measure. Because it focused on a narrow set of tasks, the team could clearly see whether the student model had successfully inherited the desired abilities. However, they also knew that there were limits. While emergent ability distillation allowed them to mimic certain behaviors of larger models, there were still some abilities that simply couldn’t be replicated in smaller models.</p><p>As the team continued their journey through the world of LLM compression, they realized that knowledge distillation was a powerful tool in their arsenal. It allowed them to create models that were not only compact but also retained much of the power and functionality of their larger counterparts. By combining the lessons they had learned from pruning with the capabilities of knowledge distillation, they were able to build an LLM that was efficient, effective, and ready for the real world.</p><h3>LLM Quantization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*ZNEyBkoYE4MweJEEByZSfQ.png" /></figure><p>Having explored pruning and knowledge distillation, the team’s journey to compress their large language model (LLM) continued. While they had made significant progress, there was still more work to be done. The model, though more compact, was still hefty. They needed a way to make it even leaner, especially if they wanted it to run efficiently on everyday devices like laptops and desktop computers.</p><p>That’s when they discovered <strong>quantization</strong> — a compression technique that promised to reduce the model’s size even further. Unlike pruning, which removed parts of the model, and distillation, which transferred knowledge to a smaller version, quantization focused on the very essence of how the model stored its data.</p><p>Large models like GPT-3 typically store their parameters as floating-point values. Even at half-precision, each parameter occupied two bytes, which added up quickly when dealing with billions of parameters. For a model as large as GPT-3, this meant hundreds of gigabytes of memory were required. The team realized that this was a significant reason why their model was still so massive.</p><p>Quantization, however, offered a clever solution. It involved converting these floating-point values into smaller integers — sometimes even single-byte values. This drastic reduction in size allowed open-source LLMs to run on devices that most people used every day, from laptops to desktops. The team was inspired by examples like <strong>GPT4All</strong> and <strong>Llama.cpp</strong>, which had successfully leveraged quantization to make powerful models more accessible.</p><p>As they dug deeper, they discovered that quantization wasn’t just a one-size-fits-all solution. There were different approaches to applying it, each with its own advantages and challenges.</p><p>The first approach they encountered was <strong>quantization-aware training (QAT)</strong>. In this method, quantization was integrated into the model’s training process from the very beginning. The idea was to let the model learn low-precision representations as it was being trained, which helped mitigate the loss of precision that could occur when parameters were compressed. While QAT promised high-quality results, it came with a major downside — it required training the model from scratch. For the team, this was a daunting task. Training a large model from the ground up would be resource-intensive and costly, something they weren’t fully prepared for.</p><p>Fortunately, there was another option — <strong>quantization-aware fine-tuning (QAFT)</strong>. This technique offered a middle ground. Instead of starting from scratch, they could take a pre-trained, high-precision model and fine-tune it to maintain its quality with lower-precision weights. Techniques like <strong>QLoRA</strong> and <strong>parameter-efficient and quantization-aware adaptation (PEQA)</strong> were commonly used for QAFT. This approach allowed the team to adapt their model without sacrificing too much of its original performance. It was like fine-tuning a machine to run smoothly with fewer parts, ensuring efficiency without a complete overhaul.</p><p>But the simplest method of all was <strong>post-training quantization (PTQ)</strong>. In PTQ, the team could transform the model’s parameters into lower-precision data types after the model had already been trained. This method didn’t require any retraining, making it incredibly efficient and straightforward. The team liked the idea of simplifying the model without altering its architecture. However, they also knew that PTQ had its limitations. While it was easy to implement, it might not preserve the model’s accuracy as effectively as the other techniques.</p><p>Despite the trade-offs, quantization became a crucial part of the team’s compression journey. By carefully choosing the right approach, they were able to make their model even more compact, bringing them closer to their goal of creating an accessible and efficient app. Whether through the sophisticated techniques of QAT and QAFT or the simplicity of PTQ, quantization allowed the team to reduce the size of their LLM without losing sight of the bigger picture — delivering a powerful, real-time translation app to people across the globe.</p><p>As they implemented quantization and saw their model become even more streamlined, the team realized that their once-giant LLM was now truly ready for the world. The journey had been long and filled with challenges, but each step had brought them closer to their vision.</p><h3>Problems solved by compressed LLM’s</h3><p>With their LLM now pruned, distilled, and quantized, the team marveled at the transformation. What once seemed like an unwieldy giant had become a nimble, efficient tool ready to solve real-world problems. But as they prepared to deploy their compressed model, they paused to consider a crucial question: <strong>What problems could they now solve with their newly compressed LLM?</strong></p><p>The answer was clear — <strong>compressing LLMs opens up a world of possibilities</strong>. The team’s app was no longer confined to powerful cloud servers or high-end hardware. Instead, it could run on everyday devices like smartphones, tablets, and laptops. This was a game-changer. People could use their app on the go, translating languages in real-time, no matter where they were. Whether it was travelers navigating foreign cities or businesses communicating with global clients, the app had the potential to bridge language barriers seamlessly.</p><p>Moreover, <strong>compressed LLMs could power applications in environments with limited resources</strong>. Imagine using LLMs in rural areas, where internet connectivity and powerful hardware are scarce. Compressed models could bring advanced AI capabilities to places where they were previously unimaginable. From educational tools that work offline to medical applications that assist doctors in remote clinics, the possibilities were endless.</p><p>The team also recognized that <strong>compression made LLMs more energy-efficient</strong>. Running massive models consumed significant amounts of power, both in terms of electricity and the energy needed to cool the hardware. By compressing their LLM, they could reduce its carbon footprint, contributing to a more sustainable future. This was particularly important as AI became increasingly integrated into various industries, and the need for eco-friendly solutions grew.</p><h3>Downsides of compressing LLM’s</h3><p>But as excited as the team was about these benefits, they also understood that <strong>compressing LLMs came with trade-offs</strong>. One of the main disadvantages of compression was the <strong>potential loss of accuracy</strong>. As they pruned parameters, distilled knowledge, and reduced precision, they knew there was a risk that their model might not perform as well as the original. While their compressed model could handle everyday tasks, it might struggle with more complex or nuanced queries.</p><p>For example, while the app could translate common phrases with ease, it might falter when dealing with idiomatic expressions or specialized terminology. This was something the team had to keep in mind as they prepared to launch their app. They knew that user feedback would be crucial in identifying areas where the model needed improvement.</p><p>Another challenge was that <strong>compressed LLMs might not be as versatile</strong> as their larger counterparts. While their model was optimized for real-time translation, it might not be able to handle other tasks with the same level of proficiency. Larger models had the advantage of being generalists, capable of performing a wide range of tasks. Compressed models, on the other hand, often had to sacrifice some of that versatility to maintain efficiency.</p><h3><strong>Real-Time Applications Of Compressed LLM’s</strong></h3><p>As the team reflected on these trade-offs, they also thought about the <strong>real-time applications where compressed LLMs could shine</strong>. Their translation app was just one example. <strong>Voice assistants</strong> were another area where compressed models could be a game-changer. By reducing the size of LLMs, companies could deploy voice assistants on devices with limited processing power, making them more accessible to a wider audience.</p><p>Similarly, <strong>chatbots and customer service applications</strong> could benefit from compressed LLMs. These models could be embedded in web applications, responding to customer queries instantly without relying on powerful backend servers. This would reduce latency and provide a smoother user experience.</p><p><strong>Personalized recommendations</strong> were yet another area where compressed LLMs could excel. From recommending movies and books to tailoring fitness plans, these models could run locally on devices, offering personalized experiences without compromising privacy or requiring constant internet access.</p><p>As the team prepared to launch their app, they knew they were at the forefront of a new era in AI. <strong>Compressed LLMs had the potential to bring powerful language models to everyone</strong>, regardless of their device or location. And while there were challenges to overcome, the possibilities far outweighed the drawbacks. Their journey with the giant AI had taught them that, sometimes,</p><blockquote><strong>making something smaller didn’t mean making it less powerful — it meant making it more accessible to the world</strong>.</blockquote><h3>Python Code Walkthrough</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AkOebH8JhtfrEl7L" /><figcaption>Photo by <a href="https://unsplash.com/@altumcode?utm_source=medium&amp;utm_medium=referral">AltumCode</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Continuing the narrative and integrating Python code into the story, let’s dive into a concrete example that aligns with the team’s journey of compressing a large language model (LLM). This example will illustrate how they used a simple compression technique, quantization, to make their LLM more efficient while maintaining its functionality.</p><p>After weeks of work, the team was ready to put their compressed model to the test. They had pruned, distilled, and quantized the model, and now it was time to see how it performed in a real-world application. To do this, they decided to write a simple Python script that would demonstrate the power of their compressed LLM.</p><p>Their goal was to take a large pre-trained model and apply quantization, converting the model’s parameters from floating-point values to lower-precision integers. This would reduce the model’s memory footprint and allow it to run more efficiently on a typical laptop.</p><p>Here’s how they did it:</p><h4>Step 1: Loading the Pre-trained Model</h4><p>First, they loaded a pre-trained language model using the transformers library. For this example, they used a smaller model, such as distilbert-base-uncased, which is already compact but still served as a good candidate for further compression.</p><pre>from transformers import AutoModelForSequenceClassification, AutoTokenizer<br>import torch<br># Load the pre-trained model and tokenizer<br>model_name = &quot;distilbert-base-uncased&quot;<br>model = AutoModelForSequenceClassification.from_pretrained(model_name)<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br># Check the model size before quantization<br>print(f&quot;Model size before quantization: {model.num_parameters()} parameters&quot;)</pre><p>This code snippet loaded the DistilBERT model and its tokenizer, which the team would use to process text inputs. They also checked the model’s size before applying any compression techniques.</p><h4>Step 2: Applying Quantization</h4><p>Next, the team applied post-training quantization to the model. This technique reduced the precision of the model’s weights without retraining, making it faster and more memory-efficient.</p><pre># Quantize the model using torch.quantization<br>quantized_model = torch.quantization.quantize_dynamic(<br>    model, {torch.nn.Linear}, dtype=torch.qint8<br>)<br># Check the model size after quantization<br>print(f&quot;Model size after quantization: {quantized_model.num_parameters()} parameters&quot;)</pre><p>Here, they used PyTorch’s quantize_dynamic function, which automatically applied quantization to the linear layers of the model. By converting the model&#39;s weights to 8-bit integers, they significantly reduced its memory usage.</p><h4>Step 3: Running Inference with the Quantized Model</h4><p>With the quantized model ready, the team ran an inference to see if the model’s performance was still acceptable after compression.</p><pre># Define a simple text input<br>text = &quot;The quick brown fox jumps over the lazy dog.&quot;<br># Tokenize the input text<br>inputs = tokenizer(text, return_tensors=&quot;pt&quot;)<br># Perform inference with the quantized model<br>with torch.no_grad():<br>    outputs = quantized_model(**inputs)<br># Print the model&#39;s predictions<br>predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)<br>print(f&quot;Model predictions: {predictions}&quot;)</pre><p>In this part of the code, the team tokenized a sample input and fed it into the quantized model. They then used the model’s output logits to calculate the predictions, ensuring the model still produced accurate results despite the reduced precision.</p><h4>Step 4: Evaluating Performance</h4><p>Finally, the team compared the performance of the original and quantized models in terms of both speed and memory usage.</p><pre>import time<br># Measure inference time with the original model<br>start_time = time.time()<br>with torch.no_grad():<br>    _ = model(**inputs)<br>print(f&quot;Inference time with the original model: {time.time() - start_time:.4f} seconds&quot;)<br># Measure inference time with the quantized model<br>start_time = time.time()<br>with torch.no_grad():<br>    _ = quantized_model(**inputs)<br>print(f&quot;Inference time with the quantized model: {time.time() - start_time:.4f} seconds&quot;)</pre><p>By timing the inference process, they could see how much faster the quantized model was compared to the original. This step confirmed that the compression had succeeded in making the model more efficient without sacrificing too much accuracy.</p><h3>Conclusion: A Triumph of Efficiency</h3><p>As the team ran their final tests, they felt a sense of accomplishment. Their journey with the giant AI had led them to a powerful realization: <strong>by compressing LLMs, they could bring the benefits of AI to more people and more places</strong>. Whether it was improving accessibility, reducing costs, or speeding up real-time applications, their work had a tangible impact.</p><p>Their story was far from over, but they knew they were on the right path. By applying techniques like quantization in Python, they had taken a critical step towards making AI more inclusive and sustainable. And with each line of code, they moved closer to a future where <strong>AI wasn’t just for the few but for everyone.</strong></p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*L8sNQAZnJwuwalAR.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*VEg4bw7nusTxbPbn.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92f4d464e750" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/making-ai-portable-compressing-large-language-models-92f4d464e750">Making AI Portable: Compressing Large Language Models</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Local RAG:From Start to Finish]]></title>
            <link>https://generativeai.pub/building-local-rag-from-start-to-finish-a61aa5d3a653?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a61aa5d3a653</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[app-development]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 14:19:10 GMT</pubDate>
            <atom:updated>2024-08-18T14:19:10.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*MK4uNOt_8vLRYw0Vbxrqdw.png" /><figcaption>Image generated by canva.ai</figcaption></figure><p>Imagine you’re a researcher preparing for an important presentation. Your desk is cluttered with papers, books, and notes — each containing valuable insights you’ve gathered over months of study. As the deadline approaches, you find yourself wishing for an assistant who could sift through all this information and provide you with precise, well-formed answers to your queries.</p><p>But what if this assistant wasn’t human? What if it was a powerful AI, right on your desktop, capable of retrieving the most relevant information from your vast collection of documents and using it to generate insightful responses? This is the idea behind a Local Retrieval-Augmented Generation (RAG) application — a tool designed to enhance your research process by combining the intelligence of a chat model with the specificity of your own data.But before this, understand how to develop a RAG Application here:<a href="https://medium.com/generative-ai/rag-app-development-from-start-to-finish-2fb9c70ae8e1">RAG App Development: From Start to Finish</a></p><p>In this article, we’ll embark on a journey to build this assistant.</p><p><em>But before we start, let’s ensure we have a solid understanding of the key concepts that will guide us along the way.</em></p><h3>Prerequisites: Equipping Our AI Research Assistant</h3><h3>1.Chat Models: The Conversational Core of Your AI Assistant</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*HebmsNWpn1-1KWIt9gGScA.png" /><figcaption>converstional models</figcaption></figure><p>Imagine you’re at your desk, preparing for that big presentation. Your AI-powered assistant is right beside you, ready to help with any question you might have. But this assistant isn’t just any old chatbot — it’s a sophisticated chat model, designed to understand not just plain text but the full context of a conversation. It’s as if you’re talking to a well-informed colleague who knows exactly what you need and how to get it for you.</p><p><strong>Understanding the Role of Chat Models</strong></p><p>In our story, the chat model is the heart of your AI assistant. Unlike traditional language models, which take in plain text and return plain text, chat models are designed to handle sequences of messages. These messages can come from different roles in a conversation — like you, the user, giving an instruction, or the AI responding with an answer. This structure allows the chat model to keep track of the conversation flow, ensuring that its responses are not only accurate but also contextually appropriate.</p><p>For example, when you ask your assistant a question, it doesn’t just see it as a string of words; it understands who’s asking, what’s been asked before, and even the system instructions that guide how it should respond. This is what makes chat models so powerful — they can differentiate between a question, an instruction, or a response, tailoring their output accordingly.</p><p><strong>Flexibility of Chat Models</strong></p><p>But here’s where it gets really interesting. Even though these chat models are designed to work with sequences of messages, they’re flexible enough to handle simple text inputs as well. Imagine you’re in a hurry and you just type a quick question. The chat model can take that string of text, treat it as a message from you (a “HumanMessage”), and then process it just as it would in a full conversation.</p><p>This flexibility means that whether you’re engaging in a complex dialogue or just asking a quick question, the chat model is ready to assist. It’s like having an assistant who can switch gears effortlessly, whether you need a detailed report or a simple fact.</p><p><strong>Tuning the Chat Models</strong></p><p>Of course, not all assistants are created equal. Depending on your needs, you might want to adjust how your AI assistant responds. This is where the various parameters of chat models come into play. Imagine you’re fine-tuning your assistant for a specific task — like making sure it doesn’t take too long to answer, or ensuring it generates a response that’s concise and to the point.</p><p>Here are some of the key parameters you can adjust:</p><ul><li><strong>Model</strong>: Think of this as choosing the specific assistant you want to work with. Different models have different strengths, so you can pick the one that best suits your needs.</li><li><strong>Temperature</strong>: This controls how creative or focused the assistant is in its responses. A higher temperature means more creative answers, while a lower temperature keeps things precise.</li><li><strong>Timeout</strong>: Just like you wouldn’t want your assistant taking forever to answer a question, you can set a timeout to ensure you get a response within a reasonable time.</li><li><strong>Max Tokens</strong>: Sometimes, you want a detailed explanation, and other times, just a quick answer. This setting lets you limit how much the assistant says in response.</li><li><strong>Stop Sequences</strong>: You can set specific points where the assistant should stop responding — like telling it to stop once it’s given you the main idea.</li><li><strong>Max Retries</strong>: If the assistant doesn’t get it right the first time, you can allow it to try again, up to a certain number of times.</li><li><strong>API Key &amp; Base URL</strong>: These are the behind-the-scenes details that connect your assistant to the right resources and make sure it can do its job.</li></ul><p><strong>Integrating and Customizing Your Assistant</strong></p><p>One of the great things about chat models is their adaptability. While 🦜️🔗 LangChain doesn’t host these models directly, it integrates seamlessly with various third-party providers. This means you can choose the chat model that best fits your project, whether you’re using OpenAI’s models, Anthropic’s, or others. Each model might have its own unique features and parameters, so you can customize your assistant to fit your exact needs.</p><p>For example, some chat models are specially tuned for tasks like tool calling — imagine asking your assistant to not only provide an answer but also perform an action, like pulling up a file or running a calculation. These fine-tuned models are often better at such tasks, making them a smart choice for more complex use cases.</p><p><strong>The Future of Chat Models: Multimodality</strong></p><p>As our story progresses, the capabilities of your AI assistant continue to grow. Picture this: instead of just handling text, your assistant starts accepting images, audio, and even video as inputs. This is the future of chat models — multimodal systems that can process and respond to a variety of data types. Although still in their early stages, these models are expanding the ways in which you can interact with your AI assistant.</p><p>Imagine showing your assistant a graph and asking it to explain the trend, or playing a sound clip and asking for an analysis. These multimodal models are like having an assistant who not only listens and talks but also sees and hears, making them incredibly versatile.</p><p>In our Local RAG application, the chat model is more than just a responder — it’s the core of the assistant, capable of handling complex conversations and adapting to your specific needs. With the right chat model in place, you’re well on your way to building an AI assistant that feels like a true partner in your research journey.</p><h3>2.Chaining Runnables: Crafting a Seamless Workflow for Your AI Assistant</h3><p>As our story continues, imagine that your AI-powered assistant is not just one entity but a team of specialists, each performing a specific task to help you prepare for your big presentation. In the world of AI, these specialists are called “runnables” — small, focused components that, when linked together, form a powerful and efficient workflow. The process of linking these components is known as chaining runnables.</p><p>Just like in a well-coordinated team, each runnable in the chain has a role. One might prepare the input, another processes it, and yet another interprets the result. The magic happens when these runnables work in harmony, passing information seamlessly from one to the next, like a relay race where each runner hands off the baton to the next.</p><p><strong>The Basics of Chaining Runnables</strong></p><p>Imagine you’re asking your assistant to do something simple, like tell you a joke about bears. First, a runnable might format your request into a prompt that the chat model can understand. The chat model then generates a response, and another runnable might parse that response into a clear, concise output. This chain of events ensures that your request is handled smoothly, and you get a witty bear joke right when you need it.</p><p>In the code world, chaining runnables is as simple as using a pipe (|) operator or the more explicit .pipe() method. Each runnable takes the output of the previous one and uses it as input for the next, creating a flow that’s easy to follow and highly efficient.</p><p><strong>Creating a RunnableSequence</strong></p><p>Think of this sequence as your assistant’s task list for the day. When you ask for something, the assistant follows the list step by step. The beauty of chaining runnables is that the resulting sequence acts as a single runnable itself, capable of being invoked, streamed, or even chained further with other sequences.</p><p>For example, when you invoke this sequence with a topic like “bears,” the assistant doesn’t just tell you any joke — it tells you a bear joke, carefully crafted and delivered based on the sequence of tasks you’ve set up. And because this sequence can stream output as soon as it’s ready, you don’t have to wait long for that punchline.</p><p><strong>Using the Pipe Operator</strong></p><p>Let’s walk through a typical scenario. You ask your assistant for a joke about bears. The first runnable might use a prompt template to format your request, like this:</p><pre>prompt = ChatPromptTemplate.from_template(&quot;tell me a joke about {topic}&quot;)</pre><p>Next, the formatted prompt is passed to a chat model, which generates the joke:</p><pre>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)</pre><p>Finally, another runnable parses the chat model’s output into a simple string:</p><pre>output_parser = StrOutputParser()</pre><p>These steps are then linked together into a runnable sequence:</p><pre>chain = prompt | model | output_parser</pre><p>Now, when you invoke this sequence with a topic like “bears,” your assistant follows the chain and delivers a joke that’s ready to make you smile.</p><p><strong>The Flexibility of Chaining with Functions</strong></p><p>What’s even more exciting is that you can add your custom logic to these chains. For example, you could use a simple function to tweak the input before it moves on to the next runnable. This allows you to tailor the assistant’s behavior even more closely to your needs.</p><p>Here’s an example:</p><pre>composed_chain_with_lambda = (<br>    chain<br>    | (lambda input: {&quot;joke&quot;: input})<br>    | analysis_prompt<br>    | model<br>    | StrOutputParser()<br>)</pre><p>With this setup, your assistant first generates a joke, then customizes the input before asking if it’s funny, ensuring that each step is precisely calibrated for the best results.</p><p><strong>Using the .pipe() Method</strong></p><p>If you prefer a more explicit approach, you can also use the .pipe() method to chain runnables. This method offers the same functionality as the pipe operator but with a more structured syntax. Here’s how it looks:</p><pre>composed_chain_with_pipe = (<br>    RunnableParallel({&quot;joke&quot;: chain})<br>    .pipe(analysis_prompt)<br>    .pipe(model)<br>    .pipe(StrOutputParser())<br>)</pre><p>This method is particularly useful when you’re working with more complex workflows, as it keeps the sequence of operations clear and easy to manage.</p><p>As our story unfolds, you’ll see how chaining runnables transforms your AI assistant into a versatile, efficient helper. Whether you’re asking for a simple joke or diving into more complex tasks, this approach ensures that each request is handled with precision and care. Your assistant isn’t just a single tool; it’s a well-orchestrated team, ready to tackle any challenge you throw its way.</p><h3>3.Embedding Models: Giving Your AI Assistant the Power of Understanding</h3><p>Continuing our story, imagine that your AI assistant, equipped with its various specialists (runnables), now needs to develop a deeper understanding of the information it encounters. How does it recognize the meaning behind words, phrases, or even entire documents? This is where embedding models come into play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*krpGtGNO0RGZymmf.png" /><figcaption>Embedding Model (source:<a href="https://python.langchain.com/v0.2/docs/concepts/#embedding-models">https://python.langchain.com/v0.2/docs/concepts/#embedding-models</a>)</figcaption></figure><p><strong>What Are Embedding Models?</strong></p><p>Think of embedding models as the assistant’s way of “visualizing” text in a meaningful way. Just as you might use a map to navigate a city, embedding models create a kind of “map” of text by transforming words and sentences into vectors — arrays of numbers that represent the text’s underlying meaning. These vectors help the assistant understand how different pieces of text relate to one another, allowing it to perform tasks like searching for similar documents or providing relevant information in response to a query.</p><p>For instance, if you’re preparing for your GRE and ask your assistant to find information on a specific topic, it uses these vector representations to search through its database and retrieve the most relevant content. The assistant isn’t just matching keywords; it’s understanding the context and meaning behind your query, ensuring that the information it finds is exactly what you need.</p><p><strong>The Role of the Embeddings Class</strong></p><p>In the world of AI, especially within the 🦜️🔗LangChain framework, the Embeddings class serves as the gateway to these powerful embedding models. This class is designed to interface with a variety of text embedding models, whether they’re hosted by well-known providers like OpenAI or Cohere, or run locally on your own hardware.</p><p>The Embeddings class offers a standardized way to interact with these models, allowing your assistant to work seamlessly across different platforms and providers. Whether your assistant is working with text from OpenAI, Hugging Face, or another source, the Embeddings class ensures that it can understand and process the information consistently.</p><p><strong>How Embedding Models Work</strong></p><p>When your assistant needs to process text, the Embeddings class provides two main methods: one for embedding documents and another for embedding queries.</p><ul><li><strong>Embedding Documents</strong>: Imagine you have a stack of notes, articles, and study guides that you want your assistant to organize. The embed_documents() method takes all these texts and converts them into vectors, creating a searchable map of the information. This way, when you later ask a question or search for something specific, the assistant can quickly find the most relevant document based on the vectors it generated.</li><li><strong>Embedding Queries</strong>: Now, suppose you’re looking for a specific concept or fact among your study materials. The embed_query() method transforms your question or search term into a vector that represents its meaning. This query vector is then used to search through the document vectors, helping your assistant find the most relevant answers or documents.</li></ul><p>The reason for having separate methods for documents and queries is that some embedding models handle them differently. For example, the way a model processes an entire document might differ from how it processes a short query, ensuring that each type of text is embedded in the most effective way possible.</p><p><strong>Why Embeddings Matter in Our Story</strong></p><p>As your AI assistant continues to assist you in various tasks, its ability to understand and retrieve relevant information becomes crucial. Whether you’re studying for an exam, preparing for a presentation, or simply organizing your notes, embedding models ensure that your assistant can find and present the information you need with remarkable accuracy.</p><p>By using embedding models, your assistant goes beyond simple keyword matching. It gains a deeper, more nuanced understanding of the text, allowing it to serve as a true knowledge partner — one that can help you navigate the complex world of information with ease.</p><p>In our story, embedding models are like the assistant’s internal compass, guiding it through the vast landscape of text and helping it find the most relevant paths to take. With these tools at its disposal, your assistant is better equipped to understand your needs and deliver the insights you’re looking for, all with the precision and understanding that only an advanced AI can provide.</p><h3>4.Vector Stores: Organizing Your AI Assistant’s Knowledge Base</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*jZzigt8tCJ-61s--Wfjf4Q.png" /><figcaption>knowledge base</figcaption></figure><p>Continuing with our story, imagine your AI assistant has learned to understand and represent text through embedding models, turning pieces of information into vectors — those handy arrays of numbers that capture the essence of the text. But now, your assistant faces a new challenge: how to store and efficiently search through all this vectorized information. This is where vector stores come into play.</p><p><strong>What Are Vector Stores?</strong></p><p>Think of vector stores as your assistant’s memory warehouse. Just like a library categorizes books to make them easy to find, a vector store organizes and stores all the embedded vectors that represent your data. But it goes beyond simple storage. A vector store also enables your assistant to search through this data quickly and effectively, ensuring it can find the most relevant information in response to your queries.</p><p>Here’s how it works: When you provide your assistant with unstructured data — such as notes, articles, or even random thoughts — it uses embedding models to convert this data into vectors. These vectors are then stored in a vector store. Later, when you ask your assistant a question or search for information, it embeds your query into a vector and compares it with the stored vectors to find the most similar matches. This way, your assistant can retrieve the information that best answers your query, even if the exact words you used aren’t present in the stored data.</p><p><strong>Storing Metadata for Smarter Searches</strong></p><p>But vector stores do more than just hold vectors. They can also store metadata — additional information about each piece of embedded data. For example, if your assistant is storing notes from different subjects, it could attach metadata indicating the subject, date, or even the source of each note. This metadata becomes incredibly useful when you need to filter search results.</p><p>Let’s say you’re only interested in study materials from the last month or want to find notes specifically related to a particular topic. The vector store can use the metadata to filter out irrelevant data, narrowing down the search results to exactly what you need. This makes your assistant’s searches not only more accurate but also more tailored to your specific needs.</p><p><strong>Using Vector Stores as Retrievers</strong></p><p>In our ongoing journey, your assistant’s ability to retrieve relevant information is key. This is where the concept of retrievers comes in. A retriever is an interface that allows your assistant to interact with the vector store in a way that’s optimized for searching and retrieving information.</p><p>Imagine you’ve set up a vector store for all your study materials. You can convert this vector store into a retriever with a simple command:</p><pre>vectorstore = MyVectorStore()<br>retriever = vectorstore.as_retriever()</pre><p>Now, your assistant can use the retriever to quickly access the stored vectors and find the most relevant matches for any query you provide. This streamlined process ensures that the assistant’s responses are not only fast but also highly relevant, giving you the precise information you’re looking for.</p><p><strong>Why Vector Stores Matter in Our Story</strong></p><p>As your AI assistant grows more capable, its ability to store and retrieve information efficiently becomes crucial. Vector stores are the backbone of this capability, ensuring that all the knowledge your assistant accumulates is well-organized and easily accessible.</p><p>Whether you’re studying, conducting research, or just trying to keep track of your thoughts, vector stores empower your assistant to handle vast amounts of information with ease. They make sure that when you ask a question, your assistant can quickly search through its memory, find the most relevant data, and present it to you in a way that makes sense.</p><p>In our story, vector stores are like the carefully organized shelves in your assistant’s library, ensuring that every piece of knowledge is stored in the right place and ready to be retrieved at a moment’s notice. With these tools at its disposal, your assistant becomes an even more powerful ally, capable of navigating the complex web of information with speed and precision.</p><h3>5.What is RAG?</h3><p>In the evolving world of AI, one of the most intriguing techniques for enhancing the capabilities of language models is Retrieval Augmented Generation (RAG). To continue our story, imagine your AI assistant has a vast knowledge base but is limited to the information it was trained on up until a certain date. What if you need it to understand and reason about data that’s private or has emerged after that date? This is where RAG becomes an invaluable tool.Learn more about it here <a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/0*qKGvrleNYV74zIQz.png" /><figcaption>Retrival And Generation Steps</figcaption></figure><p><strong>The Challenge of Static Knowledge</strong></p><p>Language models (LLMs) are powerful tools capable of reasoning and generating text across a wide range of topics. However, their knowledge is not infinite. They are bound by the data they were trained on, which includes only information available up to a specific point in time. For applications that need to access and understand more recent or private data, this limitation can be a significant hurdle.</p><p><strong>Introducing Retrieval Augmented Generation (RAG)</strong></p><p>RAG addresses this challenge by augmenting the model’s knowledge with additional, up-to-date information. Think of it like equipping your assistant with a fresh set of data that it can pull from, just as if it were getting updated information from the latest research or private documents you provide.</p><p>The process works by retrieving relevant information from an external source and incorporating it into the model’s responses. For instance, if your assistant needs to answer a question about a recent event or a piece of confidential data, RAG allows it to fetch this information dynamically and include it in its responses. This ensures that the model’s answers are both current and relevant, despite the static nature of its base knowledge.</p><p><strong>Building RAG Applications with </strong>🦜️🔗<strong>LangChain</strong></p><p>LangChain🦜️🔗 provides a robust set of tools designed to facilitate the development of RAG applications. These components are tailored to help integrate external data sources with LLMs, enabling the creation of sophisticated Q&amp;A systems and other applications that benefit from augmented knowledge.</p><p>For a deeper dive into the intricacies of RAG and how to implement it effectively, you can refer to my previous article, <a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a>. This article explores the evolution of language models, including techniques like RAG, and provides a comprehensive understanding of how these models can be utilized and enhanced.</p><p><strong>Why RAG Matters</strong></p><p>In our story, RAG represents a significant leap forward for your AI assistant. By enabling the model to access and incorporate up-to-date or private information, RAG enhances its ability to provide accurate, relevant, and timely responses. This capability transforms your assistant from a static knowledge base into a dynamic tool capable of handling the latest and most specific queries with ease.</p><p>Incorporating RAG into your AI assistant’s toolkit means it can now navigate the ever-changing landscape of information with confidence. It’s like giving your assistant an extended reach into the vast ocean of knowledge, ensuring it can always provide the most pertinent and up-to-date answers you need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*6OVjh441LU1g4YkxHwSGGQ.png" /><figcaption>setup and development</figcaption></figure><p>With these concepts in hand, we’re ready to bring our AI-powered research assistant to life. Each of these elements plays a crucial role in the story, ensuring that our Local RAG application is not just a tool, but a true partner in your research process. Now, let’s begin the journey of building this powerful assistant, step by step.</p><p>In our journey so far, we’ve explored how your AI assistant can understand and retrieve information with advanced techniques like Retrieval Augmented Generation (RAG). Now, let’s take the next step: setting up a Local RAG application. This will involve running your language models locally, ensuring that your assistant can leverage the power of RAG right from your own machine.</p><p><strong>The Shift to Local LLMs</strong></p><p>The rise of projects such as <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a>, <a href="https://github.com/ollama/ollama">Ollama</a>, and <a href="https://github.com/Mozilla-Ocho/llamafile">llamafile</a> highlights a growing trend: running large language models (LLMs) locally. This shift is driven by the need for more control, privacy, and efficiency. Running LLMs on your local hardware allows you to handle sensitive data securely and make real-time updates without relying on external servers.</p><p>LangChain🦜️🔗 supports various <a href="https://python.langchain.com/v0.2/docs/how_to/local_llms/">open-source LLM providers</a> that can be run locally, offering flexibility and customization for your RAG applications. In this guide, we’ll focus on setting up LLaMA 3.1 locally using the <a href="https://github.com/ollama/ollama">Ollama</a> provider. However, the process is similar if you choose to use other providers like <a href="https://python.langchain.com/v0.2/docs/integrations/chat/llamacpp/">LlamaCPP</a>.</p><p><strong>Setting Up LLaMA 3.1 Locally with Ollama</strong></p><p>Let’s dive into the specifics of setting up LLaMA 3.1 locally. Imagine you’re preparing your AI assistant to operate independently, right from your laptop. By following this guide, you’ll configure Ollama to run LLaMA 3.1 on your local machine, complete with local embeddings and a local LLM setup.</p><ol><li><strong>Installing Ollama</strong>: Begin by installing Ollama on your laptop. This will be the platform that allows you to run LLaMA 3.1 locally. Follow the installation instructions provided by Ollama to get started.</li><li><strong>Configuring Local Embeddings</strong>: To enhance your assistant’s ability to understand and retrieve information, set up local embeddings. These embeddings will be used to convert your documents and queries into vector representations that the LLaMA model can process.</li><li><strong>Running the Local Model</strong>: With Ollama installed and embeddings configured, you can now run LLaMA 3.1 locally. This setup allows you to process and respond to queries right on your machine, giving you full control over the model&#39;s operations and the data it handles.</li></ol><p><strong>Customizing Your Setup</strong></p><p>This guide uses a chat model wrapper that formats input prompts specifically for the local model you’re using. If you’re working directly with a text-in/text-out LLM wrapper, you might need to tailor the prompt to fit your specific model. This often involves including special tokens to ensure the model interprets the input correctly. For instance, if you’re using LLaMA 2, you would adjust the prompt to match its requirements.</p><p><strong>Why Local RAG Applications Matter</strong></p><p>Running LLaMA 3.1 locally and setting up a Local RAG application is a significant step toward enhancing your AI assistant’s capabilities. By leveraging local models, you gain greater control over the data, improve response times, and ensure privacy. Your assistant becomes a more powerful tool, capable of delivering accurate and timely information based on the latest data, all managed directly from your laptop.</p><p>By following this guide, you’re setting the stage for a robust and flexible Local RAG application. It’s like bringing your AI assistant into your own workspace, where it can operate independently and efficiently, making use of the latest techniques in RAG and local model technology.</p><h3>Setting Up Your Local RAG Application: The Setup Journey</h3><p>In our story, we’ve embarked on an exciting journey to build a Local RAG application. Now, it’s time to set up the components that will bring your AI assistant to life on your local machine. Just like preparing for a grand adventure, setting up the necessary tools and environments is crucial for a smooth and successful implementation.</p><p><strong>Starting with Ollama</strong></p><p>Our adventure begins with Ollama, the platform that will help us run our language models locally. Think of it as the foundation of your AI assistant’s new home. Here’s how we’ll get started:</p><ol><li><strong>Download and Run Ollama’s Desktop App</strong>: To begin, you’ll need to <a href="https://ollama.com/download">download</a> and install Ollama’s desktop application. This app is like the command center for your local AI models, allowing you to manage and operate them from your machine.</li><li><strong>Fetching the Models</strong>: With the app up and running, you’re ready to fetch the models you’ll need for your RAG setup. From the command line, you’ll pull the models using <a href="https://ollama.com/library">specific commands</a>. For this guide, you’ll require:</li></ol><ul><li>A general-purpose model like llama3.1:8b. You can pull this model with the command:</li><li>ollama pull llama3.1:8b</li><li>A text embedding model such as nomic-embed-text. This model can be pulled with:</li><li>ollama pull nomic-embed-text</li></ul><p>These models will be served automatically on localhost:11434 once the app is running. This is where your assistant will access its knowledge and processing power.</p><p><strong>3. Hardware Considerations</strong>: Keep in mind that the choice of models may vary depending on your hardware capabilities. Larger models may require more computational resources, so select models that are suitable for your setup.</p><p><strong>Installing Necessary Packages</strong></p><p>With Ollama set up and models fetched, we’ll now install the packages required for local embeddings, vector storage, and inference. These packages will ensure that your AI assistant can handle document loading, retrieval methods, and text splitting effectively.</p><ol><li><strong>Document Loading and Retrieval Methods</strong>: First, install the LangChain🦜️🔗 and LangChain Community packages. These are essential for handling documents and retrieving information:</li></ol><ul><li>%pip install -qU langchain langchain_community</li></ul><p><strong>2. Local Vector Store via Chroma</strong>: Next, you’ll set up a local vector store using Chroma. This tool will help store and manage the vector representations of your data:</p><ul><li>%pip install -qU langchain_chroma</li></ul><p><strong>3. Local Inference and Embeddings via Ollama</strong>: Finally, install the LangChain🦜️🔗 Ollama package for handling local inference and embeddings with your Ollama setup:</p><ul><li>%pip install -qU langchain_ollama</li></ul><p><strong>Why This Setup Matters</strong></p><p>Just like preparing for an important journey, setting up your Local RAG application involves laying a solid foundation. By installing and configuring Ollama and the necessary packages, you’re equipping your AI assistant with the tools it needs to operate efficiently and effectively on your local machine.</p><p>As you complete these setup steps, you’re not just preparing a system; you’re setting the stage for a powerful, locally run AI assistant. This setup ensures that your assistant can leverage local embeddings and models, providing you with accurate, up-to-date information right from your laptop.</p><h3>Building Your Local RAG Application: Document Loading and Embeddings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*KMPiEBO8josPUmMcncdBow.png" /><figcaption>local applications</figcaption></figure><p>With your Local RAG setup well underway, it’s time to dive into the core of your AI assistant’s capabilities. Our next adventure involves loading documents, splitting them into manageable chunks, setting up embeddings, and preparing the model for inference. Let’s walk through each step together in our ongoing journey.</p><p><strong>Loading and Splitting the Document</strong></p><p>Imagine our AI assistant is like a diligent researcher, eager to read and understand various documents. We’ll start by providing it with a <a href="https://lilianweng.github.io/posts/2023-06-23-agent/">blog post</a> by Lilian Weng on agents. This <a href="https://lilianweng.github.io/posts/2023-06-23-agent/">blog post</a> will be our source of knowledge, and we’ll teach our assistant how to handle this information effectively.</p><ol><li><strong>Fetching the Document</strong>: First, we need to fetch the blog post. We use WebBaseLoader from LangChain🦜️🔗 to do this. It’s like giving our assistant a digital book to read:</li></ol><pre>from langchain_community.document_loaders import WebBaseLoader<br><br>loader = WebBaseLoader(&quot;https://lilianweng.github.io/posts/2023-06-23-agent/&quot;)<br>data = loader.load()</pre><p><strong>2.Splitting the Document</strong>: Now that we have the entire blog post, we need to break it into smaller, more manageable pieces. This is where RecursiveCharacterTextSplitter comes in. It’s akin to dividing a book into chapters for easier reading:</p><pre>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br>text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)<br>all_splits = text_splitter.split_documents(data)</pre><p>This step ensures that our AI assistant can handle and process the information in chunks rather than a massive block of text.</p><p><strong>Setting Up Local Embeddings and Vector Storage</strong></p><p>With our document properly split, the next step is to create vector representations of these chunks and store them. This is like translating the text into a numerical format that our AI assistant can work with efficiently.</p><p><strong>Initializing the Vector Store</strong>: We use OllamaEmbeddings for creating embeddings, and Chroma to store them. This setup allows us to perform similarity searches over the embedded data:</p><pre>from langchain_chroma import Chroma<br>from langchain_ollama import OllamaEmbeddings<br><br>local_embeddings = OllamaEmbeddings(model=&quot;nomic-embed-text&quot;)<br>vectorstore = Chroma.from_documents(documents=all_splits, embedding=local_embeddings)</pre><p>Here, nomic-embed-text is the model used for generating embeddings, and Chroma is the vector store that keeps everything organized.</p><p><strong>Testing Similarity Search</strong>: Now that our vector store is set up, let’s see if it’s working as expected. We’ll test it by querying with a question to check if it retrieves relevant information:</p><pre>question = &quot;What are the approaches to Task Decomposition?&quot;<br>docs = vectorstore.similarity_search(question)</pre><p>The output will show how many relevant documents were found and display a snippet from the most relevant one:</p><pre>len(docs)  # Number of relevant documents found<br><br>docs[0]    # Display the first relevant document</pre><p><strong>Setting Up and Testing the Model</strong></p><p>Finally, we’ll set up our AI model to ensure everything is functioning properly. We’ll use Ollama with the llama3.1:8b model for this purpose.</p><ol><li><strong>Initializing the Model</strong>: Set up the model using ChatOllama:</li></ol><pre>from langchain_ollama import ChatOllama<br><br>model = ChatOllama(model=&quot;llama3.1:8b&quot;)</pre><p><strong>2.Testing the Model</strong>: To confirm that everything is working correctly, let’s run a test. We’ll ask the model to simulate a rap battle between Stephen Colbert and John Oliver:</p><pre>response_message = model.invoke(&quot;Simulate a rap battle between Stephen Colbert and John Oliver&quot;)<br>print(response_message.content)</pre><p>The response will give us a creative and entertaining rap battle scenario, showing that our model is functioning correctly and generating responses as expected.</p><p>By following these steps, you’ve successfully set up a Local RAG application that can handle document loading, embedding, and querying locally. Your AI assistant is now equipped with the tools it needs to understand and interact with the information you’ve provided, all from the comfort of your local environment.</p><h3>Adding Question-Answering to Your Local RAG Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*ru9WCuGH5lfyGP3cPbJw-g.png" /><figcaption>AI question answering</figcaption></figure><p>Now that our Local RAG application is set up and running smoothly, it’s time to incorporate question-answering capabilities. This will enable your AI assistant to not only store and retrieve information but also to respond to specific queries with contextually relevant answers. Let’s explore how to add this feature.</p><p><strong>1. Setting Up the Q&amp;A Pipeline</strong></p><p>To answer questions effectively, we’ll use a combination of retrieved context and a prompt template. The process involves defining how to format the retrieved context and how to interact with the model.</p><p><strong>Defining the Prompt Template</strong>: We’ll create a prompt template for guiding the model’s responses. This template instructs the model to use the provided context to answer questions concisely:</p><pre>from langchain_core.runnables import RunnablePassthrough<br><br>RAG_TEMPLATE = &quot;&quot;&quot;<br>You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don&#39;t know the answer, just say that you don&#39;t know. Use three sentences maximum and keep the answer concise.<br><br>&lt;context&gt;<br>{context}<br>&lt;/context&gt;<br><br>Answer the following question:<br><br>{question}&quot;&quot;&quot;<br><br>rag_prompt = ChatPromptTemplate.from_template(RAG_TEMPLATE)</pre><p>Here, RAG_TEMPLATE sets the stage for how the assistant should handle the retrieved context and the user’s question.</p><p><strong>Creating the Q&amp;A Chain</strong>: We’ll build a chain that incorporates the prompt template and processes the retrieved documents. This chain will use RunnablePassthrough to format the documents, pass them through the prompt template, and finally, invoke the model to generate the response:</p><pre>chain = (<br>    RunnablePassthrough.assign(context=lambda input: format_docs(input[&quot;context&quot;]))<br>    | rag_prompt<br>    | model<br>    | StrOutputParser()<br>)</pre><p>The RunnablePassthrough component helps in formatting the context before it’s passed to the prompt template and the model.</p><p><strong>Running the Q&amp;A Chain</strong>: We’ll test this setup by asking a question related to the documents stored in the vector store. Here’s how you can run the chain with a sample question:</p><pre>question = &quot;What are the approaches to Task Decomposition?&quot;<br><br>docs = vectorstore.similarity_search(question)<br><br># Run the Q&amp;A chain<br>response = chain.invoke({&quot;context&quot;: docs, &quot;question&quot;: question})<br>print(response)</pre><p>This will retrieve relevant documents, format them, and generate an answer based on the given question.</p><p><strong>2. Automating Context Retrieval</strong></p><p>For a more seamless experience, you can automate the retrieval of context from the vector store based on the user’s question. This eliminates the need for manual context handling and integrates the retrieval process directly into the question-answering pipeline.</p><p><strong>Creating an Automated Q&amp;A Chain</strong>: We’ll set up a chain where the vectorstore acts as a retriever. This retriever will automatically fetch relevant documents based on the question, format them, and pass them through the prompt template and model:</p><pre>retriever = vectorstore.as_retriever()<br><br>qa_chain = (<br>    {&quot;context&quot;: retriever | format_docs, &quot;question&quot;: RunnablePassthrough()}<br>    | rag_prompt<br>    | model<br>    | StrOutputParser()<br>)</pre><p><strong>Testing the Automated Chain</strong>: With the automated chain set up, you can ask questions, and the system will handle context retrieval and answer generation in one seamless process:</p><pre>question = &quot;What are the approaches to Task Decomposition?&quot;<br><br>response = qa_chain.invoke(question)<br>print(response)</pre><p>This approach ensures that the context retrieval is dynamically managed based on the question, leading to more efficient and accurate responses.</p><p>With these enhancements, your Local RAG application is now equipped with robust question-answering capabilities, seamlessly integrating document retrieval, context formatting, and response generation. Your AI assistant is ready to provide insightful answers based on the knowledge it has accumulated locally.</p><h3>Next Steps</h3><p>Congratulations! You’ve successfully built a Local RAG (Retrieval-Augmented Generation) application using all local components. This is a powerful setup that can greatly enhance your AI’s ability to reason with private or up-to-date data. To deepen your understanding and explore more advanced techniques, consider the following resources:</p><ol><li><a href="https://www.youtube.com/watch?v=-ROS6gfYIts"><strong>Video: Reliable, Fully Local RAG Agents with LLaMA 3</strong></a></li><li><a href="https://www.youtube.com/watch?v=E2shqsYwxck"><strong>Video: Building Corrective RAG from Scratch</strong></a></li><li><a href="https://python.langchain.com/v0.2/docs/concepts/#retrieval"><strong>Conceptual Guide on Retrieval</strong></a></li><li><a href="https://python.langchain.com/v0.2/docs/how_to/#qa-with-rag"><strong>How-to Guides on RAG</strong></a></li><li><a href="https://python.langchain.com/v0.2/docs/how_to/local_llms/"><strong>How to Run Models Locally</strong></a></li></ol><p>For a deeper understanding of RAG and its components, you can refer to <a href="https://medium.com/generative-ai/rag-app-development-from-start-to-finish-2fb9c70ae8e1">this previous article on RAG</a> where we discussed the fundamental aspects of Retrieval-Augmented Generation and its applications.</p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p>These resources will help you further refine your skills and expand your capabilities in building advanced RAG systems. Happy exploring!</p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*wwys8-bsfsfBYQu9.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*m2OWwYfMPUjecbV-.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a61aa5d3a653" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/building-local-rag-from-start-to-finish-a61aa5d3a653">Building Local RAG:From Start to Finish</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG App Development: From Start to Finish]]></title>
            <link>https://generativeai.pub/rag-app-development-from-start-to-finish-2fb9c70ae8e1?source=rss-f2e2bdc551f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fb9c70ae8e1</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Shreya Sri]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 00:43:51 GMT</pubDate>
            <atom:updated>2024-08-03T00:43:51.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*EL9DPhOpaYQ3L5UK8qUA4g.png" /><figcaption>Image generated by canva.ai</figcaption></figure><p>In today’s tech world, we all want our apps to be smarter and more helpful. Imagine asking an app a question and getting the perfect answer every time because it can search through tons of information quickly and use the best of it. That’s what Retrieval Augmented Generation (RAG) does.Learn more about it here <a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></p><p>Let’s start our journey in a cozy office, where a small team of developers is hard at work. Their desks are covered with laptops, coffee cups, and notebooks. They have one big goal: to create an app that uses RAG to give users great answers to their questions.</p><p>The idea for this app came from the developers’ frustration with the current AI tools that often give vague or incorrect answers. They wanted to make something better. By combining the ability to search through large amounts of information (retrieval) with the ability to generate human-like responses (generation), they came up with a plan for a RAG-powered app.</p><p>This guide will take you step-by-step through the process of building this app. We’ll start with the basics of what RAG is and why it’s useful. Then, we’ll move on to the actual coding and implementation, explaining everything in simple terms.</p><p>Imagine you’re building a tool that helps people find the best recipes. With RAG, your app doesn’t just pull up a list of recipes when someone asks for the best chocolate cake recipe. Instead, it finds the top-rated recipes from various sources and then writes out a detailed, easy-to-follow recipe based on that information.</p><p>By the end of this guide, you’ll have a solid understanding of how to create your own RAG app, from start to finish. Whether you’re a beginner or have some coding experience, this journey will show you how powerful and accessible RAG technology can be.</p><p>Let’s get started on building something amazing together!</p><h3>The Agenda</h3><p>Continuing our journey with RAG App Development, one of the most exciting uses of large language models (LLMs) is creating <strong>smart question-answering (Q&amp;A) chatbots</strong>. These chatbots can answer questions based on specific information sources using a technique called Retrieval Augmented Generation, or RAG.</p><p>This tutorial will guide you through building a Q&amp;A application using a<strong> text data source</strong>. It explain the typical Q&amp;A architecture step-by-step and provide additional resources for those interested in exploring more advanced techniques.</p><p>As we build our application, we’ll also be introduced to <a href="https://docs.smith.langchain.com/"><strong>LangSmith </strong>🦜🛠️</a>, a tool that helps us trace and understand our app’s behavior. LangSmith 🦜🛠️ becomes especially useful as our app grows and becomes more complex, ensuring we can manage and improve it efficiently.</p><p>Let’s dive into the world of RAG-powered Q&amp;A chatbots and start building something amazing!</p><h3>RAG</h3><p>Continuing our story of building a RAG-powered app, let’s understand what RAG actually is.</p><p>RAG, or Retrieval Augmented Generation, is a technique that <strong>enhances the knowledge</strong> of large language models (LLMs) with <strong>extra data</strong>.</p><p><strong>LLMs</strong> are great at understanding and discussing a wide range of topics, but their <strong>knowledge is limited</strong> to the public data available up to a certain point in time — the time when they were last trained. If we want to create AI applications that can understand and work with private data or information that came after the model’s training period, we need to add that specific information to the model.</p><p><strong><em>The process of finding the right information and including it in the model’s prompt is called Retrieval Augmented Generation (RAG)</em></strong>.</p><p>To help us build our Q&amp;A application, we’ll use <strong>LangChain</strong>, which provides various tools and components specifically designed for creating RAG applications.</p><p>This technique will allow our app to fetch the most relevant data and use it to provide accurate and up-to-date answers, making our Q&amp;A chatbot smart and reliable. Let’s continue this journey and see how RAG can transform our app!</p><p>In our cozy office, where ideas buzz as freely as the coffee flows, the team is excited to explore the core concepts of building a RAG-powered app. Let’s dive into the process in a creative and simple way.</p><h3>Important Concepts</h3><p>A typical RAG application has <strong>two main parts</strong>, much like preparing a delicious recipe in our kitchen of innovation:</p><ol><li><strong>Indexing</strong>: Think of this as gathering and prepping your ingredients before cooking. It’s the process of taking data from a source and organizing it, usually done offline.</li><li><strong>Retrieval and Generation</strong>: This is where the magic happens, akin to the actual cooking process. It takes a user’s query, fetches the relevant data from the index, and then uses that data to whip up a perfect answer.</li></ol><h3>The Full Sequence from Raw Data to Answer</h3><p>Here’s how the entire process works, from raw data to serving up an answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*1evcXijwMafyyKcC0oedNg.png" /><figcaption>Indexing Steps</figcaption></figure><h4>Indexing</h4><ol><li><strong>Load</strong>: First, our team needs to gather all the ingredients (data). This is done using Document Loaders, which help collect the information we need, much like fetching fresh ingredients from the market.</li><li><strong>Split</strong>: Next, we chop these ingredients into smaller, manageable pieces using Text Splitters. This makes it easier to index and search through the data, and helps fit the information into the model’s limited context window.</li><li><strong>Store</strong>: The chopped ingredients (data chunks) need to be stored properly so they can be easily accessed later. This is often done using a VectorStore and an Embeddings model, which organize the data efficiently for retrieval.</li></ol><p>Imagine our developers, like skilled chefs, carefully preparing each step to ensure the final dish is perfect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*3pjfXIrcX77JTpgOJZdRvw.png" /><figcaption>Retrival And Generation Steps</figcaption></figure><h4>Retrieval and Generation</h4><ol><li><strong>Retrieve</strong>: When a user asks a question, it’s like placing an order. The app uses a Retriever to fetch the relevant data chunks from storage, just like a chef gathering ingredients for a dish.</li><li><strong>Generate</strong>: Finally, our master chef, a ChatModel or LLM, uses the gathered ingredients (retrieved data) along with the user’s question to create a precise and delicious answer.</li></ol><p>The team in our cozy office, armed with creativity and LangChain’s components, follows these steps to build their RAG-powered Q&amp;A app. They ensure it can fetch the right information and provide accurate responses, much like serving a perfectly prepared meal to a satisfied customer.</p><p>In this kitchen of innovation, every step is crafted with care, turning raw data into a feast of knowledge. Let’s continue our journey and see how these ingredients come together to create a smart and reliable application!</p><h3>Setup: A Recipe to start with</h3><p>In our cozy office, the team is now ready to roll up their sleeves and dive into setting up their RAG-powered app. It’s time to prepare the workspace and gather the right tools for our project. Here’s how we do it:</p><h4>Jupyter Notebook: The Interactive Kitchen</h4><p>Think of Jupyter notebooks as an interactive kitchen, where we mix ingredients, test recipes, and troubleshoot any hiccups that come our way. They’re perfect for learning how to work with large language models (LLMs) because they let us experiment and see results in real-time. If something goes awry — unexpected output, an API hiccup, or any other tech hiccup — Jupyter notebooks are like our safety net, helping us understand and fix issues on the fly.</p><p>To get started, you’ll want to set up <strong>Jupyter notebooks</strong>. They’re the ideal environment for this guide and many others in our documentation. You can find detailed instructions on how to install Jupyter notebooks here.</p><h4>Installation: Gathering Our Tools</h4><p>Just like a chef needs the right utensils, our tutorial requires specific tools to make our RAG-powered app come to life. These tools are the LangChain dependencies, which are essential for our project.</p><p>Here’s a quick list of what you need to install:</p><ul><li><strong>LangChain dependencies</strong>: These are the key ingredients that will help us build and run our app smoothly.</li></ul><p>As we gather these components, we’re setting the stage for a successful development process. With our interactive kitchen ready and our tools in place, we’re prepared to cook up something amazing with RAG technology.</p><p>Let’s get started on setting up and get ready to bring our app to life!</p><pre>pip install langchain langchain_community langchain_chroma</pre><p>For more details, check this <a href="https://python.langchain.com/v0.2/docs/how_to/installation/">Installation guide</a>.</p><h4><a href="https://docs.smith.langchain.com/">LangSmith</a>🦜🛠️: Our Trusted Sous-Chef</h4><p>As our team continues to cook up their RAG-powered app, they know that complex recipes require a careful eye on every step. Imagine LangSmith🦜🛠️ as our trusted sous-chef, helping us keep track of every detail and ensuring everything is cooking perfectly.</p><p>In the world of LangChain, many applications involve multiple steps and several calls to LLMs. Just like in a sophisticated kitchen, where each ingredient and step must be monitored closely to ensure a delicious outcome, our applications can become intricate with various processes and interactions.</p><p>LangSmith🦜🛠️ steps in as our sous-chef, allowing us to inspect and understand exactly what’s happening inside our application’s chain or agent. It helps us keep an eye on the details, track the flow of information, and troubleshoot any issues that arise.</p><p>By using LangSmith🦜🛠️, we gain the ability to see how our app is working under the hood, making it easier to manage and refine as it grows in complexity. With LangSmith🦜🛠️ on our team, we can ensure that every step of our RAG-powered app is perfectly executed, leading to a successful and reliable application.</p><p>Let’s continue our journey with LangSmith🦜🛠️ as our expert sous-chef, making sure our RAG-powered app is crafted to perfection!</p><h3>Setting Up the Ingredients: Environment Variables</h3><p>In our cozy office, it’s time to set up our virtual kitchen by preparing the environment for our RAG-powered app. Think of this step as gathering the essential spices and tools before we start cooking.</p><p>Here’s how we prepare the environment:</p><pre>import getpass<br>import os<br># Enable LangChain tracing to monitor and debug our app<br>os.environ[&quot;LANGCHAIN_TRACING_V2&quot;] = &quot;true&quot;<br># Set up the API key for LangChain<br>os.environ[&quot;LANGCHAIN_API_KEY&quot;] = getpass.getpass()</pre><ol><li><strong>Enable LangChain Tracing</strong>: Just like setting the oven to the right temperature, we enable tracing to monitor and debug our application. Setting &quot;LANGCHAIN_TRACING_V2&quot; to &quot;true&quot; helps us keep an eye on what’s happening inside our app.</li><li><strong>Set Up the API Key</strong>: To access LangChain’s features, we need an API key. Using getpass.getpass(), we securely input our API key without displaying it on the screen—like ensuring we have the right secret ingredient without revealing it to everyone.</li></ol><p>With these steps, our virtual kitchen is prepped and ready. We’re all set to dive into the exciting world of building our RAG-powered app, with everything in place to ensure a smooth cooking process.</p><h4>Indexing: Loading Our Data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*0ay8mn85XPKUdZ4Gq4vrCQ.png" /><figcaption>Loading Data</figcaption></figure><p>In our cozy office, we’re getting ready to load the content for our RAG-powered app. This is like preparing our main ingredients before cooking. Here’s how we do it:</p><h4>Loading the Blog Post Contents</h4><p>To start, we need to load the contents of a blog post. We’ll use something called `<strong>DocumentLoaders</strong>`, which help us fetch data from a source and organize it into a list of `<strong>Documents</strong>`. Each `Document` includes the page content and some metadata.</p><p>For this task, we’ll use the `<strong>WebBaseLoader</strong>`, which is like a digital chef’s tool that grabs HTML from web URLs and converts it into text using BeautifulSoup. We can customize how the HTML is parsed by specifying parameters for BeautifulSoup. In our case, we only want to keep HTML tags with specific classes like “post-content”, “post-title”, or “post-header” and ignore everything else.</p><p>Here’s the code for loading the content:</p><pre><br>import bs4<br>from langchain_community.document_loaders import WebBaseLoader<br># Set up a filter to keep only relevant HTML tags<br>bs4_strainer = bs4.SoupStrainer(class_=(&quot;post-title&quot;, &quot;post-header&quot;, &quot;post-content&quot;))<br># Create the loader to fetch and parse the web content<br>loader = WebBaseLoader(<br> web_paths=(&quot;https://lilianweng.github.io/posts/2023-06-23-agent/&quot;,),<br> bs_kwargs={&quot;parse_only&quot;: bs4_strainer},<br>)<br># Load the documents<br>docs = loader.load()<br># Check the length of the page content<br>len(docs[0].page_content)<br># Print the first 500 characters of the page content to see what we have<br>print(docs[0].page_content[:500])<br></pre><h4>What This Does?</h4><p>1. <strong>Filtering HTML</strong>: We use `bs4.SoupStrainer` to focus on HTML tags that contain the blog’s title, headers, and content. This is like selecting only the freshest ingredients for our dish.</p><p>2.<strong> Loading the Content</strong>: The `WebBaseLoader` fetches the HTML from the given URL and uses BeautifulSoup to parse it according to our filters. It then converts it into a format we can work with.</p><p>3.<strong>Previewing the Content</strong>: Finally, we check the length of the content and print the first 500 characters to get a glimpse of what we’re working with.</p><p>With our data loaded and ready, we’re all set to move on to the next step in building our RAG-powered app. Let’s continue preparing our ingredients and get ready to create something amazing!</p><h4>Explore More:</h4><p>DocumentLoader: Object that loads data from a source as list of Documents.</p><ul><li><a href="https://python.langchain.com/v0.2/docs/how_to/#document-loaders">Docs</a>: Detailed documentation on how to use DocumentLoaders.</li><li><a href="https://python.langchain.com/v0.2/docs/integrations/document_loaders/">Integrations</a>: 160+ integrations to choose from.</li><li><a href="https://api.python.langchain.com/en/latest/document_loaders/langchain_core.document_loaders.base.BaseLoader.html">Interface</a>: API reference for the base interface.</li></ul><h3>Indexing: Splitting the Document</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*RzOwGyZB0BEkSupiamV5Tw.png" /><figcaption>Indexing</figcaption></figure><p>As our team continues in the cozy office kitchen, we’re now ready to prepare our document for the final stages of cooking. Since our document is quite long — over 42,000 characters — it’s too hefty for most models to handle in one go. Think of it as trying to fit a whole roast into a single pan. Instead, we need to break it down into smaller, more manageable chunks.</p><h4>Splitting the Document</h4><p>To make sure our document is in the right size for our models, we’ll split it into chunks. This helps us manage large amounts of text and ensures that our models can efficiently process and retrieve relevant information.</p><p>Here’s how we do it:</p><pre>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br># Set up the text splitter with chunk size and overlap<br>text_splitter = RecursiveCharacterTextSplitter(<br>    chunk_size=1000, chunk_overlap=200, add_start_index=True<br>)<br><br># Split the document into chunks<br>all_splits = text_splitter.split_documents(docs)<br><br># Check the number of chunks created<br>len(all_splits)<br><br># Check the length of the first chunk<br>len(all_splits[0].page_content)<br><br># View metadata of a chunk<br>all_splits[10].metadata</pre><h4>What This Does</h4><ol><li><strong>Splitting with Overlap</strong>: We use RecursiveCharacterTextSplitter to break the document into chunks of 1000 characters, with 200 characters of overlap between chunks. This overlap helps keep important context together and ensures we don’t lose any crucial information. It&#39;s like cutting our ingredients into pieces that fit perfectly in our recipe.</li><li><strong>Preserving Start Index</strong>: By setting add_start_index=True, we keep track of where each chunk starts within the original document. This helps us maintain context and reference points, much like keeping track of where each ingredient was added in our cooking process.</li><li><strong>Checking the Results</strong>: We check the number of chunks created and review the length of one chunk to ensure everything is split correctly. We also look at the metadata to see where each chunk originated from.</li></ol><p>With our document neatly sliced into manageable chunks, we’re ready for the next steps in our RAG-powered app development. Let’s continue to refine and build our application, ensuring it can handle and retrieve information efficiently!</p><h4>Explore More:</h4><p>TextSplitter: Object that splits a list of Documents into smaller chunks. Subclass of DocumentTransformers.</p><ul><li>Learn more about splitting text using different methods by reading the <a href="https://python.langchain.com/v0.2/docs/how_to/#text-splitters">how-to docs</a></li><li><a href="https://python.langchain.com/v0.2/docs/integrations/document_loaders/source_code/">Code (py or js)</a></li><li><a href="https://python.langchain.com/v0.2/docs/integrations/document_loaders/grobid/">Scientific papers</a></li><li><a href="https://api.python.langchain.com/en/latest/base/langchain_text_splitters.base.TextSplitter.html">Interface</a>: API reference for the base interface.</li></ul><p>DocumentTransformer: Object that performs a transformation on a list of Document objects.</p><ul><li><a href="https://python.langchain.com/v0.2/docs/how_to/#text-splitters">Docs</a>: Detailed documentation on how to use DocumentTransformers</li><li><a href="https://python.langchain.com/v0.2/docs/integrations/document_transformers/">Integrations</a></li><li><a href="https://api.python.langchain.com/en/latest/documents/langchain_core.documents.transformers.BaseDocumentTransformer.html">Interface</a>: API reference for the base interface.</li></ul><h3>Indexing: Storing the Chunks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*0q3gWr6vINfNbVVjvh_58g.png" /><figcaption>Storing</figcaption></figure><p>As our team continues in the cozy office kitchen, it’s time to store our prepared ingredients — our document chunks — so they’re ready for use when needed. This step is like organizing our prepped ingredients into storage containers, making it easy to grab what we need when it’s time to cook.</p><h4>Storing the Chunks</h4><p>To efficiently search and retrieve our document chunks at runtime, we’ll index them by embedding their contents and storing these embeddings in a vector database, or vector store. This process helps us quickly find and retrieve relevant chunks based on similarity to a search query.</p><p>Here’s how we set it up:</p><pre>from langchain_chroma import Chroma<br>from langchain_openai import OpenAIEmbeddings<br><br># Set up the vector store with our document splits and embeddings model<br>vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())</pre><h4>What This Does</h4><ol><li><strong>Embedding Chunks</strong>: We use OpenAIEmbeddings to convert each document chunk into a high-dimensional vector representation, or embedding. This process is akin to turning our ingredients into a format that’s easy to store and retrieve.</li><li><strong>Storing Embeddings</strong>: These embeddings are then stored in a vector store called <a href="https://python.langchain.com/v0.2/docs/integrations/vectorstores/chroma/">Chroma</a>. Think of Chroma as our pantry where all the prepped ingredients (embeddings) are neatly organized and ready to be accessed when needed.</li><li><strong>Similarity Search</strong>: When we search, we embed the search query in the same way and perform a similarity search to find chunks with embeddings most similar to the query embedding. We use cosine similarity, which measures the angle between vectors to determine their similarity. It’s like finding ingredients in our pantry that best match our recipe requirements.</li></ol><p>With our document chunks embedded and stored, we’re ready to efficiently search and retrieve relevant information at runtime. Let’s move on to the next step and continue building our smart and responsive RAG-powered app!</p><h4>Explore More</h4><p>Embeddings: Wrapper around a text embedding model, used for converting text to embeddings.</p><ul><li><a href="https://python.langchain.com/v0.2/docs/how_to/embed_text/">Docs</a>: Detailed documentation on how to use embeddings.</li><li><a href="https://python.langchain.com/v0.2/docs/integrations/text_embedding/">Integrations</a>: 30+ integrations to choose from.</li><li><a href="https://api.python.langchain.com/en/latest/embeddings/langchain_core.embeddings.Embeddings.html">Interface</a>: API reference for the base interface.</li></ul><p>VectorStore: Wrapper around a vector database, used for storing and querying embeddings.</p><ul><li><a href="https://python.langchain.com/v0.2/docs/how_to/vectorstores/">Docs</a>: Detailed documentation on how to use vector stores.</li><li><a href="https://python.langchain.com/v0.2/docs/integrations/vectorstores/">Integrations</a>: 40+ integrations to choose from.</li><li><a href="https://api.python.langchain.com/en/latest/vectorstores/langchain_core.vectorstores.VectorStore.html">Interface</a>: API reference for the base interface.</li></ul><h3>Retrieval and Generation: Retrieving Information</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*SIWxcLczF9PM_RXhGTp4LA.png" /><figcaption>Retriving Information</figcaption></figure><p>With our document chunks neatly stored in the vector store, it’s time to put our application into action. Imagine we’re at the heart of our cozy office kitchen, where we’re about to serve up the perfect dish based on a specific request. This step involves creating the logic to search through our stored chunks, retrieve the most relevant ones, and then generate a well-crafted response.</p><h4>Retrieving Relevant Documents</h4><p>We’ll start by defining the logic for searching through our indexed documents. LangChain provides a Retriever interface, which allows us to search and fetch relevant documents based on a user’s query.</p><p>Here’s how we set up the retrieval process:</p><pre># Set up the retriever to use similarity search<br>retriever = vectorstore.as_retriever(search_type=&quot;similarity&quot;, search_kwargs={&quot;k&quot;: 6})<br><br># Perform a search to find relevant documents<br>retrieved_docs = retriever.invoke(&quot;What are the approaches to Task Decomposition?&quot;)<br><br># Check the number of documents retrieved<br>len(retrieved_docs)<br><br># Print the content of the first retrieved document<br>print(retrieved_docs[0].page_content)</pre><h4>What This Does</h4><ol><li><strong>Create a Retriever</strong>: We convert our Chroma vector store into a Retriever using the as_retriever() method. This allows us to use the vector store’s similarity search capabilities to find relevant documents. It’s like setting up our kitchen tools to find the best ingredients for our dish.</li><li><strong>Perform a Search</strong>: We use the retriever.invoke() method to search for documents related to a specific query. For example, asking &quot;What are the approaches to Task Decomposition?&quot; is like placing an order for a particular dish.</li><li><strong>Review Retrieved Documents</strong>: We check how many documents were retrieved and examine the content of the first one. This helps us understand what relevant information was found and how well it matches the query.</li></ol><p>With this retrieval process in place, we’re ready to move on to the next step — generating responses based on the retrieved documents. Our application is now set to handle user queries efficiently and provide meaningful answers!</p><h4>Explore More:</h4><ul><li><a href="https://python.langchain.com/v0.2/docs/how_to/#retrievers">Docs</a>: Further documentation on the interface and built-in retrieval techniques. Some of which include:</li><li>MultiQueryRetriever <a href="https://python.langchain.com/v0.2/docs/how_to/MultiQueryRetriever/">generates variants of the input question</a> to improve retrieval hit rate.</li><li>MultiVectorRetriever instead generates <a href="https://python.langchain.com/v0.2/docs/how_to/multi_vector/">variants of the embeddings</a>, also in order to improve retrieval hit rate.</li><li>Max marginal relevance selects for <a href="https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversity_Based_LTMIR_1998.pdf">relevance and diversity</a> among the retrieved documents to avoid passing in duplicate context.</li><li>Documents can be filtered during vector store retrieval using metadata filters, such as with a <a href="https://python.langchain.com/v0.2/docs/how_to/self_query/">Self Query Retriever</a>.</li><li><a href="https://python.langchain.com/v0.2/docs/integrations/retrievers/">Integrations</a>: Integrations with retrieval services.</li><li><a href="https://api.python.langchain.com/en/latest/retrievers/langchain_core.retrievers.BaseRetriever.html">Interface</a>: API reference for the base interface.</li></ul><h3>Retrieval and Generation: Generating Responses</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*UituPIIdD2vPz8wH5CqzRA.png" /><figcaption>Generating Responses</figcaption></figure><p>With our documents now indexed and retrievable, it’s time to combine all the pieces into a smooth, working chain that can take a user’s question, pull relevant information, and generate a coherent response. Let’s think of it as putting together a final dish after gathering all our ingredients.</p><h4>Setting Up the Model</h4><p>We’ll use the gpt-3.5-turbo OpenAI model to generate our answers, but you can choose from various language models depending on your needs. Here’s how to set it up:</p><pre>import getpass<br>import os<br><br># Setting the API key for OpenAI<br>os.environ[&quot;OPENAI_API_KEY&quot;] = getpass.getpass()<br><br>from langchain_openai import ChatOpenAI<br><br># Initialize the chat model<br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)</pre><h4>Creating the Prompt</h4><p>We’ll use a pre-defined prompt from the LangChain prompt hub. This prompt guides the model on how to use the retrieved context to answer the question. It’s like setting up our recipe to ensure the dish turns out just right.</p><pre>from langchain import hub<br># Pull the prompt template from the hub<br>prompt = hub.pull(&quot;rlm/rag-prompt&quot;)<br># Test the prompt with example inputs<br>example_messages = prompt.invoke(<br>    {&quot;context&quot;: &quot;filler context&quot;, &quot;question&quot;: &quot;filler question&quot;}<br>).to_messages()<br>print(example_messages[0].content)</pre><h4>Building the Chain</h4><p>Now, we’ll put everything together in a chain. This chain will take a question, retrieve the relevant documents, format them, generate a response, and parse the output. It’s like cooking where each step builds on the previous one to create a complete dish.</p><pre>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.runnables import RunnablePassthrough<br><br>def format_docs(docs):<br>    return &quot;\n\n&quot;.join(doc.page_content for doc in docs)<br><br># Define the chain<br>rag_chain = (<br>    {&quot;context&quot;: retriever | format_docs, &quot;question&quot;: RunnablePassthrough()}<br>    | prompt<br>    | llm<br>    | StrOutputParser()<br>)<br><br># Use the chain to answer a question<br>for chunk in rag_chain.stream(&quot;What is Task Decomposition?&quot;):<br>    print(chunk, end=&quot;&quot;, flush=True)</pre><h4>Understanding the Chain</h4><ol><li><strong>Retrieve Documents</strong>: The retriever fetches the most relevant document chunks based on the user’s question. It’s like selecting the best ingredients for our recipe.</li><li><strong>Format Documents</strong>: The format_docs function prepares the retrieved content into a format suitable for the model, ensuring all necessary details are included.</li><li><strong>Generate Response</strong>: The model generates a response based on the formatted prompt. This is where the magic happens, turning raw data into a meaningful answer.</li><li><strong>Parse Output</strong>: The StrOutputParser extracts the final answer from the model’s output, ready to be served to the user.</li></ol><p>By following these steps, you now have a complete setup for a RAG application that can retrieve relevant information and generate precise answers based on user queries.</p><h3>Next Steps: Expanding Your Q&amp;A Application</h3><p>You’ve now built a robust Q&amp;A application that efficiently retrieves and generates answers based on indexed data. But the world of RAG applications is rich with features and possibilities. Here’s how you can take your application to the next level:</p><h4>1. Return Sources</h4><p>Enhance transparency and trustworthiness by returning source documents along with answers. This feature allows users to see which documents were used to generate the answer, adding context and credibility.</p><ul><li><strong>Explore</strong>: Learn how to modify your existing chains to include source documents in the response. This can be particularly useful for applications where source verification is crucial.</li></ul><h4>2. Streaming</h4><p>Improve user experience by streaming outputs and intermediate steps. Streaming allows users to see results as they are being processed, which can be especially useful for long or complex queries.</p><ul><li><strong>Explore</strong>: Implement streaming capabilities in your chain to provide real-time updates and feedback during the question-answering process.</li></ul><h4>3. Add Chat History</h4><p>Incorporate chat history to make your application more interactive and context-aware. This feature allows the application to remember past interactions and provide more relevant responses based on the conversation’s history.</p><ul><li><strong>Explore</strong>: Integrate mechanisms to store and manage chat history, and modify your prompts to take past interactions into account.</li></ul><h4>4. Retrieval Conceptual Guide</h4><p>Deepen your understanding of retrieval techniques by exploring high-level overviews of different retrieval strategies. This will help you optimize the retrieval process and choose the best approach for your application’s needs.</p><ul><li><strong>Explore</strong>: Study various retrieval techniques, such as sparse vs. dense retrieval, and their implications for performance and accuracy.</li></ul><h4>5. Build a Local RAG Application</h4><p>For enhanced control and privacy, consider building a local RAG application using all local components. This approach allows you to deploy and run your application entirely on your own infrastructure, bypassing external dependencies.</p><ul><li><strong>Explore</strong>: Set up local instances of your data storage, retrieval, and generation components, and test your application in a fully local environment.</li></ul><p>These next steps will help you build a more powerful and versatile Q&amp;A application, capable of handling a wider range of use cases and providing a richer user experience. Whether you’re looking to enhance transparency, improve real-time interactions, or deploy locally, there’s plenty to explore and implement.</p><h3>Conclusion</h3><p>And there you have it — the process of setting up a Retrieval Augmented Generation (RAG) application is complete! From loading and indexing your data to retrieving relevant documents and generating insightful responses, each step plays a crucial role in crafting a responsive, intelligent system. By leveraging tools like LangChain and models like GPT-4, you’ve created a powerful Q&amp;A application that can handle user queries with precision and context-aware answers. Whether you’re building a chatbot, a search engine, or any other AI-powered tool, mastering these techniques will enhance your ability to deliver meaningful interactions. As you continue to develop and refine your RAG application, remember that each step — from indexing to retrieval and generation — can be fine-tuned for better performance. Happy building, and may your AI applications thrive in their cozy digital workspace!</p><h3>Other Resources</h3><ol><li><a href="https://medium.com/generative-ai/fine-tuning-language-models-a-hands-on-guide-a592f208757f">Fine-Tuning Language Models: A Hands-On Guide</a></li><li><a href="https://generativeai.pub/fine-tuning-llms-with-custom-datasets-a-deep-dive-into-customizing-natural-language-processing-09c29ed16f68">Fine-Tuning LLMs with Custom Datasets: A Deep Dive into Customizing Natural Language Processing</a></li><li><a href="https://medium.com/generative-ai/llms-and-rag-s-the-fusion-of-marvels-c1f9d5ff0357">LLM’s and RAG’s the Fusion of Marvels</a></li><li><a href="https://medium.com/generative-ai/cuda-boosts-gpts-a-revolutionary-approach-to-language-modeling-and-generation-2dbd2a0fa8bf">CUDA Boosts GPTs: A Revolutionary Approach to Language Modeling and Generation</a></li><li><a href="https://medium.com/p/f10db82167f9">From LLaMA 1 to LLaMA 3: A Comprehensive Model Evolution</a></li></ol><p><a href="https://medium.com/@shreyasri.ankala">Read more such interesting articles</a></p><p><a href="https://www.biodrop.io/shreyasri258"><strong>Connect with me</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*6LnHbjzCjqJUeFvf.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*FmAUwCnrTTFn1Pux.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fb9c70ae8e1" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/rag-app-development-from-start-to-finish-2fb9c70ae8e1">RAG App Development: From Start to Finish</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>