<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tal Waitzenberg on Medium]]></title>
        <description><![CDATA[Stories by Tal Waitzenberg on Medium]]></description>
        <link>https://medium.com/@talon8080?source=rss-55ec5990cf02------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BOmFKnB8tSWKQMRJvzWTKQ.png</url>
            <title>Stories by Tal Waitzenberg on Medium</title>
            <link>https://medium.com/@talon8080?source=rss-55ec5990cf02------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:51:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@talon8080/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mastering RAG Chabots: The Road from a POC to Production]]></title>
            <link>https://medium.com/@talon8080/mastering-rag-chabots-the-road-from-a-poc-to-production-793fb9c3580c?source=rss-55ec5990cf02------2</link>
            <guid isPermaLink="false">https://medium.com/p/793fb9c3580c</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tal Waitzenberg]]></dc:creator>
            <pubDate>Tue, 28 May 2024 15:27:51 GMT</pubDate>
            <atom:updated>2024-05-28T15:27:51.107Z</atom:updated>
            <content:encoded><![CDATA[<p>In my “Mastering RAG Chatbots” series, we’ve been exploring the details of building advanced Retrieval-Augmented Generation (RAG) models for production environments, addressing critical challenges along the way. Throughout our journey so far, we’ve covered specific topics that help facilitate the way to production, such as conventional RAG architectures, intent classification, guardrails, and prompt routing strategies. However, in this post, we’ll shift our focus to a higher-level perspective, concentrating more on product design aspects rather than solely research and development considerations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*41LbtKRtO4sAFCECv1WxHg.png" /><figcaption>The RAG Application Production Triangle</figcaption></figure><p>Transforming a promising prototype into a reliable, scalable solution requires careful planning and execution. At the heart of this journey lies the “triangle of <strong>evaluation</strong>, <strong>feedback</strong>, and <strong>improvement</strong>” — a continuous cycle that ensures your RAG application remains robust, relevant, and responsive to user needs. In this post, we’ll dive deep into each stage of this triangle, emphasizing the importance of proactive planning before embarking on the development process. By establishing a well-defined evaluation framework, gathering insightful feedback, and implementing targeted improvements, you can navigate the road to production with confidence, delivering a chatbot that exceeds expectations and provides a a high quality user experience.</p><p>Transitioning a RAG chatbot from proof-of-concept to production requires careful planning of the evaluation, feedback, and improvement cycle. We need to define clear metrics to measure the chatbot’s accuracy and performance, and implement ways to collect valuable user feedback. This feedback, combined with metric evaluations, will guide the necessary steps to improve the chatbot. Some improvements can be automated, while others may need manual input.</p><p>The goal is to create an ongoing cycle of evaluation, feedback, and improvement. As this cycle continues, the chatbot becomes increasingly robust, accurate, and aligned with user needs, making it truly production-ready. Consistent iteration based on real-world performance is key to delivering an exceptional user experience.</p><h4>Evaluation Stage</h4><p>For effective evaluation, we’ll define KPIs to measure the chatbot’s accuracy across different answer types — informative responses, conversational exchanges, and guardrail outputs. Beyond individual answers, we’ll also assess entire conversations. Key quality characteristics include:</p><ul><li><strong>Accuracy</strong>: Is the information correct ?</li><li><strong>Completeness</strong>: Does the answer cover all relevant points ?</li><li><strong>Conciseness</strong>: Is it free from irrelevant details ?</li><li><strong>Fluency</strong>: Is the phrasing clear and understandable ?</li><li><strong>Structure</strong>: Is the information properly formatted ?</li></ul><p>To determine conversation quality, we’ll seek user feedback at the end (e.g. thumbs up/down on helpfulness). These metrics will guide our feedback collection process, allowing us to focus areas for improvement iteratively.</p><p>We can also add more quality characteristics, as it really depends on the use case. For example, click-through rates for links attached to RAG answers, or counter metrics like the percentage of queries the bot doesn’t answer — both those filtered before reaching the engine and in-scope queries the engine couldn’t provide an answer for.</p><p>We can also collect additional health measurements from the application to determine its performance and identify potential issues. Some key metrics include:</p><p><strong>Application Responses Data:</strong></p><ul><li>Average number of words per answer</li><li>Average number of queries per user per day/week/month</li><li>Time to first word of the answer supplied to customer</li><li>Time to completed answer supplied to customer</li><li>Service availability (target: 0.99 uptime)</li></ul><p><strong>Users Impact:</strong></p><ul><li>Bi-weekly/monthly satisfaction feedback from customers</li></ul><p>Monitoring these measurements alongside the quality characteristics provides a comprehensive view into the chatbot’s operations. This data-driven approach enables us to proactively address bottlenecks, optimize response times, and ensure a high-quality user experience.</p><h4>Feedback Stage</h4><p>In the feedback stage, we’ll leverage the KPIs defined during the evaluation step to collect targeted feedback from users. For each answer received, users will be prompted to provide a yes/no rating on various quality aspects:</p><ol><li>Is the answer accurate? (Target: 90% of answers)</li><li>Is the answer complete? (Target: 80% of answers)</li><li>Is the answer concise? (Target: 80% of answers)</li><li>Is the answer fluent? (Target: 90% of answers)</li><li>Is the answer well-structured? (Target: 80% of answers)</li></ol><p>By setting clear success criteria for each quality metric, we can quantify user satisfaction and pinpoint areas that require improvement. This granular feedback, combined with broader metrics like overall conversation helpfulness, provides a comprehensive view of the chatbot’s performance. Systematically collecting and analyzing this user input is crucial for driving the next stage: implementing targeted enhancements to improve the chatbot’s capabilities.</p><p>We decided to make the feedback binary (yes/no) to avoid creating overly complex data that could be difficult to analyze and take actionable improvements from. Asking users to rate answers on a scale from 1 to 5, for example, presents a couple of challenges. First, some users may try to avoid this more complicated rating task. Additionally, most users are not experienced enough to provide nuanced ratings consistently. However, determining whether an answer is accurate or not (yes/no) is a much more straightforward task for users. By keeping the feedback simple and binary, we ensure higher participation rates and more reliable data to work with. This focused approach allows us to quickly identify areas that objectively require improvement without getting slowed down by subjective interpretations of multi-point rating scales.</p><p>I also encourage adding an optional free text comment field to the feedback form. While the binary yes/no answers help simplify the complexity of the feedback data for analysis, they may not capture all the nuances users want to express. A free text field gives users the ability to provide more informative responses and convey their true feelings or observations. This additional qualitative feedback offers another valuable dimension that can be analyzed to expose potential blind spots and drive further improvements. Of course, the free text comment would be optional and not required from users providing feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*0G5KfR_0Vyoy-Te7DHQzyA.png" /><figcaption>The Feedback Form Example</figcaption></figure><h4>Improvement Stage</h4><p>The improvement stage utilizes the feedback collected according to the evaluation KPIs to drive enhancements in the system’s performance. Improvements can be implemented through various techniques like model fine-tuning, prompt engineering, and optimizing information retrieval methods. The approach depends on factors such as the volume of training data collected, and the specific areas requiring improvement, such as increasing answer accuracy, mitigating unwanted topic responses (guardrails), reducing hallucinations, or addressing incomplete answers.</p><p>In the initial production stages when training data was limited, we focused on improving model answers by defining user intents and enhancing information retrieval. User intents were mapped using a semantic routing approach, like discussed in my previous <a href="https://medium.com/@talon8080/mastering-rag-chabots-semantic-router-user-intents-ef3dea01afbc">post</a>, with dedicated prompts for each intent. This prompt hierarchy allowed for agile improvements by refining existing prompts, adding new intents, or splitting intents through prompt engineering. Concurrently, we employed hybrid search techniques combining semantic and keyword similarity to prioritize the most relevant information sources, thereby improving the model’s access to accurate and complete information.</p><p>These techniques facilitated a gradual rollout process, starting small to refine the application before scaling up. We began with an internal release to a dedicated, trained user base, guiding them on application usage and collecting quality feedback. After iterative improvements, we expanded to a larger internal audience. Once the internal release achieved a satisfactory performance level, we replicated this phased rollout with production customers, onboarding them gradually while continuously enhancing the application until full-scale deployment.</p><h4>Summary</h4><p>In this post we discussed on the critical cycle of evaluation, feedback, and improvement for transitioning a RAG chatbot from proof-of-concept to production-ready. We covered defining clear KPIs in the evaluation stage to measure answer and conversation quality. The feedback stage emphasized simple binary user inputs supplemented by optional comments to collect reliable data.</p><p>In the improvement stage, techniques like prompt engineering, model fine-tuning, and hybrid information retrieval were highlighted to address specific areas of improvement identified through feedback. A gradual rollout strategy was recommended, starting with a small internal release before incrementally expanding the user base.</p><p>Patience is required! this iterative approach ensures a positive user experience from launch. Carefully executing this cycle transforms a promising RAG model into a robust, user-centric chatbot. I hope you enjoyed reading this post, stay tuned :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793fb9c3580c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering RAG Chabots: Semantic Router — User Intents]]></title>
            <link>https://medium.com/@talon8080/mastering-rag-chabots-semantic-router-user-intents-ef3dea01afbc?source=rss-55ec5990cf02------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef3dea01afbc</guid>
            <category><![CDATA[chatbot-design]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Tal Waitzenberg]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 16:11:56 GMT</pubDate>
            <atom:updated>2024-07-18T13:47:44.374Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering RAG Chatbots: Semantic Router — User Intents</h3><p>In my third post in the “Mastering RAG Chatbots” series, we will explore the use of semantic routers for intent classification in Retrieval-Augmented Generation (RAG) chatbot applications.</p><p>Building conversational AI applications requires understanding the user’s intent behind their query. A semantic router helps disambiguate intent and route queries appropriately. User intent represents the goal or task the user wants to accomplish — are they looking for instructions, recommendations, or specific product details? Understanding intent is key to providing relevant responses in a RAG application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*__DD4jObhrqkXP2c" /><figcaption>Controlling User Intents Flow</figcaption></figure><p>A semantic router analyzes queries semantically to classify intent. Based on this, it controls which knowledge resources to retrieve from, which prompts, tools or functions to use for generating a tailored response. Properly routing by intent ensures coherent answers mapped to what the user is truly asking.</p><p>While powerful, semantic routing for intent control faces key challenges like determining optimal similarity thresholds, handling no-match cases gracefully, properly accounting for sub-intents and variants, resolving ambiguous utterances across multiple intents, and ongoing maintenance of intent taxonomies and training data as the application evolves. When implemented effectively though, it elevates the user experience by precisely understanding needs.</p><p>This post dives into the key advantages that semantic routing provides, tackling key challenges using semantic router for intent classification.</p><h4>Mapping User Intents to Routes</h4><p>Building on the “RAG gateway” example from the previous <a href="https://medium.com/@talon8080/mastering-rag-chatbots-semantic-router-rag-gateway-part-1-0773cf4e70ad">post</a>, we want to define routable intents for our Vacation Recommendations RAG application. The “recommendations” intent indicates the user is looking for vacation suggestions, so we’ll route these queries to retrieve and generate relevant recommendation ideas from our data. For the “how-to” intent, the user is asking for instructions, so we’ll route to access guides and FAQs to provide helpful guidance. The “locality” intent captures when users reference a specific location or ask about nearby options, directing these to geo-based data sources. Finally, a “default” route will handle general queries with no clearly defined intent, attempting to provide a useful response. Defining these intents and their corresponding routing policies allows us to appropriately steer queries to the right knowledge sources and response generation flows within the RAG application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KX7BXb1pYU0BABuJJt7vkw.png" /><figcaption>Queries Flow Diagram of User Intents</figcaption></figure><p>For the recommendations intent route, we want to ensure any suggestions provided to the user are backed by our own data sources, not just the LLM’s general knowledge. Our goal is to give users the best recommendation responses tailored to our domain. If we don’t have a reliable data source to draw recommendations from, we’ll avoid making generic suggestions and simply state that we don’t have any recommendations available. To achieve this, we’ll define a set of recommendation utterances in a JSON file to accurately identify when a user is seeking a recommendation versus other types of queries. This allows us to precisely route recommendation requests to our curated data pipeline.</p><pre>{<br>  &quot;name&quot;: &quot;recommendations&quot;,<br>  &quot;score_threshold&quot;: 0.7,<br>  &quot;utterances&quot;: [<br>    &quot;Can you recommend a good beach destination for a family vacation?&quot;,<br>    &quot;I&#39;m looking for suggestions on romantic getaways for couples.&quot;,<br>    &quot;What are some popular outdoor adventure vacation ideas?&quot;,<br>    &quot;Do you have any recommendations for budget-friendly vacations?&quot;,<br>    &quot;I&#39;d like to plan a trip for my upcoming anniversary. Can you suggest some options?&quot;,<br>    &quot;Can you recommend a family-friendly resort with activities for kids?&quot;,<br>    &quot;I&#39;m interested in a cultural vacation. What cities would you recommend visiting?&quot;,<br>    &quot;Do you have any suggestions for a relaxing spa vacation?&quot;,<br>    &quot;I&#39;m looking for vacation ideas that combine hiking and sightseeing.&quot;,<br>    &quot;Can you recommend an all-inclusive resort with good reviews?&quot;<br>  ]<br>}</pre><p>Leveraging a semantic router for intent classification offers several key advantages over traditional monolithic prompt approaches. By breaking down long, complex prompts into focused sub-prompts mapped to specific intents, we can improve accuracy and reduce ambiguity. The “divide and conquer” approach allows us to precisely target the language model’s capabilities for each intent type.</p><p>Additionally, a router architecture provides flexibility to add, edit or remove intents on-the-fly without disrupting the entire system. We can continuously refine and enhance prompts for different areas as needed. This modularity also enables targeted evaluation and improvement processes — pinpointing strengths and weaknesses across various intents. Over time, this focused iteration promotes consistent progress toward an increasingly robust and capable conversational AI system.</p><h4>Intent Routing Challenges</h4><p>Implementing effective intent routing with a semantic router faces several challenges:</p><ul><li><em>Determining the optimal similarity threshold for each route</em></li><li><em>Handling no-match cases</em></li><li><em>Sub-intents and Variants</em></li><li><em>Ambiguous cross-intent utterances</em></li></ul><p>Let’s explore some strategies to overcome this key intent routing challenges.</p><p><strong>Determining Optimal Similarity Threshold &amp; Handling No-Match Cases</strong></p><p>To find the optimal similarity thresholds for our intent routes, we build a training set containing relevant user queries labeled with their corresponding intents. Importantly, we also include queries without a matching intent to account for no-match scenarios. The data is split into train and validation sets, ensuring a balanced distribution of intents. We then perform a random search optimization using the training set to identify thresholds that route the validation queries to the correct intent labels with the highest accuracy. This process can be executed whenever adding or editing intents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-wdnZVkqsoFYgdAP2MO0g.png" /><figcaption>Intents Threshold Optimization Process</figcaption></figure><p>Including no-intent queries helps prevent overfitting to only existing intents. It allows the router to appropriately route unmatched cases to a fallback flow. Maintaining a balanced validation set ensures the thresholds are robust across all intent types. This data-driven approach to threshold tuning, combined with graceful no-match handling, enhances the router’s reliability in production settings.</p><p><strong>Sub-Intent Variations &amp; Ambiguous Cross Intent</strong></p><p>For certain major intents, further granularity may be required through sub-intent classification. In such cases, we can employ an inner router architecture. For example, building on the guardrails router concept from the previous <a href="https://medium.com/@talon8080/mastering-rag-chatbots-semantic-router-rag-gateway-part-1-0773cf4e70ad">post</a>, we could use it as an inner router for intent classification after the initial gateway routing we used for guardrails. This nested approach allows us to first identify a broad intent category, then dive deeper to disambiguate specific sub-intents within that category using the inner router’s semantic understanding capabilities. The diagram below illustrates this concept of using the guardrails router for an additional sub-intent resolution stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjWmImkAOW3r7JY7wgqUdQ.png" /><figcaption>Guardrails Router + Inner Intet Classification Router</figcaption></figure><h4>Summary</h4><p>This post explored using a semantic router for intent classification in conversational AI apps like chatbots — a cool concept with great advantages. The router analyzes queries to figure out the user’s intent, then routes it to the proper response pipeline. This allows breaking down complex queries into focused prompts per intent for better accuracy, plus flexibility to easily tweak intents without overhauling everything. The modular setup aids continuous improvement too. But nailing the implementation has challenges, like determining optimal similarity thresholds, gracefully handling no-matches, accounting for sub-intent nuances with nested routers, resolving ambiguities, and enabling ongoing refinement. The post covered strategies for tackling these hurdles to build robust intent routing capabilities that elevate the conversational experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef3dea01afbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering RAG Chatbots: Semantic Router — RAG gateway— Part 1]]></title>
            <link>https://medium.com/@talon8080/mastering-rag-chatbots-semantic-router-rag-gateway-part-1-0773cf4e70ad?source=rss-55ec5990cf02------2</link>
            <guid isPermaLink="false">https://medium.com/p/0773cf4e70ad</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Tal Waitzenberg]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 22:18:59 GMT</pubDate>
            <atom:updated>2024-03-29T06:58:50.900Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering RAG Chatbots: Semantic Router — RAG gateway</h3><p>Welcome to the second post in the “Mastering RAG Chatbots” series, where we delve into the powerful concept of semantic router for building advanced Retrieval-Augmented Generation (RAG) applications tailored for production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5vBpgYpOigyDFReWQt8nKg.png" /></figure><p>In this post, we will explore the concept of semantic router — intelligent gateway that leverage semantic similarity to streamline query management and enhance the capabilities of advanced RAG models for production environments. As these natural language processing systems grow increasingly sophisticated, handling diverse topics and use cases becomes a challenge. Semantic router emerge as a powerful solution, acting as intelligent gateway that direct incoming queries to the appropriate response or action pipeline based on their semantic content, thereby enabling the building of modular, scalable RAG applications suited for production.</p><p>By using a semantic router in your RAG application, you gain better control over handling queries. This smart component uses semantic similarity to route incoming requests to the right response pipeline, action, or tool. With a semantic router, you can avoid answering questions outside your RAG’s intended scope. It also allows you to seamlessly integrate actions like triggering specific functions, using external tools, or implementing FAQ’s caching routes for reducing costs and efficiency. Overall, the semantic router helps build modular RAG solutions suited for production. The semantic router is the first step to adding guardrails to our application, ensuring it operates within defined boundaries.</p><h4>The Concept of Semantic Router in RAG applications</h4><p>Let’s dive into the practical implementation of a semantic router for a Vacation Recommendations RAG model. By leveraging the power of semantic similarity, we will explore how a semantic router can intelligently manage the flow of incoming queries, routing them to the appropriate response pipeline or action executor. One of the main advantages of using a semantic router is that it significantly reduces the need for prompt engineering work. When trying to restrict a language model using prompt engineering, we risk altering or breaking well-crafted prompts, which can change the behavior and flavor of good responses. Each modification to the prompt affects the probabilities of the next words, potentially leading to partial or poor answers. By controlling unwanted topics outside the prompt, we can avoid these permanent costs (as system prompts are attached to each prompt sent to the LLM), allowing our RAG application to focus solely on providing accurate answers within its domain.</p><p>We will begin by illustrating a typical message flow, showcasing how queries are processed and how we can tailor our responses or actions based on the identified route. Our goal is to ensure that our Vacation Recommendations RAG remains focused on its intended domain, avoiding unrelated topics such as coding or investments.</p><p>Furthermore, we will show how a semantic router can serve as an alternative to traditional language model agents, enabling seamless integration of external tools and functionalities. This includes the creation of dynamic routes for frequently asked questions (FAQs), facilitating cost savings and efficient query handling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1bGXbG_qBh8S7XZKDcUzA.png" /><figcaption>Semantic Router Gateway — Queries Flow</figcaption></figure><p>The routes are:</p><ul><li><strong>Coding and Investments routes</strong>: Unwanted topics that our RAG should avoid responding to.</li><li><strong>Cached FAQ route</strong>: A dynamic route for frequently asked questions, leveraging caching mechanisms for cost efficiency.</li><li><strong>Specific action route</strong>: A route dedicated to performing actions using external tools or integrations. In this specific example we want to use a flights finder tool.</li><li><strong>Fallback route</strong>: The default route for querying the Vacation Recommendations RAG when no specific action or tool is required.</li></ul><blockquote>By adopting a “<strong>negative perspective</strong>” approach, we will focus on creating routes for unwanted topics at first, rather than attempting to overly restrict the domain of questions our RAG can answer. Our preference is to initially allow answering a wider scope of queries, even if some are unwanted, and then gradually block off specific topics as necessary. This proactive approach ensures that we avoid inadvertently filtering out valid queries that users rightfully expect our system to handle. We want to prevent users from feeling that our RAG’s scope is too narrow or that the quality is compromised by an overly restrictive domain boundary. The goal is to provide a comprehensive and high-quality experience, while retaining the ability to dynamically manage and refine the system’s scope as needed.</blockquote><h4>Aurelio.AI Sematic Router python Library</h4><p>We will utilize the Aurelio.AI Python library and extend its capabilities for implementing our semantic router. Additionally, we will work with Langchain.</p><p>Install the semantic-router library with this command:</p><pre>pip install -qU semantic-router</pre><p>Or visit their github and clone the library:</p><p><a href="https://github.com/aurelio-labs/semantic-router">https://github.com/aurelio-labs/semantic-router</a></p><p>running this example we will also use pytorch, transformers, langchain libraries.</p><h4>Unrealted Topics Implementation</h4><p>To handle topics we want to avoid answering and save costs, we’ll create a dummy LLM with predefined responses. We’ll extend the route class in the semantic-router library to support our dummy LLM. Additionally, we’ll implement a semantic router chain using Langchain to catch unrelated topics and return a predefined answer to the user. This approach allows us to generate hardcoded responses for unwanted topics, preventing our RAG model from attempting to answer them and potentially providing inaccurate or irrelevant information. By offloading these topics to a separate component with predefined responses, we can ensure our RAG model stays focused on its core domain while still providing a graceful response to the user.</p><p>Let’s create our DummyLLM class (I’m using tikti:</p><pre>class DummyLLM(LLM):<br>    &quot;&quot;&quot;Dummy LLM for hardcoded responses.&quot;&quot;&quot;<br><br>    responses: List[str]<br>    i: int = 0<br><br>    @property<br>    def _llm_type(self) -&gt; str:<br>        return &quot;dummy-llm&quot;<br><br>    def _call(<br>            self,<br>            prompt: str,<br>            stop: Optional[List[str]] = None,<br>            run_manager: Optional[CallbackManagerForLLMRun] = None,<br>            **kwargs: Any,<br>    ) -&gt; str:<br>        response = self.responses[self.i]<br>        if self.i &lt; len(self.responses) - 1:<br>            self.i += 1<br>        else:<br>            self.i = 0<br>        return response<br><br>    @property<br>    def _identifying_params(self) -&gt; Mapping[str, Any]:<br>        return {&quot;responses&quot;: self.responses}</pre><p>Now let’s wrap Aurelio.AI Route class to support our Dummy LLM and pre-defined responses:</p><pre>class ExtendedRoute(Route):<br>    &quot;&quot;&quot;This class extends the Route class to include a dummy llm for testing purposes.&quot;&quot;&quot;<br>    responses: Optional[List[str]]<br>    langchain_llm: Optional[Union[DummyLLM, LLM]]<br><br>    def __init__(self, config: Dict[str, Any]):<br>        # initialize parent<br>        super().__init__(**config)<br><br>        # extract relevance data from config<br>        self.responses = config.get(&#39;responses&#39;, [])<br>        if self.responses:<br>            # initialize dummy llm with pre-defined responses<br>            self.langchain_llm = DummyLLM(responses=self.responses)</pre><p>The ExtendedRoute class will accept a configuration dictionary containing all the relevant information of a standard Route class, including the predefined responses we want to generate when a query matches that specific route. This configuration will be represented by a JSON file, allowing us to define and customize the behavior of each route individually. By decoupling the route configuration from the code, we gain flexibility and maintainability, making it easier to update or modify the predefined responses without needing to alter the core implementation.</p><p>Here is an example of the Coding route configuration file:</p><pre>{<br>  &quot;name&quot;: &quot;coding&quot;,<br>  &quot;score_threshold&quot;: 0.65,<br>  &quot;utterances&quot;: [<br>    &quot;Can you help me code a function in Python that calculates the factorial of a number?&quot;,<br>    &quot;I&#39;m trying to write a Python script to automate security alerts. Can you guide me?&quot;,<br>    &quot;What&#39;s the best way to check if a number is prime using Python code?&quot;,<br>    &quot;How do I use loops in JavaScript?&quot;,<br>    &quot;Do you have any tips for coding a responsive website using HTML and CSS?&quot;,<br>    &quot;What&#39;s the best practice for handling exceptions in Python?&quot;,<br>    &quot;Can you guide me on how to implement a sorting algorithm in Java?&quot;,<br>    &quot;I&#39;m learning C++. Could you explain how pointers are used?&quot;,<br>    &quot;How can I use Python to extract data from a website?&quot;,<br>    &quot;I need help writing SQL queries, can you assist?&quot;,<br>    &quot;How can I use JavaScript to change the content of a HTML element?&quot;,<br>    &quot;What&#39;s the best way to implement multi-threading in Java?&quot;<br>  ],<br>  &quot;responses&quot;: [<br>    &quot;I&#39;m afraid I can&#39;t assist with that. My expertise is in vacation recommendations, not coding inquiries. Feel free to ask anything about vacations though!&quot;,<br>    &quot;Oops! Looks like you&#39;re asking about coding. I&#39;m the Vacation Recommendations ChatBot and my speciality lies in the domain of vacations. Try me with any query related to that and I&#39;ll do my best to assist!&quot;,<br>    &quot;My apologies, but your query seems to be about coding. As a Vacation Recommendations ChatBot, my skills are geared towards providing assistance on finding the best tips to ypur vacation. Please do ask me any question on this topic and I&#39;ll be glad to help!&quot;<br>  ]<br>}</pre><p>The utterances are the queries the semantic router is comparing the user query to find the semantic distance. When your utterances are similar to queries from your real-world domain, the better the semantic router will route your queries. As our application will be in an evaluation process before production or some gradual rollout, these utterances will need to be optimized to match the utterances of your domain’s real world. The same goes for the threshold score, which is a score that should be optimized. We can train the semantic router, but we will discuss that in my next posts.</p><p>Now we have our Route we can create the RouteLayer using an encoder and our routes. we are using an open source HuggingFaceEncoder:</p><pre>router = RouteLayer(encoder=encoder, routes=list(routes.values()))</pre><p>and when we are asking a coding question like this:</p><pre>router(&quot;I need to write a Python script for web scraping. Can you assist?&quot;)</pre><p>we are getting:</p><p><em>RouteChoice(name=’coding’, function_call=None, similarity_score=None)</em></p><p>Nice! Now let’s push into langchain Chain class to use it in our RAG application.</p><h4>Semantic Router Chain Class</h4><pre>class SemanticRouter(Chain):<br>    &quot;&quot;&quot;Semantic Router Chain.&quot;&quot;&quot;<br>    rag_chain: Chain<br>    router_layer: RouteLayer<br>    routes: Dict[str, ExtendedRoute]<br>    input_key: str = &quot;query&quot;  #: :meta private:<br>    output_key: str = &quot;result&quot;  #: :meta private:<br><br>    class Config:<br>        &quot;&quot;&quot;Configuration for this pydantic object.&quot;&quot;&quot;<br><br>        extra = Extra.forbid<br>        arbitrary_types_allowed = True<br>        allow_population_by_field_name = True<br><br>    @property<br>    def input_keys(self) -&gt; List[str]:<br>        &quot;&quot;&quot;Input keys.<br><br>        :meta private:<br>        &quot;&quot;&quot;<br>        return [self.input_key, self.chat_history_key]<br><br>    @property<br>    def output_keys(self) -&gt; List[str]:<br>        &quot;&quot;&quot;Output keys.<br><br>        :meta private:<br>        &quot;&quot;&quot;<br>        _output_keys = [self.output_key]<br>        return _output_keys<br><br>    @classmethod<br>    def from_llm(<br>            cls,<br>            rag_chain: Chain = None,<br>            router_layer: Optional[RouteLayer] = None<br>    ) -&gt; SemanticRouter:<br>        &quot;&quot;&quot;Initialize from LLM.&quot;&quot;&quot;<br>        return cls(default_chain=rag_chain, router_layer=router_layer,<br>                   routes={route.name: route for route in router_layer.routes})<br><br>    @property<br>    def _chain_type(self) -&gt; str:<br>        &quot;&quot;&quot;Return the chain type.&quot;&quot;&quot;<br>        return &quot;semantic router chain&quot;<br><br>    def _call(<br>            self,<br>            inputs: Dict[str, Any],<br>            run_manager: Optional[CallbackManagerForChainRun] = None,<br>    ) -&gt; Dict[str, Any]:<br>        &quot;&quot;&quot;Run chain on input query.&quot;&quot;&quot;<br>        query = inputs[self.input_key]<br><br>        # send user query to the router layer<br>        route_choice = self.router_layer(query)<br><br>        # check if a route as been found<br>        if route_choice.name:<br>            # execute the route llm to get a response<br>            response = self.routes[route_choice.name].langchain_llm.invoke(query)<br>        else:<br>            # if no route is found, use the rag chain to answer user question<br>            response = self.default_chain.invoke({&quot;query&quot;: query})[self.output_key]<br>        return {self.output_key: response}</pre><p>Here, we implemented a simple chain for our semantic router: if a matching route is found, execute the associated route-specific language model; if no matching route, fall back to our primary RAG application.</p><p>Finally, we can create our SemanticRouterchain and test it</p><pre># creates a dummy chain for demonstration purposes<br>dummy_rag_chain = LLMChain(llm=DummyLLM(responses=[&quot;Hi, I&#39;m a Dummy RAG Chain :)&quot;]), prompt=PromptTemplate.from_template(&quot;&quot;))<br><br># creating the semantic router chain<br>chain = SemanticRouter.from_llm(rag_chain=dummy_rag_chain, router_layer=router)</pre><p>Let’s put our semantic router to the test by evaluating its performance with coding-related queries.</p><pre>chain.invoke(&quot;What&#39;s your take on investing in the healthcare sector?&quot;)[&#39;result&#39;]</pre><p>The chain response:<em><br>“I’m afraid I can’t provide help with investment matters. My primary role is to offer insight on vacation recommendations. Please ask me anything about those topics, and I’d be more than happy to assist.”</em></p><p>Excellent! The router successfully identified the coding-related query and responded with one of our predefined responses.<br>Now, let’s test the router’s capability by providing a valid query related to vacation recommendations.</p><pre>chain.invoke(&quot;what is the best bakery in paris?&quot;)[&#39;result&#39;]</pre><p>The chain response:<em><br>“Hi, I’m a Dummy RAG Chain :)”</em></p><p>Great, the router correctly identified that the query was not related to coding, and as expected, it fell back to our Dummy RAG application for handling the vacation recommendation query.</p><p>The example highlighted how a semantic router enhances a RAG application by intelligently routing queries, handling unwanted topics with predefined responses. Also there are a lot more capabilties we can add to our router like executing actions or tools, and efficiently managing FAQs through caching. A semantic router enables better query flow control, focuses the RAG on accurate responses within its domain, avoids complex prompt engineering, improves performance, and reduces costs by sending less API calls to our LLM.</p><p>Continuing the “Mastering RAG Chatbots” series, this post explored the powerful concept of semantic routers for building advanced, production-ready RAG models. For the full code example integrating a semantic router with your RAG application using Aurelio.AI and Langchain, refer to my <a href="https://github.com/talon8080/semantic-router/blob/main/semantic_router.ipynb">GitHub notebook</a>. Hope you enjoyed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0773cf4e70ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering RAG Chatbots: Building Advanced RAG as a Conversational AI Tool with LangChain]]></title>
            <link>https://medium.com/@talon8080/mastering-rag-chatbots-building-advanced-rag-as-a-conversational-ai-tool-with-langchain-d740493ff328?source=rss-55ec5990cf02------2</link>
            <guid isPermaLink="false">https://medium.com/p/d740493ff328</guid>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Tal Waitzenberg]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 22:14:53 GMT</pubDate>
            <atom:updated>2024-03-11T22:28:47.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*Ok5uPTnHHr8WB09oNHl1tQ.png" /></figure><p>In the rapidly evolving landscape of generative AI, Retrieval Augmented Generation (RAG) models have emerged as powerful tools for leveraging the vast knowledge repositories available to us. However, simply building a RAG model is not enough; the true challenge lies in harnessing its full potential and integrating it seamlessly into real-world applications. This blog post, part of my “Mastering RAG Chatbots” series, delves into the fascinating realm of transforming your RAG model into a conversational AI assistant, acting as an invaluable tool to answer user queries.</p><p>Through this post, we will explore a simple yet valuable approach to endowing your RAG application with the ability to engage in natural conversations. By defining and implementing a decision mechanism, we will determine when to rely on the RAG’s knowledge retrieval capabilities and when to respond with more casual, conversational responses. Leveraging the power of LangChain, a robust framework for building applications with large language models, we will bring this vision to life, empowering you to create truly advanced conversational AI tools that seamlessly blend knowledge retrieval and natural language interaction.</p><h4><strong>Designing the Conversation Flow</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EP0UpwM8cTe6569NPxZcFg.png" /><figcaption><strong>Conversational RAG Flow</strong></figcaption></figure><p>The conversation flow is a crucial component that governs when to leverage the RAG application and when to rely on the chat model. This decision-making process comprises five distinct steps:</p><p><strong>Question Reshaping Decision —</strong> The first step evaluates whether the user’s question requires reshaping or can be processed as is. This decision is made by prompting the LLM with the user’s question and relevant context.</p><p><strong>Standalone Question Generation —</strong> If reshaping is necessary, this step transforms the original question into a standalone query that encapsulates all the required context and information. The LLM is prompted to generate a self-contained question that can be understood and answered without relying on external factors.</p><p><strong>Inner Router Decision —</strong> Once the question is reshaped into a suitable format, the inner router determines the appropriate path for obtaining a comprehensive answer. This decision-making process involves the following steps:</p><ol><li><strong>Vectorstore Relevance Check</strong>: The inner router first checks the vectorstore for relevant sources that could potentially answer the reshaped question. If relevant sources are found, the query is forwarded to the RAG application for generating a response based on the retrieved information.</li><li><strong>LLM Evaluation</strong>: If no relevant sources are found in the vectorstore, the reshaped question is prompted to the LLM. The LLM evaluates the query and determines the best path based on the information needed to provide a suitable answer.</li></ol><ul><li><em>RAG Application Route</em>: Despite the absence of relevant sources in the vectorstore, the LLM may still recommend utilizing the RAG application. In such cases, the RAG application is invoked, and a &quot;no answer&quot; response is returned, indicating that the query cannot be satisfactorily addressed with the available information.</li><li><em>Chat Model Route:</em> If the LLM deems the chat model&#39;s capabilities sufficient to address the reshaped question, the query is processed by the chat model, which generates a response based on the conversation history and its inherent knowledge.</li></ul><p>This approach ensures that the inner router leverages the strengths of both the vectorstore, the RAG application, and the chat model. By first checking for relevant sources and then involving the LLM’s decision-making capabilities, the system can provide comprehensive answers when possible or gracefully indicate the lack of sufficient information to address the query.</p><p><strong>Chat Model —</strong> If the inner router decides that the chat model can handle the question effectively, it processes the query based on the conversation history and generates a response accordingly.</p><p><strong>RAG Application —</strong> When the question demands the retrieval and synthesis of information from a broad knowledge base, the RAG application is invoked. This application utilizes a vector store to search for relevant information and generate an answer tailored to the user’s query.</p><p>By leveraging the capabilities of an LLM to make decisions in steps 1–3, the conversation flow can dynamically adapt to the user’s input, ensuring that questions are processed efficiently and leveraging the strengths of both the chat model and the RAG application to provide accurate and contextually relevant responses.</p><h4><strong>Crafting Prompts with the COSTAR Framework</strong></h4><p>To generate effective prompts for each step in the conversation flow, we’ll utilize the COSTAR framework. COSTAR (Context, Objective, Style, Tone, Audience, Response) offers a structured approach to prompt creation, ensuring all key aspects influencing an LLM’s response are considered for tailored and impactful output.</p><p>The COSTAR prompt structure is:</p><pre># Context #<br>[Provide relevant background information and context for the prompt]<br><br>#########<br><br># Objective #<br>[Clearly state the goal or purpose of the prompt]<br><br>#########<br><br># Style #<br>[Specify the desired writing style for the response]<br><br>#########<br><br># Tone #<br>[Describe the intended tone or voice for the response]<br><br>#########<br><br># Audience #<br>[Identify the target audience for the response]<br><br>#########<br><br># Response #<br>[Specify the type of response expected from the LLM]</pre><p>Utilizing the COSTAR framework ensures that our prompts are comprehensive, clear, and aligned with the intended purpose, enabling the LLM to generate high-quality responses that enhance the overall conversation experience.</p><h4>Implementing Our Conversational Flow as a Chain in LangChain</h4><p>To begin, let’s implement our custom ConversationalRagChainusing the from_llm method of a LangChain Chain.</p><pre>@classmethod<br>def from_llm(<br>        cls,<br>        rag_chain: Chain,<br>        llm: BaseLanguageModel,<br>        callbacks: Callbacks = None,<br>        **kwargs: Any,<br>) -&gt; ConversationalRagChain:<br>    &quot;&quot;&quot;Initialize from LLM.&quot;&quot;&quot;<br><br>    # create the rephrasing chain<br>    rephrasing_chain = LLMChain(llm=llm, prompt=REPHRASING_PROMPT, callbacks=callbacks)<br><br>    # create the standalone question chain<br>    standalone_question_chain = LLMChain(llm=llm, prompt=STANDALONE_PROMPT, callbacks=callbacks)<br><br>    # router decision chain<br>    router_decision_chain = LLMChain(llm=llm, prompt=ROUTER_DECISION_PROMPT, callbacks=callbacks)<br><br>    return cls(<br>        rag_chain=rag_chain,<br>        rephrasing_chain=rephrasing_chain,<br>        standalone_question_chain=standalone_question_chain,<br>        router_decision_chain=router_decision_chain,<br>        yaml_output_parser=YamlOutputParser(pydantic_object=ResultYAML),<br>        callbacks=callbacks,<br>        **kwargs,<br>    )</pre><p>In this method, we receive two inputs: the RAG chain (which you wish to transform into a conversational RAG, I will demonstrate how to implement an advanced RAG in my next posts), and the LLM (which can be any valid LLM from OpenAI, Google, Anthropic, or any other provider, including open-source models).</p><p>I have created three new chains: rephrasing_chain, standalone_question_chain, and router_decision_chain, which will be utilized in the _call method.</p><p>Additionally, I will be employing LangChain’s output parser to obtain the decisions of the chains in a YAML format, using a pydantic object.</p><p>Now let’s create the prompts templates we used in the from_llmmethod using COSTAR framework:</p><ul><li>REPHARSING_PROMPT</li></ul><pre># Context #<br>This is part of a conversational AI system that determines whether to use a retrieval-augmented generator (RAG) or a chat model to answer user questions. <br><br>#########<br><br># Objective #<br>Evaluate the given user question and determine if it requires reshaping according to chat history to provide necessary context and information for answering, or if it can be processed as is.<br><br>#########<br><br># Style #<br>The response should be clear, concise, and in the form of a straightforward decision - either &quot;Reshape required&quot; or &quot;No reshaping required&quot;.<br><br>#########<br><br># Tone #<br>Professional and analytical.<br><br>#########<br><br># Audience #<br>The audience is the internal system components that will act on the decision.<br><br>#########<br><br># Response #<br>If the question should be rephrased return response in YAML file format:<br>```<br>    result: true<br>```<br>otherwise return in YAML file format:<br>```<br>    result: false<br>```<br><br>##################<br><br># Chat History #<br>{chat_history}<br><br>#########<br><br># User question #<br>{ question }<br><br>#########<br><br># Your Decision in YAML format # </pre><ul><li>STANDALONE_PROMPT</li></ul><pre># Context #<br>This is part of a conversational AI system that determines whether to use a retrieval-augmented generator (RAG) or a chat model to answer user questions. <br><br>#########<br><br># Objective #<br>Take the original user question and chat history, and generate a new standalone question that can be understood and answered without relying on additional external information.<br><br>#########<br><br># Style #<br>The reshaped standalone question should be clear, concise, and self-contained, while maintaining the intent and meaning of the original query.<br><br>#########<br><br># Tone #<br>Neutral and focused on accurately capturing the essence of the original question.<br><br>#########<br><br># Audience #<br>The audience is the internal system components that will act on the decision.<br><br>#########<br><br># Response #<br>If the original question requires reshaping, provide a new reshaped standalone question that includes all necessary context and information to be self-contained.<br>If no reshaping is required, simply output the original question as is.<br><br>##################<br><br># Chat History #<br>{chat_history}<br><br>#########<br><br># User original question #<br>{ question }<br><br>#########<br><br># The new Standalone question #</pre><ul><li>ROUTER_DECISION_PROMPT</li></ul><pre># Context #<br>This is part of a conversational AI system that determines whether to use a retrieval-augmented generator (RAG) or a chat model to answer user questions. <br><br>#########<br><br># Objective #<br>Evaluate the given question and decide whether the RAG application is required to provide a comprehensive answer by retrieving relevant information from a knowledge base, or if the chat model&#39;s inherent knowledge is sufficient to generate an appropriate response.<br><br>#########<br><br># Style #<br>The response should be a clear and direct decision, stated concisely.<br><br>#########<br><br># Tone #<br>Analytical and objective.<br><br>#########<br><br># Audience #<br>The audience is the internal system components that will act on the decision.<br><br>#########<br><br># Response #<br>If the question should be rephrased return response in YAML file format:<br>```<br>    result: true<br>```<br>otherwise return in YAML file format:<br>```<br>    result: false<br>```<br><br>##################<br><br># Chat History #<br>{chat_history}<br><br>#########<br><br># User question #<br>{ question }<br><br>#########<br><br># Your Decision in YAML format #</pre><p>With the prompts defined using the COSTAR framework, we can proceed to implement the _call method and complete the ConversationalRagChain. This method will encapsulate the entire conversation flow, leveraging the prompts and decision-making processes we&#39;ve established.</p><pre>def _call(self, inputs: Dict[str, Any], run_manager: Optional[CallbackManagerForChainRun] = None) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;Call the chain.&quot;&quot;&quot;<br>    chat_history = inputs[self.chat_history_key]<br>    question = inputs[self.input_key]<br><br>    # chat history is empty, so don&#39;t rephrase the question and start a new conversation and direct to RAG chain.<br>    if not chat_history:<br>        answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br>        return {self.output_key: answer}<br><br>    # first check for question rephrasing and then check for standalone question<br>    needs_rephrasing = self.rephrasing_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question})[&#39;text&#39;].strip()<br>    <br>    # parse the llm response using the output parser<br>    rephrasing_decision = self.yaml_output_parser.parse(needs_rephrasing)<br><br>    # If the question need rephrasing, return the original question<br>    if rephrasing_decision.result:<br>        # Combine previous chat history and question into a single prompt to generate a standalone question.<br>        standalone_question = self.standalone_question_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question})[&#39;text&#39;].strip()<br><br>    # check if there are relevant sources in the vectorstore<br>    docs = self.rag_chain._get_docs(question)<br>    if docs:<br>        # use the RAG model<br>        answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br>    else:<br>        # send the question to inner router decision, Inner routing --&gt; use chat model or RAG model<br>        routing_decision = self.router_decision_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question},)[&quot;text&quot;].strip()<br>        <br>        # parse the llm response using the output parser<br>        routing_decision = self.yaml_output_parser.parse(routing_decision)<br>        <br>        if routing_decision.result:<br>            # Use chat model<br>            answer = self.llm.invoke(chat_history + [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}],).content<br>        else:<br>            # get the response from the RAG chain<br>            answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br><br>    return {self.output_key: answer}</pre><p>After defining the prompts and implementing the _call method, we can construct the ConversationalRagChain class:</p><pre>class ConversationalRagChain(Chain):<br>    &quot;&quot;&quot;Chain that encpsulate RAG application enablingnatural conversations&quot;&quot;&quot;<br>    rag_chain: Chain<br>    rephrasing_chain: LLMChain<br>    standalone_question_chain: LLMChain<br>    router_decision_chain: LLMChain<br>    yaml_output_parser: YamlOutputParser<br>    <br>    # input\output parameters<br>    input_key: str = &quot;query&quot;  <br>    chat_history_key: str = &quot;chat_history&quot; <br>    output_key: str = &quot;result&quot;<br><br>    @property<br>    def input_keys(self) -&gt; List[str]:<br>        &quot;&quot;&quot;Input keys.&quot;&quot;&quot;<br>        return [self.input_key, self.chat_history_key]<br><br>    @property<br>    def output_keys(self) -&gt; List[str]:<br>        &quot;&quot;&quot;Output keys.&quot;&quot;&quot;<br>        return [self.output_key]<br><br>    @property<br>    def _chain_type(self) -&gt; str:<br>        &quot;&quot;&quot;Return the chain type.&quot;&quot;&quot;<br>        return &quot;ConversationalRagChain&quot;<br><br>    @classmethod<br>    def from_llm(<br>            cls,<br>            rag_chain: Chain,<br>            llm: BaseLanguageModel,<br>            callbacks: Callbacks = None,<br>            **kwargs: Any,<br>    ) -&gt; ConversationalRagChain:<br>        &quot;&quot;&quot;Initialize from LLM.&quot;&quot;&quot;<br>    <br>        # create the rephrasing chain<br>        rephrasing_chain = LLMChain(llm=llm, prompt=REPHRASING_PROMPT, callbacks=callbacks)<br>    <br>        # create the standalone question chain<br>        standalone_question_chain = LLMChain(llm=llm, prompt=STANDALONE_PROMPT, callbacks=callbacks)<br>    <br>        # router decision chain<br>        router_decision_chain = LLMChain(llm=llm, prompt=ROUTER_DECISION_PROMPT, callbacks=callbacks)<br>    <br>        return cls(<br>            rag_chain=rag_chain,<br>            rephrasing_chain=rephrasing_chain,<br>            standalone_question_chain=standalone_question_chain,<br>            router_decision_chain=router_decision_chain,<br>            yaml_output_parser=YamlOutputParser(pydantic_object=ResultYAML),<br>            callbacks=callbacks,<br>            **kwargs,<br>        )<br><br>    def _call(self, inputs: Dict[str, Any], run_manager: Optional[CallbackManagerForChainRun] = None) -&gt; Dict[str, Any]:<br>        &quot;&quot;&quot;Call the chain.&quot;&quot;&quot;<br>        chat_history = inputs[self.chat_history_key]<br>        question = inputs[self.input_key]<br>    <br>        # chat history is empty, so don&#39;t rephrase the question and start a new conversation and direct to RAG chain.<br>        if not chat_history:<br>            answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br>            return {self.output_key: answer}<br>    <br>        # first check for question rephrasing and then check for standalone question<br>        needs_rephrasing = self.rephrasing_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question})[&#39;text&#39;].strip()<br>        <br>        # parse the llm response using the output parser<br>        rephrasing_decision = self.yaml_output_parser.parse(needs_rephrasing)<br>    <br>        # If the question need rephrasing, return the original question<br>        if rephrasing_decision.result:<br>            # Combine previous chat history and question into a single prompt to generate a standalone question.<br>            standalone_question = self.standalone_question_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question})[&#39;text&#39;].strip()<br>    <br>        # check if there are relevant sources in the vectorstore<br>        docs = self.rag_chain._get_docs(question)<br>        if docs:<br>            # use the RAG model<br>            answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br>        else:<br>            # send the question to inner router decision, Inner routing --&gt; use chat model or RAG model<br>            routing_decision = self.router_decision_chain.invoke({&quot;chat_history&quot;: chat_history, &quot;question&quot;: question},)[&quot;text&quot;].strip()<br>            <br>            # parse the llm response using the output parser<br>            routing_decision = self.yaml_output_parser.parse(routing_decision)<br>            <br>            if routing_decision.result:<br>                # Use chat model<br>                answer = self.llm.invoke(chat_history + [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}],).content<br>            else:<br>                # get the response from the RAG chain<br>                answer = self.rag_chain.invoke({&quot;query&quot;: question})[&#39;result&#39;].strip()<br>    <br>        return {self.output_key: answer}</pre><h4>Final Words</h4><p>This blog post demonstrated a simple approach to transform a RAG model into a conversational AI tool using LangChain. We designed a conversational flow to determine when to leverage the RAG application or chat model, utilizing the COSTAR framework to craft effective prompts.</p><p>Stay tuned for more insightful posts in my &quot;<strong>Mastering RAG Chatbots</strong>&quot; series, where I am exploring advanced conversational AI techniques and real-world applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d740493ff328" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>