<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nir Bar on Medium]]></title>
        <description><![CDATA[Stories by Nir Bar on Medium]]></description>
        <link>https://medium.com/@nirbar_21376?source=rss-4c106944d383------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*o9KgK0-JGEDkRCXN</url>
            <title>Stories by Nir Bar on Medium</title>
            <link>https://medium.com/@nirbar_21376?source=rss-4c106944d383------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:55:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nirbar_21376/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Production-Ready AI Agents with LangGraph: A Real-Life Use Case]]></title>
            <link>https://medium.com/cyberark-engineering/building-production-ready-ai-agents-with-langgraph-a-real-life-use-case-7bda34c7f4e4?source=rss-4c106944d383------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bda34c7f4e4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Nir Bar]]></dc:creator>
            <pubDate>Mon, 14 Oct 2024 11:21:54 GMT</pubDate>
            <atom:updated>2024-10-19T05:50:20.493Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_atx3GMtBAlay9-QWDbOdg.jpeg" /></figure><p>In my recent blog post, <a href="https://medium.com/cyberark-engineering/a-developer-guide-for-creating-a-multi-modal-chatbot-using-langchain-agents-9003ba0ffb4d">A Developer Guide for Creating a Multi-Modal Chatbot Using LangChain Agents</a>, I discussed the role of AI Agents and demonstrated an implementation using the LangChain framework. While it’s suitable for proofs of concept (POCs), it falls short for production environments. In this post, I will offer a solution that is more adaptable to a production-grade product. You’ll learn how to create a scalable, efficient system that’s better suited for real-world applications, giving you the tools to build more robust AI solutions.</p><h4>The Challenge with LangChain Agents in a Production Environment</h4><p>The main challenge with LangChain Agents is that they grant full control to a single LLM through one large prompt, which manages the entire workflow. In production environments, more granular control is often required, especially as tasks evolve.</p><p>So how can we best divide the process into smaller, focused prompts, each responsible for a specific part of the task? This method that I discuss in this post makes it easier to debug and fine-tune each individual component in the flow. Additionally, no single LLM is perfect for every task. It’s often better to divide the process into smaller tasks and assign the most suitable LLM for each, optimizing both performance and cost.</p><h4>How the LangGraph Library Helps You Build Production-Ready Systems</h4><p>This is where the LangGraph library comes in. It offers fine-grained control over an agent’s flow and state, essential for building robust, production-ready systems. LangGraph extends LangChain by enabling stateful, multi-actor applications with cyclical graphs, making it easier to build sophisticated, reliable agent runtimes.</p><p>The main key features in LangGraph are:</p><ul><li><strong>Cycles and Branching</strong>. Implement loops and conditionals in your apps.<br>In LangGraph, each node represents an LLM agent, and the edges are the communication channels between these agents. This structure allows for clear and manageable workflows, where each agent performs specific tasks and passes information to other agents as needed.</li><li><strong>Persistent State Management</strong>. Automatically save the state after each step in the graph. Pause and resume the graph execution at any point to support error recovery, human-in-the-loop workflows, time travel and more.</li><li><strong>Human-in-the-Loop</strong>. Interrupt graph execution and give the user the option to approve or edit the following action planned by the agent.</li><li><strong>Streaming Support</strong>. Stream outputs as they are produced by each node (including token streaming).</li><li><strong>Integration with LangChain and LangSmith</strong>: LangGraph integrates seamlessly with<a href="https://github.com/langchain-ai/langchain/"> LangChain</a> and<a href="https://docs.smith.langchain.com/"> LangSmith</a> (but does not require them).</li></ul><h4>What Did I Develop?</h4><p>I built an application that helps plan your next vacation or business trip. Enter a prompt, and the app fetches real-time flight and hotel options, displaying them on a user-friendly web page. If desired, you can also send this information via email.</p><p>You can find the full code in the <a href="https://github.com/nirbar1985/ai-travel-agent">Github repository</a>.</p><p>The <strong>AI travel agent</strong> uses two main tools:</p><ul><li>A tool that interacts with the Google Flights API.</li><li>A tool that interacts with the Google Hotels API.</li></ul><p>Additionally, the app uses the SendGrid API to send emails.</p><h4>Running the Application</h4><p>For example, if you input the following prompt:</p><p><em>I want to travel to Amsterdam from Madrid from 1 to 7 of October, find me flights and 4-star hotels.</em></p><p>The app provides relevant flight and hotel options based on real-time data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5XCNVNvKeViwH6ll7IFabg.png" /></figure><p>You’ll receive an output that includes logos and links for easy reference.</p><p><em>Note: The results are sourced from the Google Flights and Google Hotels APIs, with no intention to promote any specific brand.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mYMnN-qLr0u6JsnHOaqxqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jTAMD-nAC5XoAJlYi9Pg_A.png" /></figure><p>There’s also an option to send all the travel data via email. I’ll explain how this is implemented in the technical section (Hint: it involves the human-in-the-loop feature).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykiNePgcTjeZL0peJaaJmg.png" /></figure><p>Ohh nice, I got all the travel data rendered to HTML in an email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6W0qNvXx-Tc8q7ju-z-L0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4h-2qqCkrVZeDiz1oTgZg.png" /></figure><h4>How Did the Agent Handle the Travel Request?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQZTr8HIRPp3ZNWw-0Co5w.png" /></figure><p>Let’s break down how an AI agent processes a user’s travel request in a few simple steps. Imagine a user enters the following prompt:</p><p><em>“I want to travel to Amsterdam from Madrid from 1 to 7 of October, find me flights and 4-star hotels.”</em></p><p>Here’s what happens next:</p><ol><li><strong>User’s Request Sent to the AI (LLM)</strong>: The request is sent to a powerful language model (LLM) that supports tool usage. The LLM recognizes two tasks in the request: finding flights and hotels.</li></ol><p>2.<strong> Task Breakdown</strong>: The LLM breaks the request into two smaller tasks:</p><ul><li><em>Find flights: </em>The system uses a tool specifically designed to search for flights.</li><li><em>Find hotels: </em>The system also calls a tool that helps find 4-star hotels in Amsterdam.</li></ul><p>3. <strong>Tools Activated</strong>: The agent invokes these tools, providing the correct data (like dates, locations, etc.) as parameters. Each tool runs and outputs the flight and hotel options in a structured format.</p><p>4. <strong>Processing the Results</strong>: The LLM is called again to summarize the results and present them in an easy-to-read format.</p><p>5. <strong>Email Option</strong>: The user is then given an option to email this information. If the user decides to send it via email, they enter details in a form (like the recipient’s email).</p><p>7. <strong>Email Sent and Process Ends</strong>: Using the previously collected data, the agent resumes from where it left off. It invokes the email-sending function, which delivers the information to the provided email address.</p><p>Once the email is sent, the agent finishes its task, completing the flow.</p><h3>How to Implement this Type of Agent</h3><h4>How Does the Graph Look?</h4><p>The input to the Mermaid editor is generated using the following code:</p><pre>print(self.graph.get_graph().draw_mermaid())</pre><p>To visualize the graph, take the output from this code and enter it into the <a href="https://mermaid.live/edit">Mermaid editor online</a>. (Note that the exact visualization may differ slightly depending on the version of LangGraph you’re using).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*ZT6ufH2DpfZMExBUOJ2o0Q.png" /></figure><p>The code snippet below defines a class that implements the agent, which is responsible for managing tasks and invoking tools using a language model (LLM).</p><pre>TOOLS = [flights_finder, hotels_finder]<br><br>class AgentState(TypedDict):<br>    messages: Annotated[list[AnyMessage], operator.add]<br><br>class Agent:<br><br>    def __init__(self):<br>        self._tools = {t.name: t for t in TOOLS}<br>        self._tools_llm = ChatOpenAI(model=&#39;gpt-4o&#39;).bind_tools(TOOLS)<br><br>        builder = StateGraph(AgentState)<br>        builder.add_node(&#39;call_tools_llm&#39;, self.call_tools_llm)<br>        builder.add_node(&#39;invoke_tools&#39;, self.invoke_tools)<br>        builder.add_node(&#39;email_sender&#39;, self.email_sender)<br>        builder.set_entry_point(&#39;call_tools_llm&#39;)<br><br>        builder.add_conditional_edges(&#39;call_tools_llm&#39;, Agent.exists_action, {&#39;more_tools&#39;: &#39;invoke_tools&#39;, &#39;email_sender&#39;: &#39;email_sender&#39;})<br>        builder.add_edge(&#39;invoke_tools&#39;, &#39;call_tools_llm&#39;)<br>        builder.add_edge(&#39;email_sender&#39;, END)<br>        memory = MemorySaver()<br>        self.graph = builder.compile(checkpointer=memory, interrupt_before=[&#39;email_sender&#39;])<br><br>        print(self.graph.get_graph().draw_mermaid())<br><br>    @staticmethod<br>    def exists_action(state: AgentState):<br>        result = state[&#39;messages&#39;][-1]<br>        if len(result.tool_calls) == 0:<br>            return &#39;email_sender&#39;<br>        return &#39;more_tools&#39;<br><br>    def email_sender(self, state: AgentState):<br>        print(&#39;Sending email&#39;)<br>        email_llm = ChatOpenAI(model=&#39;gpt-4o&#39;, temperature=0.1)  # Instantiate another LLM<br>        email_message = [SystemMessage(content=EMAILS_SYSTEM_PROMPT), HumanMessage(content=state[&#39;messages&#39;][-1].content)]<br>        email_response = email_llm.invoke(email_message)<br>        print(&#39;Email content:&#39;, email_response.content)<br><br>        message = Mail(from_email=os.environ[&#39;FROM_EMAIL&#39;], to_emails=os.environ[&#39;TO_EMAIL&#39;], subject=os.environ[&#39;EMAIL_SUBJECT&#39;],<br>                       html_content=email_response.content)<br>        try:<br>            sg = SendGridAPIClient(os.environ.get(&#39;SENDGRID_API_KEY&#39;))<br>            response = sg.send(message)<br>            print(response.status_code)<br>            print(response.body)<br>            print(response.headers)<br>        except Exception as e:<br>            print(str(e))<br><br>    def call_tools_llm(self, state: AgentState):<br>        messages = state[&#39;messages&#39;]<br>        messages = [SystemMessage(content=TOOLS_SYSTEM_PROMPT)] + messages<br>        message = self._tools_llm.invoke(messages)<br>        return {&#39;messages&#39;: [message]}<br><br>    def invoke_tools(self, state: AgentState):<br>        tool_calls = state[&#39;messages&#39;][-1].tool_calls<br>        results = []<br>        for t in tool_calls:<br>            print(f&#39;Calling: {t}&#39;)<br>            if not t[&#39;name&#39;] in self._tools:  # check for bad tool name from LLM<br>                print(&#39;\n ....bad tool name....&#39;)<br>                result = &#39;bad tool name, retry&#39;  # instruct LLM to retry if bad<br>            else:<br>                result = self._tools[t[&#39;name&#39;]].invoke(t[&#39;args&#39;])<br>            results.append(ToolMessage(tool_call_id=t[&#39;id&#39;], name=t[&#39;name&#39;], content=str(result)))<br>        print(&#39;Back to the model!&#39;)<br>        return {&#39;messages&#39;: results}</pre><h3>Key Concepts of the Implementation Code</h3><p>Let’s break down the key concepts of the code provided:</p><h4>1. Tools Setup:</h4><ul><li>The variable TOOLS contains a list of tools, such as flights_finder and hotels_finder, that the agent can use. These tools represent actions the agent can perform to assist the user. I elaborated in my previous <a href="#">post</a> on how to implement such tools.</li></ul><h4>2. AgentState:</h4><ul><li>AgentState is a State that holds the current status of the agent in the form of a messages object. This list stores all communications (messages) exchanged between the agent and the user or tools.</li><li>Each message is added sequentially using operator.add, meaning new information is appended to the existing messages as the conversation progresses.</li></ul><h4><strong>3. Agent Class:</strong></h4><ul><li>The Agent class controls how the agent operates by invoking tools, processing user inputs, and managing workflows. It sets up a StateGraph, which is essentially a blueprint of how the tasks (or nodes) are connected and executed.</li></ul><h4><strong>4. Initialization (__init__ method):</strong></h4><h4><strong>Binding Tools:</strong></h4><ul><li>The tools are organized into a dictionary where each tool can be accessed by name.</li><li>The agent also binds these tools to a Language Model (LLM), specifically ChatOpenAI, so that the model can decide which tool to invoke based on user input.</li></ul><h4><strong>Building the StateGraph:</strong></h4><ul><li>The agent creates a StateGraph, which defines the flow of operations the agent will follow. In LangGraph terms, this is a Graph, where each step in the agent’s process is represented by a Node (e.g., calling tools, invoking them, or sending an email).</li><li>The StateGraph has three nodes:</li></ul><ol><li><strong>call_tools_llm</strong>: The agent calls the LLM to decide which tool to use based on the task.</li><li><strong>invoke_tools</strong>: The agent activates the chosen tools to carry out specific tasks like finding flights or hotels.</li><li><strong>email_sender</strong>: The agent sends the results to the user via email.</li></ol><ul><li>The starting node is call_tools_llm.</li></ul><h4><strong>Conditional Edges and Transitions:</strong></h4><ul><li>The graph has <strong>conditional edges</strong>, meaning it decides what to do next based on the agent’s current state (i.e., the messages). For instance, if more tools are required, it moves to the invoke_tools node. If no more tools are needed, it transitions to the email_sender node.</li></ul><h4><strong>Memory Management:</strong></h4><ul><li>The agent uses a <strong>Checkpointer</strong> (via <strong>MemorySaver</strong>) to save its progress. This allows the agent to remember its state and resume operations if interrupted.</li></ul><h4>5. Decision and Action Methods:</h4><h4>exists_action Method:</h4><p>This method checks the most recent message in the state to determine the next action. If no more tool calls are needed (i.e., there are no tool calls left in the message), the agent transitions to sending an email. Otherwise, it continues invoking additional tools. In LangGraph terms, this method acts as a <strong>Decision Function</strong> to decide which path (edge) the agent should take next.</p><h4><strong>email_sender Method:</strong></h4><p>This method handles the final step, where the agent sends an email with the results. The agent instantiates another LLM to help draft the email based on the last message in the state. It uses a prompt (template) to guide the LLM in generating the email content. The email is then sent to the user using the SendGridAPIClient.</p><h4><strong>call_tools_llm Method:</strong></h4><p>In this method, the agent sends a list of messages to the LLM, which includes a System Message (a predefined prompt instructing the LLM what to do) and previous messages. The LLM then returns a new message, which contains instructions on which tool to invoke next. This new message is added to the state.</p><h4><strong>invoke_tools Method:</strong></h4><p>This method is responsible for actually invoking the tools based on the LLM’s instructions. The agent checks the last message in the state for tool calls. It loops through these tool requests, ensures the tool is valid, and then invokes the corresponding tool. The results from the tools are returned as Tool Messages and appended to the state. If an invalid tool is requested, the agent prompts the LLM to retry.</p><h3>How Are LangGraph’s Main Features Implemented in My Application?</h3><h4>1. Persistent State Management</h4><pre>memory = MemorySaver()<br>self.graph = builder.compile(checkpointer=memory, interrupt_before=[&#39;email_sender&#39;])</pre><p>A crucial part of the agent’s design is its ability to remember its last state. This is managed by a MemorySaver, which ensures that if the agent needs to pause and resume (e.g., while waiting for the user to decide to send an email), it picks up from the correct point without starting over.</p><p>There are several options for checkpointers to use in production, such as <a href="https://langchain-ai.github.io/langgraph/how-tos/persistence_postgres/">Postgres</a>,<a href="https://langchain-ai.github.io/langgraph/how-tos/persistence_mongodb/"> MongoDB</a>, and<a href="https://langchain-ai.github.io/langgraph/how-tos/persistence_redis/"> Redis</a>.</p><p>In the app.py (which handles the UI code), a unique thread_id is generated for each session to maintain the context of the user’s interactions. This thread_id is passed along with every call to the agent, ensuring the memory state is preserved across requests.</p><pre>            # Create a new thread ID<br>            thread_id = str(uuid.uuid4())<br>            st.session_state.thread_id = thread_id<br><br>            # Create a message from the user input<br>            messages = [HumanMessage(content=user_input)]<br>            config = {&#39;configurable&#39;: {&#39;thread_id&#39;: thread_id}}<br><br>            # Invoke the agent<br>            result = st.session_state.agent.graph.invoke({&#39;messages&#39;: messages}, config=config)</pre><h4>2. <strong>Human-in-the-Loop</strong></h4><p>The agent allows human intervention at a critical decision point: Deciding whether or not to send an email with the gathered travel information (flights, hotels, etc). This is managed using the Human-in-the-Loop feature, which pauses the agent’s execution right before the email is sent, allowing the user to review the results and provide necessary input.</p><p>Implemented by passing <em>interrupt_before=[‘email_sender’]</em> in the graph.compile() function.</p><p>Once the user decides to send the email and submits the data, the following code runs in the UI to complete the process:</p><pre>def send_email(sender_email, receiver_email, subject, thread_id):<br>    try:<br>        populate_envs(sender_email, receiver_email, subject)<br>        config = {&#39;configurable&#39;: {&#39;thread_id&#39;: thread_id}}<br>        st.session_state.agent.graph.invoke(None, config=config)<br>        st.success(&#39;Email sent successfully!&#39;)<br>        # Clear session state<br>        for key in [&#39;travel_info&#39;, &#39;thread_id&#39;]:<br>            if key in st.session_state:<br>                del st.session_state[key]</pre><h4>3. Using Multiple LLMs Based on Agent State</h4><p>One of LangGraph’s key advantages is the flexibility to use multiple LLMs at different stages of the workflow, rather than relying on a single LLM with one large prompt. This allows the agent to select the most suitable LLM based on the current task. For example, you can use one LLM specialized in tool invocation and task processing, and another that is better suited for generating and formatting email content in HTML.</p><h4><strong>4. </strong>Integration with LangChain and LangSmith</h4><p>Integrating LangChain into your application provides access to its comprehensive suite of tools for building complex, modular workflows with language models. By leveraging LangChain’s capabilities, you can easily incorporate different LLMs, tools and workflows into your agent’s functionality.</p><p>To integrate with LangSmith, simply add these environment variables to your .env file:</p><pre>LANGCHAIN_API_KEY=&lt;langchain_api_key&gt;<br>LANGCHAIN_TRACING_V2=true<br>LANGCHAIN_PROJECT=&lt;langchain_project_name&gt;</pre><h3>Key Takeaways from the AI Travel Agent Use Case</h3><p>By breaking tasks into smaller, manageable components, LangGraph allows for easier debugging, flexibility in tool and LLM integration and the inclusion of human-in-the-loop interactions. The AI travel agent example, which helps users find real-time flight and hotel options, showcases LangGraph’s ability to handle complex workflows, leveraging multiple tools and offering email functionality. It also highlights LangGraph’s key features, such as persistent state management, multi-step workflows, and seamless integration with LangChain and LangSmith, making it a powerful framework for building robust AI-driven applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bda34c7f4e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyberark-engineering/building-production-ready-ai-agents-with-langgraph-a-real-life-use-case-7bda34c7f4e4">Building Production-Ready AI Agents with LangGraph: A Real-Life Use Case</a> was originally published in <a href="https://medium.com/cyberark-engineering">CyberArk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Developer Guide for Creating a Multi-Modal Chatbot Using LangChain Agents]]></title>
            <link>https://medium.com/cyberark-engineering/a-developer-guide-for-creating-a-multi-modal-chatbot-using-langchain-agents-9003ba0ffb4d?source=rss-4c106944d383------2</link>
            <guid isPermaLink="false">https://medium.com/p/9003ba0ffb4d</guid>
            <category><![CDATA[dalle-3]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Nir Bar]]></dc:creator>
            <pubDate>Wed, 14 Feb 2024 16:06:16 GMT</pubDate>
            <atom:updated>2024-02-14T19:19:01.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tT9418AfqmNU8i7JCXanSA.png" /><figcaption>Auto-generated using DALL E 3</figcaption></figure><p>In the field of Generative AI, agents have become a crucial element of innovation. They empower Large Language Models (LLMs) to reason better and perform complex tasks such as interfacing with external data sources. This includes performing Google searches, calling external APIs, or generating personalized images. In my previous<a href="https://medium.com/@nirbar_21376/generative-ai-harnessing-gpts-for-multimodal-ai-in-text-and-image-creation-e2fc5913b923"> post</a>, I explained how to create a personalized GPT model using OpenAI’s GPTs, capable of generating images and text. This post shows you how to develop this type of solution, giving you complete control over your chosen LLM. You can handle your proprietary data, calls to external APIs, and more. I have created a multi-modal chatbot that utilizes LangChain, ChatGPT, DALL·E 3, and the Streamlit framework for its user interface. Finally, I will also share the open-source repository I have created, allowing you to explore and deploy the chatbot on your own.</p><h4>The Challenge of Living in the Real World</h4><p>The issue involves retrieving information from the real world that falls outside the training scope of the Large Language Models (LLM). This could anything from executing calls to a proprietary API or supplying the LLM with data (such as files or images) it hasn’t been trained on and then facilitating discussions based on this data. We anticipate that the agent can then deconstruct the task into smaller, more manageable tasks, determining the appropriate tools and the sequence to use them.</p><h4>The Role of Agents in Addressing This Challenge</h4><p>Agents are equipped with various tools, including invoking external APIs, conducting Google searches, or generating images based on specific instructions. These capabilities directly tackle the challenge we’re facing, offering a comprehensive solution. Before we look to solutions, however, it’s crucial first to comprehend the functioning of Agents in the LangChain framework.</p><p>As illustrated by the diagram, the process unfolds in the background: When presented with a user’s task or query, the agent engages the LLM for reasoning, essentially decomposing the task into smaller, intermediate steps. After this, the agent activates the appropriate tool, forwarding its output to the LLM for further analysis. This reasoning cycle continues until the issue is completely resolved and a solution is delivered to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*7nhXywMHoUy5z6rc4jl0og.png" /></figure><h3>How Three Main Tools Offer a Possible Solution</h3><p>The multi-modal chatbot I crafted is backed by an agent that uses three tools:</p><ol><li>REST Countries API Chain: enables retrieving information on countries, invoking this<a href="https://restcountries.com/"> public API</a></li><li>DALL·E 3 Image Generator: Generates an image of countries based on the country name</li><li>Google Search Tool: Useful for fetching information from the web</li></ol><p>I developed the entire chatbot in Python; here is a code snippet of the agent creation:</p><pre>def create_agent():<br><br>    tools = [countries_image_generator, get_countries_by_name, google_search]<br><br>    functions = [convert_to_openai_function(f) for f in tools]<br>    model = ChatOpenAI(model_name=&quot;gpt-3.5-turbo-0125&quot;).bind(functions=functions)<br><br>    prompt = ChatPromptTemplate.from_messages([(&quot;system&quot;, &quot;You are helpful but sassy assistant&quot;),<br>                                               MessagesPlaceholder(variable_name=&quot;chat_history&quot;), (&quot;user&quot;, &quot;{input}&quot;),<br>                                               MessagesPlaceholder(variable_name=&quot;agent_scratchpad&quot;)])<br><br> <br>    memory = ConversationBufferWindowMemory(return_messages=True, memory_key=&quot;chat_history&quot;, k=5)<br><br>    chain = RunnablePassthrough.assign(agent_scratchpad=lambda x: format_to_openai_functions(x[&quot;intermediate_steps&quot;])<br>                                      ) | prompt | model | OpenAIFunctionsAgentOutputParser()<br><br>    agent_executor = AgentExecutor(agent=chain, tools=tools, memory=memory, verbose=True)<br><br>    return agent_executor</pre><p>LangChain framework offers a comprehensive solution for agents, seamlessly integrating various components such as prompt templates, memory management, LLM, output parsing, and the orchestration of these elements within an agent executor.<br>The create_agent() function is at the heart of this approach, designed to instantiate and configure a ChatGPT agent with specific functionalities, integrating external tools and a custom processing pipeline for handling user inputs and generating responses.</p><p>Here’s a breakdown of its components:</p><ul><li><strong>Tools Integration:</strong> Defines a list of tools, which include <em>countries_image_generator</em>, <em>get_countries_by_name</em>, and <em>google_search</em>. These tools are then converted into OpenAI functions, allowing them to be called within the ChatGPT’s processing pipeline.</li><li><strong>Model Configuration: </strong>The function sets up a ChatGPT model, specifically “gpt-3.5-turbo-0125”. This model is enhanced by binding it with the previously converted OpenAI functions, enabling the model to leverage these external tools during its operations.</li><li><strong>Prompt Template: </strong>Creates<strong> </strong>a <em>ChatPromptTemplate</em> using a series of predefined messages, including a mix of system-defined roles and placeholders for dynamic content such as the user’s input, and a scratchpad for intermediate steps. This template guides the conversation flow and structure.</li><li><strong>Memory Management: </strong>Uses <em>ConversationBufferWindowMemory</em> to manage conversation history, storing the last 5 messages (controlled by the parameter k) for context. This memory is indexed by “chat_history” and is configured to return messages for use in generating responses.</li><li><strong>Processing Pipeline:</strong> Defines a processing chain that starts with the <em>RunnablePassthrough</em> for handling intermediate steps, then passes the context through the prepared prompt template, the ChatGPT model itself, and finally through an <em>OpenAIFunctionsAgentOutputParser</em>. This pipeline orchestrates the flow of data through the agent, integrating the model’s output with the function calls and parsing the results. It uses <a href="https://python.langchain.com/docs/expression_language/">LangChain Expression Language</a>, or LCEL, which is a declarative way to compose chains together easily.</li><li><strong>Agent Executor: </strong>The function’s core creates an AgentExecutor, which encapsulates the entire agent along with its tools, memory management, and the defined processing pipeline. This executor runs the agent, handles inputs, and generates outputs based on the configuration.</li></ul><h4>Creating a Custom Tool in LangChain</h4><p>Using <em>@tool</em> decorator is the simplest way to define a custom tool in the LangChain framework. The decorator uses the function name as the tool name by default, which can be overridden by passing a string as the first argument. Additionally, the decorator uses the function’s docstring as the tool’s description, so a docstring MUST be provided.<br>You can also customize the tool name and JSON args by passing them into the tool decorator (see below in the second tool, get_countries_by_name).</p><p>The agent adds the tool description as context to LLM to decide which tool to use. It is crucial to select the appropriate description.</p><p>Let me share my insights on each tool:</p><h4>1) Countries Image Generator tool</h4><pre>@tool<br>def countries_image_generator(country: str):<br>    &quot;&quot;&quot;Call this to get an image of a country&quot;&quot;&quot;<br>    res = DallEAPIWrapper(model=&quot;dall-e-3&quot;).run(<br>        f&quot;You generate image of a country representing the most typical country&#39;s characteristics, incorporating its flag. the country is {country}&quot;<br>    )<br><br>    answer_to_agent = (f&quot;Use this format- Here is an image of {country}: [{country} Image]&quot;<br>                       f&quot;url= {res}&quot;)<br>    return answer_to_agent</pre><p>I used the DallEAPIWrapper to call the DALL·E 3 model and gave it specific instructions on how I wanted the country’s images to be (e.g.,representing the most typical country’s characteristics, incorporating its flag, etc). However, I added output instructions in response to the agent to set a defined format with the country name and the generated image URL. This is crucial for recognizing that the response from the agent is an image, not just text.</p><h4>2) Get Countries By Name tool</h4><pre>def prepare_and_log_request(base_url: str, params: Optional[dict] = None) -&gt; PreparedRequest:<br>    &quot;&quot;&quot;Prepare the request and log the full URL.&quot;&quot;&quot;<br>    req = PreparedRequest()<br>    req.prepare_url(https://medium.com/base_url,%20params)<br>    print(f&#39;\033[92mCalling API: {req.url}\033[0m&#39;)<br>    return req<br><br><br>class Params(BaseModel):<br>    fields: Optional[conlist(str, min_items=1, max_items=27)] = Field(<br>        default=None,<br>        description=&#39;Fields to filter the output of the request.&#39;,<br>        examples=[&quot;name&quot;, &quot;topLevelDomain&quot;, &quot;alpha2Code&quot;, &quot;alpha3Code&quot;, &quot;currencies&quot;, &quot;capital&quot;, &quot;callingCodes&quot;, &quot;altSpellings&quot;, &quot;region&quot;, &quot;subregion&quot;, &quot;population&quot;, &quot;latlng&quot;, &quot;demonym&quot;, &quot;area&quot;, &quot;gini&quot;, &quot;timezones&quot;, &quot;borders&quot;, &quot;nativeName&quot;, &quot;numericCode&quot;, &quot;languages&quot;, &quot;flag&quot;, &quot;regionalBlocs&quot;, &quot;cioc&quot;]<br>    )<br><br><br>class PathParams(BaseModel):<br>    name: str = Field(..., description=&#39;Name of the country&#39;)<br><br><br>class RequestModel(BaseModel):<br>    params: Optional[Params] = None<br>    path_params: PathParams<br><br>@tool(args_schema=RequestModel)<br>def get_countries_by_name(path_params: PathParams, params: Optional[Params] = None):<br>    &quot;&quot;&quot;Useful for when you need to answer questions about countries. Input should be a fully formed question.&quot;&quot;&quot;<br>    BASE_URL = f&#39;https://restcountries.com/v3.1/name/{path_params.name}&#39;<br><br>    effective_params = {&quot;fields&quot;: &quot;,&quot;.join(params.fields)} if params and params.fields else None<br><br>    req = prepare_and_log_request(BASE_URL, effective_params)<br><br>    # Make the request<br>    response = requests.get(req.url)<br><br>    # Raise an exception if the request was unsuccessful<br>    response.raise_for_status()<br><br>    return response.json()</pre><p>Navigating through this particular tool presented a unique challenge. The first step involved designing a model for parameters of REST API that fetches country information by name. I developed a Pydantic model encapsulating path and query parameters to achieve this. Following that, I integrated this model with the LangChain tool decorator.</p><p>The value of this approach lies in its comprehensive submission of the argument model to the Large Language Model (LLM), including detailed descriptions of each argument. It’s not just a brief overview of the tool itself. This method significantly enhances the LLM’s ability to precisely determine the correct arguments for a function based on user prompts, ensuring a more accurate and efficient interaction.</p><h4>3) Google Search tool</h4><pre>@tool<br>def google_search(query: str):<br>    &quot;&quot;&quot;Performs a Google search using the provided query string. Choose this tool when you need to find current data&quot;&quot;&quot;<br>    return SerpAPIWrapper().run(query)</pre><p>For conducting searches on Google through an API, I utilize the built-in SerpAPIWrapper. It’s important to mention that an API key is required for its operation. Learn how to obtain this key here on my <a href="https://github.com/nirbar1985/country-compass-ai/blob/main/README.md">GitHub Readme</a>.</p><h4>The Multi-Modal Chatbot Architecture</h4><p>The diagram illustrates the structure of the multi-modal chatbot system:</p><ol><li><strong>Prompt Refinement.</strong> The initial user prompt and context of the conversation history are forwarded to the LLM (in this scenario, ChatGPT) to refine the prompt into a more precise query.</li><li><strong>Thought Process.</strong> The agent relays the refined prompt, alongside any optional tools, to the LLM for reasoning. Based on this, it decides which tool to employ. If the final answer is determined at this stage, it is directly communicated to the user.</li><li><strong>Tool Invocation</strong>. The agent executes the chosen tool.</li><li><strong>Observation</strong>. The output generated by the tool is sent back to the LLM by the agent for further reasoning.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1Rgf7mtPvPRUEa8Qa161g.png" /></figure><h4>Running the Multi-Modal ChatBot</h4><p>The first question I asked was to create an image of Holland.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7DAi5bW_0WDva_RUf6Oh9Q.png" /></figure><p>It’s amusing how it depicted the bicycle floating on water, right? We also received an image of the Netherlands, capturing its essence and flag.</p><p>Let’s take a peek behind the curtain:</p><p>The agent successfully called upon the “countries_image_generator” tool, inputting the necessary argument: the country’s name. Following that, highlighted in blue, we observe the function’s output : the return value from “countries_image_generator” showcasing the URL for the image crafted by DALL·E 3. Finally, in green, we witness the agent’s conclusive action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uof997V1A8jlZkme1k8ILg.png" /></figure><p>Afterwards, I asked: “How many tourists visited Athens last year?” The response I received was that 6.4 million tourists did. The Google search tool was utilized for this query due to the need for up-to-date information. By conducting a Google search with the query “number of tourists visited Athens last year,” we obtained the answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hpAYI6Sd-6PLn4hP54pxoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gpeLngmWRL1HLOlfbZa8w.png" /></figure><p>I followed up with more inquiries:</p><ul><li>What are the regions and sub-regions of Brazil?</li><li>What is the currency and capital there?</li></ul><p>For both questions, the <em>get_countries_by_name</em> tool was employed with the appropriate parameters. For instance, for the second question, the tool was activated using: <em>get_countries_by_name</em> with the parameters <em>{‘path_params’: {‘name’: ‘Brazil’}, ‘params’: {‘fields’: [‘currencies’, ‘capital’]}}</em>.</p><p>In addition, this multi-modal chatbot can recall previous interactions, as demonstrated by its capacity to deduce the correct country for the subsequent question regarding currency and capital.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S31I0QG2tLrmoLRYjmJAjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RObLd6f1PLZR5_5474dR4w.png" /></figure><p>Finally, I asked it to create an image of Greece:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xd5a3BqBi4qFpCQHUAxAZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdVCn_fkNRa-xHqw4BiOxQ.png" /></figure><h4>Utilizing Agents for Real-World Problem Solving</h4><p>You should now have a better understanding of the crucial role of agents in developing robust applications, and specifically how a language model can function as a cognitive engine. It intelligently decides on the sequence of actions to execute, facilitating the integration of the Large Language Model (LLM) with various external resources, including APIs, proprietary datasets, and internet searches, among other potential actions. We also showed you the importance of the LangChain framework in constructing a versatile chatbot. This chatbot is not only capable of calling external APIs and browsing the internet but can also create images, showcasing its multi-modal capabilities. (You can learn more in the <a href="https://github.com/nirbar1985/country-compass-ai">Github repository</a>).</p><p>Here’s the basic takeaway: Make sure to select the appropriate tools and a compatible LLM that enables function calls to tailor your application to your specific needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9003ba0ffb4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyberark-engineering/a-developer-guide-for-creating-a-multi-modal-chatbot-using-langchain-agents-9003ba0ffb4d">A Developer Guide for Creating a Multi-Modal Chatbot Using LangChain Agents</a> was originally published in <a href="https://medium.com/cyberark-engineering">CyberArk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing OpenAI’s GPTs for Multimodal AI in Text and Image Creation]]></title>
            <link>https://medium.com/@nirbar_21376/generative-ai-harnessing-gpts-for-multimodal-ai-in-text-and-image-creation-e2fc5913b923?source=rss-4c106944d383------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2fc5913b923</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Nir Bar]]></dc:creator>
            <pubDate>Thu, 07 Dec 2023 20:04:28 GMT</pubDate>
            <atom:updated>2023-12-08T11:48:04.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*_F5NtEcFIllKjZDCCmdr-Q.png" /></figure><p>In the following post, I will share the steps to craft a customized GPT using OpenAI GPTs, which can create images and text.<br>And by the way, it took me just 2 hours, crazy, right?<br>One of the most exciting announcements made on the first OpenAI developer day, held on November 6, 2023, was the release of GPTs.<br>GPTs allow us to create custom versions of ChatGPT that combine instructions, extra knowledge, and any combination of skills.<br>These GPTs offer a quick and easy way to build a ChatGPT extension through a “no-code” platform, simplifying the development of complex multimodal chatbots.</p><p>First, let’s understand what a multimodal chatbot is.<br>Multimodal chatbots are advanced types of chatbots that can interact with users through multiple modes of communication. Unlike traditional chatbots that primarily rely on text input and output, multimodal chatbots can understand and respond using a variety of inputs and outputs, such as text, voice, images, and videos.</p><p>GPTs are a new way for anyone to create a tailored version of ChatGPT to be more helpful in their daily life, at specific tasks, at work, or at home — and then share that creation with others. For example, GPTs can help you <a href="https://openai.com/chatgpt#do-more-with-gpts">learn the rules to any board game, help teach your kids math, or design stickers</a>.</p><p>Creating one is as easy as starting a conversation, giving it instructions and extra knowledge, and picking what it can do, like searching the web, making images, or analyzing data.</p><p>The following capabilities are supported in GPTs —</p><ul><li>DALL·E 3 — newest text-to-image model</li><li>GPT-4 with Vision — GPT-4 can accept images as inputs and generate captions, classifications, and analyses.</li><li>Web browsing</li><li>Code interpreter using Python sandbox for data analysis</li></ul><p>Moreover, you can add your proprietary APIs that this GPT can use.</p><p>This concept is similar to what has stemmed from open-source projects like Agents in LangChain, a popular framework for building LLM applications.<br>Please find the Agents section in my previous <a href="https://medium.com/p/d36b8f2d3c95">blog post</a> to elaborate more.<br>The main difference with GPTs is that you don’t need coding skills.<br><br>I utilized a Rest API that is publicly available to gather information about different countries such as their flags, currencies, and borders. This data was then used to create personalized images using the customized ChatGPT. Furthermore, I set it up to fetch current information from the internet.</p><p>You can find the public API of countries <a href="https://restcountries.com/">here</a></p><p>Creating a custom ChatGPT has several steps:</p><h3>1) Configuration using GPT Builder</h3><p>My mission was to ask questions about a country, including its flag, and to ask ChatGPT to generate an image representing the country’s characteristics, including its flag.<br>Moreover, I wanted it to be able to search the web, so I asked it to provide current data, such as population statistics.<br>After a small chat with the GPT Builder bot (under the Create tab), describing my purpose, it created for me the following parameters of my customed ChatGPT —</p><ul><li>Name — Geo Explorer</li><li>Description — Assists in leveraging REST APIs for country data and visualizing it.</li><li>Instructions — Geo Explorer is designed to assist users in creating applications that use public REST APIs for detailed country data, including flags, currency, and borders. It generates images representing a country’s characteristics, incorporating its flag, and provides current data such as population statistics. Geo Explorer guides users in accessing and using these APIs effectively, offering both detailed explanations and concise answers based on user preference. It emphasizes accuracy and up-to-date information, avoiding speculative data. Clarification is provided as needed to understand specific user requirements. Responses are tailored to be both informative and visually engaging, suitable for inquiries about various countries.</li><li>Chosen Capabilities — Web Browsing and DALL·E Image Generation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*rdPTiKIP0cnsgIorzTkn9g.png" /></figure><h3>2) Create actions</h3><p>I needed to enter an openAPI schema, but this <a href="https://restcountries.com/">API</a> doesn’t have it.<br>So I overcame this issue by asking ChatGPT to provide me with one, giving it as a context the relevant code from <a href="https://github.com/SteinRobert/python-restcountries/blob/main/restcountries/base.py">here</a></p><p>So now I can provide the following spec file-</p><pre>openapi: 3.0.0<br>info:<br>  title: RestCountry API v2<br>  version: 1.0.0<br>  description: API for retrieving information about countries by name<br>servers:<br>  - url: https://restcountries.com/v2<br>paths:<br>  /name/{name}:<br>    get:<br>      summary: Get countries by name<br>      operationId: getCountriesByName<br>      parameters:<br>        - name: name<br>          in: path<br>          required: true<br>          schema:<br>            type: string<br>          description: Name of the country<br>        - name: filters<br>          in: query<br>          required: false<br>          schema:<br>            type: array<br>            items:<br>              type: string<br>          description: Fields to filter the output of the request<br>      responses:<br>        200:<br>          description: An array of countries<br>          content:<br>            application/json:<br>              schema:<br>                type: array<br>                items:<br>                  $ref: &#39;#/components/schemas/Country&#39;<br>        404:<br>          description: Country not found<br>components:<br>  schemas:<br>    Country:<br>      type: object<br>      properties:<br>        name:<br>          type: string<br>        topLevelDomain:<br>          type: array<br>          items:<br>            type: string<br>        alpha2Code:<br>          type: string<br>        alpha3Code:<br>          type: string<br>        currencies:<br>          type: array<br>          items:<br>            type: object<br>            properties:<br>              code:<br>                type: string<br>              name:<br>                type: string<br>              symbol:<br>                type: string<br>        capital:<br>          type: string<br>        callingCodes:<br>          type: array<br>          items:<br>            type: string<br>        altSpellings:<br>          type: array<br>          items:<br>            type: string<br>        region:<br>          type: string<br>        subregion:<br>          type: string<br>        population:<br>          type: integer<br>        latlng:<br>          type: array<br>          items:<br>            type: number<br>        demonym:<br>          type: string<br>        area:<br>          type: number<br>        gini:<br>          type: number<br>        timezones:<br>          type: array<br>          items:<br>            type: string<br>        borders:<br>          type: array<br>          items:<br>            type: string<br>        nativeName:<br>          type: string<br>        numericCode:<br>          type: string<br>        languages:<br>          type: array<br>          items:<br>            type: object<br>            properties:<br>              iso639_1:<br>                type: string<br>              iso639_2:<br>                type: string<br>              name:<br>                type: string<br>              nativeName:<br>                type: string<br>        flag:<br>          type: string<br>        regionalBlocs:<br>          type: array<br>          items:<br>            type: object<br>            properties:<br>              acronym:<br>                type: string<br>              name:<br>                type: string<br>        cioc:<br>          type: string</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*Z-GuoGSmMjs2wjIw4Cn7Xg.png" /></figure><p>I did not add any authentication methods since it is a POC.<br>In case you are adding a Rest API for corporate purposes, I suggest adding it. Moreover, Enterprise customers will be able to deploy internal-only GPTs via ChatGPT Enterprise</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*znbX7EUEczIbk8NFHbQrPQ.png" /></figure><h3>3) Adding Knowledge</h3><p>I didn’t need to enrich ChatGPT with proprietary data, so I didn’t utilize the feature of uploading files to the model.</p><p>Now we are ready to go since the action is set up :)</p><h3>Testing Geo Explorer GPT</h3><p>The first question is, which countries border with Brazil?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*ylc_uTomj3jyiBNK5-yCcg.png" /></figure><p>We see that a request is sent to the API with the correct format, and we get an answer —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*7NCycigV8Q3_is1GeyOsNA.png" /></figure><p>Second question — create a picture of Brazil</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*o2pNIHFRQ-SROY0zzYuF4w.png" /></figure><p>We can see that this customed ChatGPT followed our guidelines and generated an image representing a country’s characteristics, incorporating its flag.</p><p>Third question — how many tourists visited Germany in the last month?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*oKmtclBjU1MsfvaYMfcW5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*Vc6rJeg8uPzr8fWEfKrrEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*-YZEqjyxidM5jBkyjotWKg.png" /></figure><p>We see that this time, the tool that the LLM invoked was Bing search, giving us answers on current data.</p><p>Lastly, I asked for a picture of Japan —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*FIkafHvGCAq-6Fv4BKfd1w.png" /></figure><h3>Summary</h3><p>The new product of GPTs is convenient and easy to use.<br>It allows us to enrich the LLM with data from proprietary APIs or feed it with additional files using the knowledge mechanism.<br>All of this is done without using one line of code, which is fantastic.<br>In my next blog post, I will cover how to create a GenAI application with functionality similar to the one I crafted here, using Agents and tools using the LangChain framework.</p><h3>Reference</h3><p><a href="https://openai.com/blog/introducing-gpts">OpenAI GPTs Documentation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2fc5913b923" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Empowering LLMs using the LangChain Framework]]></title>
            <link>https://medium.com/cyberark-engineering/empowering-llms-using-the-langchain-framework-d36b8f2d3c95?source=rss-4c106944d383------2</link>
            <guid isPermaLink="false">https://medium.com/p/d36b8f2d3c95</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Nir Bar]]></dc:creator>
            <pubDate>Wed, 29 Nov 2023 15:26:34 GMT</pubDate>
            <atom:updated>2023-11-29T19:13:45.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GIzUtnvBM7dkfG1I6xnkNw.jpeg" /><figcaption>Auto-generated using DALL E 2</figcaption></figure><p>In the past year, we have witnessed a massive hype surrounding Generative AI (GenAI), largely fueled by ChatGPT. LangChain has risen as the go-to framework for creating GenAI applications. LangChain allows us to create super complex applications that can search the web, access databases and leverage the reasoning capabilities of Large Language Models (LLMs) to perform autonomous tasks.</p><p>If as a developer, you’ve ever wondered:</p><ul><li>How do plugins in ChatGPT Plus work underneath the hood?</li><li>How can I develop a GenAI app that executes complex tasks based on your prompts, such as initiating a Google search, interacting with SQL agents, or accessing external APIs?</li></ul><p>Then this post is definitely for you! We will be covering why you should choose the LangChain framework, explore LangChain in numbers, go over its core modules and talk about its use cases.</p><h3>Why Should You Choose LangChain?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*co-MT9hgXMLS70nplUE6YA.png" /></figure><p>We all know that ChatGPT has impressive general knowledge. You can ask it almost anything, and you will probably get a pretty good answer. But let’s suppose you want to know something specifically from your data – a book, a file, or a database (DB) with proprietary information. LangChain allows you to connect an LLM like ChatGPT (or Hugging Face) to your specific data. It’s not limited to pasting a text snippet to a ChatGPT prompt; it can reference an entire DB filled with your data.</p><p>LangChain also provides the ability to take action based on that data. For example, actions such as sending an email, performing a Google search, and calling any external API that you want.</p><h3>Langchain in Numbers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AECa8hJMpgRlsmukhlgWw.png" /></figure><p>LangChain was released in October 2022 by Harrison Chase as an open source framework and was quickly adopted by the community as the go-to framework for creating GenAI applications. It has more than 60,000 stars in Github, a lot more than its competitor LlamaIndex, and even more than traditional open source frameworks like AngularJS. It has more than 1.2 million PyPi downloads a week and 165,000 npm weekly downloads.<br>LangChain has a high score of 88 in the Snyk scan and doesn’t contain any vulnerabilities.</p><h3>Core Modules of LangChain</h3><p>The screenshot below displays several key core modules within LangChain. Chains and Agents bring together elements like LLM Wrappers, Prompt Templates, Indexes, Memory, and more, using them as fundamental building blocks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TxZEKYMK4RYuyws2PoGEjA.png" /></figure><p>For <a href="https://python.langchain.com/docs/modules/">each of these modules</a>, LangChain provides standard and extendable interfaces:</p><ul><li><em>LLM Wrappers — </em>Enables connection to LLMs/Chat models.</li><li><em>Prompts Templates — </em>Enables avoiding hard code text, which is the input to the LLM. It is cleaner than just using ‘f’ strings in Python and lets you provide only the user prompts when invoking the chain or the agent.</li><li><em>Indexes — </em>Allows you to extract the relevant information from your data.</li></ul><h3>Use Cases for LangChain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*QZcH_x-m_V8TMWYVCTYuGQ.png" /><figcaption>Icons by Freepik</figcaption></figure><p>Combining Langchain with LLMs has many use cases.</p><p>Some of them, such as QA over documents and Code Understanding, usually first require ingestion of data which the LLM was not trained on. This data must be ingested to a vector database and only then can the relevant data be retrieved and augmented in the prompt to the LLM.</p><p>Other use cases generally require feeding the LLM with API documentation schema, such as Interacting with APIs or Agents in order to be able to call external sources such as APIs.</p><p>The use cases are:</p><h4><strong>1) QA over Documents</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ucuq2ub3HQGHOpZAm-IVnQ.png" /></figure><p>Let’s consider you are using ChatGPT API, which currently has a token limit of 4,096 tokens. And you want to ask questions about a book with 200 pages.</p><p>How do we know what text should be sent from 200 pages to LLM to extract specific entities? It might be on any page/part of the document!</p><p>That is where text chunking and similarity search help to send only relevant text to LLMs to reduce tokens, thereby reducing costs.</p><p>Now that we have a better understanding of the motivation of the data ingestion phase, let’s go over the steps:</p><p>1. Load the data from any kind (images, pdf, YouTube, S3 bucket, etc.) to the document object in LangChain. You can find the updated integration list<a href="https://integrations.langchain.com/"> here</a>.</p><p>2. Split documents into small chunks so that the LLM can later digest them in the prompt (without reaching the maximum context limit).</p><p>3. Chunks are inserted into a vector database after being transformed into vectors of floats, which are also called embeddings.</p><p>Cool, so now we have our proprietary data ingested in a database, but why do we need to store it in embeddings and not in text?</p><p>The following images try to answer this question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7Xqr3uxOXi9yVf-soJwAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vs1BCCDUpspNVYovfuPGIg.png" /></figure><p>For example, I took this great<a href="https://lilianweng.github.io/posts/2023-06-23-agent/"> blog</a> post, ingested it to a vector DB locally on my server, and started chatting on its data. ChatGPT was not trained on this data since it was posted on June 23, 2023.</p><p>I asked three questions :</p><ul><li>What is sensory memory?</li><li>Which memory provides the ability to retain impressions of sensory information?</li><li>Which type of memory typically only lasts for up to a few seconds?</li></ul><p>We got answers that contain similar content. The reason for that is because <br>prompts are first transformed into vectors (or embeddings). Since they are semantically close in the vector space, the retrieval step from the vector database returns similar neighbor chunks, thus giving the user similar answers.</p><h4>2) Chatbots</h4><p>Chatbots are one of the central LLM use cases. The core features of chatbots are that they can have long-running conversations and have access to information that users want to know about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMakD5rMh8ye4yDE0V2JwA.png" /></figure><p>The above diagram illustrates the logic of the LangChain chain called ChatbotConversationalRetrievalChain.</p><p>The chain contains 3 phases:</p><p>1. Use the chat history and the prompt to create a standalone question.<br>2. The standalone question is passed to the retriever, and only relevant documents are returned. <br>3. Augmenting the prompt with those documents and the standalone question-&gt; and getting a response back.</p><p>This is done using the Retrieval Augmented Generation (RAG) method.</p><p>You can elaborate more on it<a href="https://ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/"> here</a>.</p><h4>3) Agents</h4><p>If we can create a chatbot on top of an LLM and chat over our data, why do we need agents?</p><p>The reason is that in chains in LangChain, the sequence of actions is hardcoded (in code), and in Agents, the LLM is used as a reasoning engine to determine which actions to take and in which order. This empowers the LLMs and lets us create much more complex applications.</p><p>This became feasible due to two pivotal studies (you can read them <a href="https://arxiv.org/pdf/2201.11903v1.pdf">here</a> and <a href="https://arxiv.org/pdf/2210.03629.pdf">here</a>) that played a key role in enhancing the capability of LLMs to execute more complex tasks.</p><p>In simple words:</p><p>Guiding the LLM to solve problems like we do as humans means breaking a complex task into small intermediate tasks. In each step, after the LLM chooses which tool to call, LangChain invokes it and returns with the response to the LLM until the problem is resolved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muEDyqhRKjf4IPll3KPjfA.png" /></figure><p>We can see in the above image that the prompt was :</p><p><em>How many people live in Canada as of 2023? return the number raised by the power of 2 plus 5</em></p><p>Before the agent could resolve the task using the LLM, the Google Search tool was invoked with the following input:</p><p><em>current population of Canada 2023</em></p><p>Then, the answer to the Google Search was :<em>38,781,291 people</em></p><p>The LLM did reasoning again and decided on the next tool to be invoked — a calculator with the following input — <em>(38,781,291²) + 5</em></p><p>and then we get the final answer : <em>1503988531626686</em></p><p>The following screenshot tries to explain what happens behind the scenes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3Kca5Kifn1LkJ0CeOutsg.png" /></figure><p>There is the prompt of the user, and the LangChain agent calls the LLM to perform the reasoning, or break the task into small intermediate tasks. In this case it involves first invoking the Google Search tool and taking its output and invoking the Calculator tool, thus giving us the final resolution.</p><h4>4) Interacting with APIs</h4><p>Suppose you want an LLM to interact with external APIs. This is very useful for retrieving context for the LLM to utilize. More importantly, it allows us to interact with APIs using natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8gS0qOAU17SmHzBnzO2yg.png" /></figure><p>The concept is to provide an LLM with API documentation (for example, open API spec) and then ask questions in natural language. The LLM then concludes the REST API call and performs the call, answering the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3BdQf1aYzxw9Rg5jgqEHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBWvA60EMUPHPZskhDybYQ.png" /></figure><p>You can learn more about interacting with APIs <a href="https://python.langchain.com/docs/use_cases/apis">here</a>.</p><h4>5) Tagging</h4><p>Tagging means labeling a document with classes such as:</p><ul><li>sentiment</li><li>language</li><li>style (formal, informal, etc.)</li><li>covered topics</li><li>political tendency</li></ul><p>Tagging has many use cases, such as analyzing customer cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_lQyrW-YlCvfstLVmdqNA.png" /></figure><p>To demonstrate the behavior in the screenshot above, I used the LangChain tagging chain that extracts information from a passage based on a schema.</p><p>I ran the exact scenario from the screenshot above, and this is what I received as an answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5b2Rdtp7dnub7Bo5ZlGSdw.png" /></figure><p>You can learn more about tagging <a href="https://python.langchain.com/docs/use_cases/tagging">here</a>.</p><h4>6) Summarization</h4><p>Suppose you have a set of documents (PDFs, Notion pages, customer questions, etc), and you want to summarize the content. LLMs are a great tool for this, given their proficiency in understanding and synthesizing text.</p><p>As an example, I took this great<a href="https://lilianweng.github.io/posts/2023-06-23-agent/"> blog</a> post and used the LangChain load summarizing chain, which is provided off the shelf, and asked it to summarize the content of this blog.</p><p>Here is the answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NzdShUBEqiwz7cr5iVrx0g.png" /></figure><p>Nice right?</p><p>You can learn more about summarization <a href="https://python.langchain.com/docs/use_cases/summarization">here</a>.</p><h4>7) SQL Using Natural Language</h4><p>Enterprise data is often stored in SQL databases and LLMs make it possible to interact with SQL databases using natural language. LangChain offers SQL Chains and Agents to build and run SQL queries based on natural language prompts.</p><p>These are compatible with any SQL dialect supported by SQLAlchemy (e.g., MySQL, PostgreSQL, Oracle SQL, Databricks, SQLite).</p><p>They enable use cases such as:</p><ul><li>Generating queries that run based on natural language questions</li><li>Creating chatbots that can answer questions based on database data</li><li>Building custom dashboards based on insights a user wants to analyze</li></ul><p>You can learn more about SQL using natural language <a href="https://python.langchain.com/docs/use_cases/qa_structured/sqlg">here</a>.</p><h3><strong>The Generative AI Revolution</strong></h3><p>We are witnesses to a GenAI revolution that will transform many aspects of our lives. While the industrial revolution was primarily about advancements in tools and processes, the AI revolution is focused on the replacement of human skills. Using the LangChain framework, we can embrace this change and create really powerful applications powered by LLMs such as image generation, chatbots, sentiment analysis and code generation and many more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d36b8f2d3c95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyberark-engineering/empowering-llms-using-the-langchain-framework-d36b8f2d3c95">Empowering LLMs using the LangChain Framework</a> was originally published in <a href="https://medium.com/cyberark-engineering">CyberArk Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>