<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinayak Shanawad on Medium]]></title>
        <description><![CDATA[Stories by Vinayak Shanawad on Medium]]></description>
        <link>https://medium.com/@vinayakshanawad?source=rss-609a97998ce2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3YWGfUvxCN2_Owr2eSM87Q.jpeg</url>
            <title>Stories by Vinayak Shanawad on Medium</title>
            <link>https://medium.com/@vinayakshanawad?source=rss-609a97998ce2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:48:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinayakshanawad/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build an AI Travel Assistant (Part 3): Hierarchical Workflow with a Router Agent]]></title>
            <link>https://levelup.gitconnected.com/build-an-ai-travel-assistant-part-3-hierarchical-workflow-with-a-router-agent-aaa5f420cf7b?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/aaa5f420cf7b</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-agent-protocols]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 14:30:29 GMT</pubDate>
            <atom:updated>2025-08-18T03:06:26.186Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to create a router agent to dynamically manage agent interactions for a cohesive travel planning experience using ACP</h4><h4>Introduction</h4><p>In <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>, we built a <strong>Travel Package Agent</strong> using <strong>CrewAI</strong> to answer queries about Europe travel packages, wrapping it in an <strong>Agent Communication Protocol (ACP)</strong> server for interoperability. In <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-part-2-enhancing-travel-planning-with-smolagents-mcp-and-acp-8cec81e07f17">Part 2</a>, we extended the Travel Planning MVP by adding a <strong>Travel Guidance Agent</strong> using <strong>Smolagents</strong>, integrating it with a <strong>Model Context Protocol (MCP)</strong> server for tool access, and demonstrating sequential agent chaining. In this third article, we’ll enhance our MVP by introducing a <strong>Router Agent</strong> using the ACPCallingAgent to dynamically manage agent interactions, replacing the fixed sequential workflow with a hierarchical approach. By the end, we’ll have a system that intelligently routes queries to the appropriate agents for a seamless travel planning experience.</p><p>This article assumes we’ve followed Parts 1 and 2, with the travel_acp_project set up, including dependencies, PDF brochures in the data folder, and both ACP servers running. Let’s dive into building the router agent!</p><h3>Table of Contents</h3><p>1. <a href="#ee7c">What is a Router Agent?</a><br>2. <a href="#4fba">Objective</a><br>3. <a href="#beb7">Architecture Overview</a><br>4. <a href="#2c8e">Starting the ACP Servers</a><br>5. <a href="#fbcf">Implementing the Router Agent</a><br>6. <a href="#fccb">Running the Hierarchical Workflow</a><br>7. <a href="#de6f">Next Steps</a><br>8. <a href="#2b1f">Acknowledgements</a><br>9. <a href="#b407">Conclusion</a></p><h4>1. What is Router Agent?</h4><p>A <strong>Router Agent</strong> acts as an intelligent coordinator that analyzes user queries and dynamically selects the appropriate specialized agent to handle each part of the request. Unlike the sequential chaining in Part 2, where we manually defined the order of agent calls, the router agent uses the ACPCallingAgent from the fastacp.py module to break down complex queries into sub-goals and route them to the relevant agents (travel_agency_agent, travel_research_agent, or travel_package_agent). This approach enhances flexibility and efficiency, as the router decides the best agent for each task based on the query context.</p><p>The router leverages <strong>ACP</strong> for standardized communication and <strong>MCP</strong> for tool access, ensuring seamless integration with the agents built in Parts 1 and 2.</p><h4>2. Objective</h4><p>This article guides you through implementing a <strong>Router Agent</strong> to create a hierarchical workflow for the Travel Planning MVP. The router will:</p><ul><li>Discover agents hosted on the Travel Guidance ACP Server (port 8000) and Travel Package ACP Server (port 8001).</li><li>Use the ACPCallingAgent to intelligently route user queries to the appropriate agent.</li><li>Handle a multi-part query (e.g., finding travel agencies, determining the best time to travel, and listing packages) in a dynamic, context-aware manner.</li><li>Demonstrate the workflow with a sample query and output.</li></ul><p>This completes the core components of our Travel Planning MVP, showcasing the power of ACP for dynamic agent coordination.</p><h4>3. Architecture Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBhS7b9gE8udmayB7VqzFg.png" /><figcaption>ACP Application managing the hierarchical workflow (Image by Author)</figcaption></figure><p><strong>Components</strong>:</p><ul><li><strong>User Query</strong>: Initiates the workflow via 4-Hierarchical-Chaining.ipynb notebook.</li><li><strong>ACP Client</strong>: Runs the router agent and connects to ACP servers.</li><li><strong>Router Agent (ACPCallingAgent)</strong>: Analyzes queries, routes tasks to appropriate agents, and aggregates responses.</li><li><strong>Travel Guidance ACP Server</strong>: Hosts travel_research_agent (web searches) and travel_agency_agent (MCP tool).</li><li><strong>Travel Package ACP Server</strong>: Hosts travel_package_agent(RAG-based).</li><li><strong>MCP Server</strong>: Provides the list_travel_agencies tool via MCP.</li><li><strong>OpenAI LLM</strong>: Powers reasoning for all agents.</li><li><strong>Web Tools and PDFs</strong>: Serve as data sources.</li></ul><p>This architecture highlights the router’s role in dynamic coordination, enhancing modularity and scalability.</p><h4>4. Starting the ACP Servers</h4><p>Before running the router agent, ensure both ACP servers from Parts 1 and 2 are active:</p><ul><li><strong>Travel Package ACP Server</strong> (from <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>, hosting travel_package_agent): uv run python src/crew_agent_server.py<br>Runs on <a href="http://localhost:8001.">http://localhost:8001.</a></li><li><strong>Travel Guidance ACP Server</strong> (from <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-part-2-enhancing-travel-planning-with-smolagents-mcp-and-acp-8cec81e07f17">Part 2</a>, hosting travel_research_agent and travel_agency_agent):uv run python src/smolagents_server.py<br>Runs on <a href="http://localhost:8000.">http://localhost:8000.</a></li></ul><p>If these servers are already running from previous notebooks, we can skip this step.</p><h4>5. Implementing the Router Agent</h4><p>The ACPCallingAgent in fastacp.py serves as the router agent, dynamically calling other agents based on the query. We’ll create a script to initialize the router and run the hierarchical workflow.</p><p><strong>5.1 Define the Hierarchical Workflow</strong></p><p>Create 4-Hierarchical-Chaining.ipynb notebook to set up the router agent and execute the workflow:</p><pre>import asyncio<br>import nest_asyncio<br>from acp_sdk.client import Client<br>from smolagents import LiteLLMModel<br>from fastacp import AgentCollection, ACPCallingAgent<br>from colorama import Fore<br><br>nest_asyncio.apply()<br><br>model = LiteLLMModel(<br>    model_id=&quot;openai/gpt-4&quot;<br>)<br><br>async def run_travel_workflow() -&gt; None:<br>    # Step 1: Connect to both ACP servers<br>    async with Client(base_url=&quot;http://localhost:8001&quot;) as package, Client(base_url=&quot;http://localhost:8000&quot;) as guidance:<br>        # Step 2: Discover all agents from both servers<br>        agent_collection = await AgentCollection.from_acp(package, guidance)<br>        acp_agents = {<br>            agent.name: {&#39;agent&#39;: agent, &#39;client&#39;: client}<br>            for client, agent in agent_collection.agents<br>        }<br><br>        print(Fore.CYAN + &quot;Discovered ACP Agents:\n&quot; + &quot;\n&quot;.join(acp_agents.keys()) + Fore.RESET)<br><br>        # Step 3: Initialize the Router Agent (ACPCallingAgent)<br>        router = ACPCallingAgent(acp_agents=acp_agents, model=model)<br><br>        # Step 4: Run the full query<br>        query = (<br>            &quot;I’m based in Hyderabad. Which travel agencies can help me? &quot;<br>            &quot;When is the best time to visit Europe? &quot;<br>            &quot;Are there any packages from Hyderabad that include Switzerland?&quot;<br>        )<br><br>        result = await router.run(query)<br>        print(Fore.YELLOW + f&quot;\nFinal result:\n{result}&quot; + Fore.RESET)<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(run_travel_workflow())</pre><p>This code:</p><ul><li>Uses nest_asyncio to handle nested event loops in Jupyter environments.</li><li>Connects to both ACP servers using Client objects.</li><li>Discovers agents (travel_agency_agent, travel_research_agent, travel_package_agent) using AgentCollection.from_acp.</li><li>Initializes the ACPCallingAgent as the router with the discovered agents and a GPT-4 model.</li><li>Runs a multi-part query, allowing the router to dynamically delegate tasks.</li></ul><p><strong>5.2 How the Router Works</strong></p><p>The ACPCallingAgent:</p><ol><li><strong>Analyzes the Query</strong>: Breaks down the user’s query into sub-goals (e.g., agency lookup, travel research, package details).</li><li><strong>Routes Tasks</strong>: Calls the appropriate agent (travel_agency_agent for agencies, travel_research_agent for travel advice, travel_package_agent for packages) via ACP.</li><li><strong>Accumulates Context</strong>: Stores responses in memory (e.g., travel_agency_agent_response) to inform subsequent agent calls.</li><li><strong>Delivers Final Answer</strong>: Combines results into a structured response using the final_answer tool.</li></ol><h4>6. Running the Hierarchical Workflow</h4><p>Run the workflow: in4-Hierarchical-Chaining.ipynb notebook.</p><p><strong>Sample Output</strong></p><p>The router processes the query in steps, producing:</p><pre>Discovered ACP Agents:<br>travel_package_agent<br>travel_research_agent<br>travel_agency_agent<br><br>Final result:<br>Here are some travel agencies based in Hyderabad that you can consider:<br>1. GT Holidays Hyderabad: Provides domestic, international, and European packages. [Website](https://www.gtholidays.in/hyderabad/)<br>2. RL Tours &amp; Travels: Offers custom European, Asian, and domestic packages. [Website](https://rltours.in/)<br>3. IMAD Travel: Specialists in luxury international packages including European trips. [Website](https://imadtravel.com/)<br>4. StarEdge Holidays: Provides international packages, group tours.<br>5. Thrillophilia: Known for adventure experiences, multi-day tours, and international packages including Switzerland. [Website](https://www.thrillophilia.com)<br><br>The best time to visit Europe generally is during the spring (April–June) and autumn (September–October) seasons when the weather is good, prices are lower, and crowds are manageable.<br><br>Here are some travel packages from Hyderabad to Switzerland:<br>1. &quot;Best of Switzerland &amp; Italy&quot;: 9 days &amp; 8 nights for INR 1,43,200<br>2. &quot;Scandinavian Snow Escape&quot;: 9 days &amp; 8 nights for INR 2,91,000<br>3. &quot;Europe Golden Trio&quot;: 8 days &amp; 7 nights for INR 1,73,000<br>4. &quot;Classic Europe in Winter&quot;: 13 days &amp; 12 nights for INR 2,55,000<br>5. &quot;Western Europe In A Nutshell&quot;: 11 days &amp; 10 nights for INR 1,72,000<br>6. &quot;Classic Switzerland&quot;: 7 days &amp; 6 nights for INR 1,25,900<br><br>These packages are offered by Thrillophilia. They also offer discounts of up to 40% for different itineraries which include Group Tours, Family Tours, and Honeymoon tour options. Please check the recent prices and availability on their website before booking.</pre><p>The output shows the router agent:</p><ul><li>Calling travel_agency_agent to list Hyderabad-based agencies.</li><li>Calling travel_research_agent to recommend spring and autumn for Europe travel.</li><li>Calling travel_package_agent to detail Switzerland-inclusive packages.</li><li>Combining results into a cohesive response.</li></ul><h4>7. Next Steps</h4><p>We’ve built a hierarchical workflow with a router agent, completing the core of the Travel Planning MVP. To further enhance this system:</p><ul><li><strong>Add More Agents</strong>: Integrate agents for flight bookings or hotel recommendations using additional MCP tools.</li><li><strong>Optimize Routing</strong>: Enhance the router’s logic to handle more complex queries or prioritize agents based on performance.</li><li><strong>Deploy to Production</strong>: Use ACP’s scalability features to host the system on a cloud platform.</li></ul><h4>8. Acknowledgements</h4><p>This article was inspired by the resources provided by the <a href="https://agentcommunicationprotocol.dev/introduction/welcome">Agent Communication Protocol</a> (ACP) initiative and <a href="https://learn.deeplearning.ai/courses/acp-agent-communication-protocol/lesson/yg0qt/introduction">DeepLearning.AI</a>. Special thanks to IBM and DeepLearning.AI team. These resources were instrumental in shaping the understanding and implementation of the Travel Package Agent MVP.</p><h4>9. Conclusion</h4><p>In this third part of our series, we enhanced our Travel Planning MVP by introducing a <strong>Router Agent</strong> using the ACPCallingAgent to dynamically manage agent interactions. By leveraging <strong>ACP</strong> for standardized communication and <strong>MCP</strong> for tool access, we created a hierarchical workflow that intelligently routes queries to specialized agents, delivering a cohesive travel planning experience. This completes the core components of our MVP, showcasing the power of ACP and MCP in building modular, scalable AI systems. Follow these steps to create our own router-driven travel assistant and explore the future of AI agent ecosystems.</p><p>I hope you enjoyed this article, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this article is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/Travel-Planning-ACP">GitHub - vinayak-shanawad/Travel-Planning-ACP</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aaa5f420cf7b" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/build-an-ai-travel-assistant-part-3-hierarchical-workflow-with-a-router-agent-aaa5f420cf7b">Build an AI Travel Assistant (Part 3): Hierarchical Workflow with a Router Agent</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an AI Travel Assistant (Part 2): Enhancing Travel Planning with Smolagents, MCP, and ACP]]></title>
            <link>https://levelup.gitconnected.com/build-an-ai-travel-assistant-part-2-enhancing-travel-planning-with-smolagents-mcp-and-acp-8cec81e07f17?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cec81e07f17</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent-protocols]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 07:53:21 GMT</pubDate>
            <atom:updated>2025-08-18T03:06:28.225Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to create a Travel Guidance Agent with web search capabilities and integrate it with MCP and ACP for a robust travel planning ecosystem</h4><h4>Introduction</h4><p>In <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>, we built a <strong>Travel Package Agent</strong> using <strong>CrewAI</strong> and its Retrieval-Augmented Generation (RAG) capabilities to answer queries about Europe travel packages, wrapping it in an <strong>Agent Communication Protocol (ACP)</strong> server for seamless interoperability. In this second article, we’ll extend our Travel Planning MVP by creating a <strong>Travel Guidance Agent</strong> using <strong>Smolagents</strong>, integrating it with a <strong>Model Context Protocol (MCP)</strong> server for tool access, and hosting it in an ACP server. We’ll also demonstrate how to chain multiple agents sequentially to create a cohesive travel planning workflow. By the end, we’ll have a system that answers general travel queries, finds local travel agencies, and integrates with the Travel Package Agent from <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>.</p><p>This article assumes you’ve followed <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a> and have the travel_acp_project set up with the necessary dependencies and PDF brochures in the data folder. Let’s dive into building the next piece of our Travel Planning MVP!</p><h3>Table of Contents</h3><p>1. <a href="#98b8">What is Smolagents and MCP?</a><br>2. <a href="#551d">Objective</a><br>3. <a href="#ebf6">Architecture Overview</a><br>4. <a href="#5cc6">Building the Travel Guidance Agent with Smolagents</a><br>5. <a href="#1dcc">Integrating MCP for Tool Access</a><br>6. <a href="#9eba">Run the Travel Guidance ACP Server</a><br>7. <a href="#7013">Chaining Agents for a Travel Planning Workflow</a><br>8. <a href="#3547">Next Steps</a><br>9. <a href="#4788">Acknowledgements</a><br>10. <a href="#36dc">Conclusion</a></p><h4>1. What is Smolagents and MCP?</h4><p><strong>Smolagents:<br>Smolagents</strong> is a lightweight framework for building AI agents that can perform tasks like web searches, code execution, and tool integration. Unlike CrewAI, which focuses on orchestrating agents with predefined roles, Smolagents emphasizes flexibility and code-driven workflows. In this article, we’ll use Smolagents’ CodeAgent for web-based research and ToolCallingAgent for accessing external tools.</p><p><strong>Model Context Protocol (MCP):<br></strong>The <strong>Model Context Protocol (MCP)</strong> is an open protocol that standardizes how applications provide context to large language models (LLMs). Think of MCP as a USB-C port for AI applications: just as USB-C connects devices to peripherals, MCP connects LLMs to data sources and tools. It enables:</p><ul><li><strong>Pre-built integrations</strong>: A growing list of tools that LLMs can plug into.</li><li><strong>Custom integrations</strong>: A standardized way to build new tools for AI applications.</li><li><strong>Open protocol</strong>: Free for anyone to implement and use.</li><li><strong>Flexibility</strong>: Switch between apps while retaining context.</li></ul><p>In our case, MCP will provide a tool to list travel agencies by city, enhancing our agent’s capabilities beyond Smolagents’ web search tools. Learn more at the MCP Introduction.</p><p>Both Smolagents and MCP integrate seamlessly with ACP, enabling standardized communication across agents and systems, as introduced in <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>.</p><h4>2. Objective</h4><p>This article guides you through building a <strong>Travel Guidance Agent</strong> that answers travel-related queries (e.g., visa requirements, best travel seasons) and finds local travel agencies. The agent will:</p><ul><li>Use Smolagents’ CodeAgent with web search tools (DuckDuckGoSearchTool and VisitWebPageTool) for general travel queries.</li><li>Use Smolagents’ ToolCallingAgent with an MCP-based tool to find travel agencies by city.</li><li>Be wrapped in an ACP server for interoperability with other agents, such as the Travel Package Agent from Part 1.</li><li>Demonstrate sequential chaining with the Travel Package Agent to create a cohesive travel planning workflow.</li></ul><p>This is the second component of our Travel Planning MVP, paving the way for a future router agent to dynamically manage agent interactions.</p><h4>3. Architecture Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*MuOGVrPYhFtmw_MZ.png" /><figcaption>ACP Application managing the sequential workflow (Image by Author)</figcaption></figure><p><strong>Components</strong>:</p><ul><li><strong>User Query</strong>: Initiates the workflow.</li><li><strong>ACP Client</strong>: Sends HTTP requests to both ACP servers.</li><li><strong>Travel Guidance ACP Server</strong>: Hosts travel_research_agent (web searches) and travel_agency_agent (MCP tool).</li><li><strong>MCP Server</strong>: Provides the list_travel_agencies tool.</li><li><strong>Travel Package ACP Server</strong>: Hosts the Travel Package Agent from <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Part 1</a>.</li><li><strong>OpenAI LLM</strong>: Powers reasoning for all agents.</li><li><strong>Web Tools and PDFs</strong>: Serve as data sources for the agents.</li></ul><p>This architecture ensures modularity and interoperability via ACP and MCP.</p><h4>4. Building the Travel Guidance ACP Server with Smolagents</h4><p><strong>Prerequisites</strong></p><ul><li><strong>Python 3.12+</strong>: Ensure Python is installed.</li><li><strong>uv</strong>: A Python package and project manager. Install it following these instructions.</li><li><strong>OpenAI API Key</strong>: Create a .env file in our project root with: OPENAI_API_KEY=sk-...</li><li><strong>Existing Project</strong>: The travel_acp_projecttravel_acp_project from Part 1, with dependencies in pyproject.tomland PDF brochures in the data folder.</li><li><strong>Dependencies</strong>: Update pyproject.toml to include smolagents, mcp, and requests for this article. Run: uv sync</li></ul><p>Navigate to the project directory:</p><pre>cd travel_acp_project<br>source .venv/bin/activate  # On Windows: .venv\Scripts\activate</pre><p>We’ll create a <strong>Travel Guidance Agent</strong> using Smolagents’ CodeAgent to handle travel-related queries with web search capabilities.</p><p><strong>4.1: Define the Travel Research Agent</strong></p><p>Create src/smolagents_server.py to define the agent:</p><pre>from collections.abc import AsyncGenerator<br>from acp_sdk.models import Message, MessagePart<br>from acp_sdk.server import Context, RunYield, RunYieldResume, Server<br>from smolagents import CodeAgent, DuckDuckGoSearchTool, LiteLLMModel, VisitWebpageTool<br>import logging<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>server = Server()<br><br># Configure the LLM<br>model = LiteLLMModel(<br>    model_id=&quot;openai/gpt-4&quot;,<br>    max_tokens=2048<br>)<br><br>@server.agent()<br>async def travel_research_agent(input: list[Message], context: Context) -&gt; AsyncGenerator[RunYield, RunYieldResume]:<br>    &quot;&quot;&quot;<br>    This is a CodeAgent that supports travelers with up-to-date information about travel destinations,<br>    visa requirements, best seasons to visit, and real-time tourism insights. It uses web search and<br>    web page browsing tools to answer travel-related questions.<br>    &quot;&quot;&quot;<br>    agent = CodeAgent(<br>        tools=[DuckDuckGoSearchTool(), VisitWebpageTool()],<br>        model=model<br>    )<br><br>    prompt = input[0].parts[0].content<br>    response = agent.run(prompt)<br><br>    yield Message(parts=[MessagePart(content=str(response))])<br><br>if __name__ == &quot;__main__&quot;:<br>    server.run(port=8000)</pre><p>This code:</p><ul><li>Uses Smolagents’ CodeAgent with CDuckDuckGoSearchTool and VisitWebpageTool for web-based research.</li><li>Configures an OpenAI GPT-4 model via LiteLLMModel.</li><li>Wraps the agent in an ACP server using the @server.agent() decorator, running on port 8000.</li></ul><h4>5. Integrating MCP for Tool Access</h4><p>To enhance the Travel Guidance ACP server, we’ll add a second agent (Travel agency agent) that uses an MCP server to access a tool for listing travel agencies by city.</p><p><strong>5.1: Create the MCP Server</strong></p><p>Create src/mcpserver.py to define an MCP server with a travel agency lookup tool:</p><pre>from colorama import Fore<br>from mcp.server.fastmcp import FastMCP<br>import json<br>import requests<br><br>mcp = FastMCP(&quot;travelagencyserver&quot;)<br><br>@mcp.tool()<br>def list_travel_agencies(city: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    This tool returns a list of travel agencies located in the given city.<br>    Args:<br>        city (str): The city name to search agencies for. Example: &quot;Hyderabad&quot;<br>    Returns:<br>        str: A list of travel agencies matching the city.<br>        Example: &#39;[{&quot;name&quot;: &quot;GlobeTrotters Hyderabad&quot;, &quot;rating&quot;: 4.6, &quot;contact&quot;: &quot;9876543210&quot;}, ...]&#39;<br>    &quot;&quot;&quot;<br>    url = &#39;https://raw.githubusercontent.com/vinayak-shanawad/Travel-Planning-ACP/main/travel_agencies.json&#39;<br>    resp = requests.get(url)<br>    agencies = json.loads(resp.text)<br><br>    matched = [a for a in agencies if a[&#39;city&#39;].lower() == city.lower()]<br>    return json.dumps(matched, indent=2)<br><br>if __name__ == &quot;__main__&quot;:<br>    mcp.run(transport=&quot;stdio&quot;)</pre><p>This code:</p><ul><li>Defines an MCP server (FastMCP) with a list_travel_agencies tool that fetches agency data from a JSON file.</li><li>Runs locally using the stdio transport.</li></ul><p><strong>5.2: Update the Travel Guidance ACP Server with Travel Agency Agent</strong></p><p>Update src/smolagents_server.py to include a travel_agency_agentthat uses the MCP tool:</p><pre>from collections.abc import AsyncGenerator<br>from acp_sdk.models import Message, MessagePart<br>from acp_sdk.server import Context, RunYield, RunYieldResume, Server<br>from smolagents import (<br>    CodeAgent, DuckDuckGoSearchTool, LiteLLMModel, VisitWebpageTool,<br>    ToolCallingAgent, ToolCollection<br>)<br>from mcp import StdioServerParameters<br><br>server = Server()<br><br>model = LiteLLMModel(<br>    model_id=&quot;openai/gpt-4&quot;,<br>    max_tokens=2048<br>)<br><br>server_parameters = StdioServerParameters(<br>    command=&quot;uv&quot;,<br>    args=[&quot;run&quot;, &quot;src/mcpserver.py&quot;],<br>    env=None,<br>)<br><br>@server.agent()<br>async def travel_research_agent(input: list[Message], context: Context) -&gt; AsyncGenerator[RunYield, RunYieldResume]:<br>    &quot;&quot;&quot;<br>    This is a CodeAgent that supports travelers with up-to-date information about travel destinations,<br>    visa requirements, best seasons to visit, and real-time tourism insights. It uses web search and<br>    web page browsing tools to answer travel-related questions.<br>    &quot;&quot;&quot;<br>    agent = CodeAgent(<br>        tools=[DuckDuckGoSearchTool(), VisitWebpageTool()],<br>        model=model<br>    )<br><br>    prompt = input[0].parts[0].content<br>    response = agent.run(prompt)<br><br>    yield Message(parts=[MessagePart(content=str(response))])<br><br>@server.agent()<br>async def travel_agency_agent(input: list[Message]) -&gt; AsyncGenerator[RunYield, RunYieldResume]:<br>    &quot;&quot;&quot;<br>    A Travel Agency Agent that helps users find travel agencies based on their city using an MCP tool.<br>    &quot;&quot;&quot;<br>    with ToolCollection.from_mcp(server_parameters, trust_remote_code=True) as tool_collection:<br>        agent = ToolCallingAgent(tools=tool_collection.tools, model=model)<br>        prompt = input[0].parts[0].content<br>        response = agent.run(prompt)<br>        yield Message(parts=[MessagePart(content=str(response))])<br><br>if __name__ == &quot;__main__&quot;:<br>    server.run(port=8000)</pre><p>This update:</p><ul><li>Adds a travel_agency_agent using Smolagents’ ToolCallingAgent.</li><li>Connects to the MCP server via StdioServerParameters to access the list_travel_agencies tool.</li></ul><h4>6. Run the Travel Guidance ACP Server</h4><p>The travel_research_agent and travel_agency_agent are already wrapped in an ACP server in smolagents_server.py, running on port 8000. To activate the server: uv run python src/smolagents_server.py This hosts both agents, making them accessible via ACP’s RESTful API. The MCP server must also be running.</p><h4>7. Chaining Agents for a Travel Planning Workflow</h4><p>To demonstrate the power of ACP, we’ll chain three agents sequentially:</p><ol><li><strong>Travel Agency Agent</strong> (on Travel Guidance ACP Server): Finds travel agencies in a city.</li><li><strong>Travel Research Agent</strong> (on Travel Guidance ACP Server): Answers general travel queries using web search.</li><li><strong>Travel Package Agent</strong> (on Travel Package ACP Server from Part 1): Retrieves package details from PDFs.</li></ol><p><strong>7.1: Create the Workflow</strong></p><p>Create 3-Wrap-SmolAgent-and-MCP-into-ACP.ipynb to chain the agents:</p><pre>import nest_asyncio<br>from acp_sdk.client import Client<br>import asyncio<br>from colorama import Fore<br><br>nest_asyncio.apply()<br><br>async def run_travel_workflow() -&gt; None:<br>    async with Client(base_url=&quot;http://localhost:8000&quot;) as guidance, Client(base_url=&quot;http://localhost:8001&quot;) as package:<br>        # Step 1: Ask travel agency agent for agencies in Hyderabad<br>        run1 = await guidance.run_sync(<br>            agent=&quot;travel_agency_agent&quot;,<br>            input=&quot;I am based in Hyderabad. Are there any travel agencies near me?&quot;<br>        )<br>        agency_response = run1.output[0].parts[0].content<br>        print(Fore.CYAN + &quot;TRAVEL AGENCY AGENT RESPONSE:\n&quot; + agency_response + Fore.RESET)<br><br>        # Step 2: Ask travel research agent using agency names as context<br>        run2 = await guidance.run_sync(<br>            agent=&quot;travel_research_agent&quot;,<br>            input=f&quot;Based on these agencies: {agency_response}\nWhat is the best time to go for a Europe trip from Hyderabad?&quot;<br>        )<br>        research_response = run2.output[0].parts[0].content<br>        print(Fore.LIGHTMAGENTA_EX + &quot;TRAVEL RESEARCH AGENT RESPONSE:\n&quot; + research_response + Fore.RESET)<br><br>        # Step 3: Ask travel package agent using previous responses as context<br>        combined_context = f&quot;{agency_response}\n{research_response}&quot;<br>        run3 = await package.run_sync(<br>            agent=&quot;travel_package_agent&quot;,<br>            input=f&quot;Context: {combined_context}\nWhich Europe travel packages from Hyderabad include Switzerland?&quot;<br>        )<br>        package_response = run3.output[0].parts[0].content<br>        print(Fore.YELLOW + &quot;TRAVEL PACKAGE AGENT RESPONSE:\n&quot; + package_response + Fore.RESET)<br><br>asyncio.run(run_travel_workflow())</pre><p>Run the workflow: in 3-Wrap-SmolAgent-and-MCP-into-ACP.ipynb notebook.</p><p><strong>7.2: Sample Output</strong></p><pre>TRAVEL AGENCY AGENT RESPONSE:<br>Yes, there are several travel agencies available in Hyderabad. Here are a few:<br>1. [GT Holidays Hyderabad](https://www.gtholidays.in/hyderabad/), Rating: 4.7<br>2. [RL Tours &amp; Travels](https://rltours.in/), Rating: 4.5<br>3. [IMAD Travel](https://imadtravel.com/), Rating: Not Available<br>4. StarEdge Holidays, Rating: 4.6, Contact: Not Available<br>5. [Thrillophilia](https://www.thrillophilia.com), Rating: Not Available<br><br>TRAVEL RESEARCH AGENT RESPONSE:<br>The best time to go for a Europe trip from Hyderabad is during the spring and fall shoulder seasons - from mid-April to mid-June and from September to October.<br><br>TRAVEL PACKAGE AGENT RESPONSE:<br>Here are the Europe travel packages from Hyderabad that include Switzerland:<br><br>1. &quot;[Best of Switzerland &amp; Italy | Roman Streets to Snowy Alps](https://www.thrillophilia.com/tours/best-of-switzerland-italy-tour-from-india)&quot;: Duration of 9 days &amp; 8 nights for INR 1,43,200.<br>  <br>2. &quot;[Scandinavian Snow Escape | Northern Nights &amp; Winter Lights](https://www.thrillophilia.com/tours/scandinavian-snow-escape-from-india)&quot;: Duration of 9 days &amp; 8 nights for INR 2,91,000.<br>   <br>3. &quot;[Europe Golden Trio | Switzerland, Amsterdam &amp; Paris Tour](https://www.thrillophilia.com/tours/europe-golden-trio-tour-from-india)&quot;: Duration of 8 days &amp; 7 nights for INR 1,73,000.<br>   <br>4. &quot;[Classic Europe in Winter | From Roman Ruins to Amsterdam Canals](https://www.thrillophilia.com/tours/classic-europe-in-winter-from-india)&quot;: Duration of 13 days &amp; 12 nights for INR 2,55,000.<br>   <br>5. &quot;[Western Europe In A Nutshell](https://www.thrillophilia.com/tours/western-europe-in-a-nutshell-from-india)&quot;: Duration of 11 days &amp; 10 nights for INR 1,72,000.<br>   <br>6. &quot;[Classic Switzerland | Peaks, Lakes, and Alpine Trails](https://www.thrillophilia.com/tours/classic-switzerland-tour-from-india)&quot;: Duration of 7 days &amp; 6 nights for INR 1,25,900.<br><br>You can book your Europe tour package from Hyderabad with Thrillophilia, which guarantees a fantastic journey to Europe with great prices, attractive offers, unique activities, and personalized support.</pre><p>The workflow produces:</p><ul><li><strong>Travel Agency Agent</strong>: Lists agencies like Thrillophilia, GT Holidays, and RL Tours in Hyderabad.</li><li><strong>Travel Research Agent</strong>: Recommends spring (April–June) and autumn (September–October) for Europe travel from Hyderabad.</li><li><strong>Travel Package Agent</strong>: Details packages like “Best of Switzerland &amp; Italy” (9 days, INR 1,43,200) that include Switzerland.</li></ul><p>This sequential chaining showcases how ACP enables agents to collaborate, passing context between them for a cohesive user experience.</p><h4>8. Next Steps</h4><p>We’ve built a Travel Guidance Agent and integrated it with MCP and ACP, creating a workflow that chains multiple agents. To further enhance this MVP:</p><ul><li><strong>Add a Router Agent</strong>: Replace sequential chaining with a router agent to dynamically select the appropriate agent based on the query, improving efficiency and flexibility (to be covered in <a href="https://medium.com/@vinayakshanawad/build-an-ai-travel-assistant-part-3-hierarchical-workflow-with-a-router-agent-aaa5f420cf7b">Part 3</a>).</li><li><strong>Expand Tools</strong>: Incorporate additional MCP tools, such as flight price checkers or weather APIs.</li></ul><h4>9. Acknowledgements</h4><p>This article was inspired by the resources provided by the <a href="https://agentcommunicationprotocol.dev/introduction/welcome">Agent Communication Protocol</a> (ACP) initiative and <a href="https://learn.deeplearning.ai/courses/acp-agent-communication-protocol/lesson/yg0qt/introduction">DeepLearning.AI</a>. Special thanks to IBM and DeepLearning.AI team. These resources were instrumental in shaping the understanding and implementation of the Travel Package Agent MVP.</p><h4>10. Conclusion</h4><p>In this second part of our series, we expanded our Travel Planning MVP by building a <strong>Travel Guidance Agent</strong> using <strong>Smolagents</strong>, integrating it with a <strong>Model Context Protocol (MCP)</strong> server for tool access, and wrapping it in an <strong>ACP server</strong> for interoperability. We also demonstrated sequential chaining with the Travel Package Agent from Part 1 to create a cohesive travel planning workflow. By leveraging ACP and MCP, we’ve built a modular, scalable system ready for further enhancements, such as a router agent in Part 3. Follow these steps to create our own multi-agent travel planning system and explore the power of standardized AI communication.</p><p>I hope you enjoyed this article, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this article is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/Travel-Planning-ACP">GitHub - vinayak-shanawad/Travel-Planning-ACP</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cec81e07f17" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/build-an-ai-travel-assistant-part-2-enhancing-travel-planning-with-smolagents-mcp-and-acp-8cec81e07f17">Build an AI Travel Assistant (Part 2): Enhancing Travel Planning with Smolagents, MCP, and ACP</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an AI Travel Assistant (Part 1): Creating an Agent Powered by CrewAI and ACP]]></title>
            <link>https://levelup.gitconnected.com/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6caabbdb97b4</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent-protocols]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 10:50:49 GMT</pubDate>
            <atom:updated>2025-08-18T03:06:30.452Z</atom:updated>
            <content:encoded><![CDATA[<h4>Dive into building a travel agent that answers package queries and integrates seamlessly with other AI systems using ACP</h4><h4>Introduction</h4><p>In the rapidly evolving world of AI, agents are becoming powerful tools for automating complex tasks. However, as AI agents proliferate, connecting them across different frameworks and platforms is a challenge. Enter the <strong>Agent Communication Protocol (ACP)</strong>, a standardized way to enable seamless agent interoperability. In this article, we’ll explore ACP and guide you through building a <strong>Travel Package Agent</strong> using <strong>CrewAI</strong>, a framework for orchestrating AI agents, and wrapping it into an ACP server to make it discoverable and interoperable. By the end, we’ll have a fully functional agent that answers queries about travel packages and a clear understanding of how to deploy it using ACP.</p><h4>Table of Contents:</h4><p>1. <a href="#741e">What is ACP?</a><br>2. <a href="#ab45">Why ACP?</a><br>3. <a href="#df6a">Objective</a><br>4. <a href="#1b90">Architecture Overview</a><br>5. <a href="#5f81">Building the Travel Package Agent with CrewAI</a><br>6. <a href="#701b">Wrapping the Agent in an ACP Server</a><br>7. <a href="#a14b">Next Steps</a><br>8. <a href="#7a5e">Acknowledgements</a><br>9. <a href="#4e65">Conclusion</a></p><h4><strong>1. What is ACP?</strong></h4><p>The <a href="https://agentcommunicationprotocol.dev/introduction/welcome">Agent Communication Protocol</a> (ACP) is an open standard that enables AI agents, applications, and humans to work together seamlessly, regardless of the tools or frameworks used to build them. It offers a RESTful API that supports:</p><ul><li><strong>Multi-modal communication</strong>: Handles text, images, audio, and more.</li><li><strong>Flexible interactions</strong>: Supports both instant and long-running tasks.</li><li><strong>Real-time streaming</strong>: Allows dynamic data exchange.</li><li><strong>Discovery</strong>: Finds agents online or offline.</li><li><strong>Scalability</strong>: Works with any agent implementation.</li></ul><p>Developed under the Linux Foundation, ACP ensures interoperability and fosters collaboration across diverse AI ecosystems.</p><h4>2. Why ACP?</h4><p>AI systems often face challenges due to their fragmented nature:</p><ul><li><strong>Integration hurdles</strong>: Connecting agents across different platforms is complex.</li><li><strong>Redundant work</strong>: Teams waste effort rebuilding similar features.</li><li><strong>Scaling issues</strong>: Custom integrations limit growth.</li><li><strong>Inconsistent tools</strong>: Varying frameworks slow down development.</li></ul><p>ACP solves these by providing a unified communication standard, enabling:</p><ul><li><strong>Agent swapping</strong>: Easily upgrade or replace agents without rework.</li><li><strong>Teamwork</strong>: Coordinate specialized agents for complex tasks.</li><li><strong>System integration</strong>: Link agents with business tools like CRMs or analytics.</li><li><strong>Cross-organization collaboration</strong>: Securely connect agents across companies.</li></ul><h4><strong>3. Objective</strong></h4><p>The goal of this article is to guide you through building a <strong>Travel Package Agent</strong> that answers queries about Europe travel packages using <strong>CrewAI</strong> and its <strong>Retrieval-Augmented Generation (RAG)</strong> capabilities. We’ll then wrap this agent into an <strong>ACP server</strong> to make it discoverable and interoperable with other agents or applications. By following this tutorial, we’ll:</p><ul><li>Understand how to create a RAG-based agent with CrewAI.</li><li>Learn how to integrate it with ACP for standardized communication.</li><li>Set up a local project to run and test the agent.</li></ul><p>This is part of a broader Travel Planning MVP, where this agent (focused on travel package queries) could work alongside others, like a travel research agent or a city-based agency finder. Let’s get started!</p><h4>4. Architecture Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*wFoX6-RLnX2SOYpR.png" /><figcaption>Travel Package Agent Architecture Overview (Image by Author)</figcaption></figure><p>The Travel Package Agent MVP follows a modular architecture, integrating CrewAI’s RAG capabilities with an ACP server for seamless communication.</p><p><strong>Components</strong>:</p><ul><li><strong>User Query</strong>: The user sends a query (e.g., via a client script or application).</li><li><strong>ACP Client</strong>: Sends HTTP requests to the ACP server.</li><li><strong>ACP Server</strong>: Hosts the Travel Package Agent, handling incoming requests.</li><li><strong>CrewAI Agent</strong>: Processes queries using the RAG tool and OpenAI’s GPT-4.</li><li><strong>RAG Tool</strong>: Retrieves relevant data from PDF brochures using embeddings.</li><li><strong>PDF Brochures</strong>: Serve as the knowledge base for travel package details.</li><li><strong>OpenAI LLM</strong>: Powers the agent’s reasoning and response generation.</li></ul><p>This architecture ensures modularity, allowing the agent to integrate with other systems via ACP’s standardized API.</p><h4>5. Building the Travel Package Agent with CrewAI</h4><p><strong>Prerequisites</strong></p><ul><li><strong>Python 3.12+</strong>: Ensure Python is installed.</li><li><strong>uv</strong>: A Python package and project manager. Install it by checking this <a href="https://docs.astral.sh/uv/getting-started/installation/">link</a>.</li><li><strong>OpenAI API Key</strong>: Create a .env file in our project root with: OPENAI_API_KEY=sk-...</li><li>Create <strong>data </strong>folder: Place sample PDF brochures (e.g., Europe-Packages-from-Hyderabad.pdf) in the data folder.</li></ul><p>We’ll use <strong>CrewAI</strong> to create a <strong>Travel Package Agent</strong> that answers queries about Europe travel packages using a RAG-based approach. The agent leverages the RagTool to query travel brochures stored as PDFs.</p><p><strong>5.1 Define the Large Language Model (LLM)<br></strong>We’ll now define the large language model that we will use for our CrewAI agent. max_tokens: maximum number of tokens the model can generate in a single response.</p><pre>from crewai import LLM<br><br>llm = LLM(model=&quot;openai/gpt-4&quot;, max_tokens=1024)</pre><p>This sets up the LLM for the agent’s reasoning and response generation.</p><p><strong>5.2 Configure the RAG Tool</strong></p><p>The RagTool creates a knowledge base from PDFs, chunking text and generating embeddings.</p><pre>from crewai_tools import RagTool<br><br>config = {<br>    &quot;llm&quot;: {<br>        &quot;provider&quot;: &quot;openai&quot;,<br>        &quot;config&quot;: {<br>            &quot;model&quot;: &quot;gpt-4&quot;,<br>        }<br>    },<br>    &quot;embedding_model&quot;: {<br>        &quot;provider&quot;: &quot;openai&quot;,<br>        &quot;config&quot;: {<br>            &quot;model&quot;: &quot;text-embedding-ada-002&quot;<br>        }<br>    }<br>}<br><br>rag_tool = RagTool(<br>    config=config,<br>    chunk_size=1200,<br>    chunk_overlap=200,<br>)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Ahmedabad.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Chennai.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Delhi.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Hyderabad.pdf&quot;, data_type=&quot;pdf_file&quot;)</pre><p>This sets up the RAG tool with OpenAI embeddings and loads four PDF brochures.</p><p><strong>5.3 Define the Travel Package Agent</strong></p><p>Define the agent with a role, goal, and backstory:</p><pre>from crewai import Agent<br><br>travel_package_agent = Agent(<br>    role=&quot;Senior Travel Package Advisor&quot;,<br>    goal=&quot;Accurately answer questions about Europe travel packages including pricing, itinerary, inclusions, exclusions, and departure cities.&quot;,<br>    backstory=(<br>        &quot;You are a seasoned travel consultant specializing in curated Europe travel packages. &quot;<br>        &quot;You assist customers by interpreting and explaining offerings from detailed travel brochures &quot;<br>        &quot;for different departure cities like Ahmedabad, Delhi, Chennai, and Hyderabad.&quot;<br>    ),<br>    verbose=True,<br>    allow_delegation=False,<br>    llm=llm,<br>    tools=[rag_tool],<br>    max_retry_limit=5<br>)</pre><p>This agent handles queries like “What’s included in the 7-day Europe package?” or “What is the best time to go for a Europe trip from Hyderabad?”</p><p><strong>5.4 Define a Task and Run the Agent</strong></p><p>Let’s now test the travel package agent. For that, we need to define the agent task and pass to it the query and the agent.</p><p>To run the agent, we need to pass the agent and the task to a Crew object that we can run using the kickoff method.</p><pre>from crewai import Crew, Task<br><br>task = Task(<br>    description=&quot;The user wants to know the total cost and duration of the &#39;Best of Switzerland &amp; Italy&#39; package from Hyderabad.&quot;,<br>    expected_output=&quot;A clear summary of the &#39;Best of Switzerland &amp; Italy&#39; package including cost, duration, and inclusions.&quot;,<br>    agent=travel_package_agent<br>)<br><br>crew = Crew(agents=[travel_package_agent], tasks=[task], verbose=True)<br>task_output = crew.kickoff()<br>print(task_output)</pre><pre>─────────────────────────────────────────── Crew Execution Started ─────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Crew Execution Started                                                                                         │<br>│  Name: crew                                                                                                     │<br>│  ID: 91b71bdf-155a-45a2-93e3-266b5644648c                                                                       │<br>│  Tool Args:                                                                                                     │<br>│                                                                                                                 │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭─────────────────────────────────────────────── 🤖 Agent Started ────────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Agent: Senior Travel Package Advisor                                                                           │<br>│                                                                                                                 │<br>│  Task: The user wants to know the total cost and duration of the &#39;Best of Switzerland &amp; Italy&#39; package from     │<br>│  Hyderabad. Provide pricing, number of days, and what&#39;s included if available.                                  │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭──────────────────────────────────────────── 🔧 Agent Tool Execution ────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Agent: Senior Travel Package Advisor                                                                           │<br>│                                                                                                                 │<br>│  Thought: I need to query the knowledge base with the specifics of the &#39;Best of Switzerland &amp; Italy&#39; package    │<br>│  from Hyderabad to get the details including cost, duration, and inclusions.                                    │<br>│                                                                                                                 │<br>│  Using Tool: Knowledge base                                                                                     │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭────────────────────────────────────────────────── Tool Input ───────────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  &quot;{\&quot;query\&quot;: \&quot;Best of Switzerland &amp; Italy package from Hyderabad\&quot;}&quot;                                          │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭────────────────────────────────────────────────── Tool Output ──────────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Relevant Content:                                                                                              │<br>│  Europe. Step into comfort with Thrillophilia&#39;s Europe tours from Hyderabad, featuring 3, 4, 5-star resorts     │<br>│  that mix elegance and greatservice. Rated 4.8 by real travelers — our Europe tours from Hyderabad deliver on   │<br>│  every promise. Europe Packages Europe Trip Packages From Hyderabad Duration Price Best of Switzerland &amp; Italy  │<br>│  | Roman Streets to Snowy Alps 9 days &amp; 8 nights INR 1,43,200 Scandinavian Snow Escape | Northern Nights &amp;      │<br>│  Winter Lights 9 days &amp; 8 nights INR 2,91,000 Europe Golden Trio | Switzerland, Amsterdam &amp; Paris Tour 8 days   │<br>│  &amp; 7 nights INR 1,73,000 Classic Europe in Winter | From Roman Ruins to Amsterdam Canals 13 days &amp; 12 nights    │<br>│  INR 2,55,000 Western Europe In A Nutshell 11 days &amp; 10 nights INR 1,72,000 Classic Switzerland | Peaks,        │<br>│  Lakes, and Alpine Trails 7 days &amp; 6 nights INR 1,25,900 Book your Europe tour package from Hyderabad with      │<br>│  Thrillophilia and get ready to experience majestic views of thecontinent. Europe is a dream destination for    │<br>│  travellers, offering rich                                                                                      │<br>│                                                                                                                 │<br>│  Europe Packages from Hyderabad | Get Upto 40% Off Discover the beauty of Europe with Thrillophilia&#39;s           │<br>│  all-inclusive tour packages from Hyderabad. Whether you&#39;re travelling withfamily, friends, or as a couple, we  │<br>│  have customised itineraries specifically for you. Thrillophilia guarantees a fantastic journey to Europe with  │<br>│  great prices, attractive offers, unique activities, and personalised support. Thrillophilia offers a 53+       │<br>│  Europe tour package from Hyderabad, giving you the freedom to choose one that matches your vibe.               │<br>│  Thrillophilia&#39;s Europe trips from Hyderabad start at ₹ 117,000 — perfect for both budget and luxury            │<br>│  travelers. Whether it&#39;s a weekend break or a two-week long trip, Thrillophilia&#39;s Europe tour packages from     │<br>│  Hyderabad provide the optionto travel on your terms. Your next great trip is just a click away — and 2871+     │<br>│  reviewers from Hyderabad can vouch for it. Discover your kind of fun with Group Tours, Family Tours, or        │<br>│  Honeymoon tour options, all kicking off from...                                                                │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭───────────────────────────────────────────── ✅ Agent Final Answer ─────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Agent: Senior Travel Package Advisor                                                                           │<br>│                                                                                                                 │<br>│  Final Answer:                                                                                                  │<br>│  The &#39;Best of Switzerland &amp; Italy&#39; package from Hyderabad costs INR 1,43,200 and consists of a 9 days &amp; 8       │<br>│  nights trip. However, the detailed inclusions of the package like flights, hotels, and sightseeing are not     │<br>│  mentioned directly in the data provided.                                                                       │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭──────────────────────────────────────────────── Task Completion ────────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Task Completed                                                                                                 │<br>│  Name: 4b1c4ac6-0320-4caa-911e-77cfde1c8d5b                                                                     │<br>│  Agent: Senior Travel Package Advisor                                                                           │<br>│  Tool Args:                                                                                                     │<br>│                                                                                                                 │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯<br>╭──────────────────────────────────────────────── Crew Completion ────────────────────────────────────────────────╮<br>│                                                                                                                 │<br>│  Crew Execution Completed                                                                                       │<br>│  Name: crew                                                                                                     │<br>│  ID: 91b71bdf-155a-45a2-93e3-266b5644648c                                                                       │<br>│  Tool Args:                                                                                                     │<br>│  Final Output: The &#39;Best of Switzerland &amp; Italy&#39; package from Hyderabad costs INR 1,43,200 and consists of a 9  │<br>│  days &amp; 8 nights trip. However, the detailed inclusions of the package like flights, hotels, and sightseeing    │<br>│  are not mentioned directly in the data provided.                                                               │<br>│                                                                                                                 │<br>│                                                                                                                 │<br>╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</pre><p>The output shows the agent successfully processing a query about the “Best of Switzerland &amp; Italy” package from Hyderabad. The agent, powered by CrewAI and the RAG tool, retrieves relevant information from the provided PDFs and returns a concise response: the package costs INR 1,43,200 for a 9-day, 8-night trip. However, it notes that specific inclusions (e.g., flights, hotels, sightseeing) are not detailed in the data. The execution log includes the agent’s thought process, tool usage (querying the knowledge base), and the final output, confirming task completion.</p><h4><strong>6. Wrapping the Agent in an ACP Server</strong></h4><p>To make the agent discoverable and interoperable, we’ll wrap it in an <strong>ACP server</strong> for standardized RESTful API access.</p><p><strong>6.1 Create the ACP Server</strong></p><p>Run the following cell to copy the content of the cell to the file crew_agent_server.py which will be saved under the folder travel_acp_project/src/.</p><pre>%%writefile ../travel_acp_project/src/crew_agent_server.py<br>from collections.abc import AsyncGenerator<br>from acp_sdk.models import Message, MessagePart<br>from acp_sdk.server import RunYield, RunYieldResume, Server<br><br>from crewai import Crew, Task, Agent, LLM<br>from crewai_tools import RagTool<br><br>import nest_asyncio<br>nest_asyncio.apply()<br><br>server = Server()<br>llm = LLM(model=&quot;openai/gpt-4&quot;, max_tokens=1024)<br><br>config = {<br>    &quot;llm&quot;: {<br>        &quot;provider&quot;: &quot;openai&quot;,<br>        &quot;config&quot;: {<br>            &quot;model&quot;: &quot;gpt-4&quot;,<br>        }<br>    },<br>    &quot;embedding_model&quot;: {<br>        &quot;provider&quot;: &quot;openai&quot;,<br>        &quot;config&quot;: {<br>            &quot;model&quot;: &quot;text-embedding-ada-002&quot;<br>        }<br>    }<br>}<br>rag_tool = RagTool(config=config,  <br>                   chunk_size=1200,       <br>                   chunk_overlap=200,     <br>                  )<br><br>rag_tool.add(&quot;../data/Europe-Packages-from-Ahmedabad.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Chennai.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Delhi.pdf&quot;, data_type=&quot;pdf_file&quot;)<br>rag_tool.add(&quot;../data/Europe-Packages-from-Hyderabad.pdf&quot;, data_type=&quot;pdf_file&quot;)<br><br><br>@server.agent()<br>async def travel_package_agent(input: list[Message]) -&gt; AsyncGenerator[RunYield, RunYieldResume]:<br>    &quot;This is an agent for questions around policy coverage, it uses a RAG pattern to find answers based on policy documentation. Use it to help answer questions on coverage and waiting periods.&quot;<br><br>    travel_package_agent = Agent(<br>        role=&quot;Senior Travel Package Advisor&quot;,<br>        goal=&quot;Accurately answer questions about Europe travel packages including pricing, itinerary, inclusions, exclusions, and departure cities.&quot;,<br>        backstory=(<br>            &quot;You are a seasoned travel consultant specializing in curated Europe travel packages. &quot;<br>            &quot;You assist customers by interpreting and explaining offerings from detailed travel brochures &quot;<br>            &quot;for different departure cities like Ahmedabad, Delhi, Chennai, and Hyderabad.&quot;<br>        ),<br>        verbose=True,<br>        allow_delegation=False,<br>        llm=llm,<br>        tools=[rag_tool],<br>        max_retry_limit=5<br>    )<br>    <br>    task1 = Task(<br>         description=input[0].parts[0].content,<br>         expected_output = &quot;A comprehensive response as to the users question&quot;,<br>         agent=travel_package_agent<br>    )<br>    crew = Crew(agents=[travel_package_agent], tasks=[task1], verbose=True)<br>    <br>    task_output = await crew.kickoff_async()<br>    yield Message(parts=[MessagePart(content=str(task_output))])<br><br>if __name__ == &quot;__main__&quot;:<br>    server.run(port=8001)</pre><p>This code:</p><ul><li>Uses the @server.agent() decorator for ACP compliance.</li><li>Defines an asynchronous function to process input messages and yield responses.</li><li>Integrates the CrewAI agent with the ACP server.</li></ul><p><strong>6.2 Run the Travel Package ACP Server</strong></p><p>In the terminal we can type the following commands:</p><ul><li>cd travel_acp_project: navigate to directory where we created travel_acp_project.</li><li>uv venv: to create a virtual environment</li><li>uv sync: to install the dependencies defined in `pyproject.toml` file.</li><li>source .venv/bin/activate # On Windows: .venv\Scripts\activate: to activate the virtual environment.</li></ul><p>We can then run the server using uv run. Since this code uses an OpenAI model, we would also need to specify an openAI API key in a .env file like this: OPENAI_API_KEY=sk-….</p><p>Now to activate our configured ACP agent, we would need to run our agent server. The folder travel_acp_project has been set up for us so we can run the agent server using uv:</p><ul><li>Type uv run python src/crew_agent_server.py to run the server and activate our ACP agent.</li></ul><p>The agent is hosted at http://localhost:8001 and accessible via an ACP client.</p><p><strong>6.3 Test the Agent with an ACP Client</strong></p><p>The client queries the ACP server, and the agent responds based on the PDF data.</p><pre>import nest_asyncio<br>from acp_sdk.client import Client<br>import asyncio<br>from colorama import Fore<br><br>nest_asyncio.apply()<br><br>async def example() -&gt; None:<br>    async with Client(base_url=&quot;http://localhost:8001&quot;) as client:<br>        run = await client.run_sync(<br>            agent=&quot;travel_package_agent&quot;,<br>            input=&quot;What is the best time to go for a Europe trip from Hyderabad?&quot;<br>        )<br>        print(Fore.YELLOW + run.output[0].parts[0].content + Fore.RESET)<br><br>asyncio.run(example())</pre><pre>You can plan your Europe trip from Hyderabad at any time of the year as each season has its unique appeal. <br><br>1. Peak Season (June to August): These are the summer months with temperatures ranging from 15°C to 30°C. Destinations like Santorini are at their best, and outdoor activities in France, Italy, and Spain are popular. Music festivals, long daylight hours, and vibrant city life attract large crowds to iconic landmarks.<br><br>2. Waning Season (December to February):  The winter brings temperatures between -5°C to 10°C, which is ideal for skiing in the Swiss Alps or witnessing the Northern Lights in Norway. Festive Christmas markets in cities like Vienna and Prague provide magical experiences with fewer tourists and a cosy winter charm.<br><br>Reaching Europe from Hyderabad is convenient with several flight options from Rajiv Gandhi International Airport to major European airports like Heathrow in London, or Frankfurt in Germany. Airlines such as Lufthansa, Emirates, Etihad, Qatar Airways, and Turkish Airlines operate frequent direct and connecting flights. While there are no direct flights to places other than Germany and London, with layovers in cities like Dubai, Doha, or Istanbul, the total travel time ranges between 12 and 16 hours.</pre><p>The output shows the ACP client successfully querying the Travel Package Agent, which responds with a detailed answer about the best time to travel to Europe from Hyderabad, highlighting the appeal of peak summer (June–August) and waning winter (December–February) seasons, along with flight options from Rajiv Gandhi International Airport. The response leverages the agent’s RAG-based knowledge from PDF brochures, delivered via the ACP server’s RESTful API.</p><h4>7. Next Steps</h4><p>We’ve built a Travel Package Agent and wrapped it in an ACP server! To extend this MVP:</p><ul><li>Add agents like a <strong>Travel Research Agent</strong> (using DuckDuckGoSearchTool) or a <strong>Travel Agency Agent</strong> (for city-based lookups).</li><li>Run multiple agents on one ACP server for simplicity or chain servers for isolation.</li><li>Explore ACP’s streaming or asynchronous task features for complex workflows.</li></ul><h4>8. Acknowledgements</h4><p>This article was inspired by the resources provided by the <a href="https://agentcommunicationprotocol.dev/introduction/welcome">Agent Communication Protocol</a> (ACP) initiative and <a href="https://learn.deeplearning.ai/courses/acp-agent-communication-protocol/lesson/yg0qt/introduction">DeepLearning.AI</a>. Special thanks to IBM and DeepLearning.AI team. These resources were instrumental in shaping the understanding and implementation of the Travel Package Agent MVP.</p><h4>9. Conclusion</h4><p>In this article, we explored the <strong>Agent Communication Protocol (ACP)</strong> and its role in enabling interoperable AI agents. We built a <strong>Travel Package Agent</strong> using <strong>CrewAI</strong>’s RAG capabilities to answer queries about Europe travel packages and wrapped it in an <strong>ACP server</strong> for standardized communication. Follow these steps to create our own agents and integrate them into larger AI ecosystems using ACP.</p><p>I hope you enjoyed this article, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this article is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/Travel-Planning-ACP">GitHub - vinayak-shanawad/Travel-Planning-ACP</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6caabbdb97b4" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/build-an-ai-travel-assistant-creating-an-agent-powered-by-crewai-and-acp-6caabbdb97b4">Build an AI Travel Assistant (Part 1): Creating an Agent Powered by CrewAI and ACP</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to handle a Million Vector Embeddings in the RAG Applications]]></title>
            <link>https://medium.datadriveninvestor.com/how-to-handle-a-million-embedding-vectors-in-the-rag-application-d10b875a0218?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d10b875a0218</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[pgvector]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 09:53:04 GMT</pubDate>
            <atom:updated>2024-03-31T15:46:34.356Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explore the use of PGVector in managing a large dataset for RAG applications and the challenges we faced along the way</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6j6X8PX5dftnMF0ibri_Og.jpeg" /><figcaption>Basic RAG Pipeline (Image by Author)</figcaption></figure><h4>Table of Contents:</h4><p>1. <a href="#ddc2">Motivation</a><br>2. <a href="#aa59">Yelp Reviews Dataset</a><br>3. <a href="#6f6f">Data Preparation</a><br>4. <a href="#d945">Data Ingestion</a><br>5. <a href="#5c61">Retriever</a><br>6. <a href="#a1c8">Indexing in Vector DB</a><br>7. <a href="#bd9c">Indexing hurdles</a><br>8. <a href="#ec09">Performance benchmarking results</a><br>9. <a href="#a47e">Conclusion</a><br>10. <a href="#fa35">References</a></p><h4><strong>1. Motivation</strong></h4><p>Mastering the handling of large datasets in real time is pivotal for businesses today. In my <a href="https://medium.datadriveninvestor.com/beyond-relational-databases-taming-llm-transformer-embeddings-with-pgvector-f0463b5bcce0#7eba">previous article</a>, we discussed on storing Transformer and LLM embeddings into PGVector provided valuable insights, but there’s more to explore. Further exploration in this area enables organizations to harness the power of data efficiently, facilitating quicker decision-making and fostering innovation and progress.</p><h4><strong>2. Yelp Reviews Dataset</strong></h4><p>This is an open dataset released by <a href="https://www.yelp.com/dataset">Yelp</a> for learning purposes. It consists of millions of user reviews, business attributes, and over 200,000 pictures from multiple metropolitan areas. This is a very commonly used dataset for NLP challenges globally.</p><p><strong>Size:</strong> 2.66 GB JSON<br><strong>Number of Records:</strong> 5,200,000 reviews, 174,000 business attributes, 200,000 pictures, and 11 metropolitan areas</p><p>We will consider only 1 Million reviews from yelp_academic_dataset_review.json that contains full review text data including the user_id that wrote the review and the business_id the review is written for.</p><h4>3. Data Preparation</h4><p>Let’s create the different buckets of data based on the number of reviews like 10k, 50k, 100k, 200k, 400k, 600k, 800k, and 1Million by adjusting the num_records parameter in the following code.</p><p>We will also clean the review text data by removing handles, hashtags, URLs, special characters, etc.</p><pre>import pandas as pd<br>import re<br>import unicodedata<br>import nltk<br>nltk.download(&#39;stopwords&#39;)<br>nltk.download(&#39;wordnet&#39;)<br>import json<br><br>def clean_text(text):<br>    text = re.sub(&#39;(\s)@\w+|^@\w+&#39;, &#39;&#39;, text) # removing handles from reviews<br>    text = re.sub(&quot;#[A-Za-z0-9_]+&quot;,&quot;&quot;, text) # removing hastags<br>    text = re.sub(r&quot;http\S+&quot;, &#39;&#39;, text) #remove URLs<br>    text = (unicodedata.normalize(&#39;NFKD&#39;, text)<br>                .encode(&#39;ascii&#39;, &#39;ignore&#39;)<br>                .decode(&#39;utf-8&#39;, &#39;ignore&#39;))<br>    text = re.sub(&quot;&#39;&quot;, &quot;&quot;, text) #remove single quotes<br>    text = re.sub(&#39;&quot;&#39;, &quot;&quot;, text) #remove double quotes<br>    text = re.sub(&quot;[^a-zA-Z0-9 \.]&quot;, &quot;&quot;, text) #<br>    text = re.sub(r&#39;\.+&#39;, &quot;.&quot;, text) #replace multiple full stops<br>    text = re.sub(&#39; +&#39;, &#39; &#39;, text) #remove more than one space<br>    text.lstrip(&#39;0123456789.- &#39;).rstrip(&#39;0123456789.- &#39;)<br>    text = text.strip(&#39;0123456789.- &#39;) #remove numbers at the beginning of a review<br>    text = text.strip() #remove empty spaces from left and right<br>    return text<br><br><br>if __name__ == &quot;__main__&quot;:<br><br>    file_path = &quot;C://AI-ML-Projects//VectorDB-PGVector//data//yelp//yelp_dataset//yelp_academic_dataset_review.json&quot;<br><br>    # Define the columns you want to select<br>    columns = [&quot;review_id&quot;, &quot;user_id&quot;, &quot;text&quot;]<br><br>    # Initialize an empty list to store DataFrames<br>    dfs = []<br><br>    # Open the JSON file and read it line by line<br>    with open(file_path, &quot;r&quot;) as file:<br>        chunk_size = 100000  # Adjust the chunk size as needed<br>        lines = []<br>        for line in file:<br>            lines.append(line)<br>            # Process the chunk if it reaches the desired size<br>            if len(lines) &gt;= chunk_size:<br>                # Parse each line as JSON<br>                json_objects = [json.loads(line) for line in lines]<br>                # Convert JSON objects to DataFrame<br>                df_chunk = pd.DataFrame(json_objects)<br>                # Select the desired columns<br>                df_chunk = df_chunk[columns]<br>                # Remove double quotes from selected columns<br>                for col in columns:<br>                    df_chunk[col] = df_chunk[col].str.replace(&#39;&quot;&#39;, &#39;&#39;)<br>                # Append the chunk to the list of DataFrames<br>                dfs.append(df_chunk)<br>                # Clear the list for the next chunk<br>                lines = []<br><br>        # Process the remaining lines if any<br>        if lines:<br>            json_objects = [json.loads(line) for line in lines]<br>            df_chunk = pd.DataFrame(json_objects)<br>            df_chunk = df_chunk[columns]<br>            for col in columns:<br>                df_chunk[col] = df_chunk[col].str.replace(&#39;&quot;&#39;, &#39;&#39;)<br>            dfs.append(df_chunk)<br><br>    # Concatenate the list of DataFrames into a single DataFrame<br>    df = pd.concat(dfs)<br><br>    # Define the number of records you want to store in the output file<br>    num_records = 1000000  # Change this to the desired number of records<br><br>    # Slice the DataFrame to select the specified number of records<br>    df_sliced = df.head(num_records)<br><br>    df_sliced[&#39;cleaned_text&#39;] = df_sliced[&#39;text&#39;].apply(clean_text)<br><br>    # Write the sliced DataFrame to a new JSON file<br>    new_data = df_sliced.to_dict(orient=&#39;records&#39;)<br>    with open(&#39;C://AI-ML-Projects//VectorDB-PGVector//data//yelp//yelp_dataset//review_data_1M.json&#39;, &#39;w&#39;) as f:<br>        json.dump(new_data, f, indent=4)<br><br>    print(&quot;New JSON file created successfully.&quot;)</pre><h4>4. Data Ingestion</h4><p>In my <a href="https://medium.datadriveninvestor.com/beyond-relational-databases-taming-llm-transformer-embeddings-with-pgvector-f0463b5bcce0#7eba">previous article</a>, we demonstrated how to insert embedding vectors into the PGVector table sequentially. But it’s not an efficient way of handling large data in Production. Let’s look at the following points to improve the performance.</p><ul><li><strong>Batch Insert<br></strong>Instead of inserting documents one by one, consider batching multiple inserts into a single transaction. This can significantly reduce the overhead of individual transactions and improve overall throughput. We can use the execute_batch function provided by psycopg2 to execute parameterized SQL queries in batches.</li></ul><pre>def add_texts(<br>        self,<br>        review_ids: Iterable[int],<br>        user_ids: Iterable[int],<br>        texts: Iterable[str],<br>        metadatas: Optional[List[dict]] = None,<br>        **kwargs: Any,<br>    ) -&gt; List[str]:<br>        batch_size = 128<br><br>        for i in range(0, len(texts), batch_size):<br>            batch_texts = texts[i:i+batch_size]<br><br>            embeddings = self.embed_documents(list(batch_texts))<br><br>            data = []<br>            for review_id, user_id, document, embedding in zip(review_ids, user_ids, batch_texts, embeddings):<br>                doc_id = str(uuid.uuid4())<br>                data.append((doc_id, review_id, user_id, document, embedding))<br><br>            # Batch insert using execute_batch<br>            query = f&#39;INSERT INTO {self.table_name} (id, user_id, review_id, review, embedding) VALUES (%s, %s, %s, %s, %s) RETURNING id&#39;<br>            with self.conn.cursor() as cur:<br>                execute_batch(cur, query, data)<br>                uuids = [row[0] for row in cur.fetchall()]<br><br>        return uuids</pre><ul><li><strong>Batch Processing<br></strong>Generating embeddings for text data in batches reduces the overall computational time and memory usage compared to generating embeddings for each text individually. This is particularly beneficial for large datasets and models, as it allows for parallelization and optimized memory utilization, increasing throughput and enabling faster embedding generation, crucial for real-time or near-real-time applications.</li></ul><pre>def embed_documents(self, texts: Iterable[str]):<br>        batch_size = 64<br>        embeddings = []<br><br>        for i in range(0, len(texts), batch_size):<br>            batch_texts = texts[i:i+batch_size]<br>            batch_embeddings = self.model.encode(batch_texts, show_progress_bar=True)<br>            embeddings.extend(batch_embeddings)<br><br>        embeddings_list = [emb.tolist() for emb in embeddings]<br>        return embeddings_list</pre><p>Let’s create a CustomPGVector class so that we can allow users to store embeddings, and query embeddings into a table by providing the custom embedding size, and retrieve top k relevant documents from a vector store.</p><pre>import uuid<br>import torch<br>from typing import Any, Iterable, List, Optional<br>import logging<br>from psycopg2.extras import execute_batch<br><br>class CustomPGVector():<br>    def __init__(<br>        self,<br>        model,<br>        connection_string: str,<br>        table_name: str,<br>        embed_dim: int,<br>        logger: Optional[logging.Logger] = None,<br>    ) -&gt; None:<br>        self.table_name = table_name<br>        self.model = model<br>        self.embed_dim = embed_dim<br>        self.connection_string = connection_string<br>        self.logger = logger or logging.getLogger(__name__)<br>        self.__post_init__()<br><br>    def __post_init__(<br>        self,<br>    ) -&gt; None:<br>        &quot;&quot;&quot;<br>        Initialize the store.<br>        &quot;&quot;&quot;<br>        self.conn = psycopg2.connect(self.connection_string)<br>        self.conn.autocommit = True<br><br>    def add_texts(<br>        self,<br>        review_ids: Iterable[int],<br>        user_ids: Iterable[int],<br>        texts: Iterable[str],<br>        metadatas: Optional[List[dict]] = None,<br>        **kwargs: Any,<br>    ) -&gt; List[str]:<br>        &quot;&quot;&quot;Run more texts through the embeddings and add to the vectorstore.<br><br>        Args:<br>            texts: Iterable of strings to add to the vectorstore.<br>            kwargs: vectorstore specific parameters<br><br>        Returns:<br>            List of ids from adding the texts into the vectorstore.<br>        &quot;&quot;&quot;<br>        batch_size = 128<br><br>        for i in range(0, len(texts), batch_size):<br>            batch_texts = texts[i:i+batch_size]<br><br>            embeddings = self.embed_documents(list(batch_texts))<br><br>            data = []<br>            for review_id, user_id, document, embedding in zip(review_ids, user_ids, batch_texts, embeddings):<br>                doc_id = str(uuid.uuid4())<br>                data.append((doc_id, review_id, user_id, document, embedding))<br><br>            # Batch insert using execute_batch<br>            query = f&#39;INSERT INTO {self.table_name} (id, user_id, review_id, review, embedding) VALUES (%s, %s, %s, %s, %s) RETURNING id&#39;<br>            with self.conn.cursor() as cur:<br>                execute_batch(cur, query, data)<br>                uuids = [row[0] for row in cur.fetchall()]<br><br>        return uuids<br>    <br>    def embed_documents(self, texts: Iterable[str]):<br>        &quot;&quot;&quot;Generate embeddings for text documents.<br><br>        Args:<br>            texts: Iterable of strings to add to the vectorstore.<br><br>        Returns:<br>            List of embeddings adding into the vectorstore.<br>        &quot;&quot;&quot;<br>        batch_size = 64<br>        embeddings = []<br><br>        for i in range(0, len(texts), batch_size):<br>            batch_texts = texts[i:i+batch_size]<br>            batch_embeddings = self.model.encode(batch_texts, show_progress_bar=True)<br>            embeddings.extend(batch_embeddings)<br><br>        embeddings_list = [emb.tolist() for emb in embeddings]<br>        return embeddings_list<br>    <br>    <br>    def retrieve_top_k_relevant_docs(self, query_embedding: List[float], k: int):<br>        &quot;&quot;&quot;Retrieve top k relevant documents.<br>        We order by embedding &lt;=&gt; query_embedding which will order by nearest neighbor score.<br>        The &lt;=&gt; operator computes the Cosine distance between two vectors.<br><br>        Args:<br>            k: Number of relevant documents to return.<br><br>        Returns:<br>            List of top k relevant documents from the vectorstore.<br>        &quot;&quot;&quot; <br>        try:<br>            with self.conn.cursor() as cur:<br>                #  Cosine distance<br>                # cur.execute(&#39;SET ivfflat.probes = 31&#39;) # Improve IVF Recall on 1Mrecords<br>                cur.execute(f&#39;SELECT id, review FROM {self.table_name} ORDER BY embedding &lt;=&gt; %s LIMIT {k};&#39;, (str(query_embeddings),))<br>                # cur.execute(f&#39;EXPLAIN ANALYZE SELECT id, review FROM {self.table_name} ORDER BY embedding &lt;=&gt; %s ASC LIMIT {k};&#39;, (str(query_embeddings),))<br>                docs = cur.fetchall()<br>        except Exception as e:<br>            self.logger.exception(e)<br>            docs = []<br><br>        return docs</pre><p>Let’s store the 1 Million text embeddings into a table in PGVector and provide the required embedding size.</p><pre>import pandas as pd<br>import json<br>import time<br>from sentence_transformers import SentenceTransformer<br><br># Example usage<br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;)<br>CONNECTION_STRING = &#39;postgresql://postgres:test@localhost:5432/vector_db&#39;<br>TABLE_NAME = &quot;REVIEW&quot;<br><br>store = CustomPGVector(<br>    model=model,<br>    connection_string=CONNECTION_STRING,<br>    table_name=TABLE_NAME,<br>    embed_dim=768<br>    # table_columns=TABLE_COLUMNS<br>)<br><br># Load JSON data into a Pandas DataFrame<br>with open(&#39;C://AI-ML-Projects//VectorDB-PGVector//data//yelp//yelp_dataset//review_data_1M.json&#39;, &#39;r&#39;) as f:<br>    data = json.load(f)<br>    df = pd.DataFrame(data)<br><br>review_ids = df.review_id.to_list()<br>user_ids = df.user_id.to_list()<br>texts = df.cleaned_text.to_list()<br><br>start_time = time.time()<br>uuids = store.add_texts(review_ids, user_ids, texts)<br>end_time = time.time()<br><br>execution_time = end_time - start_time<br>print(f&quot;Execution time: {execution_time} seconds&quot;)</pre><p>If you inspect the database with pgAdmin4 or another GUI, we should see something similar to the below from REVIEW table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVGSIEy7fRhLaRqtU6Thxw.png" /><figcaption>Result set from REVIEW table (Image by Author)</figcaption></figure><h4>5. Retriever</h4><p>Let’s retrieve the top K highest-scoring documents from PGVector:<br>The key functionality:</p><ul><li>We select the id and review from the REVIEW table.</li><li>The &lt;=&gt; operator computes the cosine distance between two vectors. For cosine distance, the result ranges from 0 (indicating perfect similarity) to 2 (indicating complete dissimilarity).</li><li>We order by embedding &lt;=&gt; query_embedding which will order by nearest neighbor score.</li><li>Limit to k=10 results to get the top 10 closest matches.</li></ul><pre># Generate the query embeddings<br>query = &quot;In need of a pet-friendly cafe with vegan options and free Wi-Fi.&quot;<br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;)<br>query_embeddings = model.encode(query).tolist()<br><br># Retrieve the top K highest-scoring documents from PGVector:<br>result = store.retrieve_top_k_relevant_docs(query_embeddings, 10)<br>print(result)</pre><p>So this essentially finds the 10 embeddings out of 1 million embeddings that are most similar to query embedding, which will be the ones with the highest scores/density. The time taken to retrieve the top 10 closest vectors is 2811 ms</p><p>If you notice top results are talking about “pet-friendly cafes with vegan options” which is closest to the query.</p><pre>[(&#39;7f9f5a36-f3b0-4823-b78e-79da08e1bd11&#39;,<br>  &#39;Great place Tons of breakfast options including vegan options. Patio is dog friendly&#39;),<br> (&#39;e23cf90e-a43c-4cad-806d-bdb6d2f7f754&#39;,<br>  &#39;Amazing cafe. Great for vegans. Wonderful interior. Good music. Nice seating. And fantastic food&#39;),<br> (&#39;8096b5b3-3cbb-46df-9efe-b280a2e79e96&#39;,<br>  &#39;Dog friendlyLots of organic healthy choices. Fair trade coffee and lots of yummy juice optionsThe workers are super friendlyNOGMO found here sign Very clean placeAll of their ingredients are organic and stuff I buy at home PerfectoThe barista accidentally made me a hot coffee instead of iced so he is kindly making it over and offered the other one at no charge to anyone in the restaurant.We will definitely be back in the morning for Java and maybe back this afternoon for smoothies or iced tea The lunch menu looks amazing. Veggie burger. Hot cheeses Black bean dip. Hummus. Albacore tuna salad. Grilled cheese sandwich with creamy tomato soup. Sprouts. Avocados. Oh my Organic yogurt parfait. Organic bagel. Chicken salad. Quesadillas&#39;),<br> (&#39;9a31d10f-37a4-49de-9875-9c21f5e5f1d4&#39;,<br>  &#39;Lots of vegetarian options outdoor seating cool outdoor patio in the back dog friendly homemade vegan treats&#39;),<br> (&#39;6a261974-b2b0-459d-9fb0-fb0d8f09d260&#39;,<br>  &#39;Cute spot with a good variety of food on their menu including some vegan options&#39;),<br> (&#39;5c9c35e6-b906-45e4-8511-6eb3efa631c0&#39;,<br>  &#39;Dont believe the hype this location is NOT pet friendly. Employees were incredibly rude about having a dog. The second we walked in everyone behind the counter gave us the nastiest look and immediately the atmosphere was terrible. Prices are incredibly high 8 for a piece of toast with avocado. No wonder everyone hates millennials. I ordered a hot tea it took a ridiculously long time to come out didnt even put a lid on it Also it was 5 for a 10 oz cup. Terrible place would NOT recommend to anyone who has common sense doesnt plan on blowing 30 on coffee&#39;),<br> (&#39;9f449812-e899-4357-9177-443d10077c11&#39;,<br>  &#39;My vegan friend and I came across this cute cafe in our hunt for a lunch spot. My Americano was great but the highlight was definitely the vegan cookie that was dessert. It was chewy and delicious. My mozzarella and basil panini was tasty but not amazing I would have preferred a more flavorful spread. The ambiance is eclectic and artsy. Perfect for a leisurelypaced lunch&#39;),<br> (&#39;284b2c21-dcac-4e90-8073-e752aa91e02b&#39;,<br>  &#39;Love this vegetarian cafe. Great area with cool setting and clean fast service.Also great prices&#39;),<br> (&#39;d3c400dd-2b98-4b0d-90b8-60f3c57fb169&#39;,<br>  &#39;Love this place good coffee has vegan donuts on the weekends. Wish the Fishtown shop had WiFi Id come more often&#39;),<br> (&#39;2f47314b-61e2-4d7b-867b-62fb017045d6&#39;,<br>  &#39;This cafe is great. We are from out of town and we stopped in to ask the employees about vegan options. Immediately we had a support team of 3 people searching for vegan options around the city. We ended up eating here as they do have plenty of vegan options and they offered to cook a meal to fit exactly what we wanted. It was inexpensive came out pretty much instantly and was delicious&#39;)]</pre><h4>6. Indexing in Vector DB</h4><p>Indexing refers to the process of organizing high-dimensional vectors in a way that provides efficient querying of nearest-neighbor vectors.</p><p>There are multiple indexing methods to create vector indices, such as:</p><ul><li>Linear search algorithm</li><li>Cluster-based algorithm</li><li>Quantization</li><li>Graph-based algorithm (HNSW)</li></ul><p>HNSW is the most robust and scalable. Most widely used indexing method as well so let’s go through it in detail.</p><p><a href="https://towardsdatascience.com/similarity-search-part-4-hierarchical-navigable-small-world-hnsw-2aad4fe87d37">Similarity Search, Part 4: Hierarchical Navigable Small World (HNSW)</a></p><p><a href="https://arxiv.org/pdf/1603.09320.pdf">Hierarchical Navigable Small World</a> (HNSW) is a graph-based algorithm used for efficient approximate nearest neighbor search. Here’s an explanation of HNSW with a few key points:</p><ul><li><strong>Multilayer Graph Structure</strong>: The HNSW index is organized as a multilayer graph, where each layer represents a different level of approximation in similarity.</li><li><strong>Superior Query Performance</strong>: Compared to IVFFlat, HNSW offers better query performance, striking a balance between speed and recall when searching for nearest neighbors.</li><li><strong>Slower Build Times</strong>: However, HNSW typically has slower index build times compared to IVFFlat, requiring more time to construct the hierarchical graph structure.</li><li><strong>Higher Memory Usage</strong>: It also tends to consume more memory resources due to its multilayered graph architecture.</li><li><strong>Index Creation Without Data</strong>: Unlike IVFFlat, an HNSW index can be created without any existing data in the table, as there’s no need for a training step.</li><li><strong>Parameter Tuning</strong>: HNSW has several tunable parameters, such as the maximum number of connections per node (ef_construction), the number of neighbors to consider during the search (ef_search), and the level of approximation (m). Adjusting these parameters can balance the trade-off between search accuracy, speed, and memory usage.</li><li><strong>Applications</strong>: Widely used in information retrieval, recommendation systems, and machine learning for quick similarity searches.</li></ul><p>By default, pgvector performs an exact nearest neighbor search, which provides perfect recall.</p><p>We can add an index to use approximate nearest neighbor search, which trades some recall for speed. Unlike typical indexes, we will see different results for queries after adding an approximate index.</p><p><strong>HNSW index</strong></p><p>Let’s add the HNSW index to the REVIEW table that contains 1 Million reviews from the YELP dataset using the below query.<br>Index build time = 27 mins</p><pre>CREATE INDEX hnsw_index ON REVIEW USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);</pre><p>Let’s say we would like to retrieve the top 10 closest vectors to the query after adding the HNSW index by using Cosine similarity.</p><pre># Cosine similarity<br>cur.execute(f&#39;SELECT id, review FROM {self.table_name} ORDER BY 1 - (embedding &lt;=&gt; %s) DESC LIMIT {k};&#39;, (str(query_embeddings),))</pre><blockquote><strong>Key observations</strong></blockquote><blockquote>According to this <a href="https://github.com/pgvector/pgvector/issues/485">issue</a>, the index can’t be applied to an expression 1-(embedding &lt;=&gt; %s) hence we use the Cosine distance instead of computing the Cosine similarity.</blockquote><blockquote>Postgres only supports ASC order index scans on operators, but ASC will return the closest vectors.</blockquote><p>Let’s look at the results from the HNSW index and it takes about 1.9 ms which is a huge improvement from 2811 ms <br>If you notice top results are talking about “pet-friendly cafes with vegan options” which is closest to the query and these results are equivalent to exact nearest neighbor search results.</p><pre>[(&#39;7f9f5a36-f3b0-4823-b78e-79da08e1bd11&#39;,<br>  &#39;Great place Tons of breakfast options including vegan options. Patio is dog friendly&#39;),<br> (&#39;e23cf90e-a43c-4cad-806d-bdb6d2f7f754&#39;,<br>  &#39;Amazing cafe. Great for vegans. Wonderful interior. Good music. Nice seating. And fantastic food&#39;),<br> (&#39;8096b5b3-3cbb-46df-9efe-b280a2e79e96&#39;,<br>  &#39;Dog friendlyLots of organic healthy choices. Fair trade coffee and lots of yummy juice optionsThe workers are super friendlyNOGMO found here sign Very clean placeAll of their ingredients are organic and stuff I buy at home PerfectoThe barista accidentally made me a hot coffee instead of iced so he is kindly making it over and offered the other one at no charge to anyone in the restaurant.We will definitely be back in the morning for Java and maybe back this afternoon for smoothies or iced tea The lunch menu looks amazing. Veggie burger. Hot cheeses Black bean dip. Hummus. Albacore tuna salad. Grilled cheese sandwich with creamy tomato soup. Sprouts. Avocados. Oh my Organic yogurt parfait. Organic bagel. Chicken salad. Quesadillas&#39;),<br> (&#39;9a31d10f-37a4-49de-9875-9c21f5e5f1d4&#39;,<br>  &#39;Lots of vegetarian options outdoor seating cool outdoor patio in the back dog friendly homemade vegan treats&#39;),<br> (&#39;6a261974-b2b0-459d-9fb0-fb0d8f09d260&#39;,<br>  &#39;Cute spot with a good variety of food on their menu including some vegan options&#39;),<br> (&#39;5c9c35e6-b906-45e4-8511-6eb3efa631c0&#39;,<br>  &#39;Dont believe the hype this location is NOT pet friendly. Employees were incredibly rude about having a dog. The second we walked in everyone behind the counter gave us the nastiest look and immediately the atmosphere was terrible. Prices are incredibly high 8 for a piece of toast with avocado. No wonder everyone hates millennials. I ordered a hot tea it took a ridiculously long time to come out didnt even put a lid on it Also it was 5 for a 10 oz cup. Terrible place would NOT recommend to anyone who has common sense doesnt plan on blowing 30 on coffee&#39;),<br> (&#39;9f449812-e899-4357-9177-443d10077c11&#39;,<br>  &#39;My vegan friend and I came across this cute cafe in our hunt for a lunch spot. My Americano was great but the highlight was definitely the vegan cookie that was dessert. It was chewy and delicious. My mozzarella and basil panini was tasty but not amazing I would have preferred a more flavorful spread. The ambiance is eclectic and artsy. Perfect for a leisurelypaced lunch&#39;),<br> (&#39;284b2c21-dcac-4e90-8073-e752aa91e02b&#39;,<br>  &#39;Love this vegetarian cafe. Great area with cool setting and clean fast service.Also great prices&#39;),<br> (&#39;2f47314b-61e2-4d7b-867b-62fb017045d6&#39;,<br>  &#39;This cafe is great. We are from out of town and we stopped in to ask the employees about vegan options. Immediately we had a support team of 3 people searching for vegan options around the city. We ended up eating here as they do have plenty of vegan options and they offered to cook a meal to fit exactly what we wanted. It was inexpensive came out pretty much instantly and was delicious&#39;),<br> (&#39;7b5a2a11-69f0-466e-be67-7de59223803a&#39;,<br>  &#39;This cafe is in a cute quant area and the interior is very large with plenty of seating. They dont have very many outlets so thats something to be aware of. I used xfinity wifi while I was there and that worked fine Their menu isnt very expansive or but they do have a few vegan options. I didnt particularly find anything about it affordable. Their coffee was okay honestly. Not the biggest fan of dark roast but thats what their iced coffee tasted like. They provide oat milk for nondairy options. Lots of people doing work there and theyre pretty conducive to large groups studying together&#39;)]</pre><p><strong>How do we ensure the index method is applied during query time?</strong></p><ul><li>Use EXPLAIN ANALYZE to debug performance and we are querying the results using the respective index method. For example;</li></ul><pre>EXPLAIN ANALYZE SELECT * FROM REVIEW ORDER BY embedding &lt;=&gt; &#39;[3,1......2]&#39; LIMIT 5;</pre><pre>[(&#39;Limit  (cost=192.72..198.85 rows=10 width=536) (actual time=2.093..2.163 rows=10 loops=1)&#39;,),<br> (&#39;  -&gt;  Index Scan using hnsw_index on review  (cost=192.72..613532.72 rows=1000000 width=536) (actual time=2.092..2.160 rows=10 loops=1)&#39;,),<br> (&quot;        Order By: (embedding &lt;=&gt; &#39;[0.019593524,0.07317041,-0.011720089,...0.0026845583,-0.0026375907]&#39;::vector)&quot;,),<br> (&#39;Planning Time: 0.460 ms&#39;,),<br> (&#39;Execution Time: 2.234 ms&#39;,)]</pre><p><strong>IVFFlat index</strong></p><ul><li>It divides vectors into lists and then searches a subset of those lists that are closest to the query vector</li><li>It has faster build times and uses less memory than HNSW.</li><li>It has lower query performance (in terms of speed-recall tradeoff).</li></ul><p>Three key features to achieve good recall;</p><ol><li><strong>Index Creation</strong>: Create the index <em>after</em> the table has some data</li><li><strong>Optimal Number of Lists</strong>: Choose an appropriate number of lists — a good place to start is rows / 1000 for up to 1M rows and sqrt(rows) for over 1M rows</li><li><strong>Probe Configuration</strong>: When querying, specify an appropriate number of <a href="https://github.com/pgvector/pgvector#query-options">probes</a> (higher is better for recall, lower is better for speed) — a good place to start is sqrt(lists)</li></ol><p>Let’s add the IVFFlat index to the REVIEW table that contains 1 Million reviews from the YELP dataset using the below query.<br>Index build time = 254 secs</p><pre>CREATE INDEX ivf_index ON REVIEW USING ivfflat (embedding vector_cosine_ops) WITH (lists = 1000);</pre><p>Let’s look at the results from the IVFFlat index and it takes about 2.8 ms which is a huge improvement from 2811 ms</p><pre>[(&#39;7f9f5a36-f3b0-4823-b78e-79da08e1bd11&#39;,<br>  &#39;Great place Tons of breakfast options including vegan options. Patio is dog friendly&#39;),<br> (&#39;9a31d10f-37a4-49de-9875-9c21f5e5f1d4&#39;,<br>  &#39;Lots of vegetarian options outdoor seating cool outdoor patio in the back dog friendly homemade vegan treats&#39;),<br> (&#39;6a261974-b2b0-459d-9fb0-fb0d8f09d260&#39;,<br>  &#39;Cute spot with a good variety of food on their menu including some vegan options&#39;),<br> (&#39;9f449812-e899-4357-9177-443d10077c11&#39;,<br>  &#39;My vegan friend and I came across this cute cafe in our hunt for a lunch spot. My Americano was great but the highlight was definitely the vegan cookie that was dessert. It was chewy and delicious. My mozzarella and basil panini was tasty but not amazing I would have preferred a more flavorful spread. The ambiance is eclectic and artsy. Perfect for a leisurelypaced lunch&#39;),<br> (&#39;284b2c21-dcac-4e90-8073-e752aa91e02b&#39;,<br>  &#39;Love this vegetarian cafe. Great area with cool setting and clean fast service.Also great prices&#39;),<br> (&#39;2f47314b-61e2-4d7b-867b-62fb017045d6&#39;,<br>  &#39;This cafe is great. We are from out of town and we stopped in to ask the employees about vegan options. Immediately we had a support team of 3 people searching for vegan options around the city. We ended up eating here as they do have plenty of vegan options and they offered to cook a meal to fit exactly what we wanted. It was inexpensive came out pretty much instantly and was delicious&#39;),<br> (&#39;bbad74c9-773d-455e-9ef0-114c96774f5b&#39;,<br>  &#39;Making a PETCO trip I stopped in for mid day fuel This is a must for everyone to check out All day breakfast Vegetarian Vegan heaven The Egg sandwich GF bread spinach cheese avacado plus side salad. So Good Everything is fresh and cooked perfectly Staff friendly Happy and very attentive Cool music eclectic hipster vibe. Gem of The City&#39;),<br> (&#39;d1911a3b-e90b-4bad-86c3-bd814fc2bf44&#39;,<br>  &#39;This place is perfect if you need a place to go with dogs. Food was really good. I was able to make my meal vegan&#39;),<br> (&#39;21324d9d-cbea-4694-9491-6ff6780a6c53&#39;,<br>  &#39;Adorable cafe with excellent food. Exposed brick walls with a gritty feel yet the cuisine is clean crisp fresh and delicious. Super vegan friendly with extraordinary cafe fare I got the Vegan Curried Chicken Salad Wrap. Contrra with lettuce tomatoes cranberries carrots onions avocados on a wheat wrap. ABSOLUTELY DELICIOUS Paired with a side of VeganGluten Free Potato Salad this was an excellent lunch. They offer melts soups sandwiches sales breakfast and wraps. Awesome. GO HERE&#39;),<br> (&#39;08efc26e-a641-486d-ab8a-f5b1fd6caa85&#39;,<br>  &#39;Such a cute cafe I love that their menu labels the items that are NOT vegan rather than the ones that are. I had an amazing vegan glutenfree brownie and a vegan peanut butter rice crispy treat when I went. The surrounding area is fabulous and the cafe itself is unique and gorgeous. I cannot wait to spend hours there in the coming months&#39;)]</pre><p>If you notice top results are talking about “pet-friendly cafes with vegan options” which is closest to the query and the order of these results varies a little bit in contrast to the exact nearest neighbor search results.</p><p>Let’s specify the number of probes which is sqrt(lists) = sqrt(1000) = 31 to achieve good recall.</p><pre>#  Cosine distance<br>cur.execute(&#39;SET ivfflat.probes = 31&#39;) # Improve IVF Recall on 1M records<br>cur.execute(f&#39;SELECT id, review FROM {self.table_name} ORDER BY embedding &lt;=&gt; %s LIMIT {k};&#39;, (str(query_embeddings),))</pre><p>Here are the results using the IVFFlat index with n_probes=31 and these results are equivalent to the exact nearest neighbor search results.</p><pre>[(&#39;7f9f5a36-f3b0-4823-b78e-79da08e1bd11&#39;,<br>  &#39;Great place Tons of breakfast options including vegan options. Patio is dog friendly&#39;),<br> (&#39;e23cf90e-a43c-4cad-806d-bdb6d2f7f754&#39;,<br>  &#39;Amazing cafe. Great for vegans. Wonderful interior. Good music. Nice seating. And fantastic food&#39;),<br> (&#39;8096b5b3-3cbb-46df-9efe-b280a2e79e96&#39;,<br>  &#39;Dog friendlyLots of organic healthy choices. Fair trade coffee and lots of yummy juice optionsThe workers are super friendlyNOGMO found here sign Very clean placeAll of their ingredients are organic and stuff I buy at home PerfectoThe barista accidentally made me a hot coffee instead of iced so he is kindly making it over and offered the other one at no charge to anyone in the restaurant.We will definitely be back in the morning for Java and maybe back this afternoon for smoothies or iced tea The lunch menu looks amazing. Veggie burger. Hot cheeses Black bean dip. Hummus. Albacore tuna salad. Grilled cheese sandwich with creamy tomato soup. Sprouts. Avocados. Oh my Organic yogurt parfait. Organic bagel. Chicken salad. Quesadillas&#39;),<br> (&#39;9a31d10f-37a4-49de-9875-9c21f5e5f1d4&#39;,<br>  &#39;Lots of vegetarian options outdoor seating cool outdoor patio in the back dog friendly homemade vegan treats&#39;),<br> (&#39;6a261974-b2b0-459d-9fb0-fb0d8f09d260&#39;,<br>  &#39;Cute spot with a good variety of food on their menu including some vegan options&#39;),<br> (&#39;5c9c35e6-b906-45e4-8511-6eb3efa631c0&#39;,<br>  &#39;Dont believe the hype this location is NOT pet friendly. Employees were incredibly rude about having a dog. The second we walked in everyone behind the counter gave us the nastiest look and immediately the atmosphere was terrible. Prices are incredibly high 8 for a piece of toast with avocado. No wonder everyone hates millennials. I ordered a hot tea it took a ridiculously long time to come out didnt even put a lid on it Also it was 5 for a 10 oz cup. Terrible place would NOT recommend to anyone who has common sense doesnt plan on blowing 30 on coffee&#39;),<br> (&#39;9f449812-e899-4357-9177-443d10077c11&#39;,<br>  &#39;My vegan friend and I came across this cute cafe in our hunt for a lunch spot. My Americano was great but the highlight was definitely the vegan cookie that was dessert. It was chewy and delicious. My mozzarella and basil panini was tasty but not amazing I would have preferred a more flavorful spread. The ambiance is eclectic and artsy. Perfect for a leisurelypaced lunch&#39;),<br> (&#39;284b2c21-dcac-4e90-8073-e752aa91e02b&#39;,<br>  &#39;Love this vegetarian cafe. Great area with cool setting and clean fast service.Also great prices&#39;),<br> (&#39;d3c400dd-2b98-4b0d-90b8-60f3c57fb169&#39;,<br>  &#39;Love this place good coffee has vegan donuts on the weekends. Wish the Fishtown shop had WiFi Id come more often&#39;),<br> (&#39;2f47314b-61e2-4d7b-867b-62fb017045d6&#39;,<br>  &#39;This cafe is great. We are from out of town and we stopped in to ask the employees about vegan options. Immediately we had a support team of 3 people searching for vegan options around the city. We ended up eating here as they do have plenty of vegan options and they offered to cook a meal to fit exactly what we wanted. It was inexpensive came out pretty much instantly and was delicious&#39;)]</pre><h4>7. Indexing Hurdles</h4><p><strong>1. 400k records: IVF with lists = 400</strong><br>ERROR: memory required is 93 MB, maintenance_work_mem is 64 MB<br>SQL state: 54000<br><strong>Solution</strong>: SET maintenance_work_mem = ‘8GB’;</p><p><strong>2. 800k records: HNSW with (m = 16, ef_construction = 64)<br></strong>ERROR: memory required is 104 MB, maintenance_work_mem is 64 MB<br>SQL state: 54000<br><strong>Solution</strong>: SET maintenance_work_mem = ‘8GB’;</p><p><strong>3. 1M records: IVF with lists = 1000</strong><br>ERROR: memory required is 348 MB, maintenance_work_mem is 64 MB<br>SQL state: 54000<br><strong>Solution</strong>: SET maintenance_work_mem = ‘8GB’;</p><p><strong>4. 1M records: IVF with lists = 1000<br></strong>ERROR: could not write to file “base/pgsql_tmp/pgsql_tmp71.0.fileset/0.0”: No space left on device<br><strong>Solution</strong>: Increase the volume on the server</p><blockquote><strong>Takeaways</strong></blockquote><blockquote>Indexes build significantly faster when the graph fits into maintenance_work_mem</blockquote><blockquote>Do not set maintenance_work_mem so high that it exhausts the memory on the server.</blockquote><h4>8. Performance benchmarking results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrJommR6ADvUHdcKkGce2Q.png" /><figcaption>Query time (ms) vs. Review count (Image by Author)</figcaption></figure><p>Query time increases as the number of records increases by using the exact nearest neighbor search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZaMw8WPzMqZqodzQ7dSr3Q.png" /><figcaption>HNSW Index (m=16, ef_construction=64) — Build time (minutes) vs. Review count (Image by Author)</figcaption></figure><p>Build time increases as the number of records increases by using the HNSW index with m=16, ef_construction=64.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xZw2a76SdE9IEMyfOBVHQ.png" /><figcaption>HNSW Index (m=16, ef_construction=64) — Query time (ms) vs. Review count (Image by Author)</figcaption></figure><p>Query time is consistent for the HNSW index and as observed earlier results also look equivalent to the exact nearest neighbor search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7A7oyVuQ5-5f-8rH6cJIZg.png" /><figcaption>IVFFlat Index — Build time (minutes) vs. Review count (Image by Author)</figcaption></figure><p>Build time increases as the number of records increases by using the IVFFlat index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b7LCJIyQte6zVivAiBOtZw.png" /><figcaption>IVFFlat Index — Query time (ms) vs. Review count (Image by Author)</figcaption></figure><p>Query time increases as the number of records increases by using the IVFFlat index but it has a huge improvement in response time as compared to the exact nearest neighbor search.</p><p><strong>Which indexing (HNSW and IVFFlat) is best when applied to text embeddings in PGVector?</strong></p><p>It depends on your requirements in terms of embedding size, recall, speed, index build time, and insert speed.</p><h4>9. Conclusion</h4><p>This article delves into the intricacies of handling a million embedding vectors within the RAG application framework, presenting a comprehensive exploration from data preparation to performance benchmarking. With a motivation to empower businesses with real-time data mastery, it navigates through the challenges and solutions encountered, leveraging the Yelp Reviews dataset as a prime example.</p><h4>10. References</h4><ul><li><a href="https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf">Efficient and robust approximate nearest neighbor search using HNSW</a></li><li><a href="https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw">Indexing in PGVector</a></li><li><a href="https://www.yelp.com/dataset">Yelp review dataset</a></li><li><a href="https://www.pinecone.io/learn/series/faiss/hnsw/">HNSW from Pinecone</a></li><li><a href="https://www.youtube.com/watch?v=rhZgXNdhWDY&amp;t=2208s">Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)</a></li></ul><p>I hope you enjoyed this article, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this article is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/AI-ML-Projects/blob/main/VectorDB-PGVector/pgvector-performance.ipynb">AI-ML-Projects/VectorDB-PGVector/pgvector-performance.ipynb at main · vinayak-shanawad/AI-ML-Projects</a></p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Subscribe to DDIntel <a href="https://www.ddintel.com/"><em>here</em></a>.</p><p>DDI Services: <a href="https://linktr.ee/datadriveninvestor">https://linktr.ee/datadriveninvestor</a></p><p>Featured Articles:</p><ul><li><a href="https://www.datadriveninvestor.com/2024/03/20/accelerating-success-through-the-singapore-fund-asset-management-platform/">Introducing Singapore&#39;s Fund Platform | DataDrivenInvestor</a></li><li><a href="https://www.datadriveninvestor.com/2024/03/26/singapore-unveiling-a-tapestry-of-possibilities-and-global-connections/">Singapore: Unveiling a Tapestry of Possibilities and Global Connections | DataDrivenInvestor</a></li></ul><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d10b875a0218" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/how-to-handle-a-million-embedding-vectors-in-the-rag-application-d10b875a0218">How to handle a Million Vector Embeddings in the RAG Applications</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Slashing Python Docker build times in half with uv]]></title>
            <link>https://medium.datadriveninvestor.com/boosting-docker-build-speed-slashing-python-build-times-in-half-with-uv-eb7734600cfb?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb7734600cfb</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Fri, 16 Feb 2024 10:32:29 GMT</pubDate>
            <atom:updated>2024-02-19T03:25:07.865Z</atom:updated>
            <content:encoded><![CDATA[<h4>uv: Python packaging in Rust</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIB30a3KuiYgcvaw0rqlWg.jpeg" /><figcaption>Image created by author using Dall-E 3</figcaption></figure><p><strong><em>uv</em> </strong>is<strong> </strong>a cutting-edge Python package installer and resolver, revolutionizes installation speed thanks to its Rust-based architecture. With uv, enjoy unparalleled speed and efficiency in managing dependencies, all backed by the expertise of <a href="https://astral.sh/">Astral</a>.</p><p>Charlie Marsh, the creator of <a href="https://github.com/astral-sh/ruff">Ruff</a> announced uv yesterday, and it is designed as a <a href="https://ryxcommar.com/2024/02/15/how-to-cut-your-python-docker-builds-in-half-with-uv/">drop-in replacement</a> for pip and pip-tools workflows.</p><p>Let’s see how to use it in Docker, assuming we are using a Python base image.</p><p>Here is the requirements.txt file that contains required packages we would like to install in Docker.</p><pre>torch==2.1.0<br>pandas==2.1.3<br>nltk==3.8.1<br>sentence-transformers==2.2.2<br>hdbscan==0.8.33<br>umap-learn==0.5.3<br>langchain==0.0.147<br>spacy==3.7.2<br>openai==0.27.4<br>en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.0/en_core_web_sm-3.7.0.tar.gz<br>tiktoken==0.3.3<br>tenacity==8.2.2<br>plotly==5.14.1<br>matplotlib==3.7.1<br>requests==2.28.1<br>boto3==1.34.31</pre><p><strong>Docker file without </strong><strong>uv</strong></p><pre>FROM python:3.11-bullseye<br><br>ENV APP_HOME /app<br>WORKDIR $APP_HOME<br><br># Grab native files<br>COPY requirements.txt /app<br><br># Install dependency packages<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy source code<br>COPY ./src /app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AjzaoOrPdNARfigjVvdsiw.png" /><figcaption>Docker build time without uv (Image by Author)</figcaption></figure><p>We notice that it took <strong>456.3 secs</strong> to install all required dependencies within docker.</p><p><strong>Docker file with </strong><strong>uv</strong></p><pre>FROM python:3.11-bullseye<br><br>ENV APP_HOME /app<br>WORKDIR $APP_HOME<br><br># Grab native files<br>COPY requirements.txt /app<br><br># Install dependency packages<br>ENV VIRTUAL_ENV=/usr/local<br>RUN pip install uv &amp;&amp; uv pip install --no-cache -r requirements.txt<br><br># Copy source code<br>COPY ./src /app</pre><p>Need to install <a href="https://pypi.org/project/uv/">uv</a> package first then install required dependencies.</p><p>Notice that we are using VIRTUAL_ENVenvironment variable because uv strongly recommends using virtual environments (venvs), but it’s possible to work around this requirement by setting your ‘venv’ to match the location of the global Python installation within the Docker image. uv internally acknowledges the presence of the VIRTUAL_ENV environment variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-YVJzxaT9aOLUd80EkRESg.png" /><figcaption>Docker build time with uv (Image by Author)</figcaption></figure><p>Wow, uv has cut down the docker build time from <strong>456.3 secs </strong>to<strong> 236.8 secs.</strong>🚀🚀</p><h4>🙏Acknowledgments</h4><p>Kudos to the Astral team for their groundbreaking release of uv, which has revolutionized Docker build times with its lightning-fast performance. This innovative tool has not only simplified the development workflow but also significantly reduced build durations, enhancing efficiency across the board. A big thank you to the Astral team for their dedication to improving developer productivity!</p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Subscribe to DDIntel <a href="https://www.ddintel.com/"><em>here</em></a>.</p><p>Have a unique story to share? Submit to DDIntel <a href="https://datadriveninvestor.com/ddintelsubmission"><em>here</em></a>.</p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p><a href="https://ddintel.datadriveninvestor.com/"><em>DDIntel</em> </a>captures the more notable pieces from our <a href="https://www.datadriveninvestor.com/"><em>main site</em></a> and our popular <a href="https://medium.datadriveninvestor.com/"><em>DDI Medium publication</em></a>. Check us out for more insightful work from our community.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb7734600cfb" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/boosting-docker-build-speed-slashing-python-build-times-in-half-with-uv-eb7734600cfb">Slashing Python Docker build times in half with uv</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Relational Databases: Taming LLM and Transformer Embeddings with PGVector]]></title>
            <link>https://medium.datadriveninvestor.com/beyond-relational-databases-taming-llm-transformer-embeddings-with-pgvector-f0463b5bcce0?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0463b5bcce0</guid>
            <category><![CDATA[pgvector]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 15:29:09 GMT</pubDate>
            <atom:updated>2024-02-28T07:40:23.205Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Text to Insights: An End-to-End Guide to PGVector</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZfQfv__thv_T09vOUatBQ.png" /><figcaption>Image created by author using stability.ai</figcaption></figure><h4>Table of Contents:</h4><p>1. <a href="#6f49">Motivation</a><br>2. <a href="#324f">What is VectorStore?</a><br>3. <a href="#7658">Why VectorStore?</a><br>4. <a href="#17c0">Why PGVector?</a><br>5. <a href="#4453">Vector DB Use cases</a><br>6. <a href="#7945">PGVector Setup</a><br>7. <a href="#17f9">PGVector for Storing Embeddings</a><br>8. <a href="#46c3">Summary</a><br>9. <a href="#7eba">References</a></p><h4>1. Motivation</h4><p>In a data-rich world, efficiently storing and retrieving text embeddings is crucial for unlocking valuable insights. This article aims to bridge that gap by providing practical insights and guidance on using PGVector and addresses the following use cases.</p><ul><li>Store text embeddings into a custom table in PGVector and provide the required embedding size.</li><li>Retrieve the top K highest-scoring documents from PGVector.</li></ul><p>Let’s understand the vector store concepts before addressing the above use cases.</p><h4>2. What is VectorStore?</h4><p>VectorStore refers to a storage system optimized for saving and querying vector data, typically dense numeric vectors used in machine learning systems.</p><p><strong>Some key characteristics of a vector store:</strong></p><ul><li>Stores vectors as first-class primitive data types, not just blobs.</li><li>Optimized for vector operations like nearest neighbor search, clustering, etc. Enables both exact and approximate nearest neighbor search.</li><li>Provides APIs for adding, updating, and querying vectors and associated data.</li><li>Made for high throughput ingest and low latency queries.</li><li>Allows attaching metadata like IDs, labels, and documents to vectors.</li><li>Implemented in databases like Milvus, Pinecone, Weaviate, Qdrant, PGVector, etc.</li></ul><p>We will utilize the features of PGVector in this article.</p><h4>3. Why VectorStore?</h4><p>There are a few key reasons to use a VectorStore.</p><p><strong>1. Caching embeddings:</strong></p><ul><li>Generating embeddings on the fly with large models can be expensive.</li><li>By storing embeddings in a VectorStore, we only need to generate them once.</li><li>Subsequent lookups just retrieve the cached embeddings, avoiding costly re-computation.</li></ul><p><strong>2. Reducing the load on embedding models:</strong></p><ul><li>Calling embedding models directly from client apps can overload them with requests.</li><li>A VectorStore acts as a caching layer, reducing load on the models.</li></ul><p><strong>3. Enabling additional algorithms:</strong></p><ul><li>With embeddings in a VectorStore, we can run algorithms like similarity search, clustering, etc.</li><li>This isn’t feasible when embeddings live ephemeral in model memory.</li></ul><h4>4. Why PGVector?</h4><p>PGVector is an open-source vector similarity search for Postgres</p><ul><li>It supports: — exact and approximate nearest neighbor search — L2 distance, inner product, and cosine distance</li><li>Manages — vectors stored alongside other application data.</li><li>Leverages PostgreSQL’s reliability, durability, transactions, indexing, etc.</li><li>Queries vectors using standard SQL — easy to integrate into applications.</li><li>Supports efficient indexing of billions of vectors. Supported index types are: HNSW, IVFFlat</li><li>Near real-time insert and low latency searches. Handles vectors up to thousands of dimensions.</li></ul><h4>5. Vector DB Use cases</h4><p><strong>Use case 1: Data Clustering</strong></p><ul><li><strong>Data Pipeline:</strong> Extract text data from social media sources like tweets, posts, and comments. Feed text into SentenceTransformers to generate semantic embeddings and store the embeddings in Vector DB for persistence.</li><li><strong>ML Pipeline:</strong> Retrieve vector embeddings for a dataset from Vector DB. Feed embeddings into a clustering algorithm like the K-Means Clustering algorithm that groups similar vectors based on proximity and results are clusters that segment the data.</li></ul><p>The key benefit is Vector DB provides a scalable datastore for the embeddings, allowing large-scale clustering or other ML tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ADqHnpJYNBbovJNI7UdE0A.jpeg" /><figcaption>Vector DB in data clustering pipeline (Image by Author)</figcaption></figure><p><strong>Use case 2: Face Recognition</strong></p><ul><li><strong>Face Embedding Pipeline:</strong> Generates facial embeddings like ResNet Face Encoder and stores facial embeddings in Vector DB (PGVector). Also stores person ID and other metadata in PostgreSQL.</li><li><strong>Face Recognition App: </strong>Detects faces, generates embeddings, queries (lookup most similar embeddings) PGVector to identify a person.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuIsSp2PArLoHaPa98sPcg.jpeg" /><figcaption>Vector DB in face recogintion pipeline (Image by Author)</figcaption></figure><h4>6. PGVector Setup</h4><ul><li>Pull the PGVector database: docker pull ankane/pgvector</li><li>Start the container: docker run --name pgvector-demo -e POSTGRES_PASSWORD=test -p 5432:5432 -d ankane/pgvector</li><li>Verify that this is running with: docker ps</li><li>We can now install a GUI tool such as pgAdmin4 to inspect the database that is running in the container, or else use psql on the command-line.</li></ul><p>When connecting, we can specify the host as localhost, and the password as whatever we used in the above command, in our case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*XaHP9055V9_r95FEwh2tYw.png" /><figcaption>Register Server (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Lnqy5OOuVKeuLa7Y7duGTA.png" /><figcaption>Connection details (Image by Author)</figcaption></figure><ul><li>We will now create a database, and then add the pgvector extension to that database using this SQL command: CREATE EXTENSION vector;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*uXP9zyqVu7tr9cWp_qNqKA.png" /><figcaption>Create a Database (Image by Author)</figcaption></figure><h4>7. PGVector for Storing Embeddings</h4><p>Firstly, I have done experiments using LangChain, LlamaIndex, SQLalchemy, and psycopg2 Python APIs to store embeddings into PGVector.</p><p><strong>7.1 PGVector and LangChain with 768 dim HuggingFace embeddings</strong></p><p>We used the langchain library to add embeddings to a database, created with HuggingFace’s all-mpnet-base-v2embedding model.</p><p><strong>Limitations: </strong>We can not allow users to store embeddings into a custom table and this may not be a good choice for the data team who would like to create or use existing tables, by default langchain library will store embeddings in the following tables but it can accept the custom embedding size.</p><p>1. langchain_pg_collection: Store the collection details</p><p>2. langchain_pg_embedding: Store the embedding details.</p><p>Discussion reference: <a href="https://github.com/langchain-ai/langchain/discussions/17223">https://github.com/langchain-ai/langchain/discussions/17223</a></p><blockquote>Refer to section 5 in Jupyter Notebook added to GitHub for experimentation details.</blockquote><p><strong>7.2 PGVector and LlamaIndex with 1536 dim OpenAI embeddings</strong></p><p>We used LlamaIndex to add embeddings to database, created with OpenAI’s text-embedding-ada-002 embedding model.</p><p><strong>Limitations:</strong> We can allow users to store embeddings into a custom table by providing the custom embedding size but it creates a table with data_&lt;table_name&gt; prefix.</p><blockquote>Refer to section 6 in Jupyter Notebook added to GitHub for experimentation details.</blockquote><p><strong>7.3</strong> <strong>PGVector and HuggingFace with 768 dim embeddings using psycopg2 Python API</strong></p><p>psycopg2 is the new generation psycopg, redesigned to be faster, lighter, and better suited for concurrency. It is the recommended PostgreSQL adapter for current Python versions.</p><blockquote>Refer to section 7.2 in Jupyter Notebook added to GitHub for experimentation details.</blockquote><blockquote><strong>Note:</strong> We can also use the SQLalchemy library to achieve the following use case and refer to section 7.1 in Jupyter Notebook added to GitHub for experimentation details.</blockquote><p>Let’s create a CustomPGVector class so that we can allow users to store embeddings, and query embeddings into a custom table by providing the custom embedding size, and retrieve top k relevant documents from a vector store.</p><pre>import psycopg2<br>import uuid<br><br>from typing import Any, Iterable, List, Optional<br>import logging<br><br>class CustomPGVector():<br>    def __init__(<br>        self,<br>        model,<br>        connection_string: str,<br>        table_name: str,<br>        embed_dim: int,<br>        logger: Optional[logging.Logger] = None,<br>    ) -&gt; None:<br>        self.table_name = table_name<br>        self.model = model<br>        self.embed_dim = embed_dim<br>        self.connection_string = connection_string<br>        self.logger = logger or logging.getLogger(__name__)<br>        self.__post_init__()<br><br>    def __post_init__(<br>        self,<br>    ) -&gt; None:<br>        &quot;&quot;&quot;<br>        Initialize the store.<br>        &quot;&quot;&quot;<br>        self.cur = self.connect()<br><br><br>    def connect(self):<br>        conn = psycopg2.connect(self.connection_string)<br>        conn.autocommit = True<br>        cur = conn.cursor()<br>        return cur<br><br><br>    def add_texts(<br>        self,<br>        texts: Iterable[str],<br>        metadatas: Optional[List[dict]] = None,<br>        # ids: Optional[List[str]] = None,<br>        **kwargs: Any,<br>    ) -&gt; List[str]:<br>        &quot;&quot;&quot;Run more texts through the embeddings and add to the vectorstore.<br><br>        Args:<br>            texts: Iterable of strings to add to the vectorstore.<br>            kwargs: vectorstore specific parameters<br><br>        Returns:<br>            List of ids from adding the texts into the vectorstore.<br>        &quot;&quot;&quot;    <br>        embeddings = self.embed_documents(list(texts))<br><br>        uuids = []<br>        for document, embedding in zip(texts, embeddings):<br>            doc_id = uuid.uuid4()<br>            self.cur.execute(f&#39;INSERT INTO {self.table_name} (id, document, embedding) VALUES (%s, %s, %s) RETURNING id&#39;, (doc_id, document, embedding))<br>            uuids.append(self.cur.fetchone()[0])<br><br>        return uuids<br><br><br>    def embed_documents(self, texts: Iterable[str]):<br>        &quot;&quot;&quot;Generate embeddings for text documents.<br><br>        Args:<br>            texts: Iterable of strings to add to the vectorstore.<br><br>        Returns:<br>            List of embeddings adding into the vectorstore.<br>        &quot;&quot;&quot;    <br>        embeddings = self.model.encode(texts)<br>        return embeddings.tolist()<br><br>    def retrieve_embeddings(self, doc_ids: List[str]):<br>        &quot;&quot;&quot;Retrieve embeddings from vectorstore.<br><br>        Args:<br>            doc_ids: Iterable of strings to fetch embeddings from the vectorstore.<br><br>        Returns:<br>            List of embeddings added into the vectorstore.<br>        &quot;&quot;&quot; <br>        try:<br>            placeholder = &#39;, &#39;.join([&quot;&#39;&quot; + e + &quot;&#39;&quot; for e in doc_ids])<br>            self.cur.execute(f&#39;SELECT embedding FROM {self.table_name} WHERE ID IN ({placeholder});&#39;)<br>            embeddings = self.cur.fetchall()<br>        except Exception as e:<br>            self.logger.exception(e)<br>            embeddings = []<br><br>        return embeddings<br><br>    def retrieve_top_k_relevant_docs(self, k: int):<br>        &quot;&quot;&quot;Retrieve top k relevant documents.<br>        We order by embedding &lt;-&gt; embedding which will order by nearest neighbor score.<br>        The &lt;-&gt; operator computes the L2 distance between two vectors. By comparing to itself, it will order by smallest distance (most similar) to largest.<br><br>        Args:<br>            k: Number of relevant documents to return.<br><br>        Returns:<br>            List of top k relevant documents from the vectorstore.<br>        &quot;&quot;&quot; <br>        try:<br>            self.cur.execute(f&#39;SELECT id, document FROM {self.table_name} ORDER BY embedding &lt;-&gt; embedding LIMIT {k};&#39;)<br>            docs = self.cur.fetchall()<br>        except Exception as e:<br>            self.logger.exception(e)<br>            docs = []<br><br>        return docs</pre><p>Let’s store the few text embeddings into a custom table in PGVector and provide the required embedding size:</p><pre>from sentence_transformers import SentenceTransformer<br><br># Example usage<br>model = SentenceTransformer(&#39;all-mpnet-base-v2&#39;)<br>CONNECTION_STRING = &#39;postgresql://postgres:test@localhost:5432/vector_db&#39;<br>TABLE_NAME = &quot;documents&quot;<br><br>store = CustomPGVector(<br>    model=model,<br>    connection_string=CONNECTION_STRING,<br>    table_name=TABLE_NAME,<br>    embed_dim=768<br>    # table_columns=TABLE_COLUMNS<br>)<br><br>texts = [<br>&quot;Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.&quot;,<br>&quot;Last year COVID-19 kept us apart. This year we are finally together again.&quot;,<br>&quot;Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.&quot;,<br>&quot;With a duty to one another to the American people to the Constitution.&quot;,<br>&quot;And with an unwavering resolve that freedom will always triumph over tyranny.&quot;,<br>&quot;Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated.&quot;,<br>&quot;He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined.&quot;,<br>&quot;He met the Ukrainian people.&quot;,<br>&quot;From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.&quot;,<br>&quot;Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland.&quot;,<br>&quot;In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight.&quot;,<br>&quot;Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world.&quot;,<br>&quot;Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people.&quot;,<br>&quot;Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos.&quot;,<br>&quot;They keep moving.&quot;]<br><br>uuids = store.add_texts(texts)<br>print(uuids)</pre><pre>[&#39;3245d70d-7a92-487a-8d82-ed30083bea88&#39;, &#39;78082dd8-92e3-4206-8341-91f4807903ba&#39;, &#39;2442cd86-cc65-4c6b-bffe-0ce6a932b45c&#39;, &#39;987414da-559f-4a23-8a1e-acc332332ab0&#39;, &#39;6121d6d9-4b50-4117-a313-73369990aedd&#39;, &#39;7efb7549-5663-429f-ade2-daf1c812b2f3&#39;, &#39;50dc6f83-9ea8-4a98-8bcc-1052e2c4251c&#39;, &#39;2bf6f7c3-7870-4301-a2b5-b56efc215a53&#39;, &#39;b0cc8899-4519-4206-b106-4684d66d4a1b&#39;, &#39;a09f4de7-f7e4-4e8a-b5cb-e3d2597915d8&#39;, &#39;7cbad19a-4d12-4356-b8ab-fd7af0dcc96f&#39;, &#39;ef54de6c-cb2c-449f-b49f-aeb37b2da117&#39;, &#39;9435702c-68a9-40ed-bf7e-d5bb399d0a78&#39;, &#39;6231002f-28c5-4656-8e31-ab0b23d16ec9&#39;, &#39;3eba89e4-1bb4-4f28-ae75-5351ce8eceab&#39;]</pre><p>If you inspect the database with pgAdmin4 or another GUI, we should see something similar to the below from documents table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hfd8wkzEB9DWF63XO9gqSA.png" /><figcaption>Result set from documents table (Image by Author)</figcaption></figure><p>Let’s retrieve the added embeddings from the vector store.</p><pre>doc_id_list = [&#39;b4ceeb3a-e9dd-4869-955c-d13f5b9856cc&#39;, &#39;de2ccb67-4815-4045-be98-7138b61095f7&#39;]<br>store.retrieve_embeddings(doc_id_list)</pre><pre>[(array([ 7.78687373e-03,  7.02493936e-02, -3.64201586e-03,  2.79577896e-02,<br>          1.95163172e-02,  2.90637724e-02, -3.46862264e-02,  6.58348342e-03,<br>         -1.14316922e-02, -1.68400835e-02,  1.27797546e-02, -9.22441296e-03,<br>          3.14918645e-02, -8.96910205e-02,  2.33260784e-02, -9.64646786e-02,<br>          6.50184453e-02,  2.76170392e-02,  2.39608549e-02, -2.64637507e-02,<br>         -4.19980586e-02,  1.58222318e-02,  1.84183624e-02, -1.08865770e-02,<br>         -7.66699575e-03, -2.63356902e-02,  2.51230467e-02,  4.19513583e-02,<br>...<br>          2.44493168e-02, -3.36894318e-02, -1.79641752e-03,  1.42198168e-02,<br>          1.45910291e-34,  1.94061752e-02, -3.07820621e-03, -3.92877907e-02,<br>          9.16282833e-03,  5.92188491e-03, -2.34747995e-02, -2.07795277e-02,<br>          3.24819461e-02,  4.12504189e-02,  6.27856031e-02, -2.50972789e-02],<br>        dtype=float32),)]</pre><p>Let’s retrieve the top K highest-scoring documents from PGVector:</p><p>The key functionality:</p><ul><li>We select the id and document from the document table.</li><li>We order by embedding &lt;-&gt; embedding which will order by nearest neighbor score.</li><li>The &lt;-&gt; operator computes the L2 distance between two vectors. By comparing itself, it will order by smallest distance (most similar) to largest.</li><li>Limit to 5 results to get the top 5 matches.</li></ul><p>So this essentially finds the 5 embeddings that are most similar to themselves, which will be the ones with the highest scores/density.</p><pre>store.retrieve_top_k_relevant_docs(5)</pre><pre>[(&#39;de2ccb67-4815-4045-be98-7138b61095f7&#39;,<br>  &#39;Last year COVID-19 kept us apart. This year we are finally together again.&#39;),<br> (&#39;18e921d1-0a34-492c-8b36-f839690b9513&#39;,<br>  &#39;Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.&#39;),<br> (&#39;635e8328-99f9-4ce6-9b0e-b3ff4b8b463d&#39;,<br>  &#39;With a duty to one another to the American people to the Constitution.&#39;),<br> (&#39;45ae1d35-0cdd-4a55-86b7-9d56600fe691&#39;,<br>  &#39;And with an unwavering resolve that freedom will always triumph over tyranny.&#39;),<br> (&#39;b4ceeb3a-e9dd-4869-955c-d13f5b9856cc&#39;,<br>  &#39;Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.&#39;)]</pre><h4>8. Summary</h4><p>Unlocking the potential of PGVector, this article offers comprehensive guidance on storing and retrieving text embeddings effectively. Delving into VectorStore principles and PGVector’s features, it provides practical solutions for custom table storage and document retrieval, empowering to optimize text analytics workflows with PostgreSQL.</p><h4>9. References</h4><ul><li>PGAdmin UI installation — <a href="https://www.postgresql.org/ftp/pgadmin/pgadmin4/v8.2/windows/">https://www.postgresql.org/ftp/pgadmin/pgadmin4/v8.2/windows/</a></li><li>Postgress pgvector extension — <a href="https://bugbytes.io/posts/vector-databases-pgvector-and-langchain/">https://bugbytes.io/posts/vector-databases-pgvector-and-langchain/</a></li><li>Langchain pgvector — <a href="https://python.langchain.com/docs/integrations/vectorstores/pgvector">https://python.langchain.com/docs/integrations/vectorstores/pgvector</a></li><li>pgvector: <a href="https://github.com/pgvector/pgvector">https://github.com/pgvector/pgvector</a></li><li>pgvector DockerHub image: <a href="https://hub.docker.com/r/ankane/pgvector">https://hub.docker.com/r/ankane/pgvector</a></li></ul><p>I hope you enjoyed this article, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this article is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/AI-ML-Projects/blob/main/VectorDB-PGVector/pgvector-and-huggingface.ipynb">AI-ML-Projects/VectorDB-PGVector/pgvector-and-huggingface.ipynb at main · vinayak-shanawad/AI-ML-Projects</a></p><p>Visit us at <a href="https://www.datadriveninvestor.com/"><em>DataDrivenInvestor.com</em></a></p><p>Subscribe to DDIntel <a href="https://www.ddintel.com/"><em>here</em></a>.</p><p>Have a unique story to share? Submit to DDIntel <a href="https://datadriveninvestor.com/ddintelsubmission"><em>here</em></a>.</p><p>Join our creator ecosystem <a href="https://join.datadriveninvestor.com/"><em>here</em></a>.</p><p><a href="https://ddintel.datadriveninvestor.com/"><em>DDIntel</em> </a>captures the more notable pieces from our <a href="https://www.datadriveninvestor.com/"><em>main site</em></a> and our popular <a href="https://medium.datadriveninvestor.com/"><em>DDI Medium publication</em></a>. Check us out for more insightful work from our community.</p><p>DDI Official Telegram Channel: <a href="https://t.me/+tafUp6ecEys4YjQ1">https://t.me/+tafUp6ecEys4YjQ1</a></p><p>Follow us on <a href="https://www.linkedin.com/company/data-driven-investor"><em>LinkedIn</em></a>, <a href="https://twitter.com/@DDInvestorHQ"><em>Twitter</em></a>, <a href="https://www.youtube.com/c/datadriveninvestor"><em>YouTube</em></a>, and <a href="https://www.facebook.com/datadriveninvestor"><em>Facebook</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0463b5bcce0" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/beyond-relational-databases-taming-llm-transformer-embeddings-with-pgvector-f0463b5bcce0">Beyond Relational Databases: Taming LLM and Transformer Embeddings with PGVector</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an ML Pipeline (Part 2) — Model Registration and Serving with MLflow and KServe]]></title>
            <link>https://medium.com/analytics-vidhya/build-an-ml-pipeline-part-2-model-registration-and-serving-with-mlflow-and-kserve-a2158337e077?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2158337e077</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 10:26:49 GMT</pubDate>
            <atom:updated>2024-02-19T06:09:53.608Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build an ML Pipeline (Part 2) — Model Registration and Serving with MLflow and KServe</h3><h4>Seamless Model Deployment: MLflow and KServe Collaboration</h4><h4>🎯Goal</h4><p>In my <a href="https://medium.com/@vinayakshanawad/build-an-ml-pipeline-part-1-getting-started-with-kubeflow-v2-pipelines-74981b88db9a">previous post</a>, we looked at setting up the minikube cluster, installing Kubeflow pipelines, and creating the Kubeflow pipeline using the popular data science IRIS dataset.</p><p>Let’s continue with the MLflow setup, register the model using the MLflow model registry, and serve the model using KServe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3okI1Yaolj8w_8dD" /><figcaption>Photo by <a href="https://unsplash.com/@darya_jumelya?utm_source=medium&amp;utm_medium=referral">Darya Jum</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>🤔What is MLflow?</h4><p>MLflow is an open-source platform for managing the end-to-end ML lifecycle. It provides a range of features to help data scientists and ML engineers streamline their workflow and collaborate effectively.</p><p><strong>Some key features of MLflow include</strong></p><p><strong>1. Hyperparameter Tuning:</strong> MLflow integrates with popular hyperparameter optimization libraries like Hyperopt and Optuna, making it easy to perform hyperparameter tuning and track results.</p><p><strong>2. Experiment Tracking:</strong> MLflow allows users to track experiments, including parameters, metrics, and artifacts (model files and other output). This helps with comparing and reproducing different runs.</p><p><strong>3. Model Versioning:</strong> MLflow offers versioning for models, making it easy to track and manage different iterations of a model. This helps ensure reproducibility and simplifies model management.</p><p><strong>4. Model Registry:</strong> The Model Registry in MLflow allows you to organize and manage model versions, including staging, transitioning, and rolling back models. It adds a layer of control to the model deployment process.</p><p><strong>5. Artifact Store:</strong> MLflow supports various artifact stores, including local file systems, cloud storage (e.g., Amazon S3), and database-backed stores, for managing large model files and other artifacts.</p><p><strong>6. Model Packaging and Deployment:</strong> MLflow provides tools for packaging and deploying machine learning models as REST API endpoints, Docker containers, or other target environments. This simplifies the process of putting models into production.</p><p><strong>7. Automatic Logging:</strong> MLflow can automatically log machine learning framework information, code versions, and environment details during each run. This ensures comprehensive tracking and reproducibility.</p><h4>🛠️MLflow Setup</h4><p>Let’s install MLflow tracking server on Minikube cluster and understand how Kubeflow pipelines can interact with MLflow tracking server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uFSEPSoODTT3zbh18MHSw.jpeg" /><figcaption>MLflow Setup on Minikube (Image by Author)</figcaption></figure><h4>Prerequisites</h4><p>We need to create the service account that will be used in model deployment using KServe and secret that will be used in Kubeflow pipeline steps to register the models and load the model artifacts from S3, and MLflow tracking server to populate the model artifact details from s3 on MLflow UI.</p><p>Define the K8s manifest file to create a service account and secret.</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  namespace: kubeflow<br>  name: mlflow-sa<br>  annotations:<br>    eks.amazonaws.com/role-arn: &lt;role-arn-for-aws-s3-access&gt;<br>    serving.kserve.io/s3-endpoint: s3.&lt;region&gt;.amazonaws.com<br>    serving.kserve.io/s3-usehttps: &quot;1&quot;<br>    serving.kserve.io/s3-region: &quot;&lt;region&gt;&quot;<br>    serving.kserve.io/s3-useanoncredential: &quot;false&quot;<br><br>---<br><br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  namespace: kubeflow<br>  name: aws-credentials<br>  annotations:<br>     serving.kserve.io/s3-endpoint: s3.&lt;region&gt;.amazonaws.com<br>     serving.kserve.io/s3-usehttps: &quot;1&quot;<br>     serving.kserve.io/s3-region: &quot;&lt;region&gt;&quot;<br>     serving.kserve.io/s3-useanoncredential: &quot;false&quot;<br>type: Opaque<br>stringData:<br>  AWS_ACCESS_KEY_ID: &lt;aws-access-key&gt;<br>  AWS_SECRET_ACCESS_KEY: &lt;aws-secret-key&gt;<br>  AWS_DEFAULT_REGION: &lt;region&gt;<br><br>---<br><br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  namespace: kubeflow<br>  name: mlflow-sa<br>secrets:<br>- name: aws-credentials</pre><h4>🚀MLflow deployment</h4><p>Let’s create a Docker image that builds the MLflow tracking server then build and push into Dockerhub. (You can use my docker image from <a href="https://hub.docker.com/r/vinayaks117/mlflow-repo/tags">Dockerhub</a>)</p><pre># Defining base image<br>FROM python:3.8.2-slim<br><br># Installing MLflow from PyPi<br>RUN pip install mlflow<br><br># Defining start-up command<br>EXPOSE 5000<br>ENTRYPOINT [&quot;mlflow&quot;, &quot;server&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;5000&quot;]</pre><p>Define the K8s manifest file to create a simple PVC (mlflow-pvc.yaml) for the storage (size: 10 MB) of the metadata. But the cool thing is that MLflow supports lots of different options for these, including cloud services. We can use MySQL, PostgreSQL, etc as Metadata store. I am using S3 bucket on AWS as artifact store that contains all our model artifacts.</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  namespace: kubeflow<br>  name: mlflow-pvc <br>spec:<br>  accessModes:<br>    - ReadWriteMany<br>  resources:<br>    requests:<br>      storage: 10Mi</pre><p>Let’s create PVC using the following command.</p><pre>kubectl apply -f mlflow-pvc.yaml</pre><p>Let’s define another K8s manifest file (mlflow.yaml) for MLflow setup on Minikube.</p><pre># Creating MLflow deployment<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  namespace: kubeflow<br>  name: mlflowserver<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: mlflowserver<br>  template:<br>    metadata:<br>      labels:<br>        app: mlflowserver<br>    spec:<br>      volumes:<br>        - name: mlflow-pvc<br>          persistentVolumeClaim:<br>            claimName: mlflow-pvc<br>      containers:<br>      - name: mlflowserver<br>        image: vinayaks117/mlflow-repo:v2.0<br>        imagePullPolicy: Always<br>        args:<br>        - --host=0.0.0.0<br>        - --port=5000<br>        - --backend-store-uri=/opt/mlflow/backend<br>        - --default-artifact-root=s3://kubeflow-mlflow/experiments<br>        - --workers=2<br>        env:<br>        - name: AWS_ACCESS_KEY_ID<br>          valueFrom:<br>            secretKeyRef:<br>              name: aws-credentials<br>              key: AWS_ACCESS_KEY_ID<br>        - name: AWS_SECRET_ACCESS_KEY<br>          valueFrom:<br>            secretKeyRef:<br>              name: aws-credentials<br>              key: AWS_SECRET_ACCESS_KEY<br>        - name: AWS_DEFAULT_REGION<br>          valueFrom:<br>            secretKeyRef:<br>              name: aws-credentials<br>              key: AWS_DEFAULT_REGION<br>        ports:<br>        - name: http<br>          containerPort: 5000<br>          protocol: TCP<br>        volumeMounts:<br>        - name: mlflow-pvc<br>          mountPath: /opt/mlflow/backend<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  namespace: kubeflow<br>  name: mlflowserver<br>spec:<br>  selector:<br>    app: mlflowserver<br>  ports:<br>  - protocol: TCP<br>    port: 5000<br>    targetPort: 5000</pre><p>The syntax here will seem quite familiar to you; just pay attention to the arguments we’ll use when building our Docker image.</p><p>While you’re likely familiar with the “host” and “port” arguments, the latter two might be new. They specify where MLflow should log our model metadata for the model registry and where to log the model artifacts. In this setup, I’m utilizing a simple Persistent Volume Claim (PVC).</p><p>--backend-store-uri=/opt/mlflow/backend is used to store the metadata (model parameters, evaluation metrics, etc) in PVC.</p><p>--default-artifact-root=s3://kubeflow-mlflow/experiments is used to store the artifacts (model artifacts) in S3.</p><blockquote>Note: We are configuring AWS credentials in environment variables because we would like to populate the registered model artifacts from S3 in MLflow UI.</blockquote><h4>K8s Service</h4><p>Let’s create an internal service so that we can interact with the MLflow tracking server via internal service from Kubeflow pipelines.</p><p>Let’s create an MLflow deployment and service using the following command.</p><pre>kubectl apply -f mlflow.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*KvoHAZFftnTvUBantgvPzg.jpeg" /><figcaption>MLflow installation status (Image by Author)</figcaption></figure><p>Let’s verify that the MLflow dashboard is accessible by port-forwarding:</p><pre>kubectl port-forward -n kubeflow svc/mlflowserver 8081:5000</pre><p>Then, open the MLflow dashboard at <a href="http://localhost:8080/">http://localhost:8081/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NXiz1XhheFxpH0SwkboAZQ.png" /><figcaption>MLflow dashboard (Image by Author)</figcaption></figure><h4>🛠️KServe Setup</h4><p>Please refer to my <a href="https://medium.com/mlearning-ai/deploying-a-huggingface-bert-model-with-kserve-3e521d69e596">previous article</a> where I showed how to setup KServe on Minikube.</p><p>Let’s look at the status of all Kubeflow pipeline, MLflow server, and KServe components from K8s cluster using kubectl get pods -A command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzBu_rKEESYOtVq8XB9Ahg.png" /><figcaption>Minikube Cluster Staus (Image by Author)</figcaption></figure><p>Let’s go through the remaining steps (Model provisioning, Model evaluation, and Model deployment) from the ML pipeline as we discussed in my <a href="https://medium.com/@vinayakshanawad/build-an-ml-pipeline-part-1-getting-started-with-kubeflow-v2-pipelines-74981b88db9a">previous post</a>.</p><h4>5. Register a model in the MLflow model registry — AWS S3</h4><pre>@component(<br>    packages_to_install=[&quot;pandas&quot;, &quot;numpy&quot;, &quot;scikit-learn&quot;, &quot;mlflow&quot;, &quot;boto3&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def register_model(data_path: str, aws_access_key_id: str, aws_secret_access_key: str, aws_default_region: str) -&gt; dict:<br>    import pandas as pd<br>    import numpy as np<br>    import pickle<br>    import os<br>    import mlflow<br>    from mlflow.models import infer_signature<br>    from sklearn import datasets<br>   <br>    with open(f&#39;{data_path}/model.pkl&#39;,&#39;rb&#39;) as f:<br>        logistic_reg_model = pickle.load(f)<br>    <br>    # Infer the model signature<br>    X_test = np.load(f&#39;{data_path}/X_test.npy&#39;, allow_pickle=True)<br>    y_pred = logistic_reg_model.predict(X_test)<br>    signature = infer_signature(X_test, y_pred)<br>    <br>    # Set AWS credentials in the environment<br>    os.environ[&quot;AWS_ACCESS_KEY_ID&quot;] = aws_access_key_id<br>    os.environ[&quot;AWS_SECRET_ACCESS_KEY&quot;] = aws_secret_access_key<br>    os.environ[&quot;AWS_DEFAULT_REGION&quot;] = aws_default_region<br>    <br>    # log and register the model using MLflow scikit-learn API<br>    mlflow.set_tracking_uri(&quot;http://mlflowserver.kubeflow:5000&quot;)<br>    reg_model_name = &quot;SklearnLogisticRegression&quot;<br>    <br>    experiment_id = mlflow.create_experiment(&quot;test-1&quot;) <br>    <br>    with mlflow.start_run(experiment_id=experiment_id) as run:<br>        mlflow.log_param(&#39;max_iter&#39;, 500)<br><br>        # Log model artifact to S3<br>        artifact_path = &quot;sklearn-model&quot;      <br>        mlflow.log_artifact(local_path=f&#39;{data_path}/model.pkl&#39;, artifact_path=artifact_path)<br>        <br>        model_info = mlflow.sklearn.log_model(<br>            sk_model=logistic_reg_model,<br>            artifact_path=&quot;sklearn-model&quot;,<br>            signature=signature,<br>            registered_model_name=reg_model_name,<br>        )<br>    <br>    model_uri = f&quot;runs:/{run.info.run_id}/sklearn-model&quot;<br>    <br>    # Register model linked to S3 artifact location          <br>    mlflow.register_model(<br>        model_uri,<br>        reg_model_name<br>    )<br><br>    return {&quot;artifact_path&quot;: artifact_path, &quot;artifact_uri&quot;: run.info.artifact_uri}</pre><p>Kubernetes services are accessible through DNS internally using the name &lt;service-name&gt;.&lt;namespace&gt; , earlier we deployed Kubernetes service mlflowserver exposes this container port as a service on port 5000 and connecting to mlflowserver.kubeflow:5000 routes requests to the MLflow container on port 5000.</p><p>Hence we are using MLServer tracking URI as http://mlflowserver.kubeflow:5000 in the above step.</p><p>Notice that, we are setting AWS credentials in environment variables because Kubeflow pipeline steps run in separate pods from the base cluster and need those credentials to store model artifacts in S3.</p><p>We can observe model experiments are being tracked in MLflow UI after executing the above step in the ML pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6RpLaaK6bWVaQHGJbQcUFw.png" /><figcaption>Model experiments (Image by Author)</figcaption></figure><p>Another approach to viewing the model registry is from the perspective of the Registered Models, not the Experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdrTAbNQFf6O2MeAMJfGAA.png" /><figcaption>Registered models (Image by Author)</figcaption></figure><p>The Model Versions are how a model lineage can be traced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQv_SWJnYFEaxtWpUSDv-w.png" /><figcaption>Metadata and Artifact details (Image by Author)</figcaption></figure><p>We can see that logged parameter max_iter=500 from metadata store, logged and registered model artifacts from S3.</p><h4>6. Model evaluation</h4><p>Let’s load the model from the Model Registry and score the test data. We can observe that we are setting AWS credentials in environment variables because we need load model artifacts in S3 and use it for prediction on test data.</p><pre>@component(<br>    packages_to_install=[&quot;pandas&quot;, &quot;numpy&quot;, &quot;scikit-learn&quot;, &quot;mlflow&quot;, &quot;boto3&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def predict_on_test_data(data_path: str, model_info: dict, aws_access_key_id: str, aws_secret_access_key: str, aws_default_region: str) -&gt; str:<br>    import pandas as pd<br>    import numpy as np<br>    import pickle<br>    import os<br>    import mlflow<br>    <br>    # Set AWS credentials in the environment<br>    os.environ[&quot;AWS_ACCESS_KEY_ID&quot;] = aws_access_key_id<br>    os.environ[&quot;AWS_SECRET_ACCESS_KEY&quot;] = aws_secret_access_key<br>    os.environ[&quot;AWS_DEFAULT_REGION&quot;] = aws_default_region<br>    <br>    artifact_path = model_info[&quot;artifact_path&quot;]<br>    artifact_uri = model_info[&quot;artifact_uri&quot;]<br>    <br>    mlflow.set_tracking_uri(&quot;http://mlflowserver.kubeflow:5000&quot;)<br>    model_uri = f&quot;{artifact_uri}/{artifact_path}&quot;<br>    logistic_reg_model = mlflow.sklearn.load_model(model_uri)<br>        <br>    X_test = np.load(f&#39;{data_path}/X_test.npy&#39;,allow_pickle=True)<br>    y_pred = logistic_reg_model.predict(X_test)<br>    np.save(f&#39;{data_path}/y_pred.npy&#39;, y_pred)<br>    <br>    X_test = np.load(f&#39;{data_path}/X_test.npy&#39;,allow_pickle=True)<br>    y_pred_prob = logistic_reg_model.predict_proba(X_test)<br>    np.save(f&#39;{data_path}/y_pred_prob.npy&#39;, y_pred_prob)<br>    <br>    return model_uri</pre><h4>7. Model deployment</h4><p>We will utilize the KServe Python Client SDK that interacts with KServe control plane APIs for executing operations on a remote KServe cluster, such as creating, patching and deleting of a InferenceService instance.</p><p>Let’s create a InferenceService using <a href="https://kserve.github.io/website/0.11/sdk_docs/docs/V1beta1InferenceService/">V1beta1InferenceService</a> that is the Schema for the InferenceServices API in the following example.</p><pre>@component(<br>    packages_to_install=[&quot;kserve&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def model_serving(model_uri: str):<br>    from kubernetes import client <br>    from kserve import KServeClient<br>    from kserve import constants<br>    from kserve import utils<br>    from kserve import V1beta1InferenceService<br>    from kserve import V1beta1InferenceServiceSpec<br>    from kserve import V1beta1PredictorSpec<br>    from kserve import V1beta1SKLearnSpec<br>    import os<br><br>    namespace = utils.get_default_target_namespace()<br>    <br>    name=&#39;sklearn-iris-v2&#39;<br>    kserve_version=&#39;v1beta1&#39;<br>    api_version = constants.KSERVE_GROUP + &#39;/&#39; + kserve_version<br><br>    isvc = V1beta1InferenceService(api_version=api_version,<br>                                   kind=constants.KSERVE_KIND,<br>                                   metadata=client.V1ObjectMeta(<br>                                       name=name, namespace=namespace, annotations={&#39;sidecar.istio.io/inject&#39;:&#39;false&#39;}),<br>                                   spec=V1beta1InferenceServiceSpec(<br>                                   predictor=V1beta1PredictorSpec(<br>                                   service_account_name=&#39;mlflow-sa&#39;,<br>                                   sklearn=(V1beta1SKLearnSpec(storage_uri=model_uri)))))</pre><p>Let’s look at the model deployment status and run some inference tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAJcXHnNk6e7zW1yVmDDNg.png" /><figcaption>Model deployment status (Image by Author)</figcaption></figure><p>First, we need to do some port forwarding work so our model’s port is exposed to our local system with the command:</p><pre>kubectl port-forward -n istio-system service/istio-ingressgateway 8080:80</pre><p>We’ll use the curl command to send the input json file as input to the predict method on our InferenceService on KServe with the command:</p><pre>curl -v -H &quot;Host: sklearn-iris-v2.kubeflow.example.com&quot; -H &quot;Content-Type: application/json&quot; &quot;http://localhost:8080/v1/models/sklearn-iris-v2:predict&quot; -d @./iris-input.json</pre><p>The response will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-01ffBug2WU2hx9p9-XTw.png" /><figcaption>Model inference result (Image by Author)</figcaption></figure><h4>🤖The Complete Pipeline</h4><p>Now it’s time to put together all components and define the complete pipeline.</p><pre>from kubernetes import client, config<br>import base64<br><br>@pipeline(<br>    name=&quot;iris-pipeline&quot;,<br>)<br>def iris_pipeline(data_path: str):<br>    pvc1 = kubernetes.CreatePVC(<br>        # can also use pvc_name instead of pvc_name_suffix to use a pre-existing PVC<br>        pvc_name_suffix=&#39;-iris-mlflow-pvc&#39;,<br>        access_modes=[&#39;ReadWriteMany&#39;],<br>        size=&#39;1Mi&#39;,<br>        storage_class_name=&#39;standard&#39;<br>    )<br>    <br>    # Load Kubernetes configuration<br>    config.load_kube_config()<br><br>    # Fetch the AWS credentials from the secret<br>    secret_name = &quot;aws-credentials&quot;<br>    secret_namespace = &quot;kubeflow&quot;<br>    secret_key_id = &quot;AWS_ACCESS_KEY_ID&quot;<br>    secret_key_access = &quot;AWS_SECRET_ACCESS_KEY&quot;<br>    secret_region = &quot;AWS_DEFAULT_REGION&quot;<br><br>    v1 = client.CoreV1Api()<br>    secret = v1.read_namespaced_secret(secret_name, namespace=secret_namespace)<br>    <br>    # Convert bytes to string<br>    aws_access_key_id = base64.b64decode(secret.data[secret_key_id]).decode(&#39;utf-8&#39;)<br>    aws_secret_access_key = base64.b64decode(secret.data[secret_key_access]).decode(&#39;utf-8&#39;)<br>    aws_default_region = base64.b64decode(secret.data[secret_region]).decode(&#39;utf-8&#39;)<br>    <br>    # Data preparation<br>    prepare_data_task = prepare_data(data_path=data_path)<br>    kubernetes.mount_pvc(prepare_data_task, pvc_name=pvc1.outputs[&#39;name&#39;], mount_path=&#39;/data&#39;)<br>    <br>    # Split data into Train and Test set<br>    train_test_split_task = train_test_split(data_path=data_path) <br>    kubernetes.mount_pvc(train_test_split_task, pvc_name=pvc1.outputs[&#39;name&#39;], mount_path=&#39;/data&#39;)<br>    train_test_split_task.after(prepare_data_task)<br>    <br>    # Model training<br>    training_basic_classifier_task = training_basic_classifier(data_path=data_path)<br>    kubernetes.mount_pvc(training_basic_classifier_task, pvc_name=pvc1.outputs[&#39;name&#39;], mount_path=&#39;/data&#39;)<br>    training_basic_classifier_task.after(train_test_split_task)<br>    <br>    # Register a model in the MLflow model registry<br>    register_model_task = register_model(data_path=data_path, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, aws_default_region=aws_default_region)<br>    kubernetes.mount_pvc(register_model_task, pvc_name=pvc1.outputs[&#39;name&#39;], mount_path=&#39;/data&#39;)<br>    kubernetes.mount_pvc(register_model_task, pvc_name=&quot;mlflow-pvc&quot;, mount_path=&#39;/opt/mlflow/&#39;)<br>    register_model_task.after(training_basic_classifier_task)<br>    <br>    # Model evaluation<br>    predict_on_test_data_task = predict_on_test_data(data_path=data_path, model_info=register_model_task.output, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, aws_default_region=aws_default_region)<br>    kubernetes.mount_pvc(predict_on_test_data_task, pvc_name=pvc1.outputs[&#39;name&#39;], mount_path=&#39;/data&#39;)<br>    predict_on_test_data_task.after(register_model_task)<br>    <br>    # Model deployment<br>    model_serving_task = model_serving(model_uri=predict_on_test_data_task.output)<br>    model_serving_task.after(predict_on_test_data_task)<br>    <br>    delete_pvc1 = kubernetes.DeletePVC(pvc_name=pvc1.outputs[&#39;name&#39;]).after(model_serving_task)</pre><p>Fetch AWS credentials from Secret: As discussed in prerequisites, we created a secret that holds AWS credentials that needs to be passed in Model registration and evaluation steps.</p><p>We created pvc1 persistent volume claim with size of 1MB that can be used to store the prepared data hence we need to mount pvc1 to data preparation step and remaining steps to access to prepared data from PVC.</p><p>We will log the parameters to metadata store that is PVC in this example and log the model artifacts to AWS S3 then return the dictionary that holds the artifact_path and artifact_uri details.</p><p>We can notice that register_model_task.output object passed to model evaluation step and that holds the artifact details from model registration step.</p><p>We connect pipeline steps and make sure those execute one after another using after function from Kubeflow. Finally, we delete the pvc1 to cleanup the data.</p><h4>🎉Conclusion</h4><p>We embarked on the journey to build a comprehensive end-to-end machine learning workflow using Kubernetes, Minikube, and a trio of powerful open-source technologies — Kubeflow Pipelines, MLflow, and KServe. The objective was to seamlessly transition from data processing to model deployment within a unified environment.</p><p>By the end of this comprehensive guide, hope you will be equipped to navigate through the intricacies of building, deploying, and monitoring machine learning models using the potent combination of Kubernetes and these cutting-edge open-source tools.</p><p>I hope you enjoyed this blog post, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this post is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/AI-ML-Projects/tree/main/ML-Pipeline-with-Kubeflow-MLflow-KServe">AI-ML-Projects/ML-Pipeline-with-Kubeflow-MLflow-KServe at main · vinayak-shanawad/AI-ML-Projects</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2158337e077" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/build-an-ml-pipeline-part-2-model-registration-and-serving-with-mlflow-and-kserve-a2158337e077">Build an ML Pipeline (Part 2) — Model Registration and Serving with MLflow and KServe</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an ML Pipeline (Part 1) — Getting Started with Kubeflow V2 Pipelines]]></title>
            <link>https://medium.com/analytics-vidhya/build-an-ml-pipeline-part-1-getting-started-with-kubeflow-v2-pipelines-74981b88db9a?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/74981b88db9a</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 10:14:03 GMT</pubDate>
            <atom:updated>2023-12-04T11:50:45.467Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build an ML Pipeline (Part 1) — Getting Started with Kubeflow V2 Pipelines</h3><h4>Discovering Kubeflow: Launching into ML Pipelines</h4><h4>🎯Goal</h4><p>To build an end-to-end machine learning workflow, we will harness the power and flexibility of Kubernetes and minikube by leveraging key open-source technologies — Kubeflow Pipelines, MLflow, and KServe.</p><p>By bringing together these components on minikube, we can go from data to deployment with an integrated pipeline — training effective models with Kubeflow, registering them with MLflow for versioning and lineage, and then serving predictions robustly with KServe. This unified environment gives us an extensible platform to develop, deploy, and monitor our models with the tools purpose-built for machine learning and Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zkrJm4NwU6Q1FBzUWehUaQ.jpeg" /><figcaption>End-to-end ML Pipeline using Kubeflow, MLflow, and KServe (Image by Author)</figcaption></figure><p>Let’s focus on setting up the minikube cluster, installing Kubeflow pipelines, and creating the Kubeflow pipeline using the popular data science IRIS dataset in this post.</p><p>We will look at the MLflow setup, register the model into the MLflow model registry, and serve the model using KServe in my <a href="https://medium.com/@vinayakshanawad/build-an-ml-pipeline-part-2-model-registration-and-serving-with-mlflow-and-kserve-a2158337e077">next post</a>.</p><h4>🤔What is Minikube?</h4><p>Minikube is a tool that makes it easy to run Kubernetes locally. Here are some key things to know about minikube:</p><ul><li>It is free, open source, and available for Linux, macOS and Windows.</li><li>Minikube runs a single-node Kubernetes cluster (Master and Node processes run on ONE machine) inside a VM on your local machine.</li><li>It will have docker pre-installed and set up kubectl to communicate with it.</li><li>This provides a simple way to test and develop against a real Kubernetes cluster without needing the full production infrastructure.</li><li>Once developed, the same apps can be deployed into cloud Kubernetes clusters.</li></ul><h4>🛠️Minikube Setup</h4><p>We can reference the <a href="https://minikube.sigs.k8s.io/docs/start/">minikube documentation</a> to install the minikube cluster on our Windows machine.</p><p>Well, I have the Chocolatey package manager installed if not please install the Chocolatey and use the following command to install the minikube cluster.</p><pre>choco install minikube</pre><p>Let’s start our minikube cluster, we notice from logs that by default it uses the docker as a driver to start the cluster means that minikube will configure the Kubernetes cluster to use docker as its container runtime.</p><pre>minikube start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AXEjGQhZZ85QPBxzQFJhQQ.png" /><figcaption>Minkube start logs (Image by Author)</figcaption></figure><p>How to manage minikube cluster?</p><p>Check the minikube cluster status — minikube status</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*ovKPYLRje5asEmjQ76rLVg.png" /><figcaption>Minikube cluster status (Image by Author)</figcaption></figure><p>Stop the cluster — minikube stop</p><p>Delete all of the minikube clusters — minikube delete --all</p><p>List all the pods within the minikube cluster — minikube get pod -A</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*RUkD3Uuh9uiVkuoXQqef8w.png" /><figcaption>All pods within the Minikube cluster (Image by Author)</figcaption></figure><p>Awesome, we have a minikube cluster setup ready in our local Windows machine. Let’s setup a Kubeflow pipelines on minikube cluster.</p><h4>🤔What is Kubeflow Pipeline?</h4><p>Kubeflow Pipelines provides a platform for building and deploying portable and scalable machine-learning workflows on Kubernetes. Some key capabilities:</p><ul><li>An SDK for declaring pipelines as code — the workflows are defined as Python functions decorated with Kubeflow’s domain-specific language (DSL).</li><li>Run compute-intensive jobs as Kubernetes pods and jobs, leveraging Kubernetes for distribution and scaling.</li><li>End-to-end ML workflows — steps for data ingestion, feature engineering, model training, evaluation, etc can be orchestrated.</li><li>Notebooks to interactively develop and test pipelines.</li><li>CLI and UI to monitor pipeline runs on Kubernetes.</li><li>Portable — pipelines authored on a laptop can run on multi-node Kubernetes clusters.</li></ul><h4>🛠️Kubeflow Setup</h4><p>Now that we have the prerequisite K8s cluster installed, it’s time for the main show.</p><p>The easiest way to install Kubeflow locally is to use the manifest files from the Git repo.</p><pre>kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.0.0&quot;<br><br>kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io<br><br>kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=2.0.0&quot;</pre><p>The Kubeflow Pipelines deployment may take several minutes to complete.</p><p>Let’s check if kubeflow installed on minikube cluster.</p><pre>kubectl get pods -A</pre><p>This will list all pods across all namespaces. As long as the status is Running for all of them, you’re good to go. If any of them are not, sometimes you just need to wait a little longer. Your list should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*Yl1sAzlGEN2kvK3Dt1bmaQ.png" /><figcaption>Kubeflow installation status (Image by Author)</figcaption></figure><p>Let’s verify that the Kubeflow Pipelines UI or dashboard is accessible by port-forwarding:</p><pre>kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80</pre><p>Then, open the Kubeflow Pipelines UI at <a href="http://localhost:8080/">http://localhost:8080/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1dfCKQ4pxqpAIsLMrSyrw.png" /><figcaption>Kubeflow Pipeline UI (Image by Author)</figcaption></figure><h4>🚀Kubeflow Pipelines</h4><p>Kubeflow pipelines consist of components, each representing a step in the pipeline. Each component runs in its Docker container, allowing each step to have its set of dependencies without affecting other components.</p><p>For every component we build, we’ll either create a new Docker image or use an existing one. These images take inputs, execute specific operations, and produce outputs. Additionally, we’ll maintain a distinct Python script, pipeline.py, which converts each Docker image into a pipeline component and assembles the components into a complete pipeline.</p><p>We will create data-preparation, train-test-split, model training, model evaluation, model provisioning/register model, and model serving steps using Kubeflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/0*6MNsCXoKmh0DvPS3" /><figcaption>ML Pipeline (Image by Author)</figcaption></figure><p>Let’s go through each step in detail. Please note that we are going to build the Kubeflow pipeline using Kubeflow SDK v2.</p><p><strong>1. Import the required libraries</strong></p><pre>from kfp.dsl import component, pipeline<br>import kfp<br>from kfp import kubernetes</pre><p>We will use kfp.dsl module contains domain-specific language objects used to compose pipelines.</p><p><strong>Component</strong>: Decorator for Python-function-based components.</p><p><strong>Pipeline</strong>: Decorator used to construct a pipeline.</p><p><strong>2. Data Preparation</strong></p><p>We are using Python 3.9 image that has a Python interpreter that is compatible with KFP and specifies required libraries in the packages_to_install parameter for data preparation.</p><p>Kubeflow will pull the Python 3.9 image from the docker hub if it is not already present locally, install the specified libraries, and drop NA values from the IRIS dataset.</p><pre>@component(<br>    packages_to_install=[&quot;pandas&quot;, &quot;numpy&quot;, &quot;scikit-learn&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def prepare_data(data_path: str):<br>    import pandas as pd<br>    import os<br>    from sklearn import datasets<br>    <br>    # Load dataset<br>    iris = datasets.load_iris()<br>    df = pd.DataFrame(iris.data, columns=iris.feature_names)<br>    df[&#39;species&#39;] = iris.target<br>    <br>    df = df.dropna()<br>    df.to_csv(f&#39;{data_path}/final_df.csv&#39;, index=False)</pre><p><strong>3. Split data into Train and Test set</strong></p><p>We use a similar component as data preparation in this step.</p><pre>@component(<br>    packages_to_install=[&quot;pandas&quot;, &quot;numpy&quot;, &quot;scikit-learn&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def train_test_split(data_path: str):    <br>    import pandas as pd<br>    import numpy as np<br>    import os<br>    from sklearn.model_selection import train_test_split<br>    <br>    final_data = pd.read_csv(f&#39;{data_path}/final_df.csv&#39;)<br>    <br>    target_column = &#39;species&#39;<br>    X = final_data.loc[:, final_data.columns != target_column]<br>    y = final_data.loc[:, final_data.columns == target_column]<br>    <br>    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,stratify = y, random_state=47)<br>    <br>    np.save(f&#39;{data_path}/X_train.npy&#39;, X_train)<br>    np.save(f&#39;{data_path}/X_test.npy&#39;, X_test)<br>    np.save(f&#39;{data_path}/y_train.npy&#39;, y_train)<br>    np.save(f&#39;{data_path}/y_test.npy&#39;, y_test)</pre><p><strong>4. Model training</strong></p><p>We use a similar component as data preparation in this step and train a logistic regression model.</p><pre>@component(<br>    packages_to_install=[&quot;pandas&quot;, &quot;numpy&quot;, &quot;scikit-learn&quot;],<br>    base_image=&quot;python:3.9&quot;,<br>)<br>def training_basic_classifier(data_path: str):<br>    import pandas as pd<br>    import numpy as np<br>    import os<br>    from sklearn.linear_model import LogisticRegression<br>    <br>    X_train = np.load(f&#39;{data_path}/X_train.npy&#39;,allow_pickle=True)<br>    y_train = np.load(f&#39;{data_path}/y_train.npy&#39;,allow_pickle=True)<br>    <br>    classifier = LogisticRegression(max_iter=500)<br>    classifier.fit(X_train,y_train)<br>    import pickle<br>    with open(f&#39;{data_path}/model.pkl&#39;, &#39;wb&#39;) as f:<br>        pickle.dump(classifier, f)</pre><h4>📌Summary</h4><ol><li>The article focuses on building an end-to-end machine learning workflow using Minikube, Kubeflow Pipelines, MLflow, and KServe.</li><li>Minikube was introduced as a tool to run Kubernetes locally, providing a simple and controlled environment for testing and development.</li><li>Kubeflow Pipelines, a platform for scalable ML workflows on Kubernetes, is deployed on Minikube, offering an SDK for declaring pipelines as code.</li><li>The article highlights key Minikube commands for managing the cluster and checking its status.</li><li>Kubeflow Pipelines UI is accessed through port-forwarding, providing a user-friendly interface for monitoring pipeline runs.</li><li>The <a href="https://medium.com/@vinayakshanawad/build-an-ml-pipeline-part-2-model-registration-and-serving-with-mlflow-and-kserve-a2158337e077">upcoming posts</a> in the series promise insights into MLflow setup, model registration in the MLflow model registry, and model serving using KServe.</li></ol><p>I hope you enjoyed this blog post, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74981b88db9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/build-an-ml-pipeline-part-1-getting-started-with-kubeflow-v2-pipelines-74981b88db9a">Build an ML Pipeline (Part 1) — Getting Started with Kubeflow V2 Pipelines</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serving Hugging Face Transformers: Optimizing Custom Model Deployment with Seldon Core]]></title>
            <link>https://medium.com/@vinayakshanawad/serving-hugging-face-transformers-optimizing-custom-model-deployment-with-seldon-core-a593f6ea7549?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a593f6ea7549</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[seldon-core]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 10:51:42 GMT</pubDate>
            <atom:updated>2024-02-19T05:56:07.483Z</atom:updated>
            <content:encoded><![CDATA[<h4>Transforming Deployment: Seldon Core Meets Hugging Face</h4><h4>🎯Goal</h4><p>The objective of this post is to demonstrate how we can deploy custom NLP models on Seldon Core v2. The custom model I have is the uniqueness model defines a score between 0 to 1 which measures the level of uniqueness in the tweets. This model computes a single uniqueness score by taking a list of tweets as input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a8Wt02kxF2_KLoOZ" /><figcaption>Photo by <a href="https://unsplash.com/@billjelen?utm_source=medium&amp;utm_medium=referral">Bill Jelen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>🤔What is Seldon Core?</h4><p>Seldon Core is an open-source platform for deploying, scaling, and monitoring machine learning models in Kubernetes environments.</p><ol><li><strong>Kubernetes-Native</strong>: Seldon Core is designed to work seamlessly with Kubernetes, making it easy to deploy ML in containerized environments. It leverages Kubernetes’ scalability and flexibility for deploying models.</li><li><strong>Model Serving</strong>: It provides a framework for serving ML models, enabling you to expose your models as REST APIs or gRPC services.</li><li><strong>Scaling</strong>: Seldon Core supports automatic scaling of model deployments based on demand. It can horizontally scale model replicas to handle increased traffic, ensuring optimal performance.</li><li><strong>Custom Deployment Workflows</strong>: You can define custom deployment and scaling workflows for your models using Seldon Core, allowing you to tailor the serving infrastructure to your specific needs.</li><li><strong>Monitoring and Metrics</strong>: The platform offers monitoring and metrics capabilities, allowing you to track the performance of your deployed models and gain insights into how they are behaving in production.</li></ol><h4>🧠Inference Servers in Core v2</h4><p>Seldon V2 supports any V2 protocol inference server. At present Seldon automatically installs MLServer and Triton Inference server. Models are scheduled onto servers based on the server’s resources and whether the capabilities of the server match the requirements specified in the Model request.</p><p><strong>Inference Runtimes</strong></p><p>Inference runtimes allow you to define how your model should be used within the MLServer or Triton Inference server.</p><p>Out of the box, MLServer comes with a set of pre-packaged runtimes that let you interact with a subset of the following ML frameworks. This allows you to start serving models saved in these frameworks straight away.</p><p><em>Scikit-Learn, XGBoost, HuggingFace, LightGBM, MLflow, Alibi-Detect and Alibi-Explain</em></p><p>Even Triton Inference servers cover a wide range of runtimes.</p><p><em>TensorFlow, PyTorch, ONNX, TensorRT, OpenVINO</em></p><h4>🚀Deploying custom NLP models in Seldon Core V2</h4><p>In our use case, the Hugging Face inference runtime offered by MLServer is not enough because we need the following extra custom functionality which is not included in mlserver and mlserver-huggingface inference runtime.</p><ul><li>Having custom preprocessing and postprocessing steps.</li><li>Rollout of custom inference server with extra dependencies (Sentence Transformer package in this use case)</li></ul><p>This tutorial walks through the steps required to take a custom NLP model from your machine to a production deployment on Kubernetes.</p><p><strong>1. Use Case</strong></p><p>For this tutorial, we are going to use <a href="https://www.kaggle.com/c/nlp-getting-started/overview">Kaggle competition dataset</a>, which consists of fake and real Tweets about disasters. We need to build a uniqueness model that defines a score between 0 to 1 which measures the level of uniqueness in the tweets. This model computes a single uniqueness score by taking a list of tweets as input.</p><p><strong>2. Model Serving</strong></p><p>The next step will be to serve our model using mlserver. For that, we will first implement an extension that serves as the <em>runtime</em> to perform inference using our custom Text Uniqueness model.</p><p>Deploying custom models in Seldon Core V2 looks like the below at a high level. You can also reference my <a href="https://github.com/vinayak-shanawad/AI-ML-Projects/tree/main/SeldonCore-Custom-Model-Server">GitHub repo</a> for project structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*UteB75mbQTSYy7AwLigh1Q.png" /><figcaption>Custom MLServer Setup (Photo by Author)</figcaption></figure><p><strong>Custom inference runtime</strong></p><p>In order to get our model ready to run on MLServer we need to wrap it in a single Python class with two methods, load() and predict(). Let’s take a look at the code (found in inference.py):</p><pre>%%writefile inference.py<br>from mlserver import MLModel, types<br>from mlserver.codecs import StringCodec<br>from mlserver.utils import get_model_uri<br>from typing import Dict, List, Any<br>import json<br>import torch<br>from sentence_transformers import SentenceTransformer, util<br>import numpy as np<br>import re<br>import unicodedata<br>import logging<br><br># Configure logging<br>logging.basicConfig(level=logging.ERROR)<br>logger = logging.getLogger(__name__)<br><br>class TextUniqueness(MLModel):<br>    async def load(self) -&gt; bool:<br>        model_uri = await get_model_uri(self._settings)<br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>        <br>        # Load SentenceTransformer from disk.<br>        self.model = SentenceTransformer(model_uri, device=self.device)<br>        self.ready = True<br>        return self.ready<br>    <br>    <br>    # Logic for making predictions against our model<br>    async def predict(self, payload: types.InferenceRequest) -&gt; types.InferenceResponse:<br>        try:<br>            request = self._extract_json(payload).get(&quot;echo_request&quot;, {})<br>            tweets = request.get(&quot;tweets&quot;, [])<br><br>            # Preprocessing - clean tweets<br>            cleaned_tweets = [self.clean_tweet(tweet) for tweet in tweets]<br><br>            output_data = {&quot;success&quot;: True}<br>            <br>            # Encode all sentences<br>            embeddings = self.model.encode(list(cleaned_tweets))<br>            <br>            # Compute cosine similarity between all pairs<br>            cos_sim = util.cos_sim(embeddings, embeddings)<br>            <br>            # Add all pairs to a list with their cosine similarity score<br>            all_sentence_combinations = []<br>            for i in range(len(cos_sim)-1):<br>                for j in range(i+1, len(cos_sim)):<br>                    all_sentence_combinations.append(abs(np.float64(cos_sim[i][j].numpy())))<br><br>            output_data[&quot;uniqueness_score&quot;] = round(1-(np.mean(all_sentence_combinations)),6)<br><br>            # Postprocessing - define the uniqueness bracket<br>            output_data[&quot;uniqueness_bracket&quot;] = self.defineBuckets(output_data[&quot;uniqueness_score&quot;])<br><br>        except Exception as e:<br>            logger.error(&quot;Unexpected error: %s&quot;, str(e))<br>            output_data = dict(uniqueness_score=&#39;NULL&#39;, success=False, error=str(e), loc=&quot;inference.predict&quot;)<br><br>        response_bytes = json.dumps(output_data).encode(&quot;UTF-8&quot;)<br><br>        return types.InferenceResponse(<br>            id=payload.id,<br>            model_name=self.name,<br>            model_version=self.version,<br>            outputs=[<br>                types.ResponseOutput(<br>                    name=&quot;echo_response&quot;,<br>                    shape=[len(response_bytes)],<br>                    datatype=&quot;BYTES&quot;,<br>                    data=[response_bytes],<br>                    parameters=types.Parameters(content_type=&quot;str&quot;),<br>                )<br>            ],<br>        )<br><br>    def _extract_json(self, payload: types.InferenceRequest) -&gt; Dict[str, Any]:<br>        inputs = {}<br>        for inp in payload.inputs:<br>            inputs[inp.name] = json.loads(<br>                &quot;&quot;.join(self.decode(inp, default_codec=StringCodec))<br>            )<br>        <br>        return inputs<br><br>    def clean_tweet(self, text) -&gt; str:<br>        text = re.sub(&#39;(\s)@\w+|^@\w+&#39;, &#39;&#39;, text) # removing handles from tweets<br>        text = re.sub(&quot;#[A-Za-z0-9_]+&quot;,&quot;&quot;, text) # removing hastags<br>        text = re.sub(r&quot;http\S+&quot;, &#39;&#39;, text) # remove URLs<br>        text = (unicodedata.normalize(&#39;NFKD&#39;, text)<br>                    .encode(&#39;ascii&#39;, &#39;ignore&#39;)<br>                    .decode(&#39;utf-8&#39;, &#39;ignore&#39;))<br>        text = re.sub(&quot;&#39;&quot;, &quot;&quot;, text) # remove single quotes<br>        text = re.sub(&#39;&quot;&#39;, &quot;&quot;, text) # remove double quotes<br>        text = re.sub(&quot;[^a-zA-Z0-9 \.]&quot;, &quot;&quot;, text) #<br>        text = re.sub(r&#39;\.+&#39;, &quot;.&quot;, text) # replace multiple full stops<br>        text = re.sub(&#39; +&#39;, &#39; &#39;, text) # remove more than one space<br>        text.lstrip(&#39;0123456789.- &#39;).rstrip(&#39;0123456789.- &#39;)<br>        text = text.strip(&#39;0123456789.- &#39;) # remove numbers at the beginning of a tweet<br>        text = text.strip() # remove empty spaces from left and right<br>        return text<br><br><br>    def defineBuckets(self, score) -&gt; str:<br>        &#39;&#39;&#39;<br>        Function to get the buckets for the uniqueness score<br>        &#39;&#39;&#39;<br>        if score&gt;=0.80:<br>            bracket = &#39;Extreme&#39;<br>        elif score&gt;=0.60:<br>            bracket = &#39;High&#39;<br>        elif score&gt;=0.40:<br>            bracket = &#39;Moderate&#39;<br>        elif score&gt;=0.20:<br>            bracket = &#39;Neutral&#39;<br>        else:<br>            bracket = &#39;Low&#39; <br>        return bracket</pre><p>load() method is responsible for loading the fine-tuned model weights from model_uri which is defined in the <em>model-settings.json</em> file and will go through this file in the next steps.</p><p>predict() method where we include all our inference logic with the below steps.</p><ul><li>Preprocessing (cleaning tweets)</li><li>Extract embeddings for all sentences</li><li>Compute cosine similarity between all pairs</li><li>Compute the uniqueness score</li><li>Postprocessing (define the uniqueness bracket)</li></ul><p><strong>Settings files</strong></p><p>The next step will be to create 2 configuration files:</p><p>settings.json: holds the configuration of our server (e.g. ports, log level, etc.).</p><pre>{<br>    &quot;debug&quot;: &quot;true&quot;<br>}</pre><p>model-settings.json: holds the configuration of our model (e.g. input type, runtime to use, etc.).</p><pre>{<br>    &quot;name&quot;: &quot;text-uniqueness-model&quot;,<br>    &quot;implementation&quot;: &quot;inference.TextUniqueness&quot;,<br>    &quot;parameters&quot;: {<br>        &quot;uri&quot;: &quot;./model&quot;<br>    }<br>}</pre><p>Where name is the unique model name, uri is the location of model weights, implementation is the custom runtime Python file name and its class name.</p><p><strong>Start serving our model</strong></p><p>Now that we have our config in place, we can start the server by running mlserver start . command. This needs to be run from the same directory where our config files are located.</p><p>Since this command will start the server and block the terminal, waiting for requests, this will need to be run in the background on a separate terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*kgtac5i_1jiUoK_5YrnPrw.png" /><figcaption>MLServer logs (Photo by Author)</figcaption></figure><p><strong>3. Send test inference request (REST) to MLServer</strong></p><p>We now have our model being served by mlserver. To make sure that everything is working as expected, let’s send a request from Kaggle competition dataset.</p><pre>%%writefile test.py<br># Import required dependencies<br>import requests<br>import json<br>import json<br>from mlserver.types import InferenceResponse<br>from mlserver.codecs.string import StringRequestCodec<br><br>inputs = {<br>  &quot;tweets&quot;: [<br>    &quot;Today I learned that Leonardo DiCaprio dating a 19-year-old adult woman is the height of moral evil, mostly from the same people who believe a 4-year-old can consent to gender change.&quot;,<br>    &quot;In which @mattwalshblog leaves an elected Democrat absolutely speechless with one simple question. This clip is stunning.  https://t.co/R9hsy9hSlb&quot;,<br>    &quot;Check out my full reaction to the State of the Union address here: https://t.co/YheEJblr3F https://t.co/Mln0zPbPce&quot;,<br>    &quot;This is the message of our modern-day Satanists as well. Religion and morality are evils. Worship of “authenticity,” particularly in the sexual realm, is the highest possible good. Worship Satan by worshipping yourself.\n\nBrought to you by Pfizer and Dr. Jill Biden, gang!&quot;,<br>    &quot;Which problems, specifically? Do you mean massive spikes in teen suicidal ideation, forty percent of babies born out of wedlock, social contagions around nonsense gender and sexual identities? That ain&#39;t parental sex ed.  https://t.co/WXVtKdz6sh&quot;,<br>    &quot;I have said on air that if I knew then what I know now, I wouldn&#39;t have taken it. But no matter my personal feelings about the vaxx and the evolution of the info we were given, I NEVER supported mandates. I risked my business to oppose them. Name another company that did.&quot;,<br>    &quot;I recommended the vaxx to the elderly and the obese to reduce death rates. I recommended the vaccine for people my age because we were lied to about reduction in transmissibility.  That&#39;s why I took it personally. I never recommended it for the young. My kids are not vaxxed.&quot;,<br>    &quot;We sued the federal government and refused to force our employees to vaxx or test. I noticed you have said nothing about Fox News&#39; covid policy, which mandated that its employees vaxx or test every day. Fascinating.&quot;,<br>    &quot;Hunter Biden Is An Evil Scumbag, Not A Victim | Ep. 1659  https://t.co/SkSZWZgvOz&quot;,<br>    &quot;You shut down schools, killed small businesses en masse, forcibly masked our kids, tried to force businesses to vaxx their employees, banned us from travel, kept us away from our dying relatives...all based on science you KNEW was shoddy. F*** you.   https://t.co/GaHVN3xu2r&quot;<br>  ]<br>}<br><br>inputs_string = json.dumps(inputs)<br><br>inference_request = {<br>    &quot;inputs&quot;: [<br>        {<br>          &quot;name&quot;: &quot;echo_request&quot;,<br>          &quot;shape&quot;: [len(inputs_string)],<br>          &quot;datatype&quot;: &quot;BYTES&quot;,<br>          &quot;data&quot;: [inputs_string],<br>        }<br>    ]<br>}<br><br>inference_url = &quot;http://localhost:8080/v2/models/text-uniqueness-model/infer&quot;<br><br>response = requests.post(inference_url, json=inference_request)<br>print(f&quot;full response:\n&quot;)<br>print(response)<br># retrive text output as dictionary<br>inference_response = InferenceResponse.parse_raw(response.text)<br>raw_json = StringRequestCodec.decode_response(inference_response)<br>output = json.loads(raw_json[0])<br>print(f&quot;\ndata part:\n&quot;)<br>print(output)</pre><p>Model inference output looks like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*mB92K4c0SnvhBiwQhz1yZQ.png" /><figcaption>Inference output (Photo by Author)</figcaption></figure><h4>🚀Deploying custom NLP models to Kubernetes</h4><p>Now that we have written and tested our custom NLP model in the local MLServer, the next step is to deploy it on Kubernetes. With that goal in mind, the rough outline of steps will be to first build a custom image containing our code, push it to the container registry, and then deploy it to Kubernetes.</p><p><strong>Specifying requirements</strong></p><p>Need to place the requirements.txt file under build-server folder and MLServer install the necessary Python packages.</p><pre># %load requirements.txt<br>sentence-transformers==2.2.2</pre><p><strong>Building a custom docker image</strong></p><p>MLServer offers helpers to build a custom Docker image containing your code and additional dependencies specified in the requirements.txt file. In this example, we will use the mlserver build subcommand to create an image by navigating to the build-server folder.</p><pre>%%bash<br>mlserver build . -t &lt;docker-user-id&gt;/&lt;docker-repo-name&gt;</pre><p>Push the docker image to the docker hub.</p><pre>%%bash<br>docker push &lt;docker-user-id&gt;/&lt;docker-repo-name&gt;</pre><p>We will use Seldon Core, a versatile open-source framework, for model deployment. It combines Kubernetes’ cloud-native features with ML-specific capabilities.</p><p>This tutorial assumes you have a working Seldon Core cluster. If not, follow the <a href="https://docs.seldon.io/projects/seldon-core/en/latest/nav/installation.html">installation instructions</a>.</p><p><strong>Storing model artifacts to remote storage</strong></p><p>I am using Google Cloud Storage to store the model artifacts but you can use any cloud storage of your choice and then use the following commands to store the model artifacts (inference code, model-settings.json, and model weights) which can be used in the deployment steps to serve the text-uniqueness model.</p><pre>cd SeldonCore-Custom-Model-Server<br>gsutil cp inference.py model-settings.json gs://seldon-models/text-uniqueness<br>gsutil cp -r model gs://seldon-models/text-uniqueness/model</pre><p><strong>Creating the Deployment</strong></p><p>To create our deployment with Seldon Core we need to create 2 configuration files, one for creating a custom MLServer and the other one for model deployment.</p><p>Let’s create a custom MLServer, you can find this file named server.yaml in the config folder of this tutorial’s repository.</p><pre>apiVersion: mlops.seldon.io/v1alpha1<br>kind: Server<br>metadata:<br>  name: mlserver-text-uniqueness<br>  namespace: seldon-mesh<br>spec:<br>  serverConfig: mlserver<br>  capabilities:<br>  - text-uniqueness<br>  podSpec:<br>    containers:<br>    - image: &lt;docker-user-id&gt;/&lt;docker-repo-name&gt;<br>      name: mlserver</pre><p>Let’s create a model deployment by using model.yaml</p><pre>apiVersion: mlops.seldon.io/v1alpha1<br>kind: Model<br>metadata:<br>  name: text-uniqueness-model<br>spec:<br>  storageUri: &quot;gs://seldon-models/text-uniqueness/&quot;<br>  requirements:<br>  - text-uniqueness</pre><p>We can apply these configuration files to our Kubernetes cluster just like we would for any other Kubernetes object using:</p><pre>kubectl create -f server.yaml<br>kubectl create -f model.yaml</pre><p>To check our custom MLServer is up and running we can run:</p><pre>kubectl get pods</pre><p>We should see STATUS = Running once our deployment has been finalized.</p><p>To check our model is deployed on the K8s cluster we can run:</p><pre>kubectl get models</pre><p>Having deployed our server and model to Kubernetes and tested it, our setup now looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVvkLLAIqezANsjFzFAiFw.jpeg" /><figcaption>Custom MLServer Setup on K8s (Photo by Author)</figcaption></figure><p><strong>Test the Seldon Core hosted Transformer Model</strong></p><p>Now that our model is up and running on a Kubernetes cluster (via Seldon Core), we can send some test inference requests to make sure it’s working.</p><p>First, we need to do some port forwarding work so our model’s port is exposed to our local system with the command:</p><pre>kubectl port-forward -n seldon-mesh svc/seldon-mesh 8333:80</pre><p>To do this, we simply run the test.py file in the following way:</p><pre>python test.py</pre><p>Note: <em>This example is set up to connect to a kubernetes cluster running locally on your machine. If your cluster is local, you’ll need to change the </em><em>inference_url variable on line 36 of </em><em>test.py with port number 8333.</em></p><pre>inference_url = &quot;http://localhost:8333/v2/models/text-uniqueness-model/infer&quot;</pre><p>Model inference output looks like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*mB92K4c0SnvhBiwQhz1yZQ.png" /><figcaption>Inference output (Photo by Author)</figcaption></figure><h4>📝Summary</h4><p>In this post, I have shown you how we can deploy custom NLP models on Seldon Core v2. We also look at deploying custom NLP models on K8s by leveraging the open-source framework — Seldon Core V2.</p><h4>📚References</h4><p>I recommend going through the following references to start learning about Seldon Core and take your models from Dev to Production seamlessly.</p><ul><li><a href="https://docs.seldon.io/projects/seldon-core/en/v2/contents/about/index.html">About - Seldon Core v2</a></li><li><a href="https://mlserver.readthedocs.io/en/stable/examples/cassava/README.html">Deploying a Custom Tensorflow Model with MLServer and Seldon Core - MLServer Documentation</a></li><li><a href="https://mlserver.readthedocs.io/en/stable/examples/custom/README.html">Serving a custom model - MLServer Documentation</a></li></ul><p>I hope you enjoyed this blog post, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this post is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/AI-ML-Projects/tree/main/SeldonCore-Custom-Model-Server">AI-ML-Projects/SeldonCore-Custom-Model-Server at main · vinayak-shanawad/AI-ML-Projects</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a593f6ea7549" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Dev to Production: Deploying HuggingFace BERT with KServe]]></title>
            <link>https://medium.com/@vinayakshanawad/deploying-a-huggingface-bert-model-with-kserve-3e521d69e596?source=rss-609a97998ce2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e521d69e596</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Vinayak Shanawad]]></dc:creator>
            <pubDate>Mon, 11 Sep 2023 11:14:08 GMT</pubDate>
            <atom:updated>2024-02-19T06:02:48.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Future of NLP Deployment: BERT Models and KServe in Action</h4><p>In this post, I will demonstrate how to deploy a HuggingFace pre-trained model (BERT for text classification with the Hugging Face Transformers library) to run as a KServe-hosted model.</p><p>First, let’s understand what is KServe and why we need KServe.</p><h4>🤔What is KServe?</h4><p>KServe was initially called KFServing (KubeFlow Serving) and was designed so that model serving could be operated in a standardized way across frameworks right out of the box. There was a need for a model serving system, that could easily run on existing Kubernetes and Istio stacks and also provide model explainability, inference graph operations, and other model management functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lHSSQ5OszDYwxHqG.png" /><figcaption>Model serving using KServe (Photo by <a href="https://www.kubeflow.org/docs/external-add-ons/kserve/kserve/">Kubeflow</a> )</figcaption></figure><h4>🤷‍♂️Why KServe?</h4><ul><li><strong>KServe</strong> is a standard <strong>Model Inference Platform</strong> on <strong>Kubernetes</strong>, built for <strong>highly scalable</strong> use cases.</li><li>Provides <strong>performant, standardized inference protocol</strong> across <strong>ML frameworks (</strong>Tensorflow, XGBoost, ScikitLearn, PyTorch, and ONNX<strong>)</strong></li><li>Support modern <strong>serverless inference workload</strong> with <strong>Autoscaling</strong> including <strong>Scale to Zero on GPU</strong>.</li><li><strong>Simple</strong> and <strong>Pluggable production serving</strong> for <strong>production ML serving</strong> including <strong>prediction, pre/post-processing, monitoring, and explainability</strong>.</li><li><strong>Advanced deployments</strong> with the <strong>canary rollout, experiments, ensembles, and transformers</strong>.</li></ul><h4>🛠️Setting Up KServe</h4><p>To demo the Hugging Face model on KServe we’ll use the local (Windows OS) quick install method on a <a href="https://kubernetes.io/docs/tasks/tools/install-minikube/">minikube kubernetes cluster</a>. The standalone “quick install” installs Istio and KNative for us without having to install all of Kubeflow and the extra components that tend to slow down local demo installs.</p><p>Let’s start the minikube cluster once our local minikube installation is completed.</p><pre># Start the minikube cluster<br>minikube start<br><br># Check the status of minikube cluster<br>minikube status</pre><p>The second command should give us the output below if our cluster is healthy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*GW6XFYxpRMbafIhG7WLzoA.png" /><figcaption>Minikube cluster status (Photo by Author)</figcaption></figure><p>First, we need to get a copy of the KServe repository on our local system. Use git bash to clone the KServe repository.</p><pre>cd kubeflow<br>git clone https://github.com/kserve/kserve.git</pre><p>We can’t download the Istio 1.17.2 due to some issue, hence we can download the Istio 1.17.2 from the <a href="https://github.com/istio/istio/releases/tag/1.17.2">release page</a> for Windows. Extract the istio-1.17.2-win.zip file and place the istio-1.17.2 folder under kubeflow directory.</p><pre>cd kubeflow<br>./hack/quick_install.sh</pre><p>This will install KServe along with its core dependencies such as Knative Serving all with the same install script. This install takes around 30–60 seconds, depending on your system.</p><blockquote><strong>Note</strong>: Sometimes the installer will fail because a component still has not been completely installed, just run the installer a second time if you see the failure console logs.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*TZkD5f8-aVYI7WNdR5ob-A.png" /><figcaption>KServe installation logs (Photo by Author)</figcaption></figure><p>Once our installation is complete, we can confirm that the KServe install is working on our minikube cluster with the command.</p><pre>kubectl get pods -n kserve</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*G_Rc1bBgs6SHA8i_zAlSMg.png" /><figcaption>KServe control manager (Photo by Author)</figcaption></figure><p>We can also list the all pods under the minikube cluster.</p><pre>kubectl get pods -A</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*3b79IxiiBVO8VHzAq6ZzNA.png" /><figcaption>Pods status (Photo by Author)</figcaption></figure><h4>🚀Deploying the Custom HuggingFace Model Server on KServe</h4><p>There are two main ways to deploy a model as an InferenceService on KServe:</p><ol><li>Deploy the saved model with a pre-built model server on a pre-existing image</li><li>Deploy a saved model already wrapped in a pre-existing container as a custom</li></ol><p>Most of the time we want to deploy on a pre-built model server as this will create the least amount of work for our engineering team.</p><p>There are many pre-built model servers included with KServe out of the box. With KServe our built-in model server options are:</p><ol><li>tensorflow</li><li>sklearn</li><li>pytorch</li><li>onnx</li><li>tensorrt</li><li>xgboost</li></ol><p>Sometimes we’ll have a model that will not wire up correctly with the pre-built images. The reasons this could happen include:</p><ul><li>Model built with different dependency versions than the model server</li><li>Model not saved in file format model server expects</li><li>Model was built with a new/custom framework not yet supported by KServe</li><li>Model is in a container image that has a REST interface that is different than the <a href="https://www.tensorflow.org/tfx/serving/api_rest">Tensorflow V1 HTTP API</a> that KServe expects</li></ul><p>For any of the cases above we have 3 options for deploying our model:</p><ol><li>Wrap our custom model in our own container where our container runs its own web server to expose the model endpoint</li><li>Use the KServe Model Server as the webserver (with its standard Tensorflow V1 API) and then overload the load() and predict() methods</li><li>Deploy a pre-built container image with a custom REST API, bypassing InferenceService and sending the HTTP request directly to the predictor</li></ol><p>Of the 3 options, using Model Server and just doing custom overloads will likely be the most popular route for folks just wanting to deploy a custom model.</p><p>Given that Hugging Face has a unique Python API and a lot of dependencies, it does not work on KServe out of the box. In this case, we need to do 2 key tasks:</p><ol><li>Create a new python class that inherits from KServe Model class, with custom methods for load() and predict()</li><li>Build a custom container image and then store it in a container repository</li></ol><p>The remainder of this post will be focused on:</p><ol><li>Building a custom model Pythonkserve.Model with the Hugging Face BERT model wired in.</li><li>Building a docker container with the custom python kserve.Model and push the docker container to <a href="https://hub.docker.com/">docker hub</a></li><li>Deploy the custom InferenceService to our minikube Kubernetes cluster</li><li>Test the KServe-hosted HuggingFace model</li></ol><p>Now let’s get to work building out our custom text classification InferenceService on KServe.</p><h4>1. Building a Custom Python Model Server</h4><p>In the code below we can see our custom Model with the Hugging Face code wired into the load() and predict() methods.</p><pre>from typing import Dict <br><br>import kserve<br>import torch<br><br>from transformers import AutoModelForSequenceClassification, AutoTokenizer<br>from kserve import ModelServer<br>import logging<br><br>class KServeBERTSentimentModel(kserve.Model):<br><br>    def __init__(self, name: str):<br>        super().__init__(name)<br>        KSERVE_LOGGER_NAME = &#39;kserve&#39;<br>        self.logger = logging.getLogger(KSERVE_LOGGER_NAME)<br>        self.name = name<br>        self.ready = False<br><br>    def load(self):<br>        # Build tokenizer and model<br>        name = &quot;distilbert-base-uncased-finetuned-sst-2-english&quot;<br>        self.tokenizer = AutoTokenizer.from_pretrained(name)<br>        self.model = AutoModelForSequenceClassification.from_pretrained(name, torchscript=True)<br>        self.ready = True<br><br><br>    def predict(self, request: Dict, headers: Dict) -&gt; Dict:<br>        <br>        sequence = request[&quot;sequence&quot;]<br>        self.logger.info(f&quot;sequence:-- {sequence}&quot;)<br><br>        inputs = self.tokenizer(<br>            sequence,<br>            return_tensors=&quot;pt&quot;,<br>            max_length=128,<br>            padding=&quot;max_length&quot;,<br>            truncation=True,<br>        )<br><br>        # run prediciton<br>        with torch.no_grad():<br>            predictions = self.model(**inputs)[0]<br>            scores = torch.nn.Softmax(dim=1)(predictions)<br><br>        results = [{&quot;label&quot;: self.model.config.id2label[item.argmax().item()], &quot;score&quot;: item.max().item()} for item in scores]<br>        self.logger.info(f&quot;results:-- {results}&quot;)<br><br>        # return dictonary, which will be json serializable<br>        return {&quot;predictions&quot;: results}<br><br>if __name__ == &quot;__main__&quot;:<br><br>  model = KServeBERTSentimentModel(&quot;bert-sentiment&quot;)<br>  model.load()<br><br>  model_server = ModelServer(http_port=8080, workers=1)<br>  model_server.start([model])</pre><p>There are two things happening in the above code with respect to integrating with the model server:</p><ol><li>The Hugging Face BERT model is loaded in the load(…) method</li><li>The predict(...) method takes incoming inference input from the REST call and passes it to the Hugging Face AutoModelForSequenceClassification model instance</li></ol><p>The Hugging Face model we’re using here is the “distilbert-base-uncased-finetuned-sst-2-english”. This model and associated tokenizer are loaded from pre-trained model checkpoints included in the Hugging Face framework.</p><h4>2. Building a new Docker image for the Model Server</h4><p>Once our model serving code above is saved locally, we will build a new docker container image with the code and required dependencies packaged inside. We can see examples of the container build command and the container repository store command (here, docker hub) below.</p><p>Build the new container with our custom code and then send it over to the container repository of your choice:</p><pre># Build the container on your local machine<br>docker build -t kserve-custom-model .<br><br># Push the container to docker registry<br>docker push {username}/kserve-model-repo:v1.0</pre><p>For those that would prefer to use a pre-built version of this container and skip the coding + docker steps, just use my container up on the docker hub:</p><p><a href="https://hub.docker.com/repository/docker/vinayaks117/kserve-model-repo/general">Docker</a></p><p>Now let’s move on to deploying our <a href="https://github.com/vinayak-shanawad/KServe-Custom-Model-Server/blob/main/KServe_BERT_Sentiment_ModelServer.py">model server</a> in our container as an InferenceService on KServe.</p><h4>3. Deploying Custom Model Server on KServe with kubectl</h4><p>Given that KServe treats models as infrastructure, we deploy a model on KServe with a yaml file to describe the k8s model resource (e.g., InferenceService) as a custom object. The code listing below shows our yaml file to create our custom InferenceService object on the local k8s cluster.</p><p>We need to set four parameters to uniquely identify the model, such as:</p><ul><li>apiVersion: “serving.kserve.io/v1beta1”</li><li>kind: “InferenceService”</li><li>metadata.name: [the model’s unique name inside the namespace]</li><li>metadata.namespace: [the namespace your model will live in]</li></ul><p>Here we’re using the generic kserve-custom-model as our metadata.name and our model will be created in the default namespace.</p><p>Towards the end of the spec we ask kubernetes to schedule our container wtih 4GB of ram as Hugging Face tends to take up a lot of space in memory.</p><p>Once we have our yaml file configured we can create the Kubernetes object with Kubectl as shown below.</p><pre>kubectl apply -f deploy_bert_sentiment.yaml</pre><p>Once we run the above kubectl command, we should have a working InferenceService running on our local kubernetes cluster. We can check the status of our model with the kubectl command:</p><pre>kubectl get inferenceservices</pre><p>This should give us output as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3T4PmOfo9G9IQbDrkBtLwQ.png" /><figcaption>Custom model status (Photo by Author)</figcaption></figure><p>Deploying a custom model on KServe is not as easy as using a pre-built model server, but it’s not terrible either as we’ve seen so far.🚀🚀</p><h4>4. Test the KServe-hosted HuggingFace Model</h4><p>Now let’s make an inference call to our locally hosted Hugging Face Sentiment Analysis model on KServe. First, we need to do some port forwarding work so our model’s port is exposed to our local system with the command:</p><pre>kubectl port-forward -n istio-system service/istio-ingressgateway 8080:80</pre><p>We’ll use the curl command to send the input json file as input to the predict method on our custom Hugging Face InferenceService on KServe with the command:</p><pre>curl -v -H &quot;Host: kserve-custom-model.default.example.com&quot; http://localhost:8080/v1/models/bert-sentiment:predict  -d @./input.json</pre><p>The response will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*RDrHD39yAffnEF0u7n1wUA.png" /><figcaption>Model inference (Photo by Author)</figcaption></figure><p>This example has shown how to take a non-trivial NLP model and host it as a custom InferenceService on KServe.</p><p>I hope you enjoyed this blog post, if you have any questions, feel free to contact me on <a href="https://www.linkedin.com/in/vinayak-shanawad-4122405a/">LinkedIn</a> and share your experience in the comment section.</p><p>The complete source code for this post is available in the following link.</p><p><a href="https://github.com/vinayak-shanawad/AI-ML-Projects/tree/main/KServe-Custom-Model-Server">AI-ML-Projects/KServe-Custom-Model-Server at main · vinayak-shanawad/AI-ML-Projects</a></p><p><a href="https://medium.com/r?url=https%3A%2F%2Fmlearning.substack.com%2Fabout">Become a WRITER at MLearning.ai</a> / <a href="https://open.substack.com/pub/mlearning/p/good-bad-ai-art?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">Good-Bad AI Art</a> <a href="https://open.substack.com/pub/mlearning/p/image-to-3d-one-click-ai-tool-modeling-video-txt?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">/</a> <a href="https://open.substack.com/pub/mlearning/p/100-ai-agents-ready-to-use?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">The 100+ List AI agents</a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e521d69e596" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>