<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Guillermo Ruiz | CodeAwake on Medium]]></title>
        <description><![CDATA[Stories by Guillermo Ruiz | CodeAwake on Medium]]></description>
        <link>https://medium.com/@codeawake?source=rss-1b6b211a55fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*udyOFbEZn_g33ans1vqtMA.jpeg</url>
            <title>Stories by Guillermo Ruiz | CodeAwake on Medium</title>
            <link>https://medium.com/@codeawake?source=rss-1b6b211a55fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:06:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@codeawake/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Advanced RAG for Talent Matching in Large Organizations]]></title>
            <link>https://medium.com/@codeawake/advanced-rag-talent-matching-779cee34b0e8?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/779cee34b0e8</guid>
            <category><![CDATA[consulting]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 12:50:29 GMT</pubDate>
            <atom:updated>2025-02-04T20:31:43.204Z</atom:updated>
            <content:encoded><![CDATA[<h4>Find the perfect team from 1000+ professionals in minutes with AI</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/advanced-rag-talent-matching"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3b1nnYEdx8z-M6pz7401w.png" /></figure><p>While there are many tutorials out there showing basic <strong>Retrieval Augmented Generation (RAG)</strong> implementations with simple document Q&amp;A (including <a href="https://codeawake.com/blog/rag-from-scratch">my</a> <a href="https://codeawake.com/blog/ai-chatbot">own</a>), real-world enterprise applications demand more sophisticated solutions. How do you take RAG beyond basic document retrieval and build an <strong>intelligent system that can actually help make decisions and add value to the business?</strong></p><p>That’s exactly what I’ll explore in this post, walking you through an <strong>advanced RAG-based solution I recently designed for a large Spanish technology consulting company.</strong> The goal is to transform their project staffing process through <strong>intelligent talent matching</strong> and automatically generate client-ready presentations.</p><p>This project serves as a very practical example of <strong>how to define, structure, and build a RAG system that solves a real business problem,</strong> one that’s actively costing companies valuable time and money through inefficient manual processes.</p><h3>The Business Challenge</h3><p><strong>How do you efficiently match the right talent to increasingly complex client projects when your talent pool keeps growing and changing?</strong> What works for a staff of 20 becomes unsustainable at 200 — and a nightmare over 1000.</p><p>Picture this: you’re a senior manager at a growing consultancy. A major bank needs a new team for their digital transformation initiative. The requirements are specific: a technical team lead with architecture expertise, a frontend engineer, a backend engineer and a data science specialist, all with prior experience in the financial industry.</p><p>Somewhere in your <strong>talent pool of over a thousand engineers and contractors,</strong> the perfect candidates exist. But finding them is not a simple task. Managers spend valuable hours sifting through CVs, trying to remember who has what skills, and piecing together project histories. It’s a process that’s not just time-consuming — it’s fundamentally flawed.</p><p>And it creates a cascade of business risks:</p><ul><li><strong>Senior managers spend valuable hours on administrative tasks</strong> instead of strategic work.</li><li>Project kickoffs are delayed due to <strong>staffing bottlenecks.</strong></li><li>Client presentations contain <strong>outdated or inconsistent information</strong> about team capabilities.</li><li><strong>Suboptimal teams</strong> are formed while great candidates are overlooked.</li></ul><p>The challenge isn’t just about matching skills to requirements. It’s about <strong>transforming a complex human search and decision process into a scalable, automated system that optimizes talent matching</strong> while dramatically reducing time-to-team. Let me show you how.</p><h3>Reimagining Talent Matching with AI</h3><p>What if your team assembly process could be as intuitive as having a conversation with your most experienced project manager, but with instant access to every detail about your talent pool? This is exactly what artificial intelligence, and specifically Retrieval-Augmented Generation (RAG), makes possible.</p><p>Our solution transforms project staffing from a manual hunt into an intelligent matching system that operates in two blocks:</p><ol><li>An <strong>AI-powered selection and ranking engine</strong> that combines intelligent retrieval with generative AI to match talent to requirements.</li><li>An <strong>automated presentation generator</strong> that transforms the final selection into client-ready slides adapted to the specific project.</li></ol><p><strong>Speed alone isn’t what makes this solution so valuable — it’s also accuracy and reliability.</strong> Every recommendation comes with a clear rationale, backed by current data. No more relying on outdated information or gut feelings. Every match is traceable, every decision transparent.</p><p>And these are some of the key technical choices to power this solution and meet the client’s requirements:</p><ul><li>A <strong>hybrid retrieval system that combines full-text search and vector search</strong> for semantic matching beyond keywords. Databases like <a href="https://www.postgresql.org/">PostgreSQL</a> (with the <a href="https://github.com/pgvector/pgvector">pgvector</a> extension) and <a href="https://lancedb.com/">LanceDB</a> are great options for this.</li><li>An advanced large language model like <strong>OpenAI’s GPT-4o</strong> for intelligent processing and generation.</li><li>Integrating the solution into the tools and platforms where the company’s team already works to facilitate adoption. For example, the solution is <strong>delivered as a chatbot directly within Microsoft Teams</strong> and includes <strong>real-time synchronization with SharePoint</strong> to ensure that all CVs and talent information remain up-to-date.</li><li>And because we understand European business needs, compliance is built into the foundation. All processing occurs within <strong>EU-hosted Azure infrastructure,</strong> using OpenAI’s models in a <strong>GDPR-compliant</strong> configuration. Your data stays in Europe and your processes remain compliant.</li></ul><h3>Inside the Solution: How It Works</h3><p>After the overview of the solution, let’s now go a bit deeper to understand the inner-workings of each component in more detail.</p><h4>Data Processing and Management</h4><p>There are two big challenges to solve here:</p><ol><li>Creating an <strong>optimized representation of the source documents</strong> (the 1000+ CVs and professional profiles) for the search and selection process.</li><li>Making sure that this representation <strong>remains up to date</strong> as the source documents change.</li></ol><p>In RAG applications, generally, the more complex the system and the search queries it needs to handle, the more advanced the required data processing. Rather than just breaking the documents down into chunks and embedding them, as it’s usually done in simpler RAG apps, our system demands a more sophisticated approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PGs1n5zXG1T0dKNP" /></figure><ul><li><strong>Dimension-specific extraction:</strong> Instead of treating CVs as monolithic documents, we are going to extract and index information along the key dimensions of interest: skills, technologies, education, professional roles, experience, industries, languages, etc. This increases <strong>information density</strong> compared to the raw CVs and documents.</li><li><strong>Multiple search indexes:</strong> Create search indexes for each of the key dimensions that we can query separately and then combine together. Additionally, we are storing the information both as text for full-text search and encoded as a vector embeddings for semantic search.</li><li><strong>Structured representation:</strong> Transform each professional profile into a structured representation according to a specific <strong>JSON schema.</strong> This allows filtering and a more precise search than full-text or vector search. Here is an example of how this structured representation could look like:</li></ul><pre>{<br>  &quot;professional_level&quot;: {<br>    &quot;role&quot;: &quot;Software Engineer&quot;,<br>    &quot;level&quot;: &quot;senior&quot;,<br>    &quot;years_experience&quot;: 7,<br>    &quot;leadership&quot;: {<br>      &quot;has_managed_teams&quot;: true,<br>      &quot;team_size&quot;: 5,<br>      &quot;years_leading&quot;: 2<br>    }<br>  },<br>  &quot;skills&quot;: {<br>    &quot;frontend&quot;: {<br>      &quot;react&quot;: {<br>        &quot;level&quot;: &quot;expert&quot;,<br>        &quot;years_experience&quot;: 8,<br>      },<br>      &quot;angular&quot;: {<br>        &quot;level&quot;: &quot;intermediate&quot;,<br>        &quot;years_experience&quot;: 3<br>      }<br>    },<br>    &quot;backend&quot;: {<br>      &quot;nodejs&quot;: {<br>        &quot;level&quot;: &quot;advanced&quot;,<br>        &quot;years_experience&quot;: 6<br>      }<br>    }<br>  },<br>  &quot;industries&quot;: {<br>    &quot;fintech&quot;: {<br>     &quot;years_experince&quot;: 3,<br>     &quot;num_projects&quot;: 4,<br>     &quot;roles&quot;: [&quot;Frontend Software Engineer&quot;, &quot;Frontend Lead&quot;]<br>   }<br>  },<br>  &quot;languages&quot;: {<br>    &quot;English&quot;: &quot;Native&quot;,<br>    &quot;Spanish&quot;: &quot;C1&quot;<br>  }<br>}</pre><p>The second challenge is maintaining data freshness. To solve this, we are implementing a <strong>real-time synchronization system with SharePoint using webhooks.</strong> Whenever we receive notifications of document changes, we will perform the processing described above for new or updated CVs, and make sure to delete from the database and search indexes any obsolete data.</p><p>It’s also worth mentioning that periodic bulk processing might be a more convenient alternative for high-frequency update scenarios.</p><h4>Selection &amp; Ranking Engine</h4><p>The selection engine implements a multi-stage search and ranking process that goes beyond simple keyword matching and semantic search. This is an overview of the steps involved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sXvsZAFn2C0pFWV2" /></figure><ol><li><strong>Query processing:</strong> When the user provides the project and roles requirements in natural language, the first step is to perform query processing, analogous to the document processing described above. The main goal here is to <strong>bridge the search distance between the requirements query and the professional profiles information.</strong> This involves:<br> - Extracting requirements per dimension of interest.<br> - Generate dimension-specific search vector embeddings.<br> - Creating an ideal structured candidate profile according to the same JSON schema explained previously.</li><li><strong>Multi-index hybrid search:</strong> Perform parallel searches across the different search indexes corresponding to the key dimensions. The searches will be <strong>hybrid</strong>, combining <strong>full-text search and vector similarity search</strong> for semantic matching.</li><li><strong>Ranking:</strong> Combine the results of the different searches and return a final ranking of results (per wanted role), using a <strong>ranking algorithm</strong> like <a href="https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf">Reciprocal Rank Fusion (RRF)</a>. An interesting idea is to implement a weighted ranking, where different dimensions have different weights according to their importance for the candidate selection.</li><li><strong>Structured search and filtering:</strong> The previous vector and full-text searches can be combined with a more precise search and filtering using the structured JSON schema. This enables both <strong>hard constraints through strict filtering</strong> (e.g., “minimum 3 years React experience”, “senior level or above”) and <strong>soft constraints through a weighted scoring function that rewards desired attributes</strong> (e.g., “preferably fintech experience”, “ideally team leadership background”).</li><li><strong>LLM re-ranking:</strong> The final stage uses the large language model to perform an intelligent re-ranking of the retrieved candidates. The LLM analyzes all available information about both the candidates and project requirements to generate a final ranking with the top-k candidates for each role.</li><li><strong>Interactive final selection:</strong> The system presents multiple candidates per role to the user, including links to the full CVs and AI-generated justifications explaining each selection. This enables an iterative selection process where users can review recommendations, provide feedback, and adjust preferences until the optimal team is assembled.</li></ol><h4>Team Slides Generation</h4><p>Once the final team selection has been confirmed by the user, the system automates the creation of a client-ready presentation. This process consists of two steps:</p><ol><li>The LLM analyzes the selected profiles and then <strong>synthesizes and customizes the most relevant information for the project,</strong> highlighting the key skills, experiences and achievements for each role.</li><li>The system then <strong>automatically generates the slides for the client presentation</strong> using the generated content and a corporate PowerPoint template.</li></ol><h4>User Interface</h4><p>The entire system is accessible through a <strong>simple chat interface in Microsoft Teams,</strong> seamlessly integrated into the company’s existing workflows. The interface is built using the Microsoft Bot Framework SDK and Azure AI Bot Service, with secure authentication handled by the Microsoft Identity Platform.</p><p>The user experience is intuitive and straightforward, with a natural three-step workflow:</p><ol><li>Describe the <strong>project requirements and desired roles</strong> in natural language.</li><li>Review and refine the <strong>candidates selection</strong> interactively.</li><li>Receive the automatically generated <strong>team presentation slides.</strong></li></ol><p>This simplicity in the user experience is worth emphasizing: <strong>the entire complexity of the RAG system remains completely hidden from end users,</strong> who only need to chat naturally about their projects and staffing requirements.</p><h3>Business Impact and ROI</h3><p>The shift from manual to AI-powered project-team matching delivers immediate, measurable improvements. The <strong>time reclaimed</strong> from manual processes translates directly to <strong>cost savings.</strong> What used to take hours of wading through CVs, trying to remember previous projects and preparing client slides can now be essentially completed in minutes. <strong>Managers can focus on high-impact activities instead.</strong></p><p>But the real value goes beyond time savings. The system’s <strong>AI-powered evaluation considers many factors a human might miss,</strong> especially when dealing with hundreds of profiles. This means better matches, unexpected discoveries, and teams that truly align with project needs. The standardized matching process ensures <strong>justified decisions</strong> that stand up to scrutiny.</p><p>It also <strong>makes knowledge more accessible.</strong> No longer does team assembly depend on the collective memory of a few seasoned managers. No more “I think they worked on something similar last year.” Every project experience, every skill, and every success story becomes instantly searchable and actionable.</p><p>And with real-time SharePoint synchronization, <strong>decisions are always based on current information</strong> — no more outdated CVs or missed certifications.</p><p>But perhaps most importantly, this is just the beginning. By capturing <strong>human feedback</strong> on team selections, tracking performance metrics and analyzing actual outcomes, we can <strong>continuously refine and enhance the matching process.</strong> This practical feedback loop ensures the <strong>system grows more effective over time,</strong> adapting to the organization’s unique patterns and preferences. It’s not just a staffing tool — it’s the first step towards <strong>data-driven talent optimization.</strong></p><p>If you have any questions, comments, or face similar challenges around intelligent process automation in your organization, I’d love to hear from you at <a href="/service/mailto:guillermo@codeawake.com">guillermo@codeawake.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=779cee34b0e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Agents for E-commerce with Pydantic AI]]></title>
            <link>https://medium.com/@codeawake/ai-agents-ecommerce-d52059920f11?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d52059920f11</guid>
            <category><![CDATA[customer-support]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 10:27:02 GMT</pubDate>
            <atom:updated>2025-02-04T18:29:12.246Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to automate customer support and order management in e-commerce applications</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/ai-agents-ecommerce"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wuQna1r8B-zzVFKwOtW0ow.png" /></figure><p>E-commerce businesses face incredibly complex challenges to provide an exceptional customer service: customers expect <strong>instant, accurate responses 24/7, detailed product information</strong> at their fingertips, and <strong>efficient order management</strong> at scale. Meeting these expectations has traditionally forced businesses to choose between aggressive hiring (expensive) or service compromises (risky).</p><p>But what if there was a better way?</p><p><strong>The rise of Large Language Models (LLMs) has opened new possibilities for automating complex customer interactions.</strong> Consider these real-world scenarios:</p><ul><li>A customer asking detailed questions about product materials and care instructions at 3 AM.</li><li>Someone needing to update their shipping address on a pending order.</li><li>A request for return processing with specific item conditions.</li></ul><p>Previously, each of these required human intervention. Now, they can be handled automatically, accurately, and instantly by AI agents. <strong>Agents can understand complex context and instructions, reason and plan their actions, and use tools to handle different tasks.</strong> However, there’s a catch: given the probabilistic nature of LLMs, building production-ready AI agents that you can actually trust has been challenging.</p><h3>Pydantic AI: A Game Changer for AI Agents</h3><p><a href="https://docs.pydantic.dev/latest/">Pydantic</a> is my favourite Python library and I use it in practically every Python project I build to handle data parsing and validation, from FastAPI backends to data processing pipelines. And if you are a Python developer, you probably have used it too.</p><p>Pydantic has already proven to be essential for LLM applications that need <strong>structured outputs</strong> or <strong>function calling</strong> capabilities. It’s no coincidence that libraries like Instructor, LangChain, and LlamaIndex, as well as OpenAI’s and Anthropic’s own SDKs, use it internally. Or to quote Jason Liu’s influential talk: <a href="https://www.youtube.com/watch?v=yj-wSRJwrrc">“Pydantic is all you need.”</a></p><p>Two weeks ago, Pydantic creator <a href="https://x.com/samuel_colvin">Samuel Colvin</a> and his team released <a href="https://ai.pydantic.dev/">PydanticAI</a>, a game-changing library for building AI agents in Python.</p><p>What sets PydanticAI apart? Its <strong>simplicity and clean architecture.</strong> While other frameworks pile on abstractions, PydanticAI provides a thin, type-safe layer for building AI agents. If you’ve used FastAPI, you’ll feel right at home — the library embraces Python’s native features and builds on familiar tools, making it both powerful and approachable.</p><p>Out of the box, PydanticAI delivers a set of features that are essential for production generative AI applications: <strong>type safety, definition and validation of structured outputs, streaming support, a dependency injection system, and model flexibility</strong> (supporting different model providers like OpenAI, Anthropic, Gemini, Ollama, Groq and Mistral). Make sure to check the <a href="https://ai.pydantic.dev/">PydanticAI documentation</a> for more details.</p><p>In the following two sections, we’ll go from theory to practice and learn how to use the PydanticAI library to build two AI agents for a fashion e-commerce store:</p><ol><li>A <strong>support agent</strong> that assists customers by answering their questions using <strong>Retrieval-Augmented Generation (RAG).</strong></li><li>An <strong>order management agent</strong> that processes complex operations like updates, returns and cancellations.</li></ol><p>While these examples are slightly simplified to serve as an introduction to agent building, they mirror the patterns and approaches you’d use in real systems. You can find all the code in detail in the following <a href="https://github.com/ruizguille/ai-agents-ecommerce">GitHub repository</a>.</p><h3>Customer Support RAG Agent</h3><p>Let’s start with a scenario every e-commerce business knows well: <strong>customers asking detailed questions about products, shipping policies and store information</strong> at all hours. Even if the questions aren’t particularly complex, they require accurate, consistent answers drawn from your product catalog and company policies.</p><p><strong>Retrieval-Augmented Generation (RAG)</strong> is a powerful technique to solve this problem. It combines the capabilities of a Large Language Model (LLM) with your own knowledge base to provide accurate responses to user questions.</p><p>The core idea is simple:</p><ol><li>We give our support agent access to a <strong>knowledge base containing all the relevant information about products, prices and company policies.</strong></li><li>When a customer asks a question, <strong>the agent will search the knowledge base and deliver an accurate response based on this information,</strong> in a natural and conversational manner.</li></ol><p>Let’s see how to implement it using PydanticAI. We’ll build a customer support agent for Maison Noir, a luxury fashion e-commerce store that we will use as a fictional example for this blog post.</p><p>The core of any PydanticAI application is the Agent class, which can be configured with a specific model, dependencies and a system prompt:</p><pre>from pydantic_ai import Agent<br><br>SYSTEM_PROMPT = &quot;&quot;&quot;\<br>You are a helpful and knowledgeable assistant for the luxury fashion store Maison Noir.<br>Your role is to provide detailed information and assistance about the store and its products.<br><br>Follow these guidelines:<br>- ALWAYS search the knowledge base using the search_knowledge_base tool to answer user questions.<br>- Provide accurate product and policy information based ONLY on the information retrieved from the knowledge base. Never make assumptions or provide information not present in the knowledge base.<br>- Structure your responses in a clear, concise and professional manner, maintaining our premium brand standards<br>- Highlight unique features, materials, and care instructions when relevant.<br>- If information is not found in the knowledge base, politely acknowledge this.<br>&quot;&quot;&quot;<br><br>rag_agent = Agent(<br>    model=&#39;openai:gpt-4o&#39;,<br>    deps_type=KnowledgeBase,<br>    system_prompt=SYSTEM_PROMPT<br>)</pre><p>PydanticAI uses a clean <strong>dependency injection system</strong> to provide data and services to your agents. In our case, we’re providing a custom KnowledgeBase class as a dependency, so that we are able to search it from the agent&#39;s tools.</p><p>In RAG applications, you would normally preprocess, chunk, embed and store your documents in a vector database, and ideally combine vector search with full-text search for maximum accuracy. We have explored these ideas in previous blog posts where we <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264">build RAG from scratch</a>, and use <a href="https://medium.com/@codeawake/ai-chatbot-5bd2fa3324e3">Redis</a> or <a href="https://medium.com/@codeawake/postgresql-vector-database-6516193cba90">PostgreSQL</a> as vector databases for RAG.</p><p>For this example, we’re using a extremely simplified knowledge base that just returns our store’s information:</p><pre>class KnowledgeBase:<br>    def search(self, search_query: str) -&gt; str:<br>        &quot;&quot;&quot;Search method that would typically implement vector similarity search, full-text search, or a hybrid combination of both.&quot;&quot;&quot;<br>        return KB_INFORMATION</pre><p>The beauty of this design is that in a real application, you would only need to change the implementation of the search method to use your preferred database and search strategy. The rest of the agent code remains exactly the same, completely independent of how the knowledge retrieval is implemented.</p><p>Now we need to give our agent the ability to query the knowledge base. PydanticAI makes this straightforward with its tool system. <strong>Tools</strong> are simply functions that the agent can call when needed. We can easily define a tool using the @agent.tool decorator:</p><pre>@rag_agent.tool<br>async def search_knowledge_base(ctx: RunContext[KnowledgeBase], search_query: str) -&gt; str:<br>    &quot;&quot;&quot;Search the knowledge base to retrieve information about Maison Noir, the store and its products&quot;&quot;&quot;<br>    return ctx.deps.search(search_query)</pre><p>This is where the use of Pydantic truly shines. When we create a tool, PydanticAI automatically <strong>extracts parameters from the function signature, extracts the function description from the docstring and creates a tool schema</strong> that the LLM can understand. All of this happens automatically behind the scenes, making our code simpler and cleaner.</p><p>Notice also how the tool’s first argument gives access to the agent’s <strong>context</strong>, which includes the agent’s <strong>dependencies</strong> (in our case, the knowledge base that we want to search).</p><p>Now our agent is ready to handle customer queries. We can simply run it like this, remembering to pass the knowledge base as a dependency:</p><pre>kb = KnowledgeBase()<br>user_message = &#39;How long does international shipping take?&#39;<br>result = await rag_agent.run(user_message, deps=kb)<br>print(result.data)<br><br># International shipping durations from Maison Noir are as follows:<br># - **Europe**: 3-5 business days<br># - **North America**: 5-7 business days<br># - **Asia**: 6-8 business days<br># - **Rest of the World**: 7-10 business days</pre><p>You can also explore the new messages generated during the agent run using result.new_messages() and you will get a result like this:</p><pre>[<br>    ModelRequest(<br>        parts=[<br>            SystemPromptPart(<br>                content=&#39;You are a helpful and knowledgeable assistant...&#39;,<br>                part_kind=&#39;system-prompt&#39;<br>            ),<br>            UserPromptPart(<br>                content=&#39;How long does international shipping take?&#39;,<br>                timestamp=datetime.datetime(...),<br>                part_kind=&#39;user-prompt&#39;<br>            )<br>        ],<br>        kind=&#39;request&#39;<br>    ),<br>    ModelResponse(<br>        parts=[<br>            ToolCallPart(<br>                tool_name=&#39;search_knowledge_base&#39;,<br>                args=ArgsJson(args_json=&#39;{&quot;search_query&quot;:&quot;international shipping duration&quot;}&#39;),<br>                tool_call_id=&#39;call_Pu3t2gnUiOvqMtxnnEixa5Dq&#39;,<br>                part_kind=&#39;tool-call&#39;<br>            )<br>        ],<br>        timestamp=datetime.datetime(...),<br>        kind=&#39;response&#39;<br>    ),<br>    ModelRequest(<br>        parts=[<br>            ToolReturnPart(<br>                tool_name=&#39;search_knowledge_base&#39;,<br>                content=&#39;# MAISON NOIR - Premium Fashion House\n\n...&#39;,<br>                tool_call_id=&#39;call_Pu3t2gnUiOvqMtxnnEixa5Dq&#39;,<br>                timestamp=datetime.datetime(...),<br>                part_kind=&#39;tool-return&#39;<br>            )<br>        ],<br>        kind=&#39;request&#39;<br>    ),<br>    ModelResponse(<br>        parts=[<br>            TextPart(<br>                content=&#39;International shipping durations from Maison Noir are as follows:...&#39;,<br>                part_kind=&#39;text&#39;<br>            )<br>        ],<br>        timestamp=datetime.datetime(...),<br>        kind=&#39;response&#39;<br>    )<br>]</pre><p>As you can see above, the agent internally calls the search_knowledge_base tool to query the knowledge base and then builds the response using the retrieved information.</p><p>While querying the agent with single questions is useful, most e-commerce applications need to support full conversations with customers. PydanticAI makes this straightforward by allowing us to <strong>maintain conversation history.</strong> We just need to run the agent in a loop and make sure to update the message history with the new messages generated in every agent call:</p><pre>async def run_agent():<br>    kb = KnowledgeBase()<br>    messages = []<br>    while True:<br>  user_message = input()<br>        result = await rag_agent.run(user_message, message_history=messages, deps=kb)<br>        messages += result.new_messages()<br>  print(result.data)</pre><p>For the best possible user experience, <strong>PydanticAI supports response streaming.</strong> Instead of waiting for complete responses, customers see them being generated in real-time, creating a more natural and engaging interaction. Implementing streaming is straightforward:</p><pre>async def run_agent_streaming():<br>    kb = KnowledgeBase()<br>    messages = []<br>    while True:<br>  result_content = &#39;&#39;<br>        user_message = input()<br>        async with rag_agent.run_stream(user_message, message_history=messages, deps=kb) as result:<br>            async for chunk in result.stream_text(delta=True):<br>          result_content += chunk<br>                print(chunk, end = &#39;&#39;)<br>        messages += result.new_messages() + [<br>            ModelResponse.from_text(content=result_content, timestamp=result.timestamp())<br>        ]</pre><p>⚠️ <em>The final result message will NOT be added to result messages if you use </em><em>stream_text(delta=True), so we need to add it manually.</em></p><p>You can try the agent yourself using the code in the <a href="https://github.com/ruizguille/ai-agents-ecommerce">GitHub repository</a>. Simply clone the repository, install the dependencies and run:</p><pre>python rag.py</pre><h3>Orders Management Agent</h3><p>Now that we’ve introduced the main concepts and components of PydanticAI through our RAG agent, <strong>let’s tackle something more ambitious: order management.</strong> This agent will handle a variety of operations including:</p><ul><li>Checking order status and details.</li><li>Updating shipping addresses.</li><li>Processing order cancellations.</li><li>Handling return requests.</li><li>Escalating complex cases to human support.</li></ul><p>Here is where things get interesting. We are moving beyond simple question answering into actually <strong>executing actions that affect real orders and customers.</strong> Every operation requires careful validation, state management and communication. Let’s see how PydanticAI helps us build a robust system that handles these complexities elegantly.</p><p>First of all, we need to create a set of enums and Pydantic models for our order management system:</p><pre>from pydantic import BaseModel<br>from datetime import datetime<br>from enum import Enum<br><br>class OrderStatus(str, Enum):<br>    PENDING = &#39;pending&#39;<br>    CONFIRMED = &#39;confirmed&#39;<br>    SHIPPED = &#39;shipped&#39;<br>    DELIVERED = &#39;delivered&#39;<br>    CANCELLED = &#39;cancelled&#39;<br><br>class ReturnReason(str, Enum):<br>    WRONG_SIZE = &#39;wrong_size&#39;<br>    WRONG_COLOR = &#39;wrong_color&#39;<br>    NOT_AS_DESCRIBED = &#39;not_as_described&#39;<br>    CHANGED_MIND = &#39;changed_mind&#39;<br>    DAMAGED = &#39;damaged&#39;<br><br>class EscalationReason(str, Enum):<br>    COMPLEX_REQUEST = &#39;complex_request&#39;<br>    CUSTOMER_DISSATISFIED = &#39;customer_dissatisfied&#39;<br>    CUSTOMER_REQUEST = &#39;customer_request&#39;<br>    CANNOT_RESOLVE_SITUATION = &#39;cannot_resolve_situation&#39;<br><br>class OrderItem(BaseModel):<br>    product_code: str<br>    name: str<br>    size: str<br>    color: str<br>    quantity: int<br>    price: float<br><br>class Address(BaseModel):<br>    street: str<br>    city: str<br>    postal_code: str<br>    country: str<br><br>class Order(BaseModel):<br>    order_id: str<br>    status: OrderStatus<br>    items: list[OrderItem]<br>    total_amount: float<br>    created_at: datetime<br>    shipping_address: Address<br>    <br>    @property<br>    def can_modify(self) -&gt; bool:<br>        return self.status in [OrderStatus.PENDING, OrderStatus.CONFIRMED]<br>    <br>    @property<br>    def can_return(self) -&gt; bool:<br>        if self.status != OrderStatus.DELIVERED:<br>            return False<br>        return (datetime.now() - self.created_at).days &lt;= 30</pre><p>Notice how as well as defining data structures, we are encoding business rules directly into our Pydantic models that will help during validation. The can_modify and can_return properties ensure that orders are only modified in appropriate states and that returns are only allowed within 30 days of delivery.</p><p>Let’s now create the orders agent:</p><pre>from dataclasses import dataclass<br>from pydantic_ai import Agent<br><br>SYSTEM_PROMPT = &quot;&quot;&quot;\<br>You are an AI assistant for for the luxury fashion store Maison Noir.<br>Your role is to help clients with orders, returns, and related inquiries, using the provided tools.<br><br>Key Guidelines:<br>- Orders can only be modified or canceled when PENDING or CONFIRMED.<br>- Address changes require a valid new address with all fields.<br>- There is a 30-day return window for delivered items. Items must be in original condition with tags.<br>- Always maintain a professional, refined communication tone. Be concise in your answers and precise with details.<br><br>Escalate to a human for:<br>- Complex modifications or special requests.<br>- Dissatisfied customers.<br>- When the user specifically requests to talk with a human.<br>- Any situation you cannot fully resolve.<br>&quot;&quot;&quot;<br><br>@dataclass<br>class Deps:<br>    db: Database<br><br>orders_agent = Agent(<br>    model=&#39;openai:gpt-4o&#39;,<br>    deps_type=Deps,<br>    system_prompt=SYSTEM_PROMPT<br>)</pre><p>We are now using a dataclass for dependencies instead of passing the database directly. While not strictly necessary here, this pattern is more convenient when you need to inject multiple dependencies into your agent.</p><p>The orders agent will run in a simple loop that maintains conversation history and handles streaming responses, very similar to the run_agent_streaming function we saw previously, so there is no need to repeat it here too.</p><p>You can try it yourself by cloning the <a href="https://github.com/ruizguille/ai-agents-ecommerce">GitHub repository</a>, installing the dependencies and running:</p><pre>python orders.py</pre><p>The most interesting part of the orders agent is the toolkit that it has available to perform all the necessary functionalities. Each tool represents a specific order management operation, complete with its own validation and business logic. Let’s take a look at each one in detail.</p><h4>Order Status &amp; Details Retrieval</h4><p>Our first tool handles order status and details inquiries:</p><pre>@orders_agent.tool<br>async def get_order_details(ctx: RunContext[Deps], order_id: str) -&gt; str:<br>    &quot;&quot;&quot;Get the current status and details of an order.&quot;&quot;&quot;<br>    order = ctx.deps.db.orders.get(order_id)<br>    if not order:<br>        return &#39;Order not found&#39;<br>    <br>    items = &#39;\n&#39;.join(<br>        f&quot;- {item.name} ({item.size}, {item.color}) €{item.price}&quot;<br>        for item in order.items<br>    )<br>    address = order.shipping_address<br><br>    return (<br>        f&#39;Status: {order.status.value}\n&#39;<br>        f&#39;Order date: {order.created_at.strftime(&#39;%Y-%m-%d %H:%M&#39;)}\n&#39;<br>        f&#39;Items:\n{items}\n&#39;<br>        f&#39;Total: €{order.total_amount}\n&#39;<br>        f&#39;Shipping to: {address.street}, {address.city}, {address.postal_code} {address.country}\n&#39;<br>    )</pre><p>The tool takes an order_id parameter and uses the database dependency to fetch the corresponding order, which is automatically parsed into our Order Pydantic model. It then formats the order details into a clear, readable response. The agent can therefore use this tool to answer any questions about a specific order.</p><p>Let’s see it in action:</p><blockquote><strong>USER:</strong> What’s the status of my order 001?<br><strong>AGENT:</strong> Your order (ID: 001) is currently confirmed. Here are the details:<br>- Order Date: December 19, 2024<br>- Item: Classic Noir Biker Jacket (M, Black)<br>- Total Amount: €1499.0<br>- Shipping Address: 123 Avenue des Champs-Élysées, Paris, 75008, France</blockquote><h4>Shipping Address Updates</h4><p>Our next tool handles shipping address updates, moving beyond retrieval to actual action execution:</p><pre>@orders_agent.tool<br>async def update_shipping_address(<br>    ctx: RunContext[Deps],<br>    order_id: str,<br>    street: str,<br>    city: str,<br>    postal_code: str,<br>    country: str<br>) -&gt; str:<br>    &quot;&quot;&quot;Update the shipping address for an order if possible.&quot;&quot;&quot;<br>    order = ctx.deps.db.orders.get(order_id)<br>    if not order:<br>        return &#39;Order not found&#39;<br>    <br>    if not order.can_modify:<br>        return f&#39;Cannot modify order - current status is {order.status.value}&#39;<br>    <br>    new_address = Address(street=street, city=city, postal_code=postal_code, country=country)<br>    <br>    if ctx.deps.db.update_shipping_address(order_id, new_address):<br>        return f&#39;Successfully updated shipping address&#39;<br>    return &#39;Failed to update shipping address&#39;</pre><p>Notice how the tool performs all necessary validations, verifying that the order exists and that it can be modified. Then it creates a new Address object using our Pydantic model and uses the database dependency to update the address. Whether successful or not, it returns clear status messages that the agent can relay to the customer. Let&#39;s see some example conversations:</p><blockquote><strong>USER:</strong> I would like to update the shipping address for the order 002.<br><strong>AGENT:</strong> Unfortunately, I am unable to update the shipping address for order 002, as it has already been delivered. If you need further assistance, please let me know.</blockquote><blockquote><strong>USER:</strong> I would like to update the shipping address for the order 001.<br><strong>AGENT:</strong> Could you please provide the new shipping address, including the street, city, postal code, and country?<br><strong>USER:</strong> 14 Rue des Roses, 75008 Paris, France.<br><strong>AGENT:</strong> The shipping address for your order 001 has been successfully updated to 14 Rue des Roses, 75008 Paris, France. If you need further assistance, feel free to ask.</blockquote><h4>Handling Returns &amp; Cancellations</h4><p>One of the most challenging aspects of order management is handling returns and cancellations. Let’s see how we can implement these tools:</p><pre>@orders_agent.tool<br>async def cancel_order(ctx: RunContext[Deps], order_id: str) -&gt; str:<br>    &quot;&quot;&quot;Cancel an order if possible.&quot;&quot;&quot;<br>    order = ctx.deps.db.orders.get(order_id)<br>    if not order:<br>        return &#39;Order not found&#39;<br>    <br>    if not order.can_modify:<br>        return f&#39;Cannot modify order - current status is {order.status.value}&#39;<br><br>    if ctx.deps.db.update_order_status(order_id, OrderStatus.CANCELLED):<br>        return f&#39;Successfully cancelled order&#39;<br>    return &#39;Failed to cancel order&#39;<br><br>@orders_agent.tool<br>async def request_return(<br>    ctx: RunContext[Deps],<br>    order_id: str,<br>    reason: ReturnReason<br>) -&gt; str:<br>    &quot;&quot;&quot;Process a return request for an order.&quot;&quot;&quot;<br>    order = ctx.deps.db.orders.get(order_id)<br>    if not order:<br>        return &#39;Order not found&#39;<br>    <br>    if not order.can_return:<br>        if order.status != OrderStatus.DELIVERED:<br>            return f&#39;Cannot return order - current status is {order.status.value}&#39;<br>        return &#39;Cannot return order - outside our 30-day return window.&#39;<br><br>    return_id = &#39;RET-&#39; + uuid4().hex[:12]<br>    <br>    return (<br>        f&quot;Return request approved for order {order_id}:\n&quot;<br>        f&quot;Reason: {reason.value}\n\n&quot;<br>        f&quot;A return label with ID {return_id} has been emailed to you. Please ship items within 14 days with all original tags attached.&quot;<br>    )</pre><p>PydanticAI makes it quite simple to handle the complexity of these operations elegantly and concisely. The agent automatically knows:</p><ul><li>Which parameters each operation requires (like return reasons and order IDs).</li><li>When each tool is appropriate to use (based on order status and timing).</li><li>How to handle error conditions (invalid states, expired windows).</li><li>How to communicate results back to the customer (from success messages to explanations of why an operation can’t proceed).</li></ul><p>Let’s see an example of a return request:</p><blockquote><strong>USER:</strong> My order is damaged and I would like to get a refund.<br><strong>AGENT:</strong> I can assist you with processing a return for your damaged order. Could you please provide me with your order ID so I can help you further?<br><strong>USER:</strong> 002.<br><strong>AGENT:</strong> Your return request for the damaged order has been approved. You will receive a return label via email with ID RET-eedc4cdf4494. Please ship the item back within <strong>14</strong> days, ensuring it includes all original tags. Once we’ve received the item and verified its condition, your refund will be processed. If you need further assistance, please let me know.</blockquote><h4>Human Escalation</h4><p>Finally, <strong>a critical feature of any automated system is knowing when to hand over to human support.</strong> Our escalation tool will transfer conversations to a human during complex requests or unresolved issues, when the user specifically demands it, or when the agent detects customer dissatisfaction. The tool is also able to <strong>flag high-priority cases,</strong> ensuring urgent matters or clearly dissatisfied customers receive immediate attention.</p><p>Here’s the implementation:</p><pre>@orders_agent.tool<br>async def escalate_to_human(<br>    ctx: RunContext[Deps],<br>    reason: EscalationReason,<br>    details: str,<br>    high_priority: bool = False<br>) -&gt; str:<br>    &quot;&quot;&quot;Escalate the conversation to a human.<br>    Set high_priority=True for urgent matters or when customer is clearly dissatisfied.<br>    &quot;&quot;&quot;<br>    response_time = &#39;1 hour&#39; if high_priority else &#39;12 hours&#39;<br>    return f&#39;This matter has been escalated to our support team. We will contact you within {response_time}.&#39;</pre><p>This implementation is simplified for demonstration purposes. A production system would need to integrate with your support desk software, trigger notifications, and possibly create support tickets with appropriate priority levels and routing. Those are additional details to add inside the tool, but the fundamental agent logic would remain the same.</p><p>Let’s take a look at an example where the agent detects user frustration:</p><blockquote><strong>USER:</strong> I would like to cancel my order 002.<br><strong>AGENT:</strong> Your order with ID 002 has already been delivered and cannot be canceled. However, if you wish to return the item, please let me know, and I can assist you with the return process.<br><strong>USER:</strong> You are not being helpful.<br><strong>AGENT:</strong> I’ve escalated your concern to our support team for further assistance. You will be contacted within 1 hour. Thank you for your patience.</blockquote><p>Behind the scenes, the agent generates a detailed tool call that provides the support team with valuable context:</p><pre>ToolCallPart(<br>    tool_name=&#39;escalate_to_human&#39;,<br>    args=ArgsJson(<br>        args_json=&#39;{<br>          &quot;reason&quot;: &quot;customer_dissatisfied&quot;,<br>          &quot;details&quot;: &quot;Customer is not satisfied with the inability to cancel a delivered order and is seeking further assistance.&quot;,<br>          &quot;high_priority&quot;: true<br>      }&#39;<br>    ),<br>    tool_call_id=&#39;call_Lw115FnRP5h8UIgvvJkbdaIK&#39;,<br>    part_kind=&#39;tool-call&#39;<br>)</pre><h3>Beyond the Demo: Production Considerations</h3><p>While our example agents illustrate the core concepts, deploying AI agents in production requires additional considerations:</p><ul><li>Security, customer authentication and access control.</li><li>Integration with your existing order management systems, inventory and customer databases.</li><li>Agent performance monitoring and human feedback for continuous improvement.</li><li>Logging and audit trails.</li><li>Error handling and graceful degradation when external services are unavailable.</li><li>GDPR compliance and proper data privacy.</li></ul><p>If you’re interested in exploring how AI agents and automations could transform your e-commerce operations, I’d love to have a conversation about your specific needs and challenges. You can reach me at <a href="/service/mailto:guillermo@codeawake.com">guillermo@codeawake.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d52059920f11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL with pgvector as a Vector Database for RAG]]></title>
            <link>https://medium.com/@codeawake/postgresql-vector-database-6516193cba90?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6516193cba90</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 08:05:21 GMT</pubDate>
            <atom:updated>2024-12-02T17:46:02.777Z</atom:updated>
            <content:encoded><![CDATA[<h4>Implementing vector search and Retrieval-Augmented Generation using PostgreSQL</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/postgresql-vector-database"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*NW12oM6x6OGOp5dI6uw6Pw.png" /></figure><p>We have explored <strong>vector search</strong> and <strong>Retrieval-Augmented Generation (RAG)</strong> in previous blog posts like <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264">RAG from Scratch</a> and <a href="https://medium.com/@codeawake/ai-chatbot-5bd2fa3324e3">Building an AI Chatbot Powered by Your Data</a>. In the last one, we actually built a full-stack RAG chatbot application using Redis as the vector database.</p><p>But there is another very well-known database that many developers already have in their toolkit: <a href="https://www.postgresql.org/">PostgresSQL</a>. Wouldn’t it be great if we could just use it as a vector database too?</p><p><strong>PostgreSQL is a powerful and widely used open-source relational database.</strong> It’s also incredibly versatile, allowing you to store and manipulate JSON data (similar to NoSQL and document databases) and providing a <strong>rich set of extensions with added functionalities</strong>, such as <a href="https://postgis.net/">PostGIS</a> for geospatial data or <a href="https://github.com/citusdata/pg_cron">pgcron</a> for job scheduling.</p><p>Thanks to the <a href="https://github.com/pgvector/pgvector">pgvector</a> extension, <strong>Postgres can now also perform efficient similarity searches on vector embeddings.</strong> This opens up many possibilities for RAG and AI applications, with the added benefit of using a familiar database you might already have in your stack. It also means that you can <strong>combine relational data, JSON data and vector embeddings in a single system</strong>, enabling complex queries that involve both structured data and vector searches.</p><p>There are, of course, specialized vector databases like Qdrant, Pinecone or Weaviate that offer optimized performance for very large datasets and more advanced functionalities. However, Postgres with pgvector is a very compelling alternative if you want to <strong>keep all your application data more deeply integrated and minimize the number of database systems</strong> in your stack, reducing costs and complexity.</p><p>In this post, we’ll explore how to set up and use Postgres as a vector database and how it can be used for vector search and Retrieval-Augmented Generation applications in Python.</p><h3>Setting Up PostgreSQL and pgvector</h3><p>Before we get started, we need to install PostgreSQL, pgvector and the Python libraries we will be using:</p><ol><li>Download and install PostgreSQL following the <a href="https://www.postgresql.org/download/">instructions in the official documentation</a> for your operating system.</li><li>Install the pgvector extension, following the <a href="https://github.com/pgvector/pgvector?tab=readme-ov-file#installation">installation notes in the pgvector repository</a>.</li><li>Install the required Python packages. As well as the pgvector Python library, we will be using <a href="https://www.sqlalchemy.org/">SQLAlchemy</a> as the ORM and the <a href="https://github.com/MagicStack/asyncpg">asyncpg</a> driver to connect to Postgres asynchronously using asyncio:</li></ol><pre>pip install sqlalchemy pgvector asyncpg</pre><p>4. Create a new database and enable the pgvector extension:</p><pre>createdb rag_db<br>psql rag_db<br>CREATE EXTENSION vector;</pre><h3>Creating a Vector Database with PostgreSQL</h3><p>Now that the database is set up, let’s create a <strong>SQLAlchemy model</strong> to represent our vector data:</p><pre>from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column<br>from sqlalchemy import Text<br>from sqlalchemy.dialects.postgresql import JSONB<br>from pgvector.sqlalchemy import Vector<br><br>class Base(DeclarativeBase):<br>    pass<br><br>class Vector(Base):<br>    __tablename__ = &#39;vectors&#39;<br><br>    id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)<br>    text: Mapped[str] = mapped_column(Text)<br>    vector = mapped_column(Vector(1024))<br>    metadata_: Mapped[dict | None] = mapped_column(&#39;metadata&#39;, JSONB)<br><br>    def __repr__(self):<br>        return f&#39;Vector(id={self.id}, text={self.text[:50]}..., metadata={self.metadata_})&#39;</pre><p>We’re using <a href="https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html#using-annotated-declarative-table-type-annotated-forms-for-mapped-column">SQLAlchemy 2.0 syntax</a>, which allows us to use Python’s type hints with Mapped and mapped_column to conveniently derive database column types from the type annotations.</p><p>The Vector model defines the following columns in the vectors table:</p><ul><li>id: A unique identifier (and primary key) for each vector entry.</li><li>vector: Stores the actual vector embedding. We specify the embedding dimensions (1024 in this case) to match our chosen embedding model.</li><li>text: Contains the original text that the vector embedding represents. This allows us to easily retrieve the corresponding text when performing vector similarity searches.</li><li>metadata: Stores additional properties for each vector entry as JSON. This can include information like the source document name, chunk index within the document, or any other relevant metadata useful for vector searches or filtering.</li></ul><p>⚠️ <em>Note that we use “metadata_” as the attribute name in the model because “metadata” is a reserved keyword in SQLAlchemy models, but the actual column name in the database will be “metadata”.</em></p><p>To create the database table defined in our model, we can use the following SQLAlchemy code:</p><pre>from sqlalchemy.ext.asyncio import create_async_engine<br><br>DB_URL = &#39;postgresql+asyncpg://admin:postgres@localhost:5432/rag_db&#39;<br><br>engine = create_async_engine(DB_URL)<br><br>async def db_create():<br>    async with engine.begin() as conn:<br>        await conn.run_sync(Base.metadata.create_all)</pre><p>The &#39;postgresql+asyncpg&#39; prefix in the database URL is necessary because we are using the asyncpg driver to enable asynchronous connections with asyncio.</p><p>Once the vectors table has been created, you can use <a href="https://www.pgadmin.org/">pgAdmin</a> to explore the table structure and run queries with a user-friendly interface.</p><h3>Embedding Documents into PostgreSQL</h3><p>The next step is to process, embed and store information in our Postgres vector database. If you are familiar with RAG systems or have read our previous posts, this process involves the following steps:</p><ul><li><strong>Extracting</strong> text from the source documents.</li><li><strong>Splitting</strong> the text into smaller chunks.</li><li>Converting these chunks into <strong>vector embeddings that capture their meanings.</strong></li><li><strong>Storing</strong> the vector embeddings together with the original texts and any relevant metadata in our vector database.</li></ul><p>Let’s take a look at an example function that carries out these steps:</p><pre>engine = create_async_engine(DB_URL)<br>Session = async_sessionmaker(engine, expire_on_commit=False)<br><br>async def add_document_to_vector_db(doc_path):<br>    text = extract_text(docpath)<br>    doc_name = os.path.splitext(os.path.basename(doc_path))[0]<br>    <br>    chunks = []<br>    text_splitter = TextSplitter(chunk_size=512)<br>    text_chunks = text_splitter.split(text)<br>    for idx, text_chunk in enumerate(text_chunks):<br>        chunks.append({<br>            &#39;text&#39;: text_chunk,<br>            &#39;metadata_&#39;: {&#39;doc&#39;: doc_name, &#39;index&#39;: idx}<br>        })<br><br>    vectors = await create_embeddings([chunk[&#39;text&#39;] for chunk in chunks])<br><br>    for chunk, vector in zip(chunks, vectors):<br>        chunk[&#39;vector&#39;] = vector<br>    <br>    async with Session() as db:<br>        for chunk in chunks:<br>            db.add(Vector(**chunk))<br>        await db.commit()</pre><p>In the code above:</p><ul><li>The extract_text function is responsible for extracting text from the document. You can use libraries like <a href="https://pdfminersix.readthedocs.io/en/latest/">pdfminer</a> or <a href="https://pypdf.readthedocs.io/en/stable/">pypdf</a> to extract text from PDF files, or <a href="https://github.com/ankushshah89/python-docx2txt">docx2txt</a> for Word documents.</li><li>We use a TextSplitter to break the document into smaller chunks of text. In this example, we are creating chunks of 512 tokens. You can read more about the chunking functionality and different strategies in our <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264#910d">RAG from Scratch</a> post.</li><li>The create_embeddings function converts our text chunks into vector embeddings.</li><li>Finally, we create a database session and use the previously defined Vector model to add each chunk&#39;s vector embedding, text and metadata to our Postgres database.</li></ul><p>To create the vector embeddings, you can use any embedding model of your choice. Just make sure that the embedding model’s dimensions match the dimensions of the vector column, as defined in the Vector model (1024 in our example).</p><p>Here’s an example implementation using <a href="https://platform.openai.com/docs/guides/embeddings">OpenAI’s text-embedding-3-large embeddings</a>:</p><pre>from openai import AsyncOpenAI<br><br>client = AsyncOpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br><br>async def get_embeddings(input):<br>    res = await client.embeddings.create(input=input, model=&#39;text-embedding-3-large&#39;, dimensions=1024)<br>    return [item.embedding for item in res.data]</pre><h3>Vector Search with PostgreSQL</h3><p>Now that we have our documents embedded and stored in our PostgreSQL database, we can perform <strong>vector similarity searches to retrieve the most relevant information for our queries.</strong> This is a key step in building a Retrieval-Augmented Generation (RAG) system.</p><p>The following function shows how we can implement a basic similarity search in PostgreSQL with pgvector:</p><pre>async def vector_search(query_vector, top_k=3):<br>    async with Session() as db:<br>        query = (<br>            select(Vector.text, Vector.metadata_, Vector.vector.cosine_distance(query_vector).label(&#39;distance&#39;))<br>            .order_by(&#39;distance&#39;)<br>            .limit(top_k)<br>        )<br>        res = await db.execute(query)<br>        return [{<br>            &#39;text&#39;: text,<br>            &#39;metadata&#39;: metadata,<br>            &#39;score&#39;: 1 - distance<br>        } for text, metadata, distance in res]</pre><p>Let’s break down this function and the search process:</p><ul><li>The query_vector parameter is the vector embedding of our search query. It&#39;s generated using the same embedding model we used previously for the document chunks.</li><li>We use the cosine_distance function provided by pgvector to calculate the distance between our query vector and each vector in our database. Cosine distance is a measure of dissimilarity: <strong>the smaller the cosine distance, the bigger the similarity to the user query.</strong></li><li>We sort the results by the distance (ascending) and retrieve the top_k most similar chunks to our query.</li><li>For each of the top_k most similar chunks, we return the chunk text, metadata and a similarity score which is calculated as 1 - distance.</li></ul><p>It’s also important to highlight that <strong>pgvector performs exact nearest neighbor search by default,</strong> which provides perfect recall (all of the nearest neighbors are found), but it can be slower with large datasets.</p><p>In those cases, you can also <strong>create an index to speed up searches and trade a bit of accuracy for much faster search times.</strong> The supported index types are IVFFlat (Inverted File Flat) and HNSW (Hierarchical Navigable Small World). You can read more about indexing in pgvector <a href="https://github.com/pgvector/pgvector?tab=readme-ov-file#indexing">here</a>.</p><h3>RAG in Action</h3><p>With all the vector search functionality in place, now it’s the time to put it all together to create a basic Retrieval-Augmented Generation example, using <a href="https://platform.openai.com/docs/models/gpt-4o">OpenAI’s GPT-4o</a> model, that can answer questions about the documents we embedded in the PostgreSQL database.</p><p>First, let’s take a look at the prompts that we are using:</p><pre>SYSTEM_PROMPT = &quot;&quot;&quot;<br>You are an AI assistant that answers questions about documents in your knowledge base.<br>&quot;&quot;&quot;<br><br>RAG_PROMPT = &quot;&quot;&quot;<br>Use the following pieces of context to answer the user question.<br>You must only use the facts from the context to answer.<br>If the answer cannot be found in the context, say that you don&#39;t have enough information to answer the question and provide any relevant facts found in the context.<br><br>Context:<br>{context}<br><br>User Question:<br>{question}<br>&quot;&quot;&quot;</pre><p>And this is how we can implement the basic RAG system:</p><pre>from openai import AsyncOpenAI<br><br>client = AsyncOpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br><br>async def answer_question_with_rag(question):<br>    query_vector = await get_embedding(question)<br>    top_chunks = await vector_search(query_vector, top_k=3)<br>    context = &#39;\n\n---\n\n&#39;.join([chunk[&#39;text&#39;] for chunk in top_chunks]) + &#39;\n\n---&#39;<br>    user_message = RAG_PROMPT.format(context=context, question=question)<br>    messages=[<br>        {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: SYSTEM_PROMPT},<br>        {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_message}<br>    ]<br>    response = await client.chat.completions.create(model=&#39;gpt-4o&#39;, messages=messages)<br>    return response.choices[0].message.content</pre><p>This function encapsulates the <strong>core RAG process:</strong> it converts the user’s question to a vector, performs a similarity search in our Postgres vector database and includes the retrieved information as context for the GPT-4o model to generate an informed response.</p><p>To use this in your application, you might do something like this:</p><pre>question = &quot;What are the main challenges in renewable energy adoption?&quot;<br>answer = await answer_question_with_rag(question)<br>print(answer)</pre><p>You can now apply all the code we’ve seen to your own applications. You can embed your own documents and use the <strong>combination of PostgreSQL with pgvector and GPT-4o</strong> (or any other LLM of your choice) to answer questions based on those documents.</p><p>And you can use these ideas to build more advanced applications like AI chatbots and assistants, with a <strong>simple architecture</strong> that takes advantage of PostgreSQL’s power and versatility and <strong>all your data integrated in a single place.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6516193cba90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a React Frontend for an AI Chatbot]]></title>
            <link>https://medium.com/@codeawake/ai-chatbot-frontend-1823b9c78521?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1823b9c78521</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 07:45:40 GMT</pubDate>
            <atom:updated>2025-09-27T09:08:31.465Z</atom:updated>
            <content:encoded><![CDATA[<h4>An interactive React chatbot UI with real-time streaming, markdown rendering and auto-scrolling</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/ai-chatbot-frontend"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*aaXFEwpa5J7o71eQEx_b6g.png" /></figure><p>In our previous post, <a href="https://medium.com/@codeawake/ai-chatbot-5bd2fa3324e3">“Building an AI Chatbot Powered by Your Data”</a><strong>,</strong> we explored how to implement Retrieval-Augmented Generation (RAG) to create a production-ready AI chatbot that can answer questions about new technology trends, powered by the latest reports from leading institutions like the World Bank, World Economic Forum, McKinsey, Deloitte and the OECD.</p><p>However, there was a very important part missing to complete this full-stack chatbot application: the user interface. In this post, we are building a <strong>React frontend for our technology trends AI chatbot.</strong></p><p>This frontend is designed to be flexible and can be easily adapted for any other AI chatbot application, like a customer support bot or a coding assistant. All you need is to connect it to an API that provides two specific chat endpoints (or adjust it to your own API endpoints).</p><p>The frontend will be a simple <a href="https://react.dev/">React</a> application without additional frameworks, using <a href="https://tailwindcss.com/">Tailwind CSS</a> for styling and <a href="https://vitejs.dev/">Vite</a> as a build tool and bundler. By building the frontend from the ground up, you’ll discover that creating a functional chatbot interface doesn’t require a lot of code and the code is quite simple.</p><p>Our frontend implementation will include features like <strong>streaming responses with Server-Sent-Events</strong>, <strong>markdown support</strong> for chat responses, a <strong>mobile-friendly responsive design</strong> and <strong>error handling</strong> during chat interactions. And you can extend it with more advanced features like authentication, past chats history and even chat sharing to build your own ChatGPT-like application tailored to your specific needs.</p><p>By the end of this post, you’ll have a fully functional, customizable AI chatbot frontend that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RRvXa15qtUaOLRSY" /></figure><p>The complete source code for this project (frontend and backend) is available on this <a href="https://github.com/ruizguille/tech-trends-chatbot">GitHub repository</a>.</p><h3>Frontend Project Structure</h3><p>This is the core project structure of our React application:</p><pre>frontend/<br>├── public/                   # Public static assets<br>├── src/<br>│   ├── assets/               # Images and icons<br>│   ├── components/<br>│   │   ├── Chatbot.jsx       # Main chatbot component<br>│   │   ├── ChatInput.jsx     # User input component<br>│   │   ├── ChatMessages.jsx  # Chat messages component<br>│   │   └── Spinner.jsx       # Loading spinner component<br>│   ├── hooks/                # Custom React hooks<br>│   │   ├── useAutoScroll.js<br>│   │   └── useAutosize.js<br>│   ├── api.js                # Functions for backend API communication<br>│   ├── App.jsx               # Root App component<br>│   ├── index.css             # Global styles<br>│   ├── main.jsx              # Entry point for the React app<br>│   └── utils.js              # Utility functions (parseSSEStream)<br>├── index.html                # HTML template<br>├── tailwind.config.js        # Tailwind CSS configuration<br>└── vite.config.js            # Vite configuration</pre><h3>Main Chatbot Component</h3><p>The core component of the chatbot frontend app is the Chatbot component. It contains the main application state and renders the required subcomponents. Let&#39;s take a look at a slightly simplified version of the component code:</p><pre>import { useState } from &#39;react&#39;;<br>import { useImmer } from &#39;use-immer&#39;;<br>import ChatMessages from &#39;@/components/ChatMessages&#39;;<br>import ChatInput from &#39;@/components/ChatInput&#39;;<br><br>function Chatbot() {<br>  const [chatId, setChatId] = useState(null);<br>  const [messages, setMessages] = useImmer([]);<br>  const [newMessage, setNewMessage] = useState(&#39;&#39;);<br><br>  const isLoading = messages.length &amp;&amp; messages[messages.length - 1].loading;<br><br>  async function submitNewMessage() {<br>    // Implemented in the next section<br>  }<br><br>  return (<br>    &lt;div&gt;<br>      {messages.length === 0 &amp;&amp; (<br>        &lt;div&gt;{/* Chatbot welcome message */}&lt;/div&gt;<br>      )}<br>      &lt;ChatMessages<br>        messages={messages}<br>        isLoading={isLoading}<br>      /&gt;<br>      &lt;ChatInput<br>        newMessage={newMessage}<br>        isLoading={isLoading}<br>        setNewMessage={setNewMessage}<br>        submitNewMessage={submitNewMessage}<br>      /&gt;<br>    &lt;/div&gt;<br>  );<br>}<br><br>export default Chatbot;</pre><p><em>⚠️ In general, we are going to be focusing on the structure and core logic of the frontend components. To keep the code snippets concise and easier to read, the Tailwind CSS styling classes will be omitted, as they aren’t crucial for understanding the chatbot’s functionality. But you can check out the full project code in the </em><a href="https://github.com/ruizguille/tech-trends-chatbot"><em>GitHub repository</em></a><em>.</em></p><p>As you can see at the top of the component, our application has three main state variables:</p><ul><li>chatId: Stores the current chat session id.</li><li>messages: Holds all the messages in the current chat. Each message contains a role (user or assistant), content, loading and error property.</li><li>newMessage: Stores the current text in the chat input (before it gets submitted).</li></ul><p>Note also how we are using both useState and useImmer for state management. If you have experience with React, you will know that state must never be updated directly and all <a href="https://react.dev/learn/updating-objects-in-state">state updates</a> must be performed immutably (creating a new object or modifying a deep copy of the object). This can lead to verbose and error-prone code, especially in nested data structures like the messages array.</p><p><a href="https://immerjs.github.io/immer/">Immer</a> is a tiny and very convenient library that simplifies state updates, and provides the useImmer hook for that purpose. Immer allows you to write more concise and intuitive code by applying all updates to a temporary draft object, and takes care of creating the next state immutably for you. For instance, you can conveniently update the last object in the messages array like this:</p><pre>setMessages(draft =&gt; {<br>  draft[draft.length - 1].loading = false;<br>});</pre><p>The JSX structure of the Chatbot component is straightforward. It renders three elements:</p><ul><li>An initial welcome message (displayed if there are no messages yet).</li><li>The ChatMessages component to display the chat conversation.</li><li>The ChatInput component for user input.</li></ul><h3>Submitting New Messages &amp; Parsing the Response</h3><p>The core functionality of our chatbot lies in the submitNewMessage function:</p><pre>async function submitNewMessage() {<br>  const trimmedMessage = newMessage.trim();<br>  if (!trimmedMessage || isLoading) return;<br><br>  setMessages(draft =&gt; [...draft,<br>    { role: &#39;user&#39;, content: trimmedMessage },<br>    { role: &#39;assistant&#39;, content: &#39;&#39;, sources: [], loading: true }<br>  ]);<br>  setNewMessage(&#39;&#39;);<br><br>  let chatIdOrNew = chatId;<br>  try {<br>    if (!chatId) {<br>      const { id } = await api.createChat();<br>      setChatId(id);<br>      chatIdOrNew = id;<br>    }<br><br>    const stream = await api.sendChatMessage(chatIdOrNew, trimmedMessage);<br>    for await (const textChunk of parseSSEStream(stream)) {<br>      setMessages(draft =&gt; {<br>        draft[draft.length - 1].content += textChunk;<br>      });<br>    }<br>    setMessages(draft =&gt; {<br>      draft[draft.length - 1].loading = false;<br>    });<br>  } catch (err) {<br>    console.log(err);<br>    setMessages(draft =&gt; {<br>      draft[draft.length - 1].loading = false;<br>      draft[draft.length - 1].error = true;<br>    });<br>  }<br>}</pre><p>Let’s break down what is happening:</p><ol><li>We make sure that the input message is not empty or that a response is already loading before proceeding.</li><li>We add the user’s message to the chat and create a placeholder assistant message with the loading property set to true (useful to display the spinner while it loads).</li><li>If there is no existing chat session, we create a new one using the api.createChat function.</li><li>We then use api.sendChatMessage to send the user&#39;s message to the backend, which returns a stream as response.</li><li>We use the parseSSEStream utility function to convert the SSE stream into an async iterator of text chunks. For each new text chunk received, we update the assistant message, creating a real-time streaming effect.</li><li>Once the response finishes streaming, we update the assistant message’s loading property to false.</li><li>If there any errors in the process, we set the assistant message’s error property to true to display an error message in the chat interface.</li></ol><p>The api.js file contains the two functions ( createChat and sendChatMessage) that interact with the <a href="https://medium.com/@codeawake/ai-chatbot-5bd2fa3324e3#2d57">backend API endpoints</a>:</p><pre>const BASE_URL = import.meta.env.VITE_API_URL;<br><br>async function createChat() {<br>  const res = await fetch(BASE_URL + &#39;/chats&#39;, {<br>    method: &#39;POST&#39;,<br>    headers: { &#39;Content-Type&#39;: &#39;application/json&#39; }<br>  });<br>  const data = await res.json();<br>  if (!res.ok) {<br>    return Promise.reject({ status: res.status, data });<br>  }<br>  return data;<br>}<br><br>async function sendChatMessage(chatId, message) {<br>  const res = await fetch(BASE_URL + `/chats/${chatId}`, {<br>    method: &#39;POST&#39;,<br>    headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>    body: JSON.stringify({ message })<br>  });<br>  if (!res.ok) {<br>    return Promise.reject({ status: res.status, data: await res.json() });<br>  }<br>  return res.body;<br>}</pre><p>As you can see in the code, they both use the native <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch API</a>. The createChat function returns a JSON response with the new chat id, while sendChatMessage returns the body directly as it&#39;s a streaming response that we need to handle differently.</p><p>Finally, let’s take a look at the utility function that parses the SSE stream, taking advantage of the <a href="https://github.com/rexxars/eventsource-parser">eventsource-parser</a> library to simplify the SSE data extraction:</p><pre>import { EventSourceParserStream } from &#39;eventsource-parser/stream&#39;;<br><br>export async function* parseSSEStream(stream) {<br>  const sseStream = stream<br>    .pipeThrough(new TextDecoderStream())<br>    .pipeThrough(new EventSourceParserStream())<br>  <br>  for await (const chunk of sseStream) {<br>    if (chunk.type === &#39;event&#39;) {<br>      yield chunk.data;<br>    } <br>  }<br>}</pre><p>The function applies two transformations to the input stream: TextDecoderStream() converts the incoming bytes into text, and EventSourceParserStream() parses the individual server-sent events. It then iterates through the events and <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/yield">yields</a> each event&#39;s data (which contains a text chunk of the assistant&#39;s response).</p><p>Notice how the function is an <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function*">async generator</a>, which is why we can iterate over the text chunks with a simple for await...of loop in the submitNewMessage function.</p><h3>Displaying Chat Messages</h3><p>The ChatMessages component is responsible for rendering the message history. Let&#39;s take a look at a simplified version of the code (excluding the CSS styling classes):</p><pre>import Markdown from &#39;react-markdown&#39;;<br>import useAutoScroll from &#39;@/hooks/useAutoScroll&#39;;<br>import Spinner from &#39;@/components/Spinner&#39;;<br>import userIcon from &#39;@/assets/images/user.svg&#39;;<br>import errorIcon from &#39;@/assets/images/error.svg&#39;;<br><br>function ChatMessages({ messages, isLoading }) {<br>  const scrollContentRef = useAutoScroll(isLoading);<br>  <br>  return (<br>    &lt;div ref={scrollContentRef}&gt;<br>      {messages.map(({ role, content, loading, error }, idx) =&gt; (<br>        &lt;div key={idx}&gt;<br>          {role === &#39;user&#39; &amp;&amp; (<br>            &lt;img src={userIcon} alt=&#39;user icon&#39; /&gt;<br>          )}<br>          &lt;div&gt;<br>            &lt;div&gt;<br>              {(loading &amp;&amp; !content) ? &lt;Spinner /&gt;<br>                : (role === &#39;assistant&#39;)<br>                  ? &lt;Markdown&gt;{content}&lt;/Markdown&gt;<br>                  : &lt;div&gt;{content}&lt;/div&gt;<br>              }<br>            &lt;/div&gt;<br>            {error &amp;&amp; (<br>              &lt;div&gt;<br>                &lt;img src={errorIcon} alt=&#39;error icon&#39; /&gt;<br>                &lt;span&gt;Error generating the response&lt;/span&gt;<br>              &lt;/div&gt;<br>            )}<br>          &lt;/div&gt;<br>        &lt;/div&gt;<br>      ))}<br>    &lt;/div&gt;<br>  );<br>}<br><br>export default ChatMessages;</pre><p>The component handles different types of message visualizations:</p><ul><li>User messages are displayed with a user icon.</li><li>Assistant messages are rendered using the Markdown component provided by the <a href="https://github.com/remarkjs/react-markdown">react-markdown</a> library. This is very useful as <strong>LLM responses are often formatted in Markdown with rich text formatting, paragraphs, lists and other elements.</strong></li><li>When an assistant message is loading and has no content yet, a Spinner component is displayed.</li><li>If there are any errors processing an assistant response, an error icon and message are displayed below.</li></ul><p>To improve the user experience using the chatbot, we also implement <strong>auto-scrolling when new assistant messages are streamed</strong> with a custom useAutoScroll hook. If you are curious about the details, you can <a href="https://github.com/ruizguille/tech-trends-chatbot/blob/master/frontend/src/hooks/useAutoScroll.js">check out the full code here</a>. This is a brief breakdown of how the hook performs auto-scrolling:</p><ul><li>It defines and returns a scrollContentRef that we add to the chat messages container element. Using this ref and the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Resize_Observer_API">Resize Observer Web API</a>, we can now monitor the chat messages container for changes in size (when new content is added) and automatically scroll to the bottom if the scrollbar isn&#39;t at the bottom.</li><li>It also includes a smart disable feature, common in AI chat applications: if the user manually scrolls up while an assistant message is being streamed, it temporarily disables auto-scrolling. This allows the user to read any part of the conversation history without being interrupted by auto-scrolling.</li><li>Auto-scrolling is re-enabled when the user scrolls back to the bottom, or when a new assistant message starts streaming.</li><li>An important detail to note is that this hook assumes the entire document (HTML element) is the scrollable container and therefore uses document.documentElement for scroll measurements and scrolling operations. If your application uses a different scrollable container (e.g., a div with overflow: scroll), you would need to modify the hook to use a ref for that specific container.</li></ul><h3>User Input Interface</h3><p>The final piece of our chatbot frontend is the user input interface. This is implemented in the ChatInput component, which allows users to type and submit their messages. This is a simplified version of the code:</p><pre>import useAutosize from &#39;@/hooks/useAutosize&#39;;<br>import sendIcon from &#39;@/assets/images/send.svg&#39;;<br><br>function ChatInput({ newMessage, isLoading, setNewMessage, submitNewMessage }) {<br>  const textareaRef = useAutosize(newMessage);<br><br>  function handleKeyDown(e) {<br>    if(e.keyCode === 13 &amp;&amp; !e.shiftKey &amp;&amp; !isLoading) {<br>      e.preventDefault();<br>      submitNewMessage();<br>    }<br>  }<br>  <br>  return(<br>    &lt;div&gt;<br>      &lt;textarea<br>        ref={textareaRef}<br>        rows=&#39;1&#39;<br>        value={newMessage}<br>        onChange={e =&gt; setNewMessage(e.target.value)}<br>        onKeyDown={handleKeyDown}<br>      /&gt;<br>      &lt;button onClick={submitNewMessage}&gt;<br>        &lt;img src={sendIcon} alt=&#39;send&#39; /&gt;<br>      &lt;/button&gt;<br>    &lt;/div&gt;<br>  );<br>}<br><br>export default ChatInput;</pre><p>The ChatInput component includes a textarea element for typing messages and a send button to submit them. The textarea also includes an auto-resizing functionality using a custom useAutosize hook. This hook dynamically adjusts the height of the textarea based on its content, allowing it to grow as the user types and shrink when content is deleted. You can <a href="https://github.com/ruizguille/tech-trends-chatbot/blob/master/frontend/src/hooks/useAutosize.js">see the hook code here</a>.</p><p>The component also includes a handleKeyDown function that allows users to submit messages by simply pressing Enter (without Shift). At the same time, it preserves the textarea&#39;s native behavior of adding newlines with Shift+Enter, giving users the flexibility to format longer messages or add line breaks for clarity.</p><p>We have now completed the implementation of a full-stack AI chatbot. You can use <a href="https://github.com/ruizguille/tech-trends-chatbot">this project</a> as a starting point to build upon and customize for your specific needs.</p><p>All the techniques that we have covered in this post and the previous one — <strong>Retrieval Augmented Generation, vector databases, semantic search, asynchronous programming, structured outputs, real-time SSE streaming, markdown rendering, auto-scrolling</strong> — provide a practical framework that can help you create your own AI chatbot applications.</p><p>I hope this was helpful, and I look forward to seeing what you build next with it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1823b9c78521" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI Chatbot Powered by Your Data]]></title>
            <link>https://medium.com/@codeawake/ai-chatbot-5bd2fa3324e3?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bd2fa3324e3</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 09:38:02 GMT</pubDate>
            <atom:updated>2025-09-27T09:07:42.693Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Retrieval Augmented Generation (RAG) chatbot implementation with FastAPI, Redis and OpenAI</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/ai-chatbot"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*jL3md30ZY2inQquflbMsdg.png" /></figure><p>In a previous post, <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264">“Retrieval Augmented Generation from Scratch”</a>, we explored the fundamentals of RAG systems by building a basic local application in Python from the ground up. We covered key concepts like <strong>document loading, chunking, vector embeddings and semantic search.</strong> The application we built was capable of answering questions about the movie Inception using its screenplay as the data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rQPOzCaaXMkpATVq" /></figure><p>Now we are taking this to the next level. In this blog post, we are building a <strong>production-ready AI chatbot application</strong> that shows how these concepts can be applied in a real-wold use case. It will be a <strong>technology trends chatbot powered by the latest technology reports</strong> from top institutions like the World Bank, the WEF, McKinsey, Deloitte and the OECD.</p><p>This will be a <strong>full-stack web application</strong>, with a Python backend (covered here) and a React frontend (covered in the next post). And while this is just a specific example, the beauty of this system is that it can be <strong>customized with your own data sources and applied to different use cases.</strong></p><p>We are also going to be using more advanced techniques and technologies to create a production-ready application. A brief outline of what we’ll cover:</p><ul><li><strong>FastAPI</strong> to develop the backend API.</li><li><strong>Redis</strong> as a lightning-fast vector database for semantic search and chat storage.</li><li><strong>OpenAI’s embeddings</strong> and <strong>GPT-4o</strong> as the chatbot’s large language model (LLM).</li><li><strong>OpenAI’s tool calling</strong> and the recently released <strong>Structured Outputs</strong> feature.</li><li><strong>Conversation memory</strong> using chat history.</li><li><strong>Streaming</strong> responses using Server-Sent Events.</li><li><strong>Asynchronous Python</strong> programming with asyncio.</li><li>A <strong>React</strong> user interface (in the next post).</li></ul><p>The techniques we’ll cover can be applied to many different domains, from <strong>technology assistants</strong> to <strong>customer support</strong>, <strong>financial research</strong> or any field where specialized and up-to-date information retrieval is key.</p><p>You can find all the application code in this <a href="https://github.com/ruizguille/tech-trends-chatbot">GitHub repository</a>. While I’ll do my best to explain as much as possible in this blog post, you can check the finer details there. And if you are not familiar with some of the concepts, make sure to revise the previous RAG from scratch post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wttzm2QRVy_OLUFZzOMlLw.gif" /></figure><h3>Backend Project Structure</h3><p>Before diving into the details, let’s take a look at the backend project structure to get an overview of how the AI chatbot application is organized:</p><pre>backend/<br>│<br>├── app/<br>│   ├── assistants/<br>│   │   ├── assistant.py        # Main assistant class for API<br>│   │   ├── local-assistant.py  # Assistant class for local console app<br>│   │   ├── prompts.py          # Assistant prompts<br>│   │   └── tools.py            # Contains the assistant&#39;s QueryKnowledgeBaseTool<br>│   │<br>│   ├── utils/<br>│   │   ├── splitter.py         # Text splitter utility<br>│   │   └── sse_stream.py       # SSEStream class for server-sent events<br>│   │<br>│   ├── api.py                  # FastAPI endpoints<br>│   ├── config.py               # Pydantic Settings configuration<br>│   ├── db.py                   # Redis database code<br>│   ├── export.py               # Utility to export all chats to JSON<br>│   ├── loader.py               # Document processing and knowledge base loading<br>│   ├── main.py                 # Main FastAPI application<br>│   └── openai.py               # OpenAI API functions<br>│<br>└── pyproject.toml              # Project dependencies and configuration</pre><h3>Building the Knowledge Base</h3><p>The first thing we need in a RAG-based chatbot is a knowledge base, which contains the data that will be used to answer user questions. Let’s take a look at the main function that builds the knowledge base:</p><pre>async def load_knowledge_base():<br>    async with get_redis() as rdb:<br>        await setup_db(rdb)<br>        chunks = await process_docs()<br>        await add_chunks_to_vector_db(rdb, chunks)</pre><p>The get_redis() function opens an async connection to Redis (we will cover Redis in the next section). We then set up the database, process the source documents (in our use case, the technology reports PDFs) and store the processed chunks with their vector embeddings in the vector database.</p><p>The document processing function is similar to what we did in the previous RAG from scratch post and includes:</p><ul><li><strong>Loading and extracting</strong> the text from the source PDF files.</li><li><strong>Splitting the documents</strong> down into smaller text chunks, with some overlap to preserve cross-chunk context.</li><li><strong>Converting the text chunks into vector embeddings</strong> (using <a href="https://platform.openai.com/docs/guides/embeddings">OpenAI’s embeddings</a>) that encode their meaning.</li><li><strong>Storing</strong> the text chunks and vector embeddings in the <strong>vector database.</strong></li></ul><p>Here’s a slightly simplified version of the document processing code:</p><pre>async def process_docs(docs_dir=settings.DOCS_DIR):<br>    docs = []<br>    pdf_files = [f for f in os.listdir(docs_dir) if f.endswith(&#39;.pdf&#39;)]<br>    for filename in tqdm(pdf_files):<br>        file_path = os.path.join(docs_dir, filename)<br>        text = extract_text(file_path)<br>        doc_name = os.path.splitext(filename)[0]<br>        docs.append((doc_name, text))<br><br>    chunks = []<br>    text_splitter = TextSplitter(chunk_size=512, chunk_overlap=150)<br><br>    for doc_name, doc_text in docs:<br>        doc_id = str(uuid4())[:8]<br>        doc_chunks = text_splitter.split(doc_text)<br>        for chunk_idx, chunk_text in enumerate(doc_chunks):<br>            chunk = {<br>                &#39;chunk_id&#39;: f&#39;{doc_id}:{chunk_idx+1:04}&#39;,<br>                &#39;text&#39;: chunk_text,<br>                &#39;doc_name&#39;: doc_name,<br>                &#39;vector&#39;: None<br>            }<br>            chunks.append(chunk)<br><br>    vectors = []<br>    for batch in batchify(chunks, batch_size=64):<br>        batch_vectors = await get_embeddings([chunk[&#39;text&#39;] for chunk in batch])<br>        vectors.extend(batch_vectors)<br>        <br>    for chunk, vector in zip(chunks, vectors):<br>        chunk[&#39;vector&#39;] = vector<br>    return chunks</pre><p>If you are curious about the TextSplitter functionality, it was covered in the <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264#910d">previous RAG post</a> and you can also check the code in detail <a href="https://github.com/ruizguille/tech-trends-chatbot/blob/master/backend/app/utils/splitter.py">here</a>.</p><p>Creating the vector embeddings with OpenAI is as simple as:</p><pre>from openai import AsyncOpenAI<br><br>client = AsyncOpenAI(api_key=settings.OPENAI_API_KEY)<br><br>async def get_embeddings(input, model=settings.EMBEDDING_MODEL, dimensions=settings.EMBEDDING_DIMENSIONS):<br>    res = await client.embeddings.create(input=input, model=model, dimensions=dimensions)<br>    return [d.embedding for d in res.data]</pre><h3>Redis Database</h3><p><strong>Redis is a high-performance and very versatile in-memory database.</strong> Even though it’s more popular as a cache, it has evolved a lot in the last years and includes powerful extensions for JSON, search and even vector search. It also offers different persistence options to ensure data safety.</p><p>If you are new to Redis, their <a href="https://redis.io/docs/latest/develop/get-started/">quick start guides</a> are great to get up to speed. And make sure to check out the <a href="https://redis.readthedocs.io/en/stable/examples.html">Python library examples</a> too.</p><p>In our application, Redis plays a dual role:</p><ol><li>As a <strong>vector database for </strong><a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264#d82e"><strong>semantic search</strong></a>, allowing us to retrieve the most semantically similar chunks to a specific user question.</li><li>As a <strong>general database for chat storage.</strong></li></ol><p>In order to store vectors and perform vector searches, we need to create an index:</p><pre>async def create_vector_index(rdb):<br>    schema = (<br>        TextField(&#39;$.chunk_id&#39;, no_stem=True, as_name=&#39;chunk_id&#39;),<br>        TextField(&#39;$.text&#39;, as_name=&#39;text&#39;),<br>        TextField(&#39;$.doc_name&#39;, as_name=&#39;doc_name&#39;),<br>        VectorField(<br>            &#39;$.vector&#39;,<br>            &#39;FLAT&#39;,<br>            {<br>                &#39;TYPE&#39;: &#39;FLOAT32&#39;,<br>                &#39;DIM&#39;: settings.EMBEDDING_DIMENSIONS,<br>                &#39;DISTANCE_METRIC&#39;: &#39;COSINE&#39;<br>            },<br>            as_name=&#39;vector&#39;<br>        )<br>    )<br>    await rdb.ft(&#39;idx:vector&#39;).create_index(<br>        fields=schema,<br>        definition=IndexDefinition(prefix=[&#39;vector:&#39;], index_type=IndexType.JSON)<br>    )</pre><p>The index is of type JSON. Along with the vector embedding (the vector property) we are storing other metadata properties like chunk_id, doc_name (useful for citations) and the actual chunk text in text. The &#39;vector:&#39; prefix means that all the vector objects must be stored with keys that start with that prefix to be indexed.</p><p>It’s important to mention that it’s not necessary to index every field when you are storing data in Redis. You should only index those fields that you are going to use in the queries, to make the queries performant. For example, the index we are using to store the chat objects is much simpler, and only includes the created property in case we want to retrieve the latest chats sorted by date:</p><pre>async def create_chat_index(rdb):<br>  schema = (<br>      NumericField(&#39;$.created&#39;, as_name=&#39;created&#39;, sortable=True),<br>  )<br>  await rdb.ft(&#39;idx:chat&#39;).create_index(<br>      fields=schema,<br>      definition=IndexDefinition(prefix=[&#39;chat:&#39;], index_type=IndexType.JSON)<br>  )</pre><p>Let’s now look at some of the key functions that interact with our Redis database. This is the function used in the previous section to add chunks to the vector database:</p><pre>async def add_chunks_to_vector_db(rdb, chunks):<br>    async with rdb.pipeline(transaction=True) as pipe:<br>        for chunk in chunks:<br>            pipe.json().set(&#39;vector:&#39; + chunk[&#39;chunk_id&#39;], Path.root_path(), chunk)<br>        await pipe.execute()</pre><p>This function performs semantic search using the K-nearest neighbors (KNN) algorithm and retrieves the top_k most similar chunks:</p><pre>async def search_vector_db(rdb, query_vector, top_k=settings.VECTOR_SEARCH_TOP_K):<br>    query = (<br>        Query(f&#39;(*)=&gt;[KNN {top_k} @vector $query_vector AS score]&#39;)<br>        .sort_by(&#39;score&#39;)<br>        .return_fields(&#39;score&#39;, &#39;chunk_id&#39;, &#39;text&#39;, &#39;doc_name&#39;)<br>        .dialect(2)<br>    )<br>    res = await rdb.ft(VECTOR_IDX_NAME).search(query, {<br>        &#39;query_vector&#39;: np.array(query_vector, dtype=np.float32).tobytes()<br>    })<br>    return [{<br>        &#39;score&#39;: 1 - float(d.score),<br>        &#39;chunk_id&#39;: d.chunk_id,<br>        &#39;text&#39;: d.text,<br>        &#39;doc_name&#39;: d.doc_name<br>    } for d in res.docs]</pre><p>Notice how we are calculating the <strong>similarity score</strong> as 1 - float(d.score) because the score returned in the query is the <strong>cosine distance</strong> (the smaller the cosine distance, the bigger the similarity to the user query). And we are interested in the top_k most similar chunks (we will use top 10 in our app). Notice also how Redis requires the query vector to be a byte array, that&#39;s why we are using Numpy&#39;s np.array() and tobytes().</p><p>Finally, these are the functions used to create chats, check if a chat exists, add new messages to a chat and retrieve the last_n messages of a specific chat (useful for the conversation memory):</p><pre>async def create_chat(rdb, chat_id, created):<br>    chat = {&#39;id&#39;: chat_id, &#39;created&#39;: created, &#39;messages&#39;: []}<br>    await rdb.json().set(&#39;chat:&#39; + chat_id, Path.root_path(), chat)<br>    return chat<br><br>async def chat_exists(rdb, chat_id):<br>    return await rdb.exists(&#39;chat:&#39; + chat_id)<br><br>async def add_chat_messages(rdb, chat_id, messages):<br>    await rdb.json().arrappend(&#39;chat:&#39; + chat_id, &#39;$.messages&#39;, *messages)<br><br>async def get_chat_messages(rdb, chat_id, last_n=None):<br>    if last_n is None:<br>        messages = await rdb.json().get(&#39;chat:&#39; + chat_id, &#39;$.messages[*]&#39;)<br>    else:<br>        messages = await rdb.json().get(&#39;chat:&#39; + chat_id, f&#39;$.messages[-{last_n}:]&#39;)<br>    return [{&#39;role&#39;: m[&#39;role&#39;], &#39;content&#39;: m[&#39;content&#39;]} for m in messages] if messages else []</pre><h3>API Endpoints</h3><p>We are building the backend using <a href="https://fastapi.tiangolo.com/">FastAPI</a>, a modern and fast framework for building APIs with Python. In our application, there are two main endpoints:</p><ol><li><strong>Create a new chat session.</strong></li><li><strong>Send a message and get the chatbot response</strong> in a specific chat session.</li></ol><p>Let’s see the endpoints in code:</p><pre>from fastapi import APIRouter, Depends, HTTPException<br><br>router = APIRouter()<br><br>@router.post(&#39;/chats&#39;)<br>async def create_new_chat(rdb = Depends(get_rdb)):<br>    chat_id = str(uuid4())[:8]<br>    created = int(time())<br>    await create_chat(rdb, chat_id, created)<br>    return {&#39;id&#39;: chat_id}<br><br>@router.post(&#39;/chats/{chat_id}&#39;)<br>async def chat(chat_id: str, chat_in: ChatIn):<br>    rdb = get_redis()<br>    if not await chat_exists(rdb, chat_id):<br>        raise HTTPException(status_code=404, detail=f&#39;Chat {chat_id} does not exist&#39;)<br>    assistant = RAGAssistant(chat_id=chat_id, rdb=rdb)<br>    sse_stream = assistant.run(message=chat_in.message)<br>    return EventSourceResponse(sse_stream, background=rdb.aclose)</pre><p>The chat endpoint creates an instance of a RAGAssistant class, which handles the core logic of our chatbot, as we will see shortly. It calls the run() method with the user message and uses Server-Sent Events (SSE) to stream the assistant&#39;s responses back to the client in real-time. For that, we are using the EventSourceResponse from the <a href="https://github.com/sysid/sse-starlette">sse-starlette</a> library.</p><p>Note also that we are using <a href="https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/">FastAPI’s dependency injection system</a> in rdb = Depends(get_rdb) to manage Redis connections. Using the yield statement, we can open a Redis connection when the request starts and close it after it ends:</p><pre>async def get_rdb():<br>    rdb = get_redis()<br>    try:<br>        yield rdb<br>    finally:<br>        await rdb.aclose()</pre><p>⚠️ <em>Due to a limitation with FastAPI’s streaming responses after version 0.106, we cannot use a dependency with yield to close the Redis connection in the chat endpoint, so we do it using a background task in the EventSourceResponse.</em></p><p>With the endpoints defined, creating a FastAPI application is straightforward:</p><pre>from fastapi import FastAPI<br><br>app = FastAPI()<br>app.include_router(router)</pre><h3>RAG Assistant Class</h3><p>The RAGAssistant class contains the <strong>core logic of our RAG chatbot.</strong> It manages user conversations, persists chat history, queries the knowledge base for relevant information and uses the <strong>LLM with Retrieval Augmented Generation</strong> to answer user questions.</p><p>The class is initialized as follows:</p><pre>from openai import pydantic_function_tool<br><br>class RAGAssistant:<br>    def __init__(self, chat_id, rdb, history_size=4, max_tool_calls=3):<br>        self.chat_id = chat_id<br>        self.rdb = rdb<br>        self.sse_stream = None<br>        self.main_system_message = {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: MAIN_SYSTEM_PROMPT}<br>        self.rag_system_message = {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: RAG_SYSTEM_PROMPT}<br>        self.tools_schema = [pydantic_function_tool(QueryKnowledgeBaseTool)]<br>        self.history_size = history_size<br>        self.max_tool_calls = max_tool_calls</pre><p>A quick overview of these attributes:</p><ul><li>chat_id: Current chat session id.</li><li>rdb: Redis database connection.</li><li>sse_stream: SSE stream for real-time response streaming.</li><li>main_system_message and rag_system_message: System prompts guiding the assistant&#39;s behavior.</li><li>tools_schema: Available tools for the assistant, in our case only the QueryKnowledgeBaseTool. The pydantic_function_tool generates the tool&#39;s JSON schema.</li><li>history_size: Number of previous messages to include in each conversation interaction.</li><li>max_tool_calls: Maximum number of tool calls per interaction.</li></ul><p>One of the key features of our chatbot is its ability to provide <strong>streaming responses.</strong> This is achieved through a combination of asynchronous programming and server-sent events (SSE). Let’s take a look at the main run method (used in the chat endpoint):</p><pre>def run(self, message):<br>    self.sse_stream = SSEStream()<br>    asyncio.create_task(self._handle_conversation_task(message))<br>    return self.sse_stream</pre><p>When a new user message arrives, the assistant creates an SSEStream object to queue and handle the response stream. It then <a href="https://docs.python.org/3/library/asyncio-task.html#creating-tasks">creates an asynchronous task with asyncio</a> to handle the conversation and immediately returns the sse_stream. This allows us to <strong>start streaming the response to the client as soon as the first chunks become available,</strong> rather than waiting for the whole process to complete.</p><p>Instead of awaiting self._handle_conversation_task(message) in the run method, we launch it as a concurrent task. This approach allows the <strong>conversation handling and response streaming to run simultaneously.</strong></p><p>The _handle_conversation_task method runs the conversation step logic, handles any errors and ensures that the sse_stream is always closed at the end:</p><pre>async def _handle_conversation_task(self, message):<br>    try:<br>        await self._run_conversation_step(message)<br>    except Exception as e:<br>        print(f&#39;Error: {str(e)}&#39;)<br>        # Other error handling<br>    finally:<br>        await self.sse_stream.close()</pre><p>The _run_conversation_step logic is where we apply RAG, query the knowledge base tool and use the LLM to generate the response for the user. It&#39;s probably the most important part of the RAG chatbot and we are going to cover it in detail next.</p><h3>Implementing RAG with Tool Calling and Structured Outputs</h3><p>My last post, <a href="https://medium.com/@codeawake/structured-outputs-3d5e87d22ba5">“OpenAI’s Structured Outputs for RAG and Data Extraction”</a>, covered OpenAI’s new <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">Structured Outputs</a> feature. This feature ensures that model responses match a defined schema. The post explained how to define the schemas using <a href="https://docs.pydantic.dev/latest/">Pydantic</a> and showed helpful parsing functions provided by the <a href="https://github.com/openai/openai-python">OpenAI Python SDK</a>. It also included some examples of how to use structured outputs in RAG applications, with and without streaming.</p><p>Our AI chatbot uses these same techniques, so I recommend checking out that post if you want a deeper understanding of the details.</p><p>The RAG process is integrated into our chatbot’s conversation flow through the _run_conversation_step method in the RAGAssistant class:</p><pre>async def _run_conversation_step(self, message):<br>    user_db_message = {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message, &#39;created&#39;: int(time())}<br>    chat_messages = await get_chat_messages(self.rdb, self.chat_id, last_n=self.history_size)<br>    chat_messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message})<br>    assistant_message = await self._generate_chat_response(<br>        system_message=self.main_system_message,<br>        chat_messages=chat_messages,<br>        tools=self.tools_schema<br>    )<br>    tool_calls = assistant_message.tool_calls<br><br>    if tool_calls:<br>        chat_messages.append(assistant_message)<br>        assistant_message = await self._handle_tool_calls(tool_calls, chat_messages)<br>    <br>    assistant_db_message = {<br>        &#39;role&#39;: &#39;assistant&#39;,<br>        &#39;content&#39;: assistant_message.content,<br>        &#39;tool_calls&#39;: [<br>            {&#39;name&#39;: tc.function.name, &#39;arguments&#39;: tc.function.arguments} for tc in tool_calls<br>        ],<br>        &#39;created&#39;: int(time())<br>    }<br>    await add_chat_messages(self.rdb, self.chat_id, [user_db_message, assistant_db_message])</pre><p>Here is a breakdown of what the method does:</p><ol><li>Create a user message object with the received message and a timestamp for database storage.</li><li>Fetch the last self.history_size chat messages from the database for the current chat_id session and append the new user message.</li><li>Call the LLM (<a href="https://platform.openai.com/docs/models/gpt-4o">OpenAI’s GPT-4o</a>) with the main system message, provided tools and chat history to generate the assistant’s response. <strong>The LLM then decides whether it needs to call a tool to properly answer the user’s question.</strong> In our case we are only providing the QueryKnowledgeBaseTool.</li><li>If the assistant response contains any tool calls, run the tools and generate a new assistant response that uses the tool results. In our chatbot, this means <strong>answering user questions with the relevant information retrieved from the knowledge base.</strong></li><li>Create an assistant message object with the final assistant response and add both the initial user message and this assistant message to the database chat history.</li></ol><p>If you are curious, this is the main system prompt we are using:</p><pre>MAIN_SYSTEM_PROMPT = &quot;&quot;&quot;<br>You are a knowledgeable assistant specialized in answering questions about new technology trends, their applications in various sectors and their broader impacts.<br><br>You have access to the &#39;QueryKnowledgeBaseTool,&#39; which includes technology reports from the world&#39;s leading institutions.<br>Use this tool to query the knowledge base and answer user questions.<br><br>Do not rely on prior knowledge or make answers up.<br>Always use the provided &#39;QueryKnowledgeBaseTool&#39; to ensure your answers are grounded in the most up-to-date and accurate information available.<br><br>If a user&#39;s question seems unrelated, try to find a relevant technology angle.<br>Only if the question is completely completely outside the scope of technology, kindly remind the user of your specialization.<br>&quot;&quot;&quot;</pre><p>The _generate_chat_response method calls the LLM and handles the streaming of the LLM&#39;s response:</p><pre>async def _generate_chat_response(self, system_message, chat_messages, **kwargs):<br>     messages = [system_message, *chat_messages]<br>     async with chat_stream(messages=messages, **kwargs) as stream:<br>        async for event in stream:<br>            if event.type == &#39;content.delta&#39;:<br>                await self.sse_stream.send(event.delta)<br>        <br>        final_completion = await stream.get_final_completion()<br>        assistant_message = final_completion.choices[0].message<br>        return assistant_message</pre><p>This method checks every chunk in the response stream generated by the LLM. If it’s of type &#39;content.delta&#39; (containing new text content), we stream it to the client in real-time using the sse_stream object. At the end, we call the get_final_completion method to get the final accumulated response once it has been fully generated, including the full content and any tool calls if the model decides it needs them.</p><p>The chat_stream function is just a wrapper over the client.beta.chat.completions.stream() method provided by the OpenAI Python SDK. If you want to know more about the details of streaming with structured outputs, check <a href="https://medium.com/@codeawake/structured-outputs-3d5e87d22ba5#d2cf">this section of my last post</a>.</p><p>As mentioned earlier, we provide a single tool to the assistant for querying the knowledge base. We define the tool using Pydantic:</p><pre>from pydantic import BaseModel, Field<br><br>class QueryKnowledgeBaseTool(BaseModel):<br>    &quot;&quot;&quot;Query the knowledge base to answer user questions about new technology trends, their applications and broader impacts.&quot;&quot;&quot;<br>    query_input: str = Field(description=&#39;The natural language query input string. The query input should be clear and standalone.&#39;)<br><br>    async def __call__(self, rdb):<br>        query_vector = await get_embedding(self.query_input)<br>        chunks = await search_vector_db(rdb, query_vector)<br>        formatted_sources = [f&#39;SOURCE: {c[&#39;doc_name&#39;]}\n&quot;&quot;&quot;\n{c[&#39;text&#39;]}\n&quot;&quot;&quot;&#39; for c in chunks]<br>        return f&quot;\n\n---\n\n&quot;.join(formatted_sources) + f&quot;\n\n---&quot;</pre><p>The QueryKnowledgeBaseTool includes a __call__ method to make it callable. Once instantiated with a specific natural language query_input, it can be called to <strong>query the knowledge base and retrieve the most semantically relevant chunks to that query input.</strong> Calling the tool does the following:</p><ol><li>Converts the query input to a <strong>vector embedding</strong> that encodes its meaning.</li><li>Queries the Redis vector database to retrieve the <strong>top 10 most semantically similar chunks.</strong></li><li>Returns the relevant chunks formatted as a <strong>string of sources,</strong> each one including the document name for citations.</li></ol><p>If we want our OpenAI LLM to be able to call a tool, we need to provide a JSON schema describing what the tool does and what parameters are required to call it. The OpenAI-provided pydantic_function_tool that we used in the assistant initialization conveniently converts any Pydantic tool definition to the required JSON schema:</p><pre>self.tools_schema = [pydantic_function_tool(QueryKnowledgeBaseTool)]</pre><p>Finally, let’s see what the _handle_tool_calls method does:</p><pre>async def _handle_tool_calls(self, tool_calls, chat_messages):<br>    for tool_call in tool_calls[:self.max_tool_calls]:<br>        kb_tool = tool_call.function.parsed_arguments<br>        kb_result = await kb_tool(self.rdb)<br>        chat_messages.append(<br>            {&#39;role&#39;: &#39;tool&#39;, &#39;tool_call_id&#39;: tool_call.id, &#39;content&#39;: kb_result}<br>        )<br>    return await self._generate_chat_response(<br>        system_message=self.rag_system_message,<br>        chat_messages=chat_messages,<br>    )</pre><p>A key detail here is that, because we are using the structured outputs feature and passing the Pydantic tool definition as input, <strong>the OpenAI LLM automatically parses any tool calls into instances of our Pydantic tool model.</strong> This is handled by the OpenAI-provided client.beta.chat.completions.stream() method mentioned earlier.</p><p>As a result, when the LLM decides it needs to call the knowledge base tool, the tool_call.function.parsed_arguments property will contain an actual instance of QueryKnowledgeBaseTool, with the query_input parameter filled in. And because we made the tool callable with the __call__ method, we can simply call it to query the knowledge base.</p><p>We then append a new “tool message” to the chat containing the tool result (the top 10 relevant chunks formatted as a string of sources) and call the _generate_chat_response method again to generate the final response. <strong>The LLM can then use this tool result as context to answer the user&#39;s question. This is the core idea of Retrieval Augmented Generation.</strong></p><p>Notice that we are using a different system message this time ( self.rag_system_message) more focused on the RAG process. This is the RAG system prompt we are using:</p><pre>RAG_SYSTEM_PROMPT = &quot;&quot;&quot;<br>You are a knowledgeable assistant specialized in answering questions about new technology trends, their applications in various sectors and their broader impacts.<br>Use the sources provided by the &#39;QueryKnowledgeBaseTool&#39; to answer the user&#39;s question. You must only use the facts from the sources in your answer.<br><br>Make sure to reference and include relevant excerpts from the sources to support your answers.<br>When providing an answer, mention the specific report from which the information was retrieved (e.g., &quot;According to the [Report Name], ...&quot;).<br>Your answers must be accurate and grounded on truth.<br><br>If the information needed to answer a question is not available in the sources, say that you don&#39;t have enough information and share any relevant facts you find.<br>&quot;&quot;&quot;</pre><p>This is a powerful approach for building more complex chatbots and LLM-based applications. Rather than having a single LLM, with a single system prompt and specific tools, you can <strong>orchestrate a collection of agents with different systems prompts and tools, each specialized in a specific task.</strong></p><h3>Streaming Responses with Server-Sent Events</h3><p>Typically, the client needs to send a request to receive data from the server. However, in our AI chatbot, <strong>we want the server to stream content to the client in real-time, sending each text chunk as it becomes available.</strong> This makes the chatbot application feel fast and responsive, which is essential for a great user experience.</p><p>A possible approach is to use <a href="https://codeawake.com/blog/ai-voice-assistant-2#websockets-real-time-communication">WebSockets</a>, which create a bidirectional connection between the client and the server. However, for our use case, there’s a simpler approach that works just as well and can be implemented directly using HTTP requests: <strong>Server-Sent Events (SSE).</strong></p><p><strong>Server-Sent Events allow the sever to asynchronously send data to the client once an HTTP connection is established,</strong> which is exactly what we need. To implement SSE in Python with FastAPI, we are using the <a href="https://github.com/sysid/sse-starlette">sse-starlette</a> library.</p><p>Implementing SSE in FastAPI is as simple as passing an asynchronous generator to the EventSourceResponse that provides the data to be streamed:</p><pre>from sse_starlette.sse import EventSourceResponse<br><br>async def sse_endpoint():<br>    return EventSourceResponse(generator)</pre><p>In our AI chatbot, we are using a custom SSEStream class that simplifies the process of sending data to the stream from different parts of our application:</p><pre>import asyncio<br>from sse_starlette import ServerSentEvent<br><br>class SSEStream:<br>    def __init__(self) -&gt; None:<br>        self._queue = asyncio.Queue()<br>        self._stream_end = object()<br><br>    def __aiter__(self):<br>        return self<br><br>    async def __anext__(self):<br>        data = await self._queue.get()<br>        if data is self._stream_end:<br>            raise StopAsyncIteration<br>        return ServerSentEvent(data=data)<br><br>    async def send(self, data):<br>        await self._queue.put(data)<br><br>    async def close(self):<br>        await self._queue.put(self._stream_end)</pre><p>These are the key insights of the implementation:</p><ul><li>The class uses an <a href="https://docs.python.org/3/library/asyncio-queue.html">asyncio queue</a> to store and manage the stream data.</li><li>The send method adds new data to the queue.</li><li>The close method closes the stream by adding the _stream_end sentinel object.</li><li>The __aiter__ and __anext__ methods allow the stream to be consumed as an async generator by retrieving data from the queue and returning it as server-sent events until the sentinel object is found. This is what the EventSourceResponse uses to stream the data as it becomes available.</li></ul><p>With this context, it should now be clearer how streaming is integrated in our application. Synthesizing what we have seen in previous sections:</p><ol><li>We instantiate the stream in the run method of the RAGAssistant class like this: self.sse_stream = SSEStream().</li><li>We launch the _handle_conversation_task concurrently to handle the conversation logic while we are streaming the response.</li><li>We return the stream to the FastAPI chat endpoint to be able to immediately start streaming to the client using return EventSourceResponse(sse_stream).</li><li>In the _generate_chat_response method of the RAGAssistant class, when we receive new content chunks from the LLM model, we stream them directly to the client using await self.sse_stream.send(event.delta).</li></ol><h3>Conclusion</h3><p>We have now explored in detail the building blocks and code necessary to create a production-ready AI chatbot powered by your own data. In the next post, we’ll build the chatbot’s user interface in React.</p><p>If you are interested in diving deeper, the complete code is available in the <a href="https://github.com/ruizguille/tech-trends-chatbot">GitHub repository</a>. The README provides instructions for installation and running both the full-stack app and a local Python version for testing in your console. You can easily customize it with your own data sources and build RAG chatbots for other use cases.</p><p>It’s also important to highlight that the current RAG implementation still has limitations. More advanced use cases may require additional techniques like: <strong>hybrid search</strong> (combining vector search with keyword-based search), <strong>query rewriting, query decomposition, metadata filtering, reranking</strong> and <strong>recursive retrieval.</strong></p><p>If you have any questions, comments, or would like to implement similar AI-based applications in your business, you can reach out to me at <a href="/service/mailto:guillermo@codeawake.com">guillermo@codeawake.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bd2fa3324e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenAI’s Structured Outputs for RAG and Data Extraction]]></title>
            <link>https://medium.com/@codeawake/structured-outputs-3d5e87d22ba5?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d5e87d22ba5</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Fri, 16 Aug 2024 10:10:22 GMT</pubDate>
            <atom:updated>2024-12-02T17:57:53.756Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring how structured outputs, function calling and streaming can enhance AI applications</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/structured-outputs"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*77WSX2Xe4mkBT0pX2VhGeA.png" /></figure><p>Last week, OpenAI released a very interesting new feature: <a href="https://openai.com/index/introducing-structured-outputs-in-the-api/">Structured Outputs</a>.</p><p>When you are building applications that <strong>integrate large language models with other systems</strong> or use LLMs to <strong>call functions and execute tasks</strong>, having predictable outputs becomes crucial. And this is where OpenAI’s new feature comes into play.</p><p>Previously, the way to solve this was either to use <a href="https://platform.openai.com/docs/guides/structured-outputs/json-mode">JSON mode</a> with the output schema defined in the model prompt, or use <a href="https://platform.openai.com/docs/guides/function-calling">function calling</a> with a specific function definition and schema, assuming your LLM provider supported these capabilities.</p><p>These methods generally worked well but none of them guaranteed that the model’s response would conform to the defined schema, and it could be challenging to make it work for complex schemas. This made LLM-based applications more unpredictable and error-prone, and made complex integrations and automation systems a big challenge.</p><p>The new <strong>Structured Outputs</strong> feature changes this completely. Using a technique called <strong>constrained decoding,</strong> where the model’s output is constrained to only the tokens that are valid according to the defined schema, OpenAI’s team has managed to make structured outputs <strong>score a perfect 100% when following complex JSON schemas.</strong></p><p>In this blog post we are going to learn more about this topic and explore practical examples of how to use structured outputs for <strong>Retrieval-Augmented Generation (RAG)</strong> and <strong>data extraction.</strong> We will also explore the <a href="https://docs.pydantic.dev/latest/">Pydantic library</a>, the <a href="https://github.com/openai/openai-python">OpenAI Python SDK</a> and its helpful parsing functions for handling structured outputs. And finally, we will look at an essential feature of LLM and RAG apps in production: <strong>streaming</strong>.</p><h3>Structured Outputs with Function Calling for RAG</h3><p>One of the ways to use OpenAI’s structured outputs is via function calling, which allows us to connect models to external tools and systems. Function calling was already available before, but now by setting strict: true in the function definition we can <a href="https://platform.openai.com/docs/guides/function-calling/function-calling-with-structured-outputs">enable structured outputs</a> and ensure that the model response adheres exactly to the function schema. Let&#39;s now explore how we can apply this to a RAG application.</p><p>In a previous post, we introduced <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264">Retrieval-Augmented Generation (RAG)</a> as a very powerful technique to <strong>combine the power of LLMs with external knowledge retrieval.</strong> The core idea of RAG is to take a user query, retrieve relevant information to that query from a <strong>knowledge base</strong> (using semantic search with vector embeddings or more traditional keyword-based search methods), and then <strong>use the retrieved information as augmented context for the LLM to generate an answer to the user query.</strong></p><p>In the basic RAG system we built, we performed information retrieval for each user question in isolation without considering the conversation history. But we can build a more powerful and intelligent system using function calling. By providing the LLM with a <strong>knowledge base query tool,</strong> we can let the LLM decide when and how to use the tool.</p><p>This is more powerful for two different reasons:</p><ul><li>The LLM can <strong>selectively decide when to use the retrieval tool</strong> depending on the user message and the prompt instructions (for example, it would not make sense to retrieve information for a greetings or unrelated messages).</li><li>The LLM can <strong>intelligently rephrase the user question</strong> when necessary using the conversation history. This is critical for follow-up questions.</li></ul><p>Let’s now see a practical example of how we could use the new function calling with structured outputs feature in a <strong>Retrieval-Augmented Generation (RAG) app.</strong></p><p>To define schemas for structured outputs, we are going to use <a href="https://docs.pydantic.dev/latest/">Pydantic</a>, a great Python library for data parsing and validation. We can easily define our knowledge base query tool with the following Pydantic model:</p><pre>from pydantic import BaseModel, Field<br><br>class QueryKnowledgeBaseTool(BaseModel):<br>    &quot;&quot;&quot;Query the knowledge base to answer the user questions.&quot;&quot;&quot;<br>    query_input: str = Field(description=&#39;The natural language query input. The query input should be clear and standalone.&#39;)<br><br>    def __call__(self):<br>        chunks = search_knowledge_base(self.query_input)<br>        return &#39;\n\n---\n\n&#39;.join(chunks) + &#39;\n\n---&#39;</pre><p>In this case, the tool definition is very simple and takes a single parameter: the query input string. The model also includes a __call__ method to make the tool callable and perform the search using the query input. This is not strictly necessary, but it is very convenient to have it all encapsulated in the same class.</p><p>OpenAI’s Python SDK and its parsing helpers accept Pydantic models as input, so generating the required JSON schema to include the tool in an OpenAI’s API call is as simple as:</p><pre>from openai import pydantic_function_tool<br><br>pydantic_function_tool(QueryKnowledgeBaseTool)</pre><p>And it returns the following schema:</p><pre>{<br>  &quot;type&quot;: &quot;function&quot;,<br>  &quot;function&quot;: {<br>    &quot;name&quot;: &quot;QueryKnowledgeBaseTool&quot;,<br>    &quot;strict&quot;: true,<br>    &quot;parameters&quot;: {<br>      &quot;description&quot;: &quot;Query the knowledge base to answer the user questions.&quot;,<br>      &quot;properties&quot;: {<br>        &quot;query_input&quot;: {<br>          &quot;description&quot;: &quot;The natural language query input. The query input should be clear and standalone.&quot;,<br>          &quot;title&quot;: &quot;Query Input&quot;,<br>          &quot;type&quot;: &quot;string&quot;<br>        }<br>      },<br>      &quot;required&quot;: [<br>        &quot;query_input&quot;<br>      ],<br>      &quot;title&quot;: &quot;QueryKnowledgeBaseTool&quot;,<br>      &quot;type&quot;: &quot;object&quot;,<br>      &quot;additionalProperties&quot;: false<br>    },<br>    &quot;description&quot;: &quot;Query the knowledge base to answer the user questions.&quot;<br>  }<br>}</pre><p>Notice how it’s using the very concise Pydantic model to generate the JSON schema with:</p><ul><li>The class name QueryKnowledgeBaseTool as the function name.</li><li>The docstring as the function description.</li><li>The query_input field as the function parameter, including its description.</li><li>The new strict: true option to enable the structured outputs feature.</li></ul><p>The function and parameters descriptions are very important as they provide additional information to the LLM about what the function does and how to call it.</p><p>We can now use OpenAI’s <a href="https://platform.openai.com/docs/models/gpt-4o-mini">GPT-4o-mini</a> model with this tool to answer user questions about a company using a knowledge base:</p><pre>from openai import OpenAI<br><br>client = OpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br><br>SYSTEM_PROMPT = &quot;&quot;&quot;<br>You are an AI assistant for TechPro Solutions, a company specializing in IT services and solutions.<br>Your role is to answer user questions about the company&#39;s services and offerings accurately and professionally.<br>To ensure you provide the most up-to-date and accurate information, always use the QueryKnowledgeBaseTool to retrieve relevant information before answering user queries. You are a reliable assistant and your answers must always be based on truth.<br>&quot;&quot;&quot;<br><br>messages = [<br>    {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: SYSTEM_PROMPT},<br>    {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: &#39;What services do you provide?&#39;}<br>]<br><br>response = client.beta.chat.completions.parse(<br>    model=&#39;gpt-4o-mini&#39;,<br>    messages=messages,<br>    tools=[pydantic_function_tool(QueryKnowledgeBaseTool)],<br>)<br>content = response.choices[0].message.content<br>tool_call = response.choices[0].message.tool_calls[0]<br><br>print(content)<br># None<br><br>print(tool_call.function.name)<br># &#39;QueryKnowledgeBaseTool&#39;<br><br>print(tool_call.function.parsed_arguments)<br># query_input=&#39;What services does TechPro Solutions provide?&#39;</pre><p>As you can see above, the LLM correctly identifies that it needs to call the QueryKnowledgeBaseTool to retrieve the relevant information to answer the user query. So instead of returning a text response in the message content property as usual, it returns a tool_calls property with the function to call and the arguments to pass. Note also how it rephrases the user query to include the company name and make it clearer.</p><p>Additionally, the client.beta.chat.completions.parse() method provided by the SDK (a wrapper over the usual client.chat.completions.create() method) automatically parses the response tool call into an instance of our Pydantic model. This is possible because we are using the structured outputs feature and passing the Pydantic model as input.</p><p>As a result, the tool_call.function.parsed_arguments property contains an actual instance of QueryKnowledgeBaseTool, with the query_input parameter filled in. And thanks to the __call__ method defined earlier, we can simply call the tool to query the knowledge base with the provided query input:</p><pre># kb_tool is an instance of QueryKnowledgeBaseTool<br>kb_tool = tool_call.function.parsed_arguments<br><br>print(kb_tool.query_input)<br># &#39;What services does TechPro Solutions provide?&#39;<br><br># The __call__ method allows us to call the tool to perform the query<br>kb_result = kb_tool()</pre><p>And we can run the final RAG step by adding a message with the tool response and calling the model so that it can answer the user query with the retrieved information:</p><pre>messages.append(<br>    {&#39;role&#39;: &#39;tool&#39;, &#39;tool_call_id&#39;: tool_calls[0].id, &#39;content&#39;: kb_result}<br>)<br>response = client.beta.chat.completions.parse(<br>    model=&#39;gpt-4o-mini&#39;,<br>    messages=messages<br>)<br>content = response.choices[0].message.content<br>print(content)</pre><p>In this final step we do not need to provide tools to the model, as we just want it to give an answer to the user query using the retrieved information in the tool message. And it will output the answer in the message content property.</p><h3>Structured Outputs with JSON responses for Data Extraction</h3><p>The second way to use structured outputs is by providing a JSON schema to the new response_format API parameter. This is useful when you want to return the LLM response in a structured JSON format, which was previously done using JSON mode.</p><p>Let’s take as an example the <a href="https://medium.com/@codeawake/invoice-processing-962385333027">invoice processing automation system</a> we built in a previous post, where we used GPT-4o’s vision capabilities to extract relevant data from invoice images following a specific schema. In that system, we defined the schema in the model prompt and used JSON mode to extract the data according to the schema. We then <a href="https://medium.com/@codeawake/invoice-processing-962385333027#5cdd">validated the data with a Pydantic model</a>.</p><p>With structured outputs, we can now pass the same Pydantic model directly in the response_format parameter to define the output schema and automatically parse the response, just like we did in the function calling example. And unlike JSON mode, we are guaranteed that the model response will conform strictly to the schema.</p><p>We can do all of that in a single call with the same client.beta.chat.completions.parse() method we used before:</p><pre>messages = [<br>    {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: INVOICE_EXTRACTION_SYSTEM_PROMPT},<br>    {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: [<br>      {&#39;type&#39;: &#39;image_url&#39;, &#39;image_url&#39;: {&#39;url&#39;: f&#39;data:image/png;base64,{invoice_img}&#39;}}<br>  ]}<br>]<br><br>response = client.beta.chat.completions.parse(<br>    model=&#39;gpt-4o-mini&#39;,<br>    messages=messages,<br>    response_format=Invoice,<br>)<br><br>message = completion.choices[0].message<br>if message.parsed:<br>    print(message.parsed)</pre><p>The message.parsed property will therefore return an Invoice instance with its fields populated with the extracted data.</p><h3>Streaming Structured Outputs</h3><p>In production LLM and RAG applications, streaming is essential to provide the best user experience. By streaming the model’s output, we can start displaying results to users immediately, rather than waiting for the response to be fully generated. This significantly improves the perceived speed of the applications and provides a more responsive and interactive experience.</p><p>Let’s return to our previous RAG example and add streaming using the client.beta.chat.completions.stream() method provided by the OpenAI Python SDK:</p><pre>client = AsyncOpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br><br>SYSTEM_PROMPT = &quot;&quot;&quot;<br>You are an AI assistant for TechPro Solutions, a company specializing in IT services and solutions.<br>Your role is to answer user questions about the company&#39;s services and offerings accurately and professionally.<br>To ensure you provide the most up-to-date and accurate information, always use the QueryKnowledgeBaseTool to retrieve relevant information before answering user queries. You are a reliable assistant and your answers must always be based on truth.<br>&quot;&quot;&quot;<br><br>messages = [<br>    {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: SYSTEM_PROMPT},<br>    {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: &#39;What services do you provide?&#39;}<br>]<br><br>content = None<br>tool_calls = []<br><br>async with client.beta.chat.completions.stream(<br>    model=&#39;gpt-4o-mini&#39;,<br>    messages=messages,<br>    tools=[pydantic_function_tool(QueryKnowledgeBaseTool)],<br>) as stream:<br>    async for event in stream:<br>        if event.type == &#39;content.delta&#39;:<br>            print(event.delta, flush=True, end=&#39;&#39;)<br>        elif event.type == &#39;content.done&#39;:<br>            content = event.content<br>        elif event.type == &#39;tool_calls.function.arguments.done&#39;:<br>            tool_calls.append({&#39;name&#39;: event.name, &#39;parsed_arguments&#39;: event.parsed_arguments})<br><br>print(content)<br># None<br><br>print(tool_calls[0][&#39;name&#39;])<br># QueryKnowledgeBaseTool<br><br>print(tool_calls[0][&#39;parsed_arguments&#39;])<br># query_input=&#39;list of services provided by TechPro Solutions&#39;</pre><p>This method simplifies the parsing and handling of streaming responses, providing different events to track the generation progress:</p><ul><li>content.delta: Allows you to view and print new content as it&#39;s generated, providing immediate feedback to the user.</li><li>content.done: Signals when content generation is complete, giving you access to the full content.</li><li>tool_calls.function.arguments.done: Provides parsed tool calls as soon as their arguments are complete. And as before, the parsed_arguments property contains an instance of our QueryKnowledgeBaseTool Pydantic model.</li></ul><p>In this specific example, the LLM decides to call the knowledge base query tool instead of generating a text answer. As a result, you won’t see streaming content deltas. However, if you call the model again appending a message with the tool result (as we did in the non-streaming version), you will see the final answer streamed chunk by chunk.</p><p>There are other useful events like chunk (which provides every single response chunk as it arrives) and tool_calls.function.arguments.delta (which provides parts of a tool call&#39;s arguments as they are generated). You can learn more about the parsing methods and the streaming events in the <a href="https://github.com/openai/openai-python/blob/main/helpers.md">OpenAI Python SDK repository</a>.</p><p>For convenience, the SDK also provides a get_final_completion method that returns the final accumulated response once it has been fully generated. This allows us to easily obtain the same final results as in the non-streaming version:</p><pre>response = await stream.get_final_completion()<br><br>content = response.choices[0].message.content<br>tool_call = response.choices[0].message.tool_calls[0]<br><br>print(content)<br># None<br><br>print(tool_call.function.name)<br># QueryKnowledgeBaseTool<br><br>print(tool_call.function.parsed_arguments)<br># query_input=&#39;What services does TechPro Solutions provide?&#39;</pre><p>As we have seen in this post, combining the new structured outputs feature with function calling, JSON responses and streaming can significantly improve RAG applications and LLM-based systems in general.</p><p>This overview also serves as an introduction to some key concepts and tools we’ll use in a future blog post, where we’ll build a production-ready RAG chatbot application. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d5e87d22ba5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Invoice Processing with GPT-4o]]></title>
            <link>https://medium.com/@codeawake/invoice-processing-962385333027?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/962385333027</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[invoice-processing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Thu, 01 Aug 2024 11:42:31 GMT</pubDate>
            <atom:updated>2024-12-02T17:41:47.322Z</atom:updated>
            <content:encoded><![CDATA[<h4>Extract invoice data with AI vision, analyze financials and generate an Excel report in Python</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/invoice-processing"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*EVDh2w3OOtq7NDTnvzyLeA.png" /></figure><p>Running a business very often means dealing with a mountain of administrative tasks, paperwork, invoices and countless chores that eat up an insane amount of hours that you could be otherwise dedicating to improving and growing the business. But what is still perhaps not widely known is the <strong>incredible potential of the new AI models combined with some simple programming to automate away a lot of these tasks.</strong></p><p>In this post, I will demonstrate this idea by building a simple demo app in Python that uses the vision capabilities of OpenAI’s most advanced model, <a href="https://platform.openai.com/docs/models/gpt-4o">GPT-4o</a>, to <strong>automate the extraction of data from multiple invoices simultaneously,</strong> use that data to <strong>perform basic financial summaries,</strong> and then <strong>create an Excel report</strong> with all the data collected and the generated summaries.</p><p>While we will look at the code and the technical details, don’t worry if you are not a programmer. The concepts behind this app are relevant to anyone interested in automating and optimizing their business operations. And these techniques are not just useful for invoice processing but can be generalized to all kinds of documents and paperwork. By the end of this post, you’ll have new insights on <strong>how AI can transform routine administrative tasks, potentially saving your business time and reducing errors.</strong></p><h3>Overview of the Invoice Processing App</h3><p>Before we look at the code, let’s take a step back and see what the app is actually doing. This will give you a clearer picture of how AI can <strong>transform a manual process into an automated workflow.</strong></p><p>The app goes through three main stages:</p><ol><li>Data extraction.</li><li>Financial analysis.</li><li>Report generation.</li></ol><p>The <strong>DATA EXTRACTION</strong> stage is where the magic happens. OpenAI’s GPT-4o is a very advanced <strong>multimodal</strong> model, which means that it can <strong>process text, audio and images in real-time.</strong> This is incredibly powerful and the possibilities are endless. If you haven’t already, check out some of the examples in <a href="https://openai.com/index/hello-gpt-4o/">OpenAI’s website</a>. This means we can feed it images and ask the model to describe them, analyze them or extract information from them (such as all the relevant details in an invoice).</p><p>And not only that, you can also force the model to <strong>return the information in a structured format like JSON</strong> using the <a href="https://platform.openai.com/docs/guides/json-mode">JSON mode</a>. A simple text response is great when building a text-based assistant, but if you want to build a more complex application that integrates the model with other systems and your own code, it’s essential to have the output of the model in a structured format that machines understand and that is much easier to handle with code.</p><p>Let’s look at an example of an invoice and the extracted output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNs6PwgfgCnUQNIK" /></figure><pre>{<br>  &quot;invoice_number&quot;: &quot;DT-45678&quot;,<br>  &quot;invoice_date&quot;: &quot;2024-01-07&quot;,<br>  &quot;invoice_type&quot;: &quot;incoming&quot;,<br>  &quot;issuer&quot;: {<br>    &quot;name&quot;: &quot;DevTools Pro&quot;,<br>    &quot;address&quot;: &quot;789 Coder&#39;s Court, Austin, TX 78701&quot;,<br>    &quot;phone&quot;: &quot;+1 (512) 555-3456&quot;,<br>    &quot;email&quot;: &quot;accounts@devtoolspro.com&quot;<br>  },<br>  &quot;recipient&quot;: {<br>    &quot;name&quot;: &quot;TechNova Solutions, Inc.&quot;,<br>    &quot;address&quot;: &quot;1250 Charleston Road, Mountain View, CA 94043, United States&quot;,<br>    &quot;phone&quot;: &quot;+1 (650) 555-0123&quot;,<br>    &quot;email&quot;: &quot;info@technovasolutions.com&quot;<br>  },<br>  &quot;invoice_items&quot;: [<br>    {<br>      &quot;description&quot;: &quot;Annual Team License Renewal (25 users)&quot;,<br>      &quot;total&quot;: 8500.00<br>    },<br>    {<br>      &quot;description&quot;: &quot;Premium Support Package&quot;,<br>      &quot;total&quot;: 2000.00<br>    }<br>  ],<br>  &quot;subtotal&quot;: 10500.00,<br>  &quot;tax_rate&quot;: 8.25,<br>  &quot;tax&quot;: 866.25,<br>  &quot;total&quot;: 11366.25,<br>  &quot;terms&quot;: &quot;Payment is due within 30 days&quot;<br>}</pre><p>As you can see, it does a perfect extraction job. It’s even able to extract the invoice number that appears as vertical text on the left side. And how do we instruct the model to perform the data extraction and output the data with this particular structure? With the following prompt:</p><pre>You are an AI assistant specialized in extracting information from invoice images for the company &quot;TechNova Solutions, Inc.&quot;.<br>Your task is to analyze the given invoice image and extract the relevant information according to the specified JSON structure below.<br>If a field is not present in the invoice or cannot be determined leave it as null.<br>If the invoice date format is ambiguous, assume it&#39;s the international date format dd-mm-yyyy.<br><br>Provide the extracted information in this JSON structure:<br>```json<br>{<br>  &quot;invoice_number&quot;: &quot;string&quot;,<br>  &quot;invoice_date&quot;: &quot;YYYY-MM-DD&quot;,<br>  &quot;invoice_type&quot;: &quot;incoming | outgoing&quot;,<br>  &quot;issuer&quot;: {<br>    &quot;name&quot;: &quot;string&quot;,<br>    &quot;address&quot;: &quot;string&quot;,<br>    &quot;phone&quot;: &quot;string&quot;,<br>    &quot;email&quot;: &quot;string&quot;<br>  },<br>  &quot;recipient&quot;: {<br>    &quot;name&quot;: &quot;string&quot;,<br>    &quot;address&quot;: &quot;string&quot;,<br>    &quot;phone&quot;: &quot;string&quot;,<br>    &quot;email&quot;: &quot;string&quot;<br>  },<br>  &quot;invoice_items&quot;: [<br>    {<br>      &quot;description&quot;: &quot;string&quot;,<br>      &quot;total&quot;: &quot;number&quot;<br>    }<br>  ],<br>  &quot;subtotal&quot;: &quot;number&quot;,<br>  &quot;tax_rate&quot;: &quot;number (percentage)&quot;,<br>  &quot;tax&quot;: &quot;number&quot;,<br>  &quot;total&quot;: &quot;number&quot;,<br>  &quot;terms&quot;: &quot;string&quot;<br>}<br>```<br><br>For the &quot;invoice_type&quot; field:<br>- Set it to &quot;incoming&quot; if the company &quot;TechNova Solutions, Inc.&quot; is the recipient of the invoice.<br>- Set it to &quot;outgoing&quot; if the company &quot;TechNova Solutions, Inc.&quot; is the issuer of the invoice.<br>- Set it to null if the company &quot;TechNova Solutions, Inc.&quot; does not appear as issuer or recipient.</pre><p>You can try it yourself in <a href="https://chatgpt.com/">ChatGPT</a> with your own invoices. Notice also how we are specifying the name of the company so that the model is able to categorize the invoice as <em>“incoming”</em> (invoiced to us) or <em>“outgoing”</em> (invoiced to a client). This is very useful for the financial analysis we are doing next.</p><p>In the <strong>ANALYSIS STAGE</strong>, the application uses the extracted data to calculate totals like invoiced revenues (from the outgoing invoices), invoiced expenses (from the incoming invoices), net income, taxes and other metrics. The app also performs a month-by-month breakdown, calculating the same metrics but on a monthly basis. This step <strong>turns a pile of individual invoices into a basic financial picture of your business.</strong></p><p>Finally, in the <strong>REPORT GENERATION</strong> stage, the application takes all this processed data and creates a properly-formatted Excel report with two sheets. The ‘Invoices’ sheet shows the extracted invoice data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HlTh_4AYsSMZt02U" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1pP1aOPFl-2cvUht" /><figcaption>Excel report: Invoices sheet</figcaption></figure><p>And the ‘Summary’ sheet displays the financial summaries and an automatically generated chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SQAGrbGMilqcxlt3" /><figcaption>Excel report: Summary sheet</figcaption></figure><p>The whole process, from feeding in the invoice PDFs to getting the final Excel report, takes about 10–15 seconds. Imagine doing all of this manually!</p><p>And what about other kinds of documents and paperwork? The beauty of this approach is its flexibility. With some tweaks, you could <strong>adapt this system to process receipts, purchase orders, shipping manifests, employee timesheets, customer feedback forms, and so on.</strong> The AI can be instructed to look for different types of data depending on the document type.</p><p>In the next sections, we will take a look at the Python code used to implement each of these 3 steps. You can find all the code for the app in this <a href="https://github.com/ruizguille/invoice-processing">GitHub repo</a>.</p><h3>Data Extraction with a Vision Model</h3><p>This is the main function that performs the data extraction:</p><pre>from tqdm.asyncio import tqdm_asyncio<br><br>async def extract_invoices_data(invoices_dir):<br>    invoices_filenames = []<br>    tasks = []<br>    print(&#39;Extracting the invoices data&#39;)<br>    for filename in sorted(os.listdir(invoices_dir)):<br>        if filename.endswith(&#39;.pdf&#39;):<br>            print(filename)<br>            invoices_filenames.append(filename)<br>            pdf_path = os.path.join(invoices_dir, filename)<br>            base64_images = pdf_to_base64_images(pdf_path)<br>            task = extract_invoice_data(base64_images[0])<br>            tasks.append(task)<br>    invoices_json = await tqdm_asyncio.gather(*tasks)<br>    return invoices_filenames, invoices_json</pre><p>It loads all the PDF files in the given invoices directory, converts the PDFs to <a href="https://en.wikipedia.org/wiki/Base64">Base64</a> encoded images (the format that GPT-4o accepts) and then extracts the invoice data in JSON format using the model.</p><p>We are using Python’s <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a> library to run the code asynchronously. This allows us to extract data from all invoices simultaneously using <a href="https://docs.python.org/3/library/asyncio-task.html#asyncio.gather">asyncio.gather</a>, rather than processing them sequentially one after the other. By calling the AI model for all invoices in parallel, we significantly reduce the overall processing time. This makes a huge difference when processing a large number of invoices.</p><p>Note that we are using <a href="https://tqdm.github.io/docs/asyncio/">tqdm’s asyncio wrapper</a>, instead of asyncio.gather directly, in order to display a progress bar during the data extraction.</p><p>This is the function that converts the PDF into Base64 encoded images, using the <a href="https://github.com/Belval/pdf2image">pdf2image</a> library:</p><pre>from pdf2image import convert_from_path<br><br>def pdf_to_base64_images(pdf_path):<br>    imgs = convert_from_path(pdf_path, fmt=&#39;png&#39;)<br>    base64_imgs = []<br>    for image in imgs:<br>        buffered = BytesIO()<br>        image.save(buffered, format=&#39;PNG&#39;)<br>        img_str = base64.b64encode(buffered.getvalue()).decode()<br>        base64_imgs.append(img_str)<br>    <br>    return base64_imgs</pre><p>And this is the function that calls the GPT-4o model asynchronously to extract the data for each invoice:</p><pre>from openai import AsyncOpenAI<br><br>openai_client = AsyncOpenAI(api_key=settings.OPENAI_API_KEY)<br><br>async def extract_invoice_data(base64_img):<br>    response = await openai_client.chat.completions.create(<br>        model=&#39;gpt-4o&#39;,<br>        response_format={&#39;type&#39;: &#39;json_object&#39;},<br>        messages=[<br>            {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: SYSTEM_PROMPT.format(company=settings.COMPANY_NAME)},<br>            {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: [<br>                {&#39;type&#39;: &#39;image_url&#39;, &#39;image_url&#39;: {&#39;url&#39;: f&#39;data:image/png;base64,{base64_img}&#39;}}<br>            ]}<br>        ],<br>        temperature=0.1,<br>    )<br>    return response.choices[0].message.content</pre><p>When calling the model, we are using the same prompt shown earlier. Notice how we are using Python’s <a href="https://docs.python.org/3/library/stdtypes.html#str.format">format</a> function to add the company name (defined in the application settings) to the prompt. Notice also the use of the response_format={&#39;type&#39;: &#39;json_object&#39;} option to enable JSON mode. You can learn more about these parameters in <a href="https://platform.openai.com/docs/api-reference/chat/create">OpenAI&#39;s documentation.</a></p><h3>Invoice Data Validation and DataFrame Creation</h3><p>A crucial step when generating structured output with an AI model is ensuring that the data is correct and in the right format. In the Python ecosystem, there is an amazing library called <a href="https://docs.pydantic.dev/latest/">Pydantic</a> just for that purpose. Pydantic is widely used for data parsing and validation in backend applications, APIs development (it’s a key pillar of <a href="https://fastapi.tiangolo.com/">FastAPI</a>) and data science. It has very powerful functionalities and it’s really fast.</p><p>Pydantic uses <a href="https://docs.python.org/3/library/typing.html">Python’s type hints</a> to define data schemas. Let’s take a look at the schema we are using to validate the invoices data according to the structure specified in the prompt:</p><pre>from pydantic import BaseModel, ConfigDict<br><br>class InvoiceType(str, Enum):<br>    incoming = &#39;incoming&#39;<br>    outgoing = &#39;outgoing&#39;<br><br>class InvoiceItem(BaseModel):<br>    description: str<br>    total: float<br><br>class Company(BaseModel):<br>    name: str<br>    address: str<br>    phone: str | None = None<br>    email: str | None = None<br><br>class Invoice(BaseModel):<br>    model_config = ConfigDict(use_enum_values=True)<br>    <br>    invoice_number: str<br>    invoice_date: date<br>    invoice_type: InvoiceType | None = None<br>    issuer: Company<br>    recipient: Company<br>    invoice_items: list[InvoiceItem]<br>    subtotal: float<br>    tax_rate: float = 0<br>    tax: float = 0<br>    total: float<br>    terms: str | None = None</pre><p>If you are new to this syntax, I recommend checking out <a href="https://docs.pydantic.dev/latest/concepts/models/">Pydantic’s documentation</a> for a deeper dive. It’s also worth being familiar with Python’s type hints and FastAPI’s documentation includes a <a href="https://fastapi.tiangolo.com/python-types/">great introduction</a> to them.</p><p>Essentially, the code above is a simple way to define in Python the same structure we instructed GPT-4o to follow when extracting the invoice data. Many of the fields are defined as <a href="https://docs.pydantic.dev/latest/concepts/models/#required-fields">required</a> (for instance, invoice_number: str), while there are others that are optional and include a default value (like phone: str | None = None).</p><p>Once you have a Pydantic model defined, like Invoice above, you can perform data validation with a single line of code:</p><pre>validated_invoice = Invoice(**invoice_data)</pre><p>After validating the invoice data, the next step is to transform it into a format that makes it easier to analyze, manipulate and eventually display the results in an Excel report. This is where <a href="https://pandas.pydata.org/">Pandas</a> comes in. Pandas is a powerful data manipulation library for Python, widely used in data science and analytics.</p><p>Converting the invoice data into a <a href="https://pandas.pydata.org/docs/user_guide/dsintro.html#dataframe">Pandas DataFrame</a> has two advantages:</p><ol><li>It gives access to a wide range of functions for data aggregation, filtering and computation, which we will use for our financial analysis and summaries.</li><li>It represents data in a table-like structure, which is perfect to generate our final Excel report.</li></ol><p>Now that we have introduced the main elements of this section, let’s take a look at the actual function that parses and validates the invoice data, and then transforms it into a DataFrame:</p><pre>import json<br>import pandas as pd<br><br>def build_invoices_dataframe(invoices_filenames, invoices_json):<br>    flat_invoices = []<br>    for filename, invoice_json in zip(invoices_filenames, invoices_json):<br>        try:<br>            invoice_data = json.loads(invoice_json)<br>            validated_invoice = Invoice(**invoice_data)<br>            flat_invoice = flatten_invoice_structure(validated_invoice)<br>            flat_invoices.append(flat_invoice)<br>        except json.JSONDecodeError as e:<br>            print(f&#39;JSON parsing error in invoice {filename}: {e}&#39;)<br>        except ValidationError as e:<br>            print(f&#39;Validation error in invoice {filename}: {e}&#39;)<br>        except Exception as e:<br>            print(f&#39;Unexpected error in invoice {filename}: {e}&#39;)<br>    <br>    invoices_df = pd.DataFrame(flat_invoices)<br>    invoices_df[&#39;Invoice Date&#39;] = pd.to_datetime(invoices_df[&#39;Invoice Date&#39;])<br>    invoices_df.insert(2, &#39;Year-Month&#39;, invoices_df[&#39;Invoice Date&#39;].dt.to_period(&#39;M&#39;))<br>    return invoices_df</pre><p>A brief summary of what the function is doing:</p><ol><li><strong>JSON parsing</strong>: Loads the JSON string output from GPT-4o into a Python object.</li><li><strong>Data validation</strong>: Validates the invoice data according to our Pydantic Invoice model.</li><li><strong>Error handling</strong>: Catches any parsing or validation errors, ensuring only valid invoices are processed.</li><li><strong>Data flattening</strong>: Transforms the nested invoice structure into a flat, tabular format using flatten_invoice_structure (you can find this helper function <a href="https://github.com/ruizguille/invoice-processing/blob/master/app/processing.py#L36-L60">in the repository</a>).</li><li><strong>DataFrame creation</strong>: Creates a Pandas DataFrame from the flattened invoice data.</li><li><strong>Date processing</strong>: Converts the ‘Invoice Date’ field to a datetime format and adds a ‘Year-Month’ field for the monthly analysis.</li></ol><h3>Simple Financial Analysis with Pandas</h3><p>We can synthesize the whole workflow as: <strong>extract → validate → flatten → analyze → report.</strong> We have already completed the first three steps. Let’s now focus on the function that analyzes the data and generates the financial summaries of the invoices:</p><pre>def generate_financial_summary(invoices_df):<br>    total_s = pd.Series(dtype=&#39;object&#39;)<br><br>    # Total<br>    total_s[&#39;Period&#39;] = f&quot;{invoices_df[&#39;Invoice Date&#39;].min().strftime(&#39;%Y-%m&#39;)} to {invoices_df[&#39;Invoice Date&#39;].max().strftime(&#39;%Y-%m&#39;)}&quot;<br>    total_s[&#39;Invoices&#39;] = len(invoices_df)<br><br>    revenue = invoices_df[invoices_df[&#39;Invoice Type&#39;] == &#39;outgoing&#39;][&#39;Total&#39;].sum()<br>    expenses = invoices_df[invoices_df[&#39;Invoice Type&#39;] == &#39;incoming&#39;][&#39;Total&#39;].sum()<br>    total_s[&#39;Revenue&#39;] = revenue<br>    total_s[&#39;Expenses&#39;] = expenses<br>    total_s[&#39;Net Income&#39;] = revenue - expenses<br><br>    total_tax_collected = invoices_df[invoices_df[&#39;Invoice Type&#39;] == &#39;outgoing&#39;][&#39;Tax&#39;].sum()<br>    total_tax_paid = invoices_df[invoices_df[&#39;Invoice Type&#39;] == &#39;incoming&#39;][&#39;Tax&#39;].sum()<br>    total_s[&#39;Tax Collected&#39;] = total_tax_collected<br>    total_s[&#39;Tax Paid&#39;] = total_tax_paid<br>    total_s[&#39;Net Tax&#39;] = total_tax_collected - total_tax_paid<br><br>    # Monthly<br>    monthly_df = invoices_df.groupby(&#39;Year-Month&#39;).agg(**{<br>        &#39;Invoices&#39;: (&#39;Invoice Number&#39;, &#39;count&#39;),<br>        &#39;Revenue&#39;: (&#39;Total&#39;, lambda x: x[invoices_df[&#39;Invoice Type&#39;] == &#39;outgoing&#39;].sum()),<br>        &#39;Expenses&#39;: (&#39;Total&#39;, lambda x: x[invoices_df[&#39;Invoice Type&#39;] == &#39;incoming&#39;].sum()),<br>        &#39;Tax Collected&#39;: (&#39;Tax&#39;, lambda x: x[invoices_df[&#39;Invoice Type&#39;] == &#39;outgoing&#39;].sum()),<br>        &#39;Tax Paid&#39;: (&#39;Tax&#39;, lambda x: x[invoices_df[&#39;Invoice Type&#39;] == &#39;incoming&#39;].sum())<br>    })<br>    monthly_df = monthly_df.reset_index()<br>    monthly_df[&#39;Year-Month&#39;] = monthly_df[&#39;Year-Month&#39;].astype(str)<br>    monthly_df.insert(4, &#39;Net Income&#39;, monthly_df[&#39;Revenue&#39;] - monthly_df[&#39;Expenses&#39;])<br>    monthly_df[&#39;Net Tax&#39;] = monthly_df[&#39;Tax Collected&#39;] - monthly_df[&#39;Tax Paid&#39;]<br><br>    return total_s, monthly_df.T</pre><p>If you have never used Pandas before, the code might look confusing, but what it’s doing is actually quite simple. The function generates two main outputs: a total summary and a monthly breakdown of our invoice data.</p><p>For the <strong>total summary,</strong> we use a one-dimensional <a href="https://pandas.pydata.org/docs/user_guide/dsintro.html#series">Pandas Series</a>, perfect for storing a single row of summary data. This includes the total period covered, number of invoices, invoiced revenue, invoiced expenses, net income, and tax information. To calculate these metrics, we need to sum up the relevant data from the invoice DataFrame, distinguishing between incoming and outgoing invoices.</p><p>For the <strong>monthly summary,</strong> we use a two-dimensional Pandas DataFrame (just like for the invoice data), which is ideal to represent multiple rows (months) with multiple columns (financial metrics). We use <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html">Pandas’ groupby</a> function (which is very similar to the SQL GROUP BY statement) to aggregate the invoice data by month, calculating the same metrics as in the total summary but on a month-by-month basis.</p><p>The power of Pandas is obvious here. With only a few lines of code, we are able to <strong>perform complex calculations across the whole dataset of invoices, transforming raw invoice data into financial insights.</strong> And now we are ready to display it all in an Excel report.</p><h3>Excel Report Creation</h3><p>The final step in the invoice processing pipeline is creating an Excel report. Using Pandas and the <a href="https://xlsxwriter.readthedocs.io/">xlsxwriter</a> library, we create an Excel workbook with two sheets: the ‘Invoices’ sheet contains all the raw invoice data, and the ‘Summary’ sheet displays the financial analysis calculated in the previous step.</p><p>This is how the function looks like:</p><pre>def create_excel_report(invoices_df, total_s, monthly_df, filepath):<br>    with pd.ExcelWriter(filepath, engine=&#39;xlsxwriter&#39;, datetime_format=&#39;YYYY-MM-DD&#39;) as writer:<br>        # Write invoice data sheet<br>        invoices_df.to_excel(writer, sheet_name=&#39;Invoices&#39;, index=False)<br><br>        # Create summary sheet<br>        workbook = writer.book<br>        worksheet = workbook.add_worksheet(&#39;Summary&#39;)<br><br>        # Define formats<br>        title_format = workbook.add_format({&#39;bold&#39;: True, &#39;font_size&#39;: 14})<br>        money_format = workbook.add_format({&#39;num_format&#39;: &#39;#,##0.00&#39;})<br><br>        # Write total summary<br>        worksheet.write(&#39;A1&#39;, &#39;TOTAL&#39;, title_format)<br>        total_s.to_frame().to_excel(writer, sheet_name=&#39;Summary&#39;, startrow=1, startcol=0, header=False)<br><br>        # Write monthly summary<br>        worksheet.write(&#39;A12&#39;, &#39;MONTHLY&#39;, title_format)<br>        monthly_df.to_excel(writer, sheet_name=&#39;Summary&#39;, startrow=12, startcol=0, header=False)<br><br>        # Money formatting<br>        money_rows = list(range(3, 9)) + list(range(14, 20))<br>        for row in money_rows:<br>            worksheet.set_row(row, None, money_format)<br><br>        # Create chart<br>        chart = workbook.add_chart({&#39;type&#39;: &#39;line&#39;})<br><br>        num_cols = monthly_df.shape[1]<br>        for i, metric in enumerate([&#39;Revenue&#39;, &#39;Expenses&#39;, &#39;Net Income&#39;]):<br>            chart.add_series({<br>                &#39;name&#39;: metric,<br>                &#39;categories&#39;: [&#39;Summary&#39;, 12, 1, 12, num_cols],<br>                &#39;values&#39;: [&#39;Summary&#39;, 14 + i, 1, 14 + i, num_cols],<br>            })<br><br>        chart.set_x_axis({&#39;name&#39;: &#39;Month&#39;})<br>        chart.set_y_axis({&#39;name&#39;: &#39;Amount&#39;, &#39;major_gridlines&#39;: {&#39;visible&#39;: False}})<br><br>        worksheet.insert_chart(&#39;A23&#39;, chart)</pre><p>As you can see above, displaying a Pandas DataFrame in an Excel sheet is as simple as: invoices_df.to_excel(writer, sheet_name=&#39;Invoices&#39;, index=False). And if we want to display the data as columns and starting at a specific cell (like we do in the &#39;Summary&#39; sheet) we can do it like this: monthly_df.to_excel(writer, sheet_name=&#39;Summary&#39;, startrow=12, startcol=0, header=False).</p><p>We also apply some basic formatting to make the report more readable, such as bold titles and proper number formatting for monetary values. And at the end, we generate a line chart displaying the invoiced revenues, expenses and net income over time.</p><p>It is a very basic financial analysis and report, but you can use it as a blueprint to do much more. Depending on your business needs, you could easily extend this code to include more complex financial analyses, additional charts, conditional formatting and other worksheets with more detailed breakdowns.</p><h3>Putting It All Together</h3><p>The only missing part now is putting together the whole invoice processing pipeline using the functions created in the previous sections. And we need to call it using asyncio because it contains asynchronous code. This is how it looks like:</p><pre>async def process_invoices():<br>    invoices_filenames, invoices_json = await extract_invoices_data(settings.INVOICES_DIR)<br>    invoices_df = build_invoices_dataframe(invoices_filenames, invoices_json)<br>    total_s, monthly_df = generate_financial_summary(invoices_df)<br>    create_excel_report(invoices_df, total_s, monthly_df, settings.REPORT_FILEPATH)<br><br>def main():<br>    asyncio.run(process_invoices())</pre><p>This simple demo shows the <strong>immense potential of AI-powered automation to transform tedious business tasks.</strong> In not many lines of code, we have automated a task that would probably require hours of manual work.</p><p>Remember that you can <a href="https://github.com/ruizguille/invoice-processing">access all the code here</a>, and you can apply and extend it to your own business processes. What will you automate next?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=962385333027" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Embeddings in Action: Visualizing IMDB’s Top 250 Movies]]></title>
            <link>https://medium.com/@codeawake/movie-embeddings-829c224c68f6?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/829c224c68f6</guid>
            <category><![CDATA[imdb]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[vector-embeddings]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Mon, 22 Jul 2024 08:41:31 GMT</pubDate>
            <atom:updated>2024-12-02T17:44:03.373Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Python guide to scrape IMDB and explore semantic connections between movies with vector embeddings</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/movie-embeddings"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*SHIcjVnVhHSERZcNDJcFjQ.png" /></figure><p><a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264">In our last post</a>, we explored Retrieval Augmented Generation by building a RAG app from scratch. One of the most important concepts in that post was <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264#47ba"><strong>Vector Embeddings</strong></a>, a very powerful tool that allows us to <strong>represent information as vectors in a high-dimensional space and encode the semantics of the information.</strong> This was very useful for <a href="https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264#d82e"><strong>semantic search</strong></a>, where we used vector embeddings to find the pieces of information that were most similar in meaning to a specific query.</p><p>Given that vector embeddings are one of the most powerful and fascinating concepts in AI, I thought it would be interesting to build a simple project to gain a better intuition about them by actually visualizing the embeddings and their relationships. And to make the project more fun, we are going to be exploring connections between popular movies.</p><p>So here is the plan:</p><ul><li>First of all, we’ll scrape <a href="https://www.imdb.com/chart/top/">IMDB’s top 250 movies</a>, extracting the titles and plots.</li><li>Then we’ll convert the plots into vector embeddings using <a href="https://platform.openai.com/docs/guides/embeddings">OpenAI’s Embeddings</a>, capturing their meanings and relationships.</li><li>And finally we’ll transform these high-dimensional embeddings into 2D and 3D representations to be able to plot them and visualize the semantic relationships among them.</li></ul><p>And you can find all the code in the following <a href="https://github.com/ruizguille/movie-embeddings">GitHub repository</a>.</p><h3>Scraping</h3><p>To scrape IMDB we are going to use the amazing <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">Beautiful Soup</a> Python library, that makes parsing and extracting data from HTML pages a breeze. We are also using the <a href="https://requests.readthedocs.io/en/latest/">Requests</a> library to make the necessary HTTP requests. Our scraping work will be done in two parts:</p><ol><li>First, we’ll get the list of 250 movie titles and their URLs from IMDB’s Top 250 movies chart.</li><li>Then, we’ll extract a short and a long plot from each of those movies. This is the actual text that will be converted to vector embeddings later.</li></ol><p>Let’s begin extracting the top 250 movies from the URL <a href="https://www.imdb.com/chart/top/">https://www.imdb.com/chart/top/</a>. The simplest way I found to scrape the necessary data is to read a &lt;script&gt; tag that contains the top 250 movies data in <a href="https://json-ld.org/">JSON-LD</a> format:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q0TQW3uf0U3B9Xi-" /><figcaption>Scraping IMDB’s top 250 movies</figcaption></figure><p>This is simpler than parsing the actual HTML elements that display the movies on the webpage because these are not all displayed at once, only the first 25 are shown and the rest are loaded dynamically using JavaScript. To scrape dynamically loaded content, we would need a more advanced scraping library. The JSON-LD script, however, contains the data we need for all 250 movies and is straightforward to extract.</p><p>Let’s take a a look at the code:</p><pre>headers = {<br>    &#39;Accept&#39;: &#39;*/*&#39;,<br>    &#39;Connection&#39;: &#39;keep-alive&#39;,<br>    &#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/70.0.3538.110 Safari/537.36&#39;,<br>    &#39;Accept-Language&#39;:&#39;en-US,en;q=0.9&#39;,<br>    &#39;Cache-Control&#39;: &#39;max-age=0&#39;,<br>    &#39;Upgrade-Insecure-Requests&#39;: &#39;1&#39;<br>}<br><br>def scrape_imdb_top_250():<br>    url = &#39;https://www.imdb.com/chart/top/&#39;<br>    response = requests.get(url, headers=headers)<br>    soup = BeautifulSoup(response.text, &#39;html.parser&#39;)<br>    <br>    movies = []<br>    data = json.loads(soup.find(&#39;script&#39;, {&#39;type&#39;:&#39;application/ld+json&#39;}).text)<br>    for item in tqdm(data[&#39;itemListElement&#39;]):<br>        # Need to unescape HTML characters to extract the title properly<br>        title = html.unescape(item[&#39;item&#39;].get(&#39;alternateName&#39;, item[&#39;item&#39;][&#39;name&#39;]))<br>        url = item[&#39;item&#39;][&#39;url&#39;]<br>        short_plot, long_plot = get_movie_plot(url + &#39;plotsummary/&#39;)<br>        movies.append({&#39;title&#39;: title, &#39;short_plot&#39;: short_plot, &#39;long_plot&#39;: long_plot })<br>        time.sleep(1)<br>    return pd.DataFrame(movies)</pre><p>As you can see above, we first make a request to the top 250 movies URL using specific headers to emulate a regular web browser request and avoid web scraping blocking. We then simply find the application/ld+json script and load the JSON-LD data. The title for each movie in the list is contained under [&#39;item&#39;][&#39;name&#39;] or [&#39;item&#39;][&#39;alternateName&#39;] (for the English title of non-English movies). The URL is contained under [&#39;item&#39;][&#39;url&#39;].</p><p>For each of the 250 movies, we then scrape the movie plot subpage with the function get_movie_plot and extract the short and long plot. And, finally, we build a <a href="https://pandas.pydata.org/docs/user_guide/dsintro.html#dataframe">Pandas DataFrame</a> with the title, short plot and long plot for each movie. We use a DataFrame because it makes it easy to operate with the data and save it to a Comma Separated Values (CSV) file.</p><p>This is the code for the get_movie_plot function that scrapes the plots for each movie:</p><pre>def get_movie_plot(url):<br>    response = requests.get(url, headers=headers)<br>    soup = BeautifulSoup(response.text, &#39;html.parser&#39;)<br>    summaries = soup.select(&#39;li.ipc-metadata-list__item div.ipc-html-content-inner-div div.ipc-html-content.ipc-html-content--base div.ipc-html-content-inner-div&#39;)<br>    plot = summaries[0].get_text().strip()<br>    # Remove the storyline author &lt;span&gt; element<br>    storyline_author = summaries[1].find(&#39;span&#39;)<br>    if storyline_author:<br>        storyline_author.decompose()<br>    storyline = summaries[1].get_text().strip()<br>    return plot, storyline</pre><p>To understand what it’s doing, let’s look at the <a href="https://www.imdb.com/title/tt0068646/plotsummary/">plot subpage for The Godfather</a>. We are extracting the first one (a short plot) and the second one (a longer plot). These are the same ones that appear on the main movie page, but they are easier to scrape from the plot subpage. Notice also how we remove the unwanted author &lt;span&gt; in the second plot so that it doesn&#39;t affect the semantics of the plot when we embed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OZJamPGS4Cab5-ok" /><figcaption>Plot subpage for The Godfather</figcaption></figure><p>Finally, we can run our scraping function and save the returned Pandas DataFrame as a CSV file for later use:</p><pre>movies_df = scrape_imdb_top_250()<br>movies_df.to_csv(&#39;data/movies.csv&#39;, index=False)</pre><h3>Embedding</h3><p>As mentioned earlier, we are going to use <a href="https://platform.openai.com/docs/guides/embeddings">OpenAI’s Embeddings</a> to convert the movie plots into vector embeddings. We are using their latest and most performant model text-embedding-3-large via the OpenAI API. To get started with their API you can follow <a href="https://platform.openai.com/docs/api-reference/introduction">this guide</a> and you can also check the <a href="https://platform.openai.com/docs/api-reference/embeddings">Embeddings API reference</a> for more detailed information.</p><p>The code to create the embeddings is really simple:</p><pre>def embed_movies(movies_df, model=&#39;text-embedding-3-large&#39;):<br>    movie_plots = (movies_df[&#39;short_plot&#39;] + &#39;\n&#39; + movies_df[&#39;long_plot&#39;]).to_list()<br>    embed_res = openai_client.embeddings.create(input=movie_plots, model=model)<br>    movies_df[&#39;embedding&#39;] = [d.embedding for d in embed_res.data]<br>    return movies_df</pre><p>Notice how we are concatenating the short and long plots for each movie, embedding them and then adding them to the movies DataFrame as a new column.</p><p>We can now run the function and save the result to another CSV file, which we’ll use for analysis and plotting:</p><pre>movies_df = embed_movies(movies_df)<br>movies_df.to_csv(&#39;data/movies_embeddings.csv&#39;, index=False)</pre><p>You can find both CSV files (with and without embeddings) in the <a href="https://github.com/ruizguille/movie-embeddings/tree/master/data">data folder of the repo</a>, so you can easily plot the movie embeddings without having to create an OpenAI account and generate the embeddings if you don’t want to.</p><h3>Plotting Vector Embeddings in 2D and 3D</h3><p>Now that we have converted the movie plots to high-dimensional vector embeddings (specifically, vectors of 3072 dimensions using OpenAI’s text-embedding-3-large model), we face a challenge: How can we visualize data with 3072 dimensions?</p><p>The answer is using <a href="https://en.wikipedia.org/wiki/Dimensionality_reduction"><strong>dimensionality reduction</strong></a> techniques. Dimensionality reduction allows us to <strong>transform high-dimensional data into a lower-dimensional space while preserving as much of the important information as possible.</strong> Our goal is to reduce those 3072 dimensions to 2 or 3 dimensions that we can easily plot in Python, while still keeping as many of the relationships between the data points as possible.</p><p>Yes, much of the data will be lost when moving from 3072 to 2 or 3 dimensions, if you are wondering about it. However, with the use of powerful dimensionality reduction techniques, enough of the key information will be retained to be able to visualize the connections between movies and gain a better intuition about vector embeddings.</p><p>The technique we are going to use is <a href="https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding"><strong>t-SNE</strong> (t-Distributed Stochastic Neighbor Embedding)</a> which works really well for reducing high-dimensional data to 2 or 3 dimensions and it’s particularly good at preserving local relationships in the data. And thanks to the wonderful <a href="https://scikit-learn.org/stable/index.html">scikit-learn</a> library, we can apply it with a single line of Python.</p><p>Let’s take a look at the code to plot the movie embeddings in 2D:</p><pre>import pandas as pd<br>import numpy as np<br>from sklearn.manifold import TSNE<br>from sklearn.preprocessing import StandardScaler<br>import plotly.express as px<br>from ast import literal_eval<br><br>def plot_movies_2d(file=&#39;data/movies_embeddings.csv&#39;):<br>    df = pd.read_csv(file)<br>    embeddings = np.array(df[&#39;embedding&#39;].apply(literal_eval).tolist())<br><br>    tsne = TSNE(n_components=2)<br>    scaler = StandardScaler()<br>    embeddings_2d = tsne.fit_transform(embeddings)<br>    embeddings_2d = scaler.fit_transform(embeddings_2d)<br><br>    df[&#39;x&#39;] = embeddings_2d[:, 0]<br>    df[&#39;y&#39;] = embeddings_2d[:, 1]<br><br>    fig = px.scatter(<br>        df, x=&#39;x&#39;, y=&#39;y&#39;, text=&#39;title&#39;,<br>        hover_data={&#39;title&#39;: True, &#39;x&#39;: False, &#39;y&#39;: False},<br>        title=&#39;Movie Similarity based on Plot Embeddings&#39;<br>    )<br>    fig.update_traces(textposition=&#39;top center&#39;, textfont=dict(size=6))<br>    fig.show()</pre><p>Let’s highlight the important parts of the code:</p><ul><li>The first thing we do is read the previously stored DataFrame that includes the vector embeddings from the CSV file. We are using the slightly awkward line np.array(df[&#39;embedding&#39;].apply(literal_eval).tolist()) to convert the embeddings column in the CSV file from text to a list of lists of floats and then to a <a href="https://numpy.org/doc/stable/user/whatisnumpy.html">NumPy array</a>.</li><li>We then use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html">scikit-learn’s TSNE</a> class to apply t-SNE to the data and transform it to two dimensions, and we also use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">StandardScaler</a> class to normalize the results.</li><li>Finally, we add the x and y coordinates from the reduced 2D vectors to the original DataFrame and use the <a href="https://plotly.com/python/">Plotly library</a> to create an interactive scatter plot. Each point in the scatter plot represents a movie, with the title as a label, and its position is determined by the coordinates of the reduced 2D vector embedding.</li></ul><p>Let’s now take a look at the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZAOTqEJwl3xXNvBz" /><figcaption>2D plot of movie embeddings showing clusters of related movies</figcaption></figure><p>The plot contains only the top 150 movies to make it more legible, and I’ve added hand-drawn clusters to highlight groups of films that are close together and semantically related. This visualization gives us a fascinating way to explore the <strong>“semantic space”</strong> of movie plots, and the clusters reveal very interesting insights about how our embedding model understands and relates different movies.</p><p>You can find <strong>clusters of similar genres, themes or styles.</strong> And sometimes the clusters reveal unexpected similarities between films. A few interesting examples:</p><ul><li>The Matrix, Inception and Metropolis, despite being very different films, share interesting similarities and appear close together.</li><li>Casablanca is not clearly contained by any of the clusters but seems to be positioned somewhere in between the Noir Thrillers and the WWII / Nazi Germany films.</li><li>Pan’s Labyrinth appears to be thematically close to the Studio Ghibli and Japanese animated films.</li></ul><p>The 3D plotting code is almost identical to the 2D, so I won’t repeat it here, but you can check it in detail <a href="https://github.com/ruizguille/movie-embeddings/blob/master/plot_movies.py#L28-L47">in this file</a>. I recommend running the code yourself to fully explore and interact with the 3D plot in detail. Let’s see how it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YnYFn_cnIs4SzU7Q" /><figcaption>3D plot of movie embeddings</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K4NEA9nDc_7BiIsY" /><figcaption>3D plot of movie embeddings</figcaption></figure><p>The transition from the 2D to the 3D visualization adds a new layer of depth to the movies’ representation and relationships. With this additional dimension, we can observe <strong>more nuanced clusters and complex interconnections</strong> that were not apparent in the 2D plot.</p><p>While the 3D plot is more informative than the 2D version, it’s still a massive simplification of the original 3072-dimensional vector embeddings space. You can intuitively imagine the <strong>incredible level of detail and nuance that can be captured in 3072 dimensions.</strong> It’s like having 3072 scales or spectrums to describe each movie, each of those scales representing a different feature. This high-dimensionality is what gives vector embeddings their power, allowing them to <strong>represent complex concepts and relationships.</strong></p><p>By exploring these visualizations, it’s a lot easier to understand how vector embeddings work. In this application, we used them to cluster and explore relationships between movies. But they can similarly be used with other types of content and formats, like novels, scientific papers, emails, and even images and audio. And they can be used for applications like <strong>semantic search</strong> (as we saw in the previous post) or <strong>content recommendation systems.</strong></p><p>Vector embeddings are an incredible tool to understand how machines can represent concepts and meaning, and they open up new possibilities for <strong>AI applications that can understand and work with human-generated content.</strong></p><p><em>Originally published at </em><a href="https://codeawake.com/blog/movie-embeddings"><em>https://codeawake.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=829c224c68f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval Augmented Generation from Scratch: Inception RAG]]></title>
            <link>https://medium.com/@codeawake/rag-from-scratch-ec1a36be0264?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec1a36be0264</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 08:40:32 GMT</pubDate>
            <atom:updated>2024-12-02T17:49:33.955Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explore vector embeddings, semantic search, chunking and more building an open-source RAG app</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/rag-from-scratch"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*ZhvPbo_RypjxV6hoFS4DOw.png" /></figure><blockquote>“Downwards is the only way forwards.”</blockquote><p>As Large Language Models (LLMs) have become more intelligent and widely used, <strong>Retrieval Augmented Generation (RAG)</strong> has emerged as a very powerful technique to enhance their capabilities and help with their limitations.</p><p>To really understand how RAG works, in this post we are going to dive down and build a RAG system from scratch using Python and only open-source tools. We will use the open-source <a href="https://ai.meta.com/blog/meta-llama-3/">Llama 3 model</a>, powered by the incredibly fast <a href="https://groq.com/">Groq platform</a>, and the open-source <a href="https://blog.nomic.ai/posts/nomic-embed-text-v1">Nomic embeddings</a>. And we will not use complex libraries, frameworks like Langchain, or even vector databases. We will build the functionality from scratch with Python and some Numpy.</p><p>And our goal? A basic RAG system capable of answering questions about <strong>Christopher Nolan’s cult film Inception</strong>, using its screenplay as our knowledge base. By the end, you will have a clear understanding of how RAG works and the ability to implement it in your own projects. And you will also be aware of the issues and limitations of basic RAG and possible ways to improve it.</p><p>Let’s begin!</p><h3>What is Retrieval Augmented Generation?</h3><p>Large Language Models like GPT-4 or Llama 3 have a broad general knowledge and very advanced capabilities. They excel at understanding, processing and generating text, and are really useful as conversational assistants. But they have some limitations:</p><ul><li>Their <strong>knowledge is static</strong> and they might not know about specific topics or recent events.</li><li>They have a <strong>limited context window</strong> (the maximum text a LLM can process including input and output), so you can’t feed them all the information you want.</li><li>There’s <strong>no easy way to integrate your own personal or organization data</strong> into their knowledge, and it’s even harder if your data is always changing.</li><li>They sometimes <strong>“hallucinate”</strong> answers that sound coherent but are incorrect or even nonsensical.</li></ul><p>Retrieval Augmented Generation has proven to be a very effective technique (or rather, a family of techniques) to address these problems. <strong>RAG combines the power of LLMs with external knowledge retrieval</strong>, making it possible for language models to access relevant information when generating responses and not be constrained only by the knowledge they were trained on.</p><p>Applying RAG means that you can now <strong>integrate information about recent events, niche topics and your own personal and organization data</strong> into your LLM-based applications and make it actionable. It also means that this external knowledge base is no longer static, and it can <strong>adapt as your data changes or your business evolves.</strong></p><p>RAG also allows you to have a greater control over which data the LLM uses in each generation. For example, you might want to prioritize your organization’s more accurate and specialized data when answering questions over the LLM’s general and potentially incomplete knowledge.</p><p>In the next sections, we will see how to achieve this by building a RAG system from the ground up. The Inception screenplay will serve as the knowledge base for a basic assistant that can answer questions about the movie. And we will look in detail at the different steps of the RAG pipeline. A quick preview:</p><ul><li><strong>Loading:</strong> Import and parse the source documents.</li><li><strong>Chunking:</strong> Break the documents down into smaller, manageable pieces of text.</li><li><strong>Vector Embeddings</strong>: Convert the text chunks into numerical representations that encode their semantic meaning.</li><li><strong>Semantic Search</strong>: Use vector embeddings to find the pieces of text that are most semantically similar to a search query.</li><li><strong>Vector Store</strong>: Create a custom storage system that holds our vector embeddings and can perform semantic search.</li><li><strong>Retrieval Augmented Generation</strong>: Tie all the components together to take a user question, embed it, use the vector store to search for the most relevant information, and answer the question with the LLM using the relevant information as context.</li></ul><p>You can find all the code in this <a href="https://github.com/ruizguille/rag-from-scratch">GitHub repository</a> with an MIT license to use it as you want.</p><h3>Loading Documents</h3><p>The first step in the RAG pipeline is to load and parse the information that we want to include in our knowledge base. In our case, it’s the Inception screenplay in pdf format, but we will generalize the code to any pdf files included in the DOCS_DIR folder, and it can be easily extended to other formats as well. To extract the text from the pdf files we&#39;ll be using the <a href="https://pdfminersix.readthedocs.io/en/latest/">pdfminer</a> Python library.</p><pre>from pdfminer.high_level import extract_text<br><br>docs = []<br>for filename in os.listdir(DOCS_DIR):<br>    if filename.endswith(&#39;.pdf&#39;):<br>        file_path = os.path.join(DOCS_DIR, filename)<br>        text = extract_text(file_path)<br>        docs.append(text)</pre><h3>Tokens</h3><p><strong>LLMs</strong> (and also embedding models) <strong>process text in basic units known as tokens.</strong> Depending on the model and the tokenization process used, tokens could be words, subwords or characters. You can explore how language models tokenize text with <a href="https://platform.openai.com/tokenizer">OpenAI’s tokenizer tool</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XDm1yHEcCd6CEvrb" /><figcaption>Text tokenized by a language model</figcaption></figure><p>Because tokens are the basic units of LLM text processing, you will generally see inputs, outputs, context windows and chunk sizes measured in tokens. Keep also in mind that different models use different tokenization processes. In particular, Llama 3 uses a <a href="https://huggingface.co/learn/nlp-course/en/chapter6/5">Byte-Pair Encoding (BPE) tokenizer</a> which is based on <a href="https://github.com/openai/tiktoken">tiktoken</a> (the tokenizer used by OpenAI’s models), which is different to the SentencePiece tokenizer used by Llama 2.</p><h3>Chunking</h3><p>As seen before, large language models have a limited context window, which means that we need to be selective about the context included when answering a specific question. RAG helps solve this by <strong>splitting the source documents into chunks</strong>, converting each chunk into a vector embedding that encodes its meaning, and then using semantic search to find the most relevant chunks to the user question.</p><p>There are now two key questions: <strong>How do we split the documents? And what should be the size of the chunks?</strong></p><p>To answer the first question, there are a number of possible techniques:</p><ul><li>Simple methods that split the text by a <strong>fixed number of characters</strong>.</li><li>Recursive methods that split the text <strong>hierarchically</strong> using a list of separators or splitting functions.</li><li>Specialized methods that take into account the <strong>document format</strong> (e.g. markdown or code).</li><li>More advanced methods like <strong>semantic chunking</strong> (proposed by <a href="https://x.com/GregKamradt">Greg Kamradt</a>), where vector embeddings are used to split the text into chunks of semantically similar text.</li></ul><p>You can learn more about these methods by reading Greg’s amazing <a href="https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb">5 Levels Of Text Splitting notebook</a> and watching his <a href="https://www.youtube.com/watch?v=8OJC21T2SL4">Youtube tutorial</a>, or reading this <a href="https://www.pinecone.io/learn/chunking-strategies/">article by Pinecone on chunking strategies</a>.</p><p>The second crucial question is how large should the chunks be. <strong>Smaller chunks</strong> (for example, a single sentence) <strong>will capture well nuances in meaning</strong> and allow you to find very precise information, but may miss important context that will be scattered across neighboring chunks. <strong>Larger chunks</strong> (multiple pages of text, for instance) <strong>will include broader context and themes</strong>, but may also include irrelevant information and dilute the specific meaning of individual sentences.</p><p>This is a very important decision and unfortunately there is not a one-size-fits-all recipe. In general, you should evaluate different chunk sizes for your own application and see the one that gives you the optimal balance and the best results. In practice, somewhere between <strong>512–1024 tokens</strong> seems to be a sweet spot for many applications.</p><p>In our RAG system, we’ll be using a recursive chunking method inspired by <a href="https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/sentence_splitter/">LlamaIndex’s Sentence Splitter</a> with a chunk size of 512 tokens. Let’s take a look at some of the TextSplitter code:</p><pre>class TextSplitter:<br>    def __init__(self, chunk_size):<br>        self.chunk_size = chunk_size<br>        self.splitters = [<br>            partial(split_by_separator, sep=&#39;\n\n&#39;),<br>            partial(split_by_separator, sep=&#39;\n&#39;),<br>            split_sentences,<br>            partial(split_by_separator, sep=&#39; &#39;)<br>        ]<br>    <br>    def split(self, text):<br>        splits = self._split_recursive(text)<br>        chunks = self._merge_splits(splits)<br>        return chunks</pre><p>As you can see above, there are 4 splitting functions applied hierarchically which break down text into paragraphs (ending in a double newline), lines ending in a newline character, sentences and words, respectively. To split sentences we are using NLTK’s amazing <a href="https://www.nltk.org/api/nltk.tokenize.punkt.html">Punkt Sentence Tokenizer</a>.</p><p>The splitting is performed recursively by the _split_recursive function, starting with the largest splits (paragraphs), and only splitting into smaller units when the split size (in tokens) is larger than the specified chunk_size. Then the _merge_splits function merges the splits together into the final chunks ensuring that all chunks have a size as close as possible to the specified chunk_size but never exceeding it.</p><p>And this is how we can use the TextSplitter to chunk the documents that we loaded previously:</p><pre>chunks = []<br>text_splitter = TextSplitter(chunk_size=512)<br>for i, doc in enumerate(docs):<br>    doc_chunks = text_splitter.split(doc)<br>    chunks += doc_chunks</pre><p>If you want to look at the text splitting code and the implementation of the rest of the functions in detail, you can <a href="https://github.com/ruizguille/rag-from-scratch/blob/master/app/splitter.py">find it here</a>.</p><h3>Vector Embeddings</h3><p>Vector Embeddings are one of the most incredible and useful concepts in AI. They allow us to <strong>represent phrases, paragraphs and even entire documents as numerical vectors</strong> in a high-dimensional space. And the most important quality of this representation is that it <strong>captures semantic similarity</strong>, meaning that pieces of text that have similar meanings are closer together in this vector space.</p><p>To get a better intuition about embeddings, check the <a href="https://projector.tensorflow.org/">Tensorflow Embedding Projector</a>. This tool displays high-dimensional embeddings into a simple 3D view that captures the key properties of the data. With word embeddings, for example, you can explore which words are semantically closer to a specific word. Let’s see the plot for the word “intelligence”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F6RkC0S_XDkTprjg" /><figcaption>Words that are semantically similar to the word intelligence</figcaption></figure><p>But the power of embeddings is not limited to text. <strong>They can also represent images, audio and other formats.</strong> By converting different types of information into embeddings in the same vector space, you can capture meanings and connections across them and be able to search for content in all these formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2sBBxKODUCZWWgOm" /><figcaption>Embedding documents, images and audio in the same vector space</figcaption></figure><p>In our RAG system, we are using <a href="https://blog.nomic.ai/posts/nomic-embed-text-v1">Nomic Embed</a>, a recently released and fully open-source embedding model that outperforms well-known embedding models like OpenAI’s <em>Ada-002</em> and <em>text-embedding-3-small</em> on both short and long context tasks.</p><p>Embedding the text chunks generated in the previous step is straightforward with <a href="https://docs.nomic.ai/reference/python-api/embeddings">Nomic’s Python client</a>:</p><pre>from nomic import embed<br><br>embed_res = embed.text(<br>    texts=chunks,<br>    model=&#39;nomic-embed-text-v1.5&#39;,<br>    task_type=&#39;search_document&#39;,<br>    inference_mode=&#39;local&#39;<br>)<br><br>print(embed_res[&#39;embeddings&#39;])</pre><p>For each text chunk that you embed with Nomic’s model, you will get as output a vector of 768 coordinates that encodes the chunks’s semantic content. The inference_mode=&#39;local&#39; option means that we are <a href="https://blog.nomic.ai/posts/local-nomic-embed">running the model locally in our computer</a>, but Nomic also provides a remote API for more performant use cases.</p><h3>Semantic Search</h3><p>Now that we have split the Inception screenplay into chunks and converted them into vector embeddings that encode the meaning of those chunks, we can <strong>perform semantic search to find the text chunks that are most similar to a specific user query.</strong> But how does semantic search actually work?</p><p>If you remember, vector embeddings capture semantic similarity by grouping pieces of text that have a similar meaning closer together in the vector embedding space. So to do semantic search, we only need to convert the user query into a vector embedding too, and find which vector embeddings from the source documents are <em>closer</em> or <em>most similar</em> to the query embedding.</p><p>There are several methods to measure the <em>distance</em> or <em>similarity</em> between vectors: <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean distance</a> (the straight-line distance between two vectors), <a href="https://en.wikipedia.org/wiki/Dot_product">dot product</a> (the sum of the products of the vector components) or <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a> (a measure of the angle between the vectors).</p><p><strong>Cosine similarity</strong> is commonly used as it is invariant to vector magnitudes (it measures the angle between the vectors), it’s suitable for high-dimensional data and it’s very easy to interpret. The result of cosine similarity ranges from -1 to 1, where 1 means the angle between the vectors is 0 (they are identical in direction and as similar as possible), 0 means they are orthogonal (uncorrelated), and -1 means they point in opposite directions. Although in practice and depending on the embedding model, the actual range of values might be from 0 to 1, or narrower.</p><p>Cosine similarity is calculated by dividing the dot product of two vectors by the product of their magnitudes, and it equals the cosine of the angle between the vectors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bDzYrXb_e9dfroX_" /><figcaption>Cosine similarity mathematical equation</figcaption></figure><p>And this is how to implement it in code using the <a href="https://numpy.org/">NumPy library</a>:</p><pre>def cosine_similarity(query_vector, vectors):<br>    query_vector = np.array(query_vector)<br>    vectors = np.array(vectors)<br>    return np.dot(vectors, query_vector) / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(query_vector))</pre><p>Note how the function uses a <strong>vectorized approach</strong> to calculate the cosine similarity between the query vector and a list of vectors simultaneously. This takes advantage of <a href="https://numpy.org/doc/stable/user/whatisnumpy.html#whatisnumpy">NumPy’s efficient array operations</a> that are much faster than calculating the similarities one by one in a loop.</p><p>Using the cosine_similarity function, we can perform semantic search by calculating the similarities between the user query embedding and all the document chunks embeddings, and then selecting the chunks with the highest similarity scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*Lym4kM2i_s2uTABOQyiOAw.png" /><figcaption>Semantic search finds the chunks that are most relevant to the user query</figcaption></figure><h3>Vector Store</h3><p>A vector store is a special database designed to store and search through large collections of vector embeddings. There are several popular vector database providers like <a href="https://qdrant.tech/">Qdrant</a>, <a href="https://www.pinecone.io/">Pinecone</a>, <a href="https://www.trychroma.com/">Chroma</a>, <a href="https://weaviate.io/">Weaviate</a> or even PostgreSQL with the <a href="https://github.com/pgvector">pgvector</a> extension, most of which are also open-source. These solutions offer great performance, high scalability and advanced querying functionalities, and are very useful for large RAG systems.</p><p>However, for our simple RAG application, we are going to build a very basic but effective vector store from scratch that will also allow us to understand the key concepts. In our simple implementation, we are going to store the vector embeddings and their corresponding text chunks as a list of dictionaries like this one:<br>{&quot;vector&quot;: [0.1, 0.4, ...], &quot;text&quot;: &quot;This is the text chunk.&quot;}</p><p>Let’s take a look at its implementation:</p><pre>class VectorStore:<br>    def __init__(self):<br>        self.store = []<br><br>    def add(self, items):<br>        self.store.extend(items)<br>    <br>    def save(self, file_path=VECTOR_STORE_FILEPATH):<br>        with open(file_path, &#39;w&#39;) as f:<br>            json.dump(self.store, f)<br><br>    def load(self, file_path=VECTOR_STORE_FILEPATH):<br>        with open(file_path, &#39;r&#39;) as f:<br>            self.store = json.load(f)<br>    <br>    def query(self, vector, top_k=10):<br>        vectors = [item[&#39;vector&#39;] for item in self.store]<br>        similarities = cosine_similarity(vector, vectors)<br>        top_k_indices = np.argsort(similarities)[-top_k:][::-1]<br>        return [{**self.store[i], &#39;score&#39;: similarities[i]} for i in top_k_indices]</pre><p>The most important method is the query method and it&#39;s the one used for semantic search. It takes a vector embedding as input (for example, the embedded user query), calculates the similarities between this query vector and all the vectors in the vector store, and returns the top k most similar chunks including the score.</p><p>If you are curious about the details, NumPy’s <a href="https://numpy.org/doc/stable/reference/generated/numpy.argsort.html">argsort</a> returns the indices of the sorted similarities (in ascending order). We are interested in the top k highest similarities (in descending order), so we take the last k indices and reverse them. We then use these indices to retrieve the corresponding items from our store, returning the vector, the text chunk and the similarity score for each of the top k.</p><p>The add method simply appends a list of items (the dictionaries containing the vector embedding-text chunk pairs) to the vector store. And the save and load methods allow us to save the vector store to and load it from a JSON file.</p><p>And this is how we use our vector store to add the vector embeddings and text chunks generated in the previous step and save it for future use:</p><pre>vector_store = VectorStore()<br>vectors = [<br>    {&#39;vector&#39;: vector, &#39;text&#39;: text} for vector, text in zip(embed_res[&#39;embeddings&#39;], chunks)<br>]<br>vector_store.add(vectors)<br>vector_store.save()</pre><h3>Retrieval Augmented Generation</h3><p>We have explored all the necessary building blocks of a RAG system. Now let’s bring them together to create an application that can answer questions about Inception (or any other documents you would like use in your own app).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DFDPgYXkxxVi-21_" /><figcaption>Loading documents to a vector store and querying using Retrieval Augmented Generation</figcaption></figure><p>The core idea of RAG is very simple:</p><ul><li>Take a user’s question and convert it to a <strong>vector embedding.</strong></li><li>Use semantic search to <strong>retrieve</strong> the most relevant chunks to the user question from the vector store.</li><li>Include these chunks as context for the language model to <strong>generate</strong> an answer based on them.</li></ul><p>Let’s now see this in actual code:</p><pre>def answer_question(question, vector_store):<br>    # Embed the user&#39;s question<br>    embed_res = embed.text(<br>        texts=[question],<br>        model=&#39;nomic-embed-text-v1.5&#39;,<br>        task_type=&#39;search_query&#39;,<br>        inference_mode=&#39;local&#39;<br>    )<br>    query_vector = embed_res[&#39;embeddings&#39;][0]<br><br>    # Find the most relevant chunks in our vector store using semantic search<br>    chunks = vector_store.query(query_vector)<br><br>    # Prepare the context and prompt, and generate an answer with the LLM<br>    context = &#39;\n\n---\n\n&#39;.join([chunk[&#39;text&#39;] for chunk in chunks]) + &#39;\n\n---&#39;<br>    user_message =  USER_PROMPT.format(context=context, question=question)<br>    messages=[<br>        {&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: SYSTEM_PROMPT},<br>        {&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_message}<br>    ]<br>    chat_completion = groq_client.chat.completions.create(<br>        messages=messages, model=&#39;llama3-70b-8192&#39;<br>    )<br>    return chat_completion.choices[0].message.content</pre><p>We have already seen how to embed text and perform semantic search in our vector store using the query method. The new element here is how we use Meta’s Llama 3 language model (specifically the larger, more powerful 70B version), powered by Groq, to generate answers. And because Llama 3 is open-source, you can also deploy it in your own hardware without Groq if you prefer.</p><p>If it’s your first time using LLMs like Llama 3, you can check <a href="https://medium.com/@codeawake/ai-voice-assistant-part1-13cacc59b078#b9ed">this introduction section</a> in my previous series on building an AI voice assistant.</p><p>Essentially, we pass two messages to the LLM:</p><ul><li>A basic system message that sets the overall behavior.</li><li>A user message containing the user question, the most relevant chunks from our vector store and specific instructions for answering the question.</li></ul><p>A crucial instruction for LLMs in RAG systems is to answer the user questions using <strong>ONLY the provided context</strong>, and never making answers up if they are not found in the context. Here is the actual user prompt template:</p><pre>USER_PROMPT = &quot;&quot;&quot;<br>Use the following pieces of context to answer the user question.<br>You must only use the facts from the context to answer.<br>If the answer cannot be found in the context, say that you don&#39;t have enough information to answer the question and provide any relevant facts found in the context.<br>Don&#39;t address \&quot;the context\&quot; explicitly in your answer, answer the question like it&#39;s your own knowledge.<br><br>Context:<br>{context}<br><br>User Question:<br>{question}<br>&quot;&quot;&quot;</pre><p>Note that we are using Python’s string formatting to fill the user prompt template with the relevant context (the retrieved chunks) and the user question:</p><pre>USER_PROMPT.format(context=context, question=question)</pre><p>And to complete our RAG application, we create a main function that loads the vector store from the JSON file and starts a simple loop for handling multiple user questions:</p><pre>def main():<br>    vector_store = VectorStore()<br>    vector_store.load()<br><br>    print(&quot;Ask a question about Christopher Nolan&#39;s Inception movie:\n&quot;)<br>    while True:<br>        question = input()<br>        answer = answer_question(question, vector_store)<br>        print(answer, &#39;\n&#39;)</pre><h3>Evaluation</h3><p>Now that our RAG system is complete, let’s test it with some questions about Inception:</p><blockquote><strong>1. What’s the most resilient parasite?</strong><br> According to Cobb, the most resilient parasite is an idea. He says, “Resilient, highly contagious. Once an idea’s taken hold in the brain it’s almost impossible to eradicate. A person can cover it up, ignore it — but it stays there.”</blockquote><blockquote><strong>2. What’s the name of Cobb’s wife?</strong><br> Cobb’s wife’s name is Mal.</blockquote><blockquote><strong>3. What’s the name of the Fischer’s family business?</strong><br> The name of Fischer’s family business is Fischer Morrow.</blockquote><blockquote><strong>4. What happened to Mal?</strong><br> Mal, Cobb’s wife, died by jumping off a ledge in a hotel. This event occurred before the main storyline of the script. Cobb was unable to stop her from jumping, and her death has had a profound impact on him. Throughout the script, it is revealed that Mal’s death was a result of Cobb’s own actions, as he had performed an inception on her, planting the idea that their reality was not real. This led to her becoming convinced that they needed to “wake up” from their reality, and ultimately, to her death.</blockquote><blockquote><strong>5. What was Fisher’s relationship with his father like?</strong><br> According to the script, Robert Fischer’s relationship with his father, Maurice Fischer, was strained and disappointing. In a conversation with Peter Browning, Fischer reveals that his father told him “there’s really nothing to be said” when he went to him in grief after his mother’s death. Fischer was only 11 years old at the time. Later, when Fischer’s father was on his deathbed, he called Fischer to his side and his last words to him were “Disappointed.” This implies that Fischer’s father was emotionally distant and critical, and that Fischer felt unsuccessful in gaining his father’s approval or affection.</blockquote><p>Overall we can see that the system accurately answers a range of questions about Inception, from simple facts to more complex questions about characters’ relationships and plot points, and is able to integrate multiple pieces of information to provide more complete answers.</p><h3>Limitations of Basic RAG and Potential Improvements</h3><blockquote>“You mustn’t be afraid to dream a little bigger, darling.”</blockquote><p>Our basic built-from-scratch RAG application is a good starting point and very helpful to understand the essential elements and concepts. However, as you start applying it to real-world use cases, you will quickly find a number of issues and limitations. Let’s end this post exploring some of them and opening up possible avenues for improvement:</p><ol><li><strong>COMPLEX QUERIES:</strong> Our current system works well for simple and factual questions, but will not work so well with <strong>complex questions that require many pieces of information or a synthesis of whole documents.</strong> For example: <em>“How does Cobb’s character evolve throughout the movie?”</em> or <em>“What are the main themes of Inception?”</em>. You are always limited by the amount of information you can retrieve and by the context window of the LLM.<br><br>✅ <strong>Improvements:</strong> Break down complex questions into simpler <strong>sub-queries</strong>, use multiple rounds of <strong>iterative retrieval</strong> to gather enough relevant information before generating the answer, or use <strong>recursive summarization</strong> techniques to synthesize whole documents.</li><li><strong>SEMANTIC GAP:</strong> The effectiveness of retrieval depends heavily on how the user phrases the question and whether there is enough information in the question to connect it with the relevant chunks using semantic search. If the semantic gap is too large, these chunks may not be fetched or may rank lower than less relevant chunks that share more semantic similarities with the question.<br><br>✅ <strong>Improvements:</strong> <strong>Query rewriting</strong> or <strong>query expansion</strong> (using an LLM to rewrite the query or generate multiple variations of the query), <strong>hybrid retrieval</strong> (combining semantic search with the more traditional keyword-based search) or <strong>iterative retrieval.</strong></li><li><strong>LARGER DATASETS:</strong> As the knowledge base grows (for instance, all of Christopher Nolan’s screenplays), it will become harder to retrieve the correct chunks for specific questions and the LLM context might be polluted by chunks that might not even belong to the movie you are asking about.<br> <br>✅ <strong>Improvements:</strong> Use <strong>metadata filtering</strong> to narrow down the search space (e.g. movie title, date, characters), or use a two-step retrieval where the LLM first tries to <strong>route the question</strong> to a specific document or subset of documents.</li><li><strong>RETRIEVAL QUALITY:</strong> Some retrieved chunks might not be relevant or the order might not be the optimal one. This is a common issue with vector databases and large datasets, as they use approximate similarity search techniques that trade off some accuracy for speed.<br> <br>✅ <strong>Improvements:</strong> Implement a <strong>reranking step</strong> after initial retrieval with a specialized model that is able to sort the chunks more accurately.</li><li><strong>CONVERSATION MEMORY</strong>: Our current system treats each question independently and is not suitable for follow-up questions or questions that reference previous interactions. For example, asking <em>“Who’s Ariadne?”</em> followed by <em>“What’s her role in the team?”<br> <br></em>✅ <strong>Improvements:</strong> Implement a <strong>conversational RAG system</strong> that includes the conversation history in the LLM context and use the LLM to rephrase the question based on the previous interactions.</li><li><strong>SCALABILITY:</strong> The basic vector store we built from scratch works well for simple use cases but lacks the efficiency and performance required for applications with large datasets.<br> <br>✅ <strong>Improvements:</strong> Implement more advanced <strong>search indexes</strong> and <strong>similarity search algorithms</strong> (like <a href="https://www.pinecone.io/learn/series/faiss/hnsw/">HNSW</a>), or use one of the popular vector databases mentioned earlier.</li></ol><p>As you can see, even though we covered a lot in this post, there is much more to explore in the world of Retrieval Augmented Generation, especially when it comes to applying it to real-world use cases. And because it’s a new and rapidly growing field, there is also new research emerging constantly. In future posts, we will continue exploring more sophisticated RAG systems and techniques.</p><p>Meanwhile, the code for this project is available on <a href="https://github.com/ruizguille/rag-from-scratch">GitHub</a>. And if you have any questions or comments, you can reach me at <a href="/service/mailto:guillermo@codeawake.com">guillermo@codeawake.com</a>.</p><p><em>Originally published at </em><a href="https://codeawake.com/blog/rag-from-scratch"><em>https://codeawake.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec1a36be0264" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI Voice Assistant with Groq and Deepgram (Part 3)]]></title>
            <link>https://medium.com/@codeawake/ai-voice-assistant-part3-6041d718e0a7?source=rss-1b6b211a55fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/6041d718e0a7</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Guillermo Ruiz | CodeAwake]]></dc:creator>
            <pubDate>Sun, 07 Jul 2024 13:06:31 GMT</pubDate>
            <atom:updated>2025-09-27T09:01:40.278Z</atom:updated>
            <content:encoded><![CDATA[<h4>An interactive React user interface</h4><p><em>Originally published at </em><a href="https://codeawake.com/blog/ai-voice-assistant-3"><em>https://codeawake.com</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*S9ONaHoBLzTCSPaHriveVg.png" /></figure><p>This is the third and final part of this series on building a fast AI voice assistant using Python, React, Deepgram, Llama 3, and Groq. Here’s a quick recap:</p><ul><li><a href="https://medium.com/@codeawake/ai-voice-assistant-part1-13cacc59b078">Part 1: A local Python voice assistant app.</a></li><li><a href="https://medium.com/@codeawake/ai-voice-assistant-part2-d53374332861">Part 2: A Python backend with FastAPI &amp; WebSockets.</a></li><li><strong>Part 3: An interactive React user interface.</strong></li></ul><p>In Part 1, we built a local Python application that demonstrated the core features of an AI voice assistant. And in Part 2, we built the Python backend for the complete voice assistant web app.</p><p>In this final part, we’ll create a <strong>React user interface</strong> that allows users to interact with a fully functional AI voice assistant. Users will be able to start and stop conversations, view real-time transcripts and listen to the AI assistant’s responses to their own voice queries.</p><h3>1. High-Level Overview</h3><p>The frontend for the voice assistant app is built using <a href="https://react.dev/">React</a>. React is a popular JavaScript library for building dynamic user interfaces. React’s <a href="https://react.dev/learn/reacting-to-input-with-state#how-declarative-ui-compares-to-imperative">declarative syntax</a> and component-based architecture makes it very easy to create complex UIs.</p><p>The main building blocks of a React application are <a href="https://react.dev/learn/your-first-component">components</a>. A component is essentially a JavaScript function that represents a reusable piece of the user interface, combining its own logic and markup (the structure and appearance of the UI). Components can be as small as a single button or as large as an entire page layout.</p><p>In React, components define their UI using a syntax extension called <a href="https://react.dev/learn/writing-markup-with-jsx">JSX</a>. JSX allows you to write HTML-like markup inside your JavaScript code. When a component renders, it returns this JSX markup, which React then efficiently displays in the browser.</p><h4>1.2. User Interface</h4><p>Let’s take a look at the final user interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vSrLWbea9XkfQA0P" /><figcaption>Voice assistant user interface</figcaption></figure><p>The user interface is designed to be simple and intuitive. It includes:</p><ul><li>A button to start/stop the conversation.</li><li>A button to mute/unmute the microphone.</li><li>The list of conversation messages (including the real-time transcript when the user speaks).</li><li>A small animation at the top that indicates when the assistant is active.</li></ul><h4>1.3. Client-Server Interaction and Web APIs</h4><p><a href="https://medium.com/@codeawake/ai-voice-assistant-part2-d53374332861#1823">As discussed in Part 2</a>, we use WebSockets to create a real-time, bidirectional communication channel between the client (React frontend) and the server (Python backend). To better understand how the frontend and backend interact and what needs to be implemented in the frontend, it’s a good idea to revisit the <a href="https://medium.com/@codeawake/ai-voice-assistant-part2-d53374332861#1c45">client-server data flow section in Part 2</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-oFluKwv9MAtUR11" /></figure><p>Additionally, to interact with the user’s microphone and be able to play the assistant’s audio responses, we need to use two web APIs:</p><ul><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API"><strong>MediaStream Recording API:</strong></a> Allows us to access the user’s microphone (with their permission) and record the audio input. The <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder">MediaRecorder</a> interface captures the audio stream from the microphone, which is then sent to the server in chunks through the WebSocket connection.</li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API"><strong>Media Source API:</strong></a> Allows us to play the assistant’s audio response as it is streamed from the server. The <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaSource">MediaSource</a> interface acts as a container for the audio data, while the <a href="https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer">SourceBuffer</a> interface is used to append the incoming audio chunks to the MediaSource.</li></ul><h3>2. Technical Deep Dive</h3><p>Let’s now dive into the frontend code. You can find it all in the frontend/ folder of this <a href="https://github.com/ruizguille/voice-assistant">GitHub repository</a>.</p><p>We are using the <a href="https://nextjs.org/">Next.js framework</a> in this application, which follows a specific project structure. However, you can easily adapt the code and use React without a framework with a modern bundler like <a href="https://vitejs.dev/guide/">Vite</a>.</p><h4>2.1. Application Structure and Core Elements</h4><p>The voice assistant frontend is a relatively simple application, it consists of a single main component called VoiceAssistant. This component encapsulates all the necessary logic for managing the conversation state, handling user interactions, communicating with the backend via WebSocket, and controlling the microphone and audio playback.</p><p>Following a top-down approach, let’s take a look at the structure and core elements of the VoiceAssistant component, shown in this simplified version of the code:</p><pre>function VoiceAssistant() {<br>  // State variables<br>  const [conversation, dispatch] = useReducer(conversationReducer, initialConversation);<br>  const [isRunning, setIsRunning] = useState(false);<br>  const [isListening, setIsListening] = useState(false);<br>  // Refs<br>  const wsRef = useRef(null);<br>  const mediaRecorderRef = useRef(null);<br>  const mediaSourceRef = useRef(null);<br>  const sourceBufferRef = useRef(null);<br>  const audioElementRef = useRef(null);<br>  const audioDataRef = useRef([]);<br>  const messagesEndRef = useRef(null);<br><br>  // Open WebSocket connection and set up event listeners<br>  function openWebSocketConnection() { }<br><br>  // Close WebSocket connection<br>  function closeWebSocketConnection() { }<br><br>  // Request microphone access and start recording<br>  async function startMicrophone() { }<br><br>  // Pause or resume microphone recording<br>  function toggleListening() { }<br><br>  // Stop recording and release microphone<br>  function stopMicrophone() { }<br><br>  // Initialize MediaSource and Audio element to play audio responses<br>  function startAudioPlayer() { }<br><br>  // Stop audio playback and release resources<br>  function stopAudioPlayer() { }<br><br>  async function startConversation() {<br>    dispatch({ type: &#39;reset&#39; });  // Reset the conversation messages<br>    try {<br>      openWebSocketConnection();<br>      await startMicrophone();<br>      startAudioPlayer();<br>      setIsRunning(true);<br>      setIsListening(true);<br>    } catch (err) {<br>      console.log(&#39;Error starting conversation:&#39;, err);<br>      endConversation();<br>    }<br>  }<br><br>  function endConversation() {<br>    closeWebSocketConnection();<br>    stopMicrophone();<br>    stopAudioPlayer();<br>    setIsRunning(false);<br>    setIsListening(false);<br>  }<br> <br>  // Simplified JSX (HTML-like markup) returned by the component<br>  return (<br>    &lt;div&gt;<br>      &lt;div&gt;{/* CSS animation */}&lt;/div&gt;<br>      &lt;div&gt;<br>        {/* Start/end conversation button */}<br>        &lt;button onClick={isRunning ? endConversation : startConversation}&gt;<br>          {isRunning ? &#39;End conversation&#39; : &#39;Start conversation&#39;}<br>        &lt;/button&gt;<br>        {/* Mute/unmute microphone button */}<br>        &lt;button onClick={toggleListening}&gt;<br>          &lt;Image src={isListening ? micIcon : micOffIcon}/&gt;<br>        &lt;/button&gt;<br>      &lt;/div&gt;<br>      &lt;div&gt;<br>        {/* Conversation messages */}<br>        {conversation.messages.map(message =&gt; (<br>          &lt;div&gt;{message.content}&lt;/div&gt;<br>        ))}<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  );<br>}</pre><p>It’s a very simplified version of the application code but it gives us a very clear overview of what’s happening. A few notes:</p><ul><li>We are using Refs with the useRef hook to reference DOM elements and store mutable values without triggering component re-renders. For example, wsRef stores the WebSocket instance and mediaRecorderRef stores the MediaRecorder instance. You can <a href="https://react.dev/learn/referencing-values-with-refs">learn more about Refs in the React docs</a>.</li><li>The JSX shown is minimal and doesn’t include all the application elements. It also contains no styling, as the focus here is on the component’s structure and core functionality. If you are not familiar with the JSX syntax, you can <a href="https://react.dev/learn/writing-markup-with-jsx">learn more about JSX here</a>.</li><li>The startConversation and endConversation functions orchestrate the main flow of the application by calling the necessary secondary functions.</li></ul><h4>2.2. Application State</h4><p>In a React application, the <a href="https://react.dev/learn/state-a-components-memory">state</a> is used to store and manage data that can change over time. The state of our voice assistant application is defined at the top of the VoiceAssistant component, with the useState and useReducer hooks:</p><pre>const initialConversation = { messages: [], finalTranscripts: [], interimTranscript: &#39;&#39; };<br><br>const [conversation, dispatch] = useReducer(conversationReducer, initialConversation);<br>const [isRunning, setIsRunning] = useState(false);<br>const [isListening, setIsListening] = useState(false);</pre><p>isRunning and isListening are simple boolean variables that indicate whether the conversation is currently running and whether the microphone is active (unmuted), respectively.</p><p>The conversation state variable holds all the conversation data, including the list of messages that store the conversation history as well as the final transcripts and the current interim transcript of the user&#39;s speech. This variable is managed using a reducer, with the useReducer hook.</p><p>A reducer is a function that takes the current state and an action as arguments, and returns the next state based on the action. To update the state, we use dispatch to dispatch one of the actions. This allows us to centralize and manage complex state logic outside of the component in a single function. You can <a href="https://react.dev/learn/extracting-state-logic-into-a-reducer">learn more about reducers here</a>.</p><p>A reducer is an ideal choice for the conversation state, which involves updating multiple state properties at once ( messages, finalTranscripts and interimTranscript) based on different actions (&#39;reset&#39;, &#39;transcript_interim&#39;, &#39;transcript_final&#39; and &#39;assistant&#39;). The &#39;reset&#39; action is used to reset the conversation state when a new conversation starts, and the other three actions are used to handle the corresponding JSON messages received from the server. This was all described in <a href="https://medium.com/@codeawake/ai-voice-assistant-part2-d53374332861#1c45">section 1.3 of Part 2</a>.</p><p>This is the code for the conversationReducer function:</p><pre>function conversationReducer(state, action) {<br>  switch(action.type) {<br>    // Reset the conversation state to its initial values<br>    case &#39;reset&#39;: {<br>      return { messages: [], finalTranscripts: [], interimTranscript: &#39;&#39; };<br>    }<br>    // Update the interim transcript with the latest content<br>    case &#39;transcript_interim&#39;: {<br>      return {<br>        ...state,<br>        interimTranscript: action.content<br>      };<br>    }<br>    // Append the final transcript to the list of final transcripts<br>    // and reset the interim transcript<br>    case &#39;transcript_final&#39;: {<br>      return {<br>        ...state,<br>        finalTranscripts: [...state.finalTranscripts, action.content],<br>        interimTranscript: &#39;&#39;<br>      };<br>    }<br>    // Add the user message (joining the previous final transcripts) and the<br>    // assistant message to the list of messages<br>    case &#39;assistant&#39;: {<br>      const newMessages = [];<br>      if (state.finalTranscripts.length &gt; 0) {<br>        newMessages.push({ role: &#39;user&#39;, content: state.finalTranscripts.join(&#39; &#39;) });<br>      }<br>      newMessages.push({ role: &#39;assistant&#39;, content: action.content });<br>      return {<br>        ...state,<br>        messages: [...state.messages, ...newMessages],<br>        finalTranscripts: [],<br>        interimTranscript: &#39;&#39;<br>      };<br>    }<br>    default: {<br>      return state;<br>    }<br>  }<br>}</pre><h4>2.3. WebSocket Connection</h4><p>The WebSocket connection is managed by openWebSocketConnection and closeWebSocketConnection:</p><pre>function openWebSocketConnection() {<br>  wsRef.current = new WebSocket(process.env.NEXT_PUBLIC_WEBSOCKET_URL);<br>  wsRef.current.binaryType = &#39;arraybuffer&#39;;<br><br>  function handleJsonMessage(jsonData) {<br>    const message = JSON.parse(jsonData);<br>    if (message.type === &#39;finish&#39;) {<br>      endConversation();<br>    } else {<br>      // If user interrupts while audio is playing, skip the audio currently playing<br>      if (message.type === &#39;transcript_final&#39; &amp;&amp; isAudioPlaying()) {<br>        skipCurrentAudio();<br>      }<br>      dispatch(message);<br>    }<br>  }<br>  <br>  wsRef.current.onmessage = (event) =&gt; {<br>    if (event.data instanceof ArrayBuffer) {<br>      handleAudioStream(event.data);<br>    } else {<br>      handleJsonMessage(event.data);<br>    }<br>  };<br><br>  wsRef.current.onclose = () =&gt; {<br>    endConversation();<br>  }<br>}<br><br>function closeWebSocketConnection() {<br>  if (wsRef.current) {<br>    wsRef.current.close();<br>  }<br>}</pre><p>In openWebSocketConnection, we create a new WebSocket instance using the Websocket constructor of the <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">WebSocket API</a> and store it in the wsRef ref. Once created, we can start sending data through the WebSocket using wsRef.current.send(), as we will see shortly.</p><p>The onmessage event handler is responsible for processing the data received from the server. <a href="https://medium.com/@codeawake/ai-voice-assistant-part2-d53374332861#1c45">As we saw in Part 2</a>, the server sends either JSON messages (interim transcripts, final transcripts, assistant responses or conversation end messages) or binary data (the assistant&#39;s response audio stream). We handle the received data with handleAudioStream for binary data and handleJsonMessage for JSON messages.</p><p>handleJsonMessage parses the JSON data and either ends the conversation if the message type is &#39;finish&#39; or dispatches the corresponding action to update the conversation state using the conversationReducer we have just seen. If the user interrupts the assistant while the audio is playing, we call skipCurrentAudio to stop the current audio playback.</p><p>The onclose event handler is invoked when the WebSocket connection is closed by the server, and it simply ends the conversation.</p><p>And closeWebSocketConnection is used to close the WebSocket connection when the conversation ends if it&#39;s still active.</p><h4>2.4. Microphone</h4><p>The microphone recording functionality is implemented in the startMicrophone, stopMicrophone and toggleListening functions, and uses the <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API">MediaStream Recording Web API</a>:</p><pre>async function startMicrophone() {<br>  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });<br>  mediaRecorderRef.current = new MediaRecorder(stream);<br>  mediaRecorderRef.current.addEventListener(&#39;dataavailable&#39;, e =&gt; {<br>    if (e.data.size &gt; 0 &amp;&amp; wsRef.current.readyState == WebSocket.OPEN) {<br>      wsRef.current.send(e.data);<br>    }<br>  });<br>  mediaRecorderRef.current.start(250);<br>}<br><br>function stopMicrophone() {<br>  if (mediaRecorderRef.current &amp;&amp; mediaRecorderRef.current.stream) {<br>    mediaRecorderRef.current.stop();<br>    mediaRecorderRef.current.stream.getTracks().forEach(track =&gt; track.stop());<br>  }<br>}<br><br>function toggleListening() {<br>  if (isListening) {<br>    mediaRecorderRef.current.pause();<br>  } else {<br>    mediaRecorderRef.current.resume();<br>  }<br>  setIsListening(!isListening);<br>}</pre><p>startMicrophone requests access to the user&#39;s microphone using <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia">getUserMedia</a> and creates a new <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder">MediaRecorder</a> instance with the microphone audio stream. It then adds a <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/dataavailable_event">dataavailable</a> event listener, which listens to the user&#39;s audio data received from the microphone and sends it to the server via the WebSocket connection using wsRef.current.send().</p><p>toggleListening pauses and resumes the microphone recording when the mute/unmute microphone button is clicked.</p><p>stopMicrophone stops the microphone and cleans up the resources when the conversation ends.</p><h4>2.5. Audio Player</h4><p>The audio player functionality uses the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API">Media Source Web API</a> to play the assistant’s audio response as it is streamed from the server. The main elements are:</p><ul><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaSource">MediaSource</a>: Acts as a container for the audio data. It represents the source of the media data for an HTML media element like &lt;audio&gt;.</li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer">SourceBuffer</a>: Represents a chunk of media data that is appended to the MediaSource.</li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement">Audio Element</a>: The HTML audio element that is used to play the audio data from the MediaSource.</li><li>audioDataRef: A ref that references an array used as a buffer to store the incoming audio data chunks from the server until they are appended to the SourceBuffer. This is necessary because the SourceBuffer may not be immediately ready to receive new data when the audio chunks arrive from the server.</li></ul><p>Now, let’s take a look at the startAudioPlayer function:</p><pre>function startAudioPlayer() {<br>  mediaSourceRef.current = new MediaSource();<br><br>  mediaSourceRef.current.addEventListener(&#39;sourceopen&#39;, () =&gt; {<br>    if (!MediaSource.isTypeSupported(&#39;audio/mpeg&#39;)) return;<br><br>    sourceBufferRef.current = mediaSourceRef.current.addSourceBuffer(&#39;audio/mpeg&#39;);<br>    sourceBufferRef.current.addEventListener(&#39;updateend&#39;, () =&gt; {<br>      if (audioDataRef.current.length &gt; 0 &amp;&amp; !sourceBufferRef.current.updating) {<br>        sourceBufferRef.current.appendBuffer(audioDataRef.current.shift());<br>      }<br>    });<br>  });<br><br>  const audioUrl = URL.createObjectURL(mediaSourceRef.current);<br>  audioElementRef.current = new Audio(audioUrl);<br>  audioElementRef.current.play();<br>}</pre><p>This is what is happening:</p><ol><li>A new MediaSource instance is created and an event listener is added to listen for the &#39;sourceopen&#39; event, which fires when the MediaSource is ready to receive data.</li><li>When the &#39;sourceopen&#39; event is triggered, the code checks if the &#39;audio/mpeg&#39; codec is supported and a new SourceBuffer is created and added to the MediaSource using addSourceBuffer().</li><li>An event listener is added to the SourceBuffer to listen for the &#39;updateend&#39; event, which fires whenever a SourceBuffer update operation completes and it&#39;s ready to receive new data. We then take the next audio chunk from the audio data buffer referenced by audioDataRef and append it to the SourceBuffer.</li><li>A URL is created for the MediaSource using URL.createObjectURL(), and a new Audio element is created with this URL as its src.</li><li>Finally, the Audio element starts playing using the play() method, waiting for the first audio chunks to be received from the server.</li></ol><p>Now it’s a good time to go back to two functions that we skipped earlier in the WebSocket onmessage event handler: handleAudioStream and skipCurrentAudio.</p><pre>function handleAudioStream(streamData) {<br>  audioDataRef.current.push(new Uint8Array(streamData));<br>  if (sourceBufferRef.current &amp;&amp; !sourceBufferRef.current.updating) {<br>    sourceBufferRef.current.appendBuffer(audioDataRef.current.shift());<br>  }<br>}<br><br>function skipCurrentAudio() {<br>  audioDataRef.current = [];<br>  const buffered = sourceBufferRef.current.buffered;<br>  if (buffered.length &gt; 0) {<br>    if (sourceBufferRef.current.updating) {<br>      sourceBufferRef.current.abort();<br>    }<br>    audioElementRef.current.currentTime = buffered.end(buffered.length - 1);<br>  }<br>}</pre><p>handleAudioStream is called when new audio stream data is received from the server. As mentioned earlier, rather than appending the data directly to the SourceBuffer and risk losing some chunks, we first add it to the audioDataRef array and only update SourceBuffer if it&#39;s not currently being updated. If it is, then the updateend event listener we defined earlier will make sure to update SourceBuffer once the current operation ends and it&#39;s ready to receive new data.</p><p>By using this approach, the audio data is continuously appended to the SourceBuffer as it arrives from the server, allowing for smooth playback of the audio stream. And the audioDataRef buffer ensures that no audio data is lost if the SourceBuffer is not ready to receive new data when the audio chunks arrive from the server.</p><p>The other function mentioned, skipCurrentAudio, is used to allow the user to interrupt the assistant&#39;s audio response. It just clears the audioDataRef buffer, aborts any SourceBuffer updates in progress and sets the currentTime of the Audio element forward, effectively skipping the remaining audio.</p><p>Finally, the stopAudioPlayer function simply cleans up the resources when the conversation ends:</p><pre>function stopAudioPlayer() {<br>  if (audioElementRef.current) {<br>    audioElementRef.current.pause();<br>    URL.revokeObjectURL(audioElementRef.current.src);<br>    audioElementRef.current = null;<br>  }<br><br>  if (mediaSourceRef.current) {<br>    if (sourceBufferRef.current) {<br>      mediaSourceRef.current.removeSourceBuffer(sourceBufferRef.current);<br>      sourceBufferRef.current = null;<br>    }<br>    mediaSourceRef.current = null;<br>  }<br><br>  audioDataRef.current = [];<br>}</pre><h4>2.6 Running the Code</h4><p>Now that we have completed both the backend and frontend of the voice assistant app, it’s time to finally run it. To do so, follow these steps in your terminal or command-line interface:</p><ol><li>Clone the <a href="https://github.com/ruizguille/voice-assistant">GitHub repository</a> containing the code:</li></ol><pre>git clone https://github.com/ruizguille/voice-assistant.git</pre><p>2. Run the backend server by following the <a href="https://codeawake.com/blog/ai-voice-assistant-2#running-the-code">instructions from Part 2.</a></p><p>3. In a new terminal window, navigate to the frontend folder and install the JavaScript dependencies:</p><pre>cd voice-assistant/frontend<br>npm install</pre><p>4. Create a .env file in the frontend folder copying the .env.example file provided that includes the required environment variable: NEXT_PUBLIC_WEBSOCKET_URL.</p><p>5. Start the frontend server and visit http://localhost:3000 in your browser to access the application:</p><pre>npm run dev</pre><h3>3. Conclusion</h3><p>In this 3-part series, we’ve explored a wide range of technologies (including Python, React, Groq, Deepgram, WebSockets, asynchronous programming, FastAPI, and Web APIs) to build a <strong>fully functional, real-time voice assistant that showcases the potential of AI in web development.</strong></p><p>There is, of course, room for improvement. For example, the audio player doesn’t support iOS on iPhone due to the lack of Media Source API support. And another issue is that the assistant’s audio response is occasionally captured by the microphone and interpreted as user speech, leading to unintended behavior. This is a common challenge in voice assistant systems, and addressing it could involve more advanced echo cancellation and audio processing techniques.</p><p>It’s also worth noting that the recent <a href="https://openai.com/index/hello-gpt-4o/">OpenAI Spring Update</a> introduced their new model <strong>GPT-4o</strong>, which is able to accept as input and generate as output any combination of text, audio, image and video. This could revolutionize the way voice assistants are built, reducing latency and enabling more advanced functionalities, such as considering the speaker’s tone, handling multiple speakers, and expressing emotions in the response.</p><p>I hope this series has been helpful. The code for this project is available on <a href="https://github.com/ruizguille/voice-assistant">GitHub</a> under the MIT license, so feel free to explore, experiment, and build upon it.</p><p>And if you have any questions, comments, or would like to learn more about what we do at <a href="https://codeawake.com/">CodeAwake</a>, don’t hesitate to reach out to me at <a href="/service/mailto:guillermo@codeawake.com">guillermo@codeawake.com</a>.</p><p><em>Originally published at </em><a href="https://codeawake.com/blog/ai-voice-assistant-3"><em>https://codeawake.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6041d718e0a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>