<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jayant Pal on Medium]]></title>
        <description><![CDATA[Stories by Jayant Pal on Medium]]></description>
        <link>https://medium.com/@jayant017?source=rss-782051e88a53------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Vxjvn-SahnAufzSZOUKoKQ@2x.jpeg</url>
            <title>Stories by Jayant Pal on Medium</title>
            <link>https://medium.com/@jayant017?source=rss-782051e88a53------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:12:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jayant017/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build an MCP server locally and integrate it with Claude Desktop]]></title>
            <link>https://jayant017.medium.com/build-an-mcp-server-locally-and-integrate-it-with-claude-desktop-99f69ae96e08?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/99f69ae96e08</guid>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[mcps]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 08:48:03 GMT</pubDate>
            <atom:updated>2026-03-15T08:48:03.156Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uI_lM0M6ReJVfLpi" /><figcaption>Photo by <a href="https://unsplash.com/@plhnk?utm_source=medium&amp;utm_medium=referral">Paul Hanaoka</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For a long time, AI interactions followed the same pattern — you open up a chat, ask something, get an answer. Helpful, but limited. The model had no real connection to the tools and systems you actually work with every day.</p><p>That changed with <strong>Model Context Protocol (MCP)</strong>. AI models can now go beyond just answering questions — they can connect to external tools, fetch live data, and trigger actions in a structured, predictable way. Instead of building custom integrations for every use case, developers can expose capabilities through an MCP server, and AI clients can discover and use them on the fly.</p><p>When I first came across MCP, one question immediately came to my mind : <em>how easy is it to actually build and use a local MCP Server?</em> Do you need a complex cloud setup? Do you need to containerise everything?</p><p>Turns out, it’s much simpler than it sounds.</p><p>In this article, we’ll go end-to-end — building a <strong>local MCP server </strong>from scratch<strong> </strong>and connect it to <strong>Claude Desktop</strong>. Not just a hello-world server — we’ll build something actually useful: a server that looks up official documentation for any library you ask about, scrapes the relevant content, and feeds Claude the context it needs to give you accurate, grounded answers.</p><p>By the end, you’ll have Claude talking directly to a tool running on your own machine. Let’s get into it.</p><h3><strong>Model Context Protocol</strong></h3><p><strong>MCP</strong> is an open standard introduced by Anthropic. It gives AI models a structured way to interact with external tools and data sources.</p><p>Before MCP, connecting an AI to an external tool meant building a custom integration every single time. MCP changed that. It defines a common protocol so any AI client can discover and talk to any MCP server, without any custom code.</p><p><strong>Here’s the basic idea:</strong></p><ul><li>You run an <strong>MCP server</strong> that exposes tools or data.</li><li>An <strong>MCP client</strong> (like Claude Desktop) connects to it.</li><li>The AI can now call those tools as part of a conversation — naturally, without any manual switching.</li></ul><p><strong>A quick look at how it works:</strong></p><pre>User prompt → Claude → MCP Client → MCP Server → Tool/Data → Response back to Claude</pre><p>The AI decides <em>when</em> to call a tool, <em>what</em> to pass to it, and <em>how</em> to use the result — all on its own.</p><p>This entire setup will be done in 3 phases :</p><p>Phase 1 : Build MCP Server</p><p>Phase 2 : Build &amp; Test With MCP Client</p><p>Phase 3 : Configure &amp; Test with Claude</p><blockquote><strong><em>Phase 1 : Build MCP Server</em></strong></blockquote><p>We&#39;ll build this MCP server in <strong>3 stages</strong>, each one building on the previous.</p><p><strong><em>Stage 1 </em>: Setup Web Search using Serper.</strong></p><p>We’ll use <a href="https://serper.dev/">Serper</a><strong> API</strong> to search the web. It wraps Google Search results into a clean JSON response and gives you <strong>2500 free queries</strong> to start with.</p><pre>SERPER_URL= &quot;https://google.serper.dev/search&quot;<br><br>async def search_web(query: str) -&gt; dict | None:<br>    payload = json.dumps({&quot;q&quot;: query, &quot;num&quot;: 2})<br>    headers = {<br>    &#39;X-API-KEY&#39;: os.getenv(&quot;SERPER_API_KEY&quot;),<br>    &#39;Content-Type&#39;: &#39;application/json&#39;<br>    }  <br>    async with httpx.AsyncClient() as client:<br>        response = await client.post(<br>            SERPER_URL, headers=headers, data=payload, timeout=30.0<br>        )<br>        response.raise_for_status()<br>        return response.json()</pre><ul><li>&quot;num&quot;: 2 — we only fetch the <strong>top 2 results</strong>.</li><li>httpx.AsyncClient — we use <strong>async HTTP</strong> so the server stays non-blocking.</li><li>raise_for_status() — throws an error immediately if the request fails.</li><li>Returns the raw JSON response — a list of search results with titles, URLs and snippets.</li></ul><p>This gives us the URLs we need. Stage 2 will take those URLs and scrape the actual content from them.</p><p><strong><em>Stage 2</em> : Setup Web Scraper using <em>trafilatura</em></strong></p><p>We’ll setup web scraper using <em>trafilatura</em>, a Python package built specifically for extracting clean, readable text from web pages. It strips out ads, navbars, footers — all the noise.</p><pre>def clean_html_to_txt(html: str) -&gt; str | None:<br>    try:  <br>        extracted = trafilatura.extract(<br>            html,<br>            include_comments=False,<br>            include_tables=False,<br>            favor_recall=False,<br>        )<br>        if extracted:<br>            return extracted<br>    except Exception as e:<br>        raise e<br><br>async def fetch_url(/service/url: str) -&gt; str | None:<br>    async with httpx.AsyncClient() as client:<br>        response = await client.get(url, timeout=30.0)<br>        cleaned_response = clean_html_to_txt(response.text)<br>        return cleaned_response</pre><ul><li>clean_html_to_txt— takes raw HTML and extracts only the meaningful content using trafilatura.</li><li>include_comments = False ignores comment sections</li><li>include_tables = False skips tables</li><li>favor_recall = False prioritizes precision over quantity.</li><li>fetch_url— fetches the raw HTML from a given URL asynchronously, then passes it straight to <em>clean_html_to_txt</em>.</li></ul><p>Instead of <em>trafilatura, </em>you can build an AI Web Scrapper that cleans all HTML components and returns just valid text.</p><p>Alright, together these two functions take a URL and return clean, structured text — ready to be used as context.</p><p><strong><em>Stage 3</em> : Setting up the final tool</strong></p><p>This is where everything comes together. We define the actual MCP tool that Claude will call.</p><p>We use <a href="https://gofastmcp.com/getting-started/welcome">FastMCP</a> and create an MCP server with name &quot;docs&quot;.</p><pre>from fastmcp import FastMCP<br><br>mcp = FastMCP(&quot;docs&quot;)</pre><p>Next, we define a mapping of libraries to their official documentation. For now, we will start small and give support of the following libraries to this tool.</p><pre>docs_urls = {<br>    &quot;langchain&quot;: &quot;python.langchain.com/docs&quot;,<br>    &quot;llama-index&quot;: &quot;docs.llamaindex.ai/en/stable&quot;,<br>    &quot;openai&quot;: &quot;platform.openai.com/docs&quot;,<br>    &quot;uv&quot;: &quot;docs.astral.sh/uv&quot;,<br>    &quot;claude&quot; : &quot;https://platform.claude.com/docs&quot;,<br>    &quot;groq&quot; : &quot;https://console.groq.com/docs/overview&quot;<br>}</pre><p>Now the tool itself:</p><pre>from fastmcp import FastMCP<br>mcp = FastMCP(&quot;docs&quot;)<br><br>@mcp.tool()<br>async def get_docs(query: str, library: str):<br>    &quot;&quot;&quot;<br>    Search the latest docs for a given query and library.<br>    Supports langchain, openai, llama-index and uv.<br><br>    Args:<br>        query: The query to search for (e.g. &quot;Publish a package with UV&quot;)<br>        library: The library to search in (e.g. &quot;uv&quot;)<br><br>    Returns:<br>        Summarized text from the docs with source links.<br>    &quot;&quot;&quot;<br>    if library not in docs_urls:<br>        raise ValueError(f&quot;Library {library} not supported by this tool&quot;)<br>    <br>    query = f&quot;site:{docs_urls[library]} {query}&quot;<br>    results = await search_web(query)<br><br>    if len(results[&quot;organic&quot;]) == 0:<br>        return &quot;No results found&quot;<br>    <br>    text_parts = []<br>    for result in results[&quot;organic&quot;]:<br>        link = result.get(&quot;link&quot;, &quot;&quot;)<br>        raw = await fetch_url(https://medium.com/link)<br>        if raw:<br>            labeled = f&quot;SOURCE: {link}\n{raw}&quot;<br>            text_parts.append(labeled)<br>    <br>    return &quot;\n\n&quot;.join(text_parts)</pre><ul><li>@mcp.tool() — this decorator registers get_docs as a tool that Claude can discover and call. Also make sure that the name is descriptive.</li><li><strong>The docstring matters</strong> — this isn’t just code documentation. Claude actually reads this docstring to understand what the tool does, what arguments to pass, and when to use it. This is the tool’s instruction manual for AI.</li><li><em>Library validation</em><strong><em> </em></strong>— if the library isn’t in our mapping, we fail fast with a clear error.</li><li>site:{docs_url} — we prefix the query with a Google site: filter, so Serper only returns results from that library&#39;s official documentation.</li><li>We loop through the results, scrape each URL using <em>fetch_url</em>, and label each chunk with its source link</li><li>Everything is returned as a single structured text block — Claude uses this as context to answer your query</li></ul><p>Finally, we need to run the server locally:</p><pre>def main():<br>    mcp.run(transport=&quot;stdio&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>mcp.run(transport=&quot;stdio&quot;) — runs the MCP server using stdio transport. This is the transport mode Claude Desktop expects when connecting to a locally running MCP server.</p><blockquote><strong><em>Phase 2 : Build &amp; test with MCP Client</em></strong></blockquote><p>Before connecting to Claude Desktop, let’s test our server with a simple Python MCP client.</p><p>First, tell the client where the MCP server lives:</p><pre>from mcp import StdioServerParameters<br><br>server_params = StdioServerParameters(<br>    command=&quot;uv&quot;,<br>    args=[&quot;run&quot;, &quot;mcp_server.py&quot;],<br>    env=None,<br>)</pre><p>StdioServerParameters tells the client how to <strong>launch and connect</strong> to our MCP server — in this case, by running mcp_server.py using uv.</p><p>Now, the client itself:</p><pre>from mcp.client.stdio import stdio_client<br>from mcp import ClientSession<br>import os<br>from dotenv import load_dotenv<br>from groq import Groq<br><br>def get_response_from_llm(user_prompt, system_prompt, model):<br>    api_key = os.getenv(&quot;GROQ_API_KEY&quot;)<br><br>    groq_client = Groq(api_key=api_key)<br>    chat_completion = groq_client.chat.completions.create(<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_prompt},<br>            ],<br>            model=model,<br>        )<br>    return chat_completion.choices[0].message.content<br><br>async def main():<br>    async with stdio_client(server_params) as (read_stream, write_stream):<br>        async with ClientSession(read_stream, write_stream) as session:<br><br>            await session.initialize()<br><br>            tools_response = await session.list_tools()<br>            print(&quot;Available tools:&quot;, [t.name for t in tools_response.tools])<br><br>            query = &quot;How to use chromadb with langchain?&quot;<br>            library = &quot;langchain&quot;<br>            res = await session.call_tool(&quot;get_docs&quot;, arguments={&quot;query&quot;: query, &quot;library&quot;: library})<br><br>            context = res.content<br>            user_prompt_with_context = f&quot;Query: {query}, Context: {context}&quot;<br><br>            # Get a human readable response from LLM<br>            SYSTEM_PROMPT = &quot;&quot;&quot;<br>            Answer ONLY using the provided context. If info is missing say you don&#39;t know.<br>            Keep every &#39;SOURCE:&#39; line exactly; list sources at the end.<br>            &quot;&quot;&quot;<br>            answer = get_response_from_llm(<br>                user_prompt=user_prompt_with_context,<br>                system_prompt=SYSTEM_PROMPT,<br>                model=&quot;openai/gpt-oss-20b&quot;<br>            )<br>            print(&quot;ANSWER: &quot;, answer)<br></pre><ul><li>session.initialize() — establishes the connection with the MCP server.</li><li>session.list_tools() — discovers all tools the server exposes.</li><li>session.call_tool() — calls get_docs with your query and library.</li><li>The raw context returned by the tool is then passed to an LLM with a strict <strong><em>system prompt</em></strong><em> </em>— answer only from the provided context(always cite sources).</li><li>get_response_from_llm — A simple utility function that sends a prompt to <a href="https://groq.com/"><strong>Groq’</strong></a><strong>s </strong>API(you can get API key for free)<strong> </strong>and returns the response.</li></ul><p>Let’s test it out. Run the client script and you should see something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUGRYDbyWYDaJ6cATZTzpA.png" /></figure><p>Available tools: [&#39;get_docs&#39;] — the client discovered our tool. This confirms the server and client are connected and talking to each other.</p><ul><li>The answer is structured and clean — pulled directly from LangChain’s official documentation, with clear steps and code snippets.</li></ul><blockquote><strong><em>Phase 3 : Configure &amp; Test with Claude</em></strong></blockquote><p><strong><em>Step 1 — Download Claude Desktop</em></strong></p><p>First you need to download Claude Desktop from <a href="https://claude.com/download">Claude</a>. Create an account if you don’t have one.</p><p><strong><em>Step 2 — Edit the Claude Desktop config</em></strong></p><p>Open Claude Desktop → go to <strong>Settings → Developer</strong> → click <strong>Edit Config</strong>.</p><p>This opens a JSON config file. This is where you tell Claude Desktop about your MCP server.</p><pre>{<br>  &quot;mcpServers&quot;: {<br>  &quot;mcp-docs&quot;: {<br>    &quot;command&quot; : &quot;/Users/jayant/Desktop/Workspace/Projects/MCP_AI_WebScrapping/.venv/bin/python&quot;,<br>    &quot;args&quot; : [&quot;/Users/jayant/Desktop/Workspace/Projects/MCP_AI_WebScrapping/mcp_server.py&quot;],<br>    &quot;cwd&quot; : &quot;/Users/jayant/Desktop/Workspace/Projects/MCP_AI_WebScrapping&quot;,<br>    &quot;env&quot; : {<br>      &quot;SERPER_API_KEY&quot;: &quot;your_serper_api_key&quot;,<br>      &quot;GROQ_API_KEY&quot; :  &quot;your_groq_api_key&quot;<br>      }<br>    }<br>  }<br>}</pre><ul><li>mcpServers — a dictionary of all your MCP servers. Have multiple servers? Just add more keys.</li><li>command — the Python executable from your <strong>virtual environment</strong></li><li>args — the path to your mcp_server.py</li><li>cwd — the working directory of your project. Claude Desktop will run the server from here.</li><li>env — your API keys.</li></ul><p><strong><em>Step 3 — Enable the Tool in Claude Desktop</em></strong></p><p>Once you restart Claude Desktop, click on the <strong>Connectors</strong> option in the chat interface. You should see mcp-docs listed and ready to toggle on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*grnxrVhGQF1t5-Sp8hNb3Q.png" /></figure><p>Your MCP server is now connected and active.</p><p><strong><em>Step 4 — Test it</em></strong></p><p>Ask Claude anything about a supported library. For example:</p><blockquote>“How to use Chroma DB with LangChain?”</blockquote><p>Claude will automatically recognise that it needs to call the get_docs tool. It will prompt you for permission first — click <strong>Always Allow</strong> to let it run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aRVC2uxBgDzqZ1DHLaKA-w.png" /></figure><p><strong><em>Step 5 — The Response</em></strong></p><p>Claude fetches the official LangChain docs, scrapes the content, and returns a clean, structured answer — directly from the source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drezoYOZll0_l46eBgeMdw.png" /></figure><p>And that’s it. Your local MCP server is integrated with Claude Desktop. 🎉</p><h3>Conclusion</h3><p>That’s a fully working MCP server — built locally, connected to Claude Desktop, and answering from official documentation.</p><p>What we built is simple but powerful:</p><ul><li>A <strong>web search</strong> layer that targets only official docs</li><li>A <strong>scraper</strong> that extracts clean, readable content</li><li>A <strong>tool</strong> that wires it all together and plugs directly into Claude</li></ul><p>You can extend this same pattern to build tools that interact with databases, internal APIs, file systems, or any external service. MCP makes it straightforward to expose any capability to an AI client — without complex infrastructure or custom integrations.</p><p>If you found this useful, try adding more libraries to docs_urls or building an entirely new tool on top of this setup.</p><p>The full code is available on <a href="https://github.com/jayantkp/MCP_Server_with_Claude"><strong>GitHub</strong></a>.</p><p>Let me know if you have any questions/suggestions.</p><h3>References</h3><p><a href="https://gofastmcp.com/getting-started/welcome">Welcome to FastMCP - FastMCP</a></p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99f69ae96e08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Personalised AI Agent with LangGraph, Groq, Tavily, FastAPI and Streamlit]]></title>
            <link>https://jayant017.medium.com/building-a-personalised-ai-agent-with-langgraph-groq-tavily-fastapi-and-streamlit-2e4e215e8ea9?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e4e215e8ea9</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sun, 24 Aug 2025 17:42:57 GMT</pubDate>
            <atom:updated>2025-08-24T17:42:57.395Z</atom:updated>
            <content:encoded><![CDATA[<p>AI agents have taken the world by storm in 2025. But what exactly are AI agents and why are they generating so much excitement and traction across industries?</p><p>In this article, we will learn about what an AI agent is and how to build an end-to-end production ready AI Agent using LangGraph, setup the backend using FastAPI and deploy it using Streamlit.</p><blockquote><strong>So what exactly are AI Agents?</strong></blockquote><p>In essence, an AI agent is an intelligent software system that is capable enough to take decisions and actions autonomously to accomplish its tasks or goals. Unlike traditional software that follows strict instructions, these systems are smart enough to perceive their environment, and adapt dynamically to changing environments without needing constant human input.</p><p>Before diving into AI Agents, it’s important to understand a key complementary terminology known as <strong>Retrieval Augmented Generation</strong> or RAG in short. RAG enhances the capabilities of LLMs by integrating them with powerful retrieval systems that can fetch up-to-date and relevant information from external data sources.</p><p>For more information on RAG and how to leverage it to build an AI application, you can refer to my <a href="https://medium.com/@jayant017/leveraging-rag-for-chatbot-development-using-gemma-2b-it-faiss-vectordb-and-streamlit-689ee7d060f9">article</a>.</p><blockquote><strong><em>LangGraph</em></strong></blockquote><p>LangGraph is an open-source framework created by LangChain designed specifically for building, managing and scaling complex AI agent workflows. LangGraph allows us to connect LLMs to connect to various tools and APIs directly using it’s graph based workflows.</p><p>Here I am using <strong><em>Groq </em></strong>as the LLM provider integrated with LangGraph for efficient language understanding and generation. Groq supports a wide variety of LLMs, including open-weight versions of Meta’s Llama 3.</p><p>Additionally I will also be using <strong><em>Tavily </em></strong>to connect my LLM with external tools and data sources. It is a powerful web access layer designed specifically for AI Agents, offering scalable APIs for real-time search.</p><p><em>We will develop this AI agent into 3 different phases:</em></p><p><strong><em>Phase 1</em></strong>: Creating the AI Agent using LangGraph</p><p><strong><em>Phase 2</em></strong>: Setting up the backend using FastAPI and Pydantic schema</p><p><strong><em>Phase 3</em></strong>: Setting up the frontend with Streamlit.</p><p>The architecture of the entire workflow looks like this -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJETg4lVmE4NTb7GKdN51Q.png" /></figure><blockquote><strong>Phase 1</strong></blockquote><p>In this phase, first we will setup the LLM and tools. We would also need the API keys for <a href="https://groq.com/">Groq</a> and <a href="https://www.tavily.com/">Tavily</a> which are available for free in their website.</p><pre># Setting up the LLM and tools<br>from langchain_groq import ChatGroq<br>from langchain_community.tools.tavily_search import TavilySearchResults</pre><p>Next we will setup the AI Agent with search tool functionality using LangGraph.</p><pre>from langgraph.prebuilt import create_react_agent<br>from langchain_core.messages.ai import AIMessage<br><br>def get_response_from_agent(llm_id, query, allow_search, system_prompt, provider):<br>    <br>    if provider == &#39;Groq&#39;:<br>        llm = ChatGroq(model=llm_id)<br><br>    tools = [TavilySearchResults(max_results = 2)] if allow_search else []<br><br>    agent = create_react_agent(<br>        model = llm,<br>        tools = tools,<br>        prompt = system_prompt<br>    )<br>    <br>    state = {&quot;messages&quot;:query}<br>    response = agent.invoke(state)<br>    messages  = response.get(&quot;messages&quot;)<br>    ai_messages = [message.content for message in messages if isinstance(message,AIMessage)]<br>    return ai_messages[-1]</pre><p>Here we are creating a function that defines the llm based on the provider. We also connect the llm with internet for search functionality using Tavily. We can also connect our own custom tools with the agent.</p><blockquote><strong>Phase 2</strong></blockquote><p>Next we will setup the backend using FastAPI and Pydantic for schema validation. This is to ensure that the agent response always follows a proper structure and standard.</p><p>First we need to import a <em>BaseModel</em> from Pydantic. Next we create a class <em>RequestState</em> that inherits the property of <em>BaseModel</em>. We then define the structure that will be used by the back-end to validate once it gets input from front-end payload.</p><pre>from pydantic import BaseModel<br>from typing import List<br><br>class RequestState(BaseModel):<br>    model_name : str<br>    model_provider : str<br>    system_prompt : str<br>    messages : List[str]<br>    allow_search : bool</pre><p>Next we need to setup an endpoint which will enable the front-end to send the payload to the back-end.</p><p>Here we create an instance for the endpoint. We then create a decorator that receives a post request from the front-end and returns a response using the response functionality that we created in the previous step.</p><pre>from fastapi import FastAPI<br>from ai_agent import get_response_from_agent<br><br>ALLOWED_MODEL_NAMES = [&quot;llama-3.1-8b-instant&quot;,&quot;gemma2-9b-it&quot;,&quot;llama-3.3-70b-versatile&quot;]<br><br>app = FastAPI(title=&quot;LangGraph AI Agent&quot;)<br><br>@app.post(&quot;/chat&quot;)<br>def chat_endpoint(request : RequestState):<br>    if request.model_name not in ALLOWED_MODEL_NAMES:<br>        return {&quot;error&quot;:&quot;Invalid model name. Kindly select a valid AI model&quot;}<br>    <br>    llm_id = request.model_name<br>    query = request.messages<br>    allow_search = request.allow_search<br>    system_prompt = request.system_prompt<br>    provider = request.model_provider<br><br>    response = get_response_from_agent(llm_id, query, allow_search, system_prompt, provider)<br>    return response<br><br>if __name__ == &quot;__main__&quot;:<br>    import uvicorn<br>    uvicorn.run(app, host = &quot;127.0.0.1&quot;, port = 1200)</pre><p>We finally host this endpoint using <em>uvicorn</em>.</p><blockquote><strong>Phase 3</strong></blockquote><p>Finally we setup the front-end using Streamlit. We setup the page config, title and other functionalities for the UI. Here we have the option to select 3 models — two from Llama and one from Gemma.</p><p>We first take-in the user input to define the Agent role that he/she wants. Next the option for agent to allow web search or not and finally the user query.</p><pre>import streamlit as st<br>import requests<br><br>st.set_page_config(page_title = &quot;LangGraph Agentic AI&quot;, layout=&quot;centered&quot;)<br>st.title(&quot;AI Chatbot Agent&quot;)<br><br>system_prompt = st.text_area(&quot;Define your Agent&quot;, height = 70, placeholder=&quot;Enter your system prompt here&quot;)<br><br>MODEL_NAMES_GROQ = [&quot;llama-3.1-8b-instant&quot;,&quot;gemma2-9b-it&quot;,&quot;llama-3.3-70b-versatile&quot;]<br><br>provider = &quot;Groq&quot;<br>selected_model = st.selectbox(&quot;Select Groq Model&quot;,MODEL_NAMES_GROQ)<br><br>allow_web_search = st.checkbox(&quot;Allow Web Search&quot;)<br><br>user_query = st.text_area(&quot;Enter your query&quot;, height=150, placeholder=&quot;Ask anything..&quot;)</pre><p>Finally, we connect this UI with the back-end using the endpoint we created in the previous step.</p><pre>API_URL = &quot;http://127.0.0.1:1200/chat&quot;<br><br>if st.button(&quot;Ask Agent&quot;):<br><br>    if user_query.strip():<br>        payload = {&quot;model_name&quot;:selected_model,<br>                   &quot;model_provider&quot;:provider,<br>                   &quot;system_prompt&quot;:system_prompt,<br>                   &quot;messages&quot;:[user_query],<br>                   &quot;allow_search&quot;:allow_web_search<br>                   }<br>        response = requests.post(API_URL,json = payload<br>        if response.status_code == 200:<br>            response_data = response.json()<br>            if &quot;error&quot; in response_data:<br>                st.error(response_data[&#39;error&#39;])<br>            else:<br>                st.subheader(&quot;Agent Response&quot;)<br>                st.markdown(response_data)</pre><p>This is how the final UI looks like. As we can see, we can define the agent role, select the model we want, allow web search and finally enter the user query to get the response from the agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LxpVkc6pPyu2rMDvlBPWQA.png" /></figure><h3><strong>Conclusion</strong></h3><p>We have successfully created a personalised AI agent using LangGraph that can handle perform multiple actions based on the user query. The code can be found at my <a href="https://github.com/jayantkp/LangGraph_AIAgent">Github</a>.</p><p>Let me know if you have any questions/suggestions.</p><h3><strong>References</strong></h3><p><a href="https://www.langchain.com/langgraph">LangGraph: Agent Orchestration Framework for Reliable AI Agents</a></p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e4e215e8ea9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG using LangChain : Part 5-Hypothetical Document Embeddings(HyDE)]]></title>
            <link>https://jayant017.medium.com/rag-using-langchain-part-5-hypothetical-document-embeddings-hyde-050f57dfc252?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/050f57dfc252</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[hyde]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Mon, 24 Jun 2024 08:07:33 GMT</pubDate>
            <atom:updated>2024-06-24T08:07:33.165Z</atom:updated>
            <content:encoded><![CDATA[<h3>RAG using LangChain : Part 5-Hypothetical Document Embeddings(HyDE)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SByf-xDBfXHuBFqX" /><figcaption>Photo by <a href="https://unsplash.com/@donramxn?utm_source=medium&amp;utm_medium=referral">Ramón Salinero</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>Retrievers</strong></blockquote><p>In the previous articles, we discussed about various retrievers that can be used in a RAG pipeline. In retrieval process, we are comparing the query embeddings with the document embeddings and retrieving the most similar documents. Essentially, we are matching the query with the content that has the information that can answer the query. So it’s a question-answer comparison.</p><p>Now, there can be several issues that can occur during retrieval process.</p><ul><li>The document base may be too big and hence the retrieved documents might not contain important information.</li><li>If the retriever itself is not good, the LLM will either hallucinate or not respond to the user query.</li><li>Since we are comparing question to answer, the query embedding sometimes might not be sufficient enough to retrieve the correct documents.</li></ul><p>So, to ensure that the retrieval process is effective, instead of question-answer comparison, we need to have answer-answer comparison. This is where <strong>Hypothetical Document Embeddings</strong> can be used.</p><h3>What is Hypothetical Document Embedding(HyDE) ?</h3><p>HyDE uses a LLM to generate a “fake” hypothetical document for a given user query. It then embeds the document which is then used to look up for real documents that are similar to the hypothetical document.</p><p>The underlying concept here is that the hypothetical document may be closer to the real documents in the embedding space than the query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/0*P4T6hzUSgeyXCtcD.png" /><figcaption>Source: <a href="https://wfhbrian.com/revolutionizing-search-how-hypothetical-document-embeddings-hyde-can-save-time-and-increase-productivity/">https://wfhbrian.com/revolutionizing-search-how-hypothetical-document-embeddings-hyde-can-save-time-and-increase-productivity/</a></figcaption></figure><p><strong>HyDE in practice using LangChain</strong></p><p>Import all necessary dependencies.</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain_community.document_loaders import WikipediaLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.embeddings import HuggingFaceBgeEmbeddings<br>from langchain.vectorstores import Chroma</pre><p>Setting up the LLM and creating a document using <em>WikipediaLoader</em>, loading and splitting it using <em>RecursiveCharacterTextSplitter</em></p><pre>import os<br><br>with open(&#39;../../openai_api_key.txt&#39;) as f:<br>    api_key = f.read()<br>os.environ[&#39;OPENAI_API_KEY&#39;] = api_key<br><br>chat = ChatOpenAI()<br><br>chunk_size = 300<br>chunk_overlap = 100<br><br># loading data<br>loader = WikipediaLoader(query=&quot;Steve Jobs&quot;, load_max_docs=5)<br>documents = loader.load()<br><br># text splitting<br>text_splitter = RecursiveCharacterTextSplitter(chunk_size = chunk_size, chunk_overlap = chunk_overlap)<br>docs = text_splitter.split_documents(documents=documents)</pre><p>Let’s set up the embedding model</p><pre>embedding_function = HuggingFaceBgeEmbeddings(<br>    model_name = &quot;BAAI/bge-small-en-v1.5&quot;,<br>    model_kwargs = {&#39;device&#39;:&#39;cpu&#39;},<br>    encode_kwargs = {&#39;normalize_embeddings&#39;:True}<br>)</pre><p>Now let’s create a vector store and set up the base retriever using it.</p><pre># creating vector store<br>db = Chroma.from_documents(documents = docs,embedding=embedding_function,persist_directory = &quot;output/steve_jobs_for_hyde.db&quot;)<br><br># create the retriever<br>base_retriever = db.as_retriever(search_kwargs = {&quot;k&quot;:5})</pre><p>Creating a prompt template for generating HyDE</p><pre>from langchain.prompts.chat import SystemMessagePromptTemplate, ChatPromptTemplate<br><br>def get_hypo_doc(query):<br>    template = &quot;&quot;&quot;Imagine you are an expert writing a detailed explanation on the topic: &#39;{query}&#39;<br>    Your response should be comprehensive and include all key points that would be found in the top search result.&quot;&quot;&quot;<br><br>    system_message_prompt = SystemMessagePromptTemplate.from_template(template = template)<br>    chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt])<br>    messages = chat_prompt.format_prompt(query = query).to_messages()<br>    response = chat(messages = messages)<br>    hypo_doc = response.content<br>    return hypo_doc</pre><p>Getting the relevant Hypothetical documents</p><pre>query = &#39;When was Steve Jobs fired from Apple?&#39;<br>print(get_hypo_doc(query=query))<br><br><br># OUTPUT<br><br>Steve Jobs was fired from Apple on September 17, 1985. The decision to remove Jobs from his position as the head of the Macintosh division was made by then-CEO John Sculley, who had been brought in by Jobs himself to help run the company. Jobs&#39; management style and clashes with other executives led to his ousting from the company he co-founded.<br>Following his departure from Apple, Jobs went on to found NeXT Inc., a computer platform development company, and later acquired The Graphics Group, which would eventually become Pixar Animation Studios. Jobs returned to Apple in 1997 when the company acquired NeXT, and he eventually became CEO once again, leading the company to become one of the most successful tech companies in the world.<br>The date of Steve Jobs&#39; firing from Apple, September 17, 1985, is seen as a pivotal moment in his career and the history of Apple Inc. It marked a period of struggle for both Jobs and the company, but ultimately led to Jobs&#39; growth as a leader and his eventual triumphant return to Apple.</pre><p>We will be using this hypothetical answer to answer the user query.</p><pre>matched_doc = base_retriever.get_relevant_documents(query = get_hypo_doc(query))<br>print(matched_doc)<br><br># OUTPUT<br>[Document(page_content=&quot;In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets,&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Steve_Jobs&#39;, &#39;summary&#39;: &#39;Steven Paul Jobs (February 24, 1955 – October 5, 2011) was an American businessman, inventor, and investor best known for co-founding the technology company Apple Inc. Jobs was also the founder of NeXT and chairman and majority shareholder of Pixar. He was a pioneer of the personal computer revolution of the 1970s and 1980s, along with his early business partner and fellow Apple co-founder Steve Wozniak.\nJobs was born in San Francisco in 1955 and adopted shortly afterwards. He attended Reed College in 1972 before withdrawing that same year. In 1974, he traveled through India, seeking enlightenment before later studying Zen Buddhism. He and Wozniak co-founded Apple in 1976 to further develop and sell Wozniak\&#39;s Apple I personal computer. Together, the duo gained fame and wealth a year later with production and sale of the Apple II, one of the first highly successful mass-produced microcomputers. Jobs saw the commercial potential of the Xerox Alto in 1979, which was mouse-driven and had a graphical user interface (GUI). This led to the development of the unsuccessful Apple Lisa in 1983, followed by the breakthrough Macintosh in 1984, the first mass-produced computer with a GUI. The Macintosh launched the desktop publishing industry in 1985 with the addition of the Apple LaserWriter, the first laser printer to feature vector graphics and PostScript.\nIn 1985, Jobs departed Apple after a long power struggle with the company\&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer graphics division of Lucasfilm that eventually spun off independently as Pixar, which produced the first 3D computer-animated feature film Toy Story (1995) and became a leading animation studio, producing over 27 films since.\nIn 1997, Jobs returned to Apple as CEO after the company\&#39;s acquisition of NeXT. He was largely responsible for reviving Apple, which was on the verge of bankruptcy. He worked closely with British designer Jony Ive to develop a line of products and services that had larger cultural ramifications, beginning with the &quot;Think different&quot; advertising campaign, and leading to the iMac, iTunes, Mac OS X, Apple Store, iPod, iTunes Store, iPhone, App Store, and iPad. In 2003, Jobs was diagnosed with a pancreatic neuroendocrine tumor. He died of respiratory arrest related to the tumor in 2011; in 2022, he was posthumously awarded the Presidential Medal of Freedom.&#39;, &#39;title&#39;: &#39;Steve Jobs&#39;})]</pre><p>We can see that using this hypothetical answer, the LLM can generate a correct answer to the user query without hallucination. Now, all the steps that we performed is a manual process to create Hypothetical Embeddings, wherein we are defining a prompt template to define hypothetical answer and then performing similarity search between the answer and the document chunks.</p><p>We can also create Hypothetical Documents using LangChain’s predefined functions.</p><p><strong>HyDE from Chains</strong></p><pre>from langchain.chains import HypotheticalDocumentEmbedder<br><br>hyde_embedding_function = HypotheticalDocumentEmbedder.from_llm(llm = chat, base_embeddings = embedding_function, prompt_key = &#39;web_search&#39; )</pre><p>The <em>HypotheticalDocumentEmbedder </em>class takes care of creating hypothetical answers, embedding them and retrieving similar chunks.</p><p>Default prompts: <em>[‘web_search’, ‘sci_fact’, ‘arguana’, ‘trec_covid’, ‘fiqa’, ‘dbpedia_entity’, ‘trec_news’, ‘mr_tydi’]</em></p><ul><li><em>web_search</em>: This key is likely used for general web search tasks where the goal is to retrieve the most relevant documents from the web based on a user’s query.</li><li><em>sci_fact</em>: This could be related to scientific fact verification, where the system retrieves documents that can confirm or refute a scientific claim.</li></ul><p>Creating the database with Hypothetical Document Embedding function</p><pre>doc_db = Chroma.from_documents(docs, hyde_embedding_function,persist_directory=&#39;output/steve_job_hyde_chains&#39;)</pre><p>Getting the matched documents..</p><pre>matched_docs_new = doc_db.similarity_search(query)<br><br>for doc in matched_docs_new:<br>    print(doc.page_content)<br>    print(&#39; &#39;)<br><br><br># OUTPUT<br><br>In 1997, Jobs returned to Apple as CEO after the company&#39;s acquisition of NeXT. He was largely responsible for reviving Apple, which was on the verge of bankruptcy. He worked closely with British designer Jony Ive to develop a line of products and services that had larger cultural ramifications,<br> <br>In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets,<br> <br>On October 5, 2011, at the age of 56, Steve Jobs, the CEO of Apple, died due to complications from a relapse of islet cell neuroendocrine pancreatic cancer. Powell Jobs inherited the Steven P. Jobs Trust, which as of May 2013 had a 7.3% stake in The Walt Disney Company worth about $12.1 billion,<br> <br>conducted by Sorkin. The film covers fourteen years in the life of Apple Inc. co-founder Steve Jobs, specifically ahead of three press conferences he gave during that time - the formal unveiling of the Macintosh 128K on January 24, 1984; the unveiling of the NeXT Computer on October 12, 1988; and</pre><p><strong>Conclusion</strong></p><p><strong>Hypothetical Document Embeddings)HyDE </strong>overcomes the limitations of RAG by improving the retrieval accuracy and reducing hallucinations. It works better than other retrievers in most the cases only when the LLM has some knowledge about the asked question. If the LLM has no clue of the asked question, the results can be quite messy. So, look at the responses of the LLM for the sample set of the questions before proceeding with it.</p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me</a> on LinkedIn if you have any questions.</p><p><strong>References</strong></p><p><a href="https://python.langchain.com/v0.1/docs/templates/hyde/">https://python.langchain.com/v0.1/docs/templates/hyde</a>/</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=050f57dfc252" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Q&A Chatbot using OpenAI, LangChain, ChromaDB and Gradio]]></title>
            <link>https://jayant017.medium.com/rag-q-a-chatbot-using-openai-langchain-chromadb-and-gradio-536945dd92f9?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/536945dd92f9</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Wed, 05 Jun 2024 11:58:26 GMT</pubDate>
            <atom:updated>2024-06-05T11:58:26.613Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5c5iD_hb4DRVWDXU" /><figcaption>Photo by <a href="https://unsplash.com/@inakihxz?utm_source=medium&amp;utm_medium=referral">Iñaki del Olmo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Retrieval-Augmented Generation(</strong>RAG) emerges as a promising approach that handles the limitations of Large Language Models(LLMs) mainly hallucinating information and inconsistent outputs. Using RAG, we can give the model access to specific information that can be used by the model as context to generate responses that are more factually correct and more consistent.</p><p>Let’s take a look at the RAG pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*Ch9FXZgeanLjSOZY.png" /></figure><p>The process begins with a data source which can be text data such as PDFs that are loaded and stored as documents. During preprocessing, the documents are segmented or chunked into pieces that align with the model’s context window.</p><p>The segmented documents are fed into an embedding model, which generates vector representations of the text. These vectors are then saved in a vector database that can be used during the generation process.</p><p>Upon receiving a user query, the system employs the same embedding model to transform the query into a vector representation. This vector is then used to search the vector database for entries containing similar vector representations, indicating potentially relevant information.</p><p>The <strong><em>retrieval </em></strong>step in RAG essentially identifies relevant information from the vector database. We will talk more about this step as this forms an essential component in RAG.</p><p>The retrieved information becomes the context that provides the LLM with additional knowledge relevant to the user’s query. Finally, the original query is <strong><em>augmented</em></strong>, or enriched, with this context before being fed to the LLM for response <strong><em>generation</em></strong>, which ensures that the output is factually more correct.</p><blockquote><strong>What is Gradio?</strong></blockquote><p>Gradio is an open source Python library that simplifies the process of creating user interfaces for ML models, APIs, etc. In just a few lines of code, we can build a web interface that allows people to interact with the model.</p><p>Let’s get our hands dirty and start building a Q&amp;A chatbot using RAG capabilities.</p><p>The first step is to import all necessary dependencies.</p><pre>import gradio as gr<br>import os<br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.embeddings import HuggingFaceEmbeddings<br>from langchain.chat_models import ChatOpenAI<br><br>from langchain.retrievers.document_compressors import LLMChainExtractor<br>from langchain.retrievers.multi_query import MultiQueryRetriever<br>from langchain.retrievers import ContextualCompressionRetriever<br>from langchain.prompts.chat import ChatPromptTemplate, HumanMessagePromptTemplate<br>from langchain.vectorstores import Chroma</pre><p>We will discuss about all the dependencies as we move forward.</p><p>We would need to have the OpenAI API key to call the <strong><em>ChatOpenAI </em></strong>model. You can get it in their website.</p><p>Let’s create an instance of the chat model. Also let’s define the embedding function that we will be using to store the vector representations in the vector store.</p><p>BGE models on HuggingFaceare one of the best open source embedding models.</p><pre>chat = ChatOpenAI()<br>embedding_function = HuggingFaceEmbeddings(model_name = &quot;BAAI/bge-large-en-v1.5&quot;,model_kwargs={&#39;device&#39;: &#39;cpu&#39;},encode_kwargs={&quot;normalize_embeddings&quot;: True})</pre><p>Now let’s define a function <em>add_docs()</em> that would take in a file(say a PDF), load and split it using <a href="https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html"><em>RecursiveCharacterTextSplitter</em></a> with chunk size 500 and chunk overlap 100. Then it would embed the documents using the embedding function and store it in Chroma VectorDB.</p><p>We will be using the same database during generation.</p><pre>def add_docs(path):<br><br>    loader = PyPDFLoader(file_path=path)<br>    docs = loader.load_and_split(text_splitter=RecursiveCharacterTextSplitter(chunk_size = 500,<br>                                                                                chunk_overlap = 100,<br>                                                                                length_function = len,<br>                                                                                is_separator_regex=False))<br>    model_vectorstore = Chroma<br>    db = model_vectorstore.from_documents(documents=docs,embedding= embedding_function, persist_directory=&quot;output/general_knowledge&quot;)<br>    return db</pre><blockquote><strong>Retrieval Process</strong></blockquote><p>For the retrieval process, we will use a combination of Contextual Compressor and Multi Query Retriever. I have discussed about these in my <a href="https://jayant017.medium.com/rag-using-langchain-part-4-retrievers-79908145e28c">article</a>.</p><p>We will define a function that first loads the vector store. It takes in the query from the user, creates the context using the query, augments the context to the prompt and generates a response. Since we are creating a chatbot, we also append the response to the chat history.</p><p>We use prompting techniques in LangChain to create the prompt templates for generation.</p><pre>def answer_query(message, chat_history):<br>    base_compressor = LLMChainExtractor.from_llm(chat)<br>    db = Chroma(persist_directory = &quot;output/general_knowledge&quot;, embedding_function=embedding_function)<br>    base_retriever = db.as_retriever()<br>    mq_retriever = MultiQueryRetriever.from_llm(retriever = base_retriever, llm=chat)<br>    compression_retriever = ContextualCompressionRetriever(base_compressor=base_compressor, base_retriever=mq_retriever)<br><br>    matched_docs = compression_retriever.get_relevant_documents(query = message)<br><br>    context = &quot;&quot;<br><br>    for doc in matched_docs:<br>        page_content = doc.page_content<br>        context+=page_content<br>        context += &quot;\n\n&quot;<br>    template = &quot;&quot;&quot;<br>    Answer the following question only by using the context given below in the triple backticks, do not use any other information to answer the question.<br>    If you can&#39;t answer the given question with the given context, you can return an emtpy string (&#39;&#39;)<br>    Context: ```{context}```<br>    ----------------------------<br>    Question: {query}<br>    ----------------------------<br>    Answer: &quot;&quot;&quot;<br>    <br>    human_message_prompt = HumanMessagePromptTemplate.from_template(template=template)<br>    chat_prompt = ChatPromptTemplate.from_messages([human_message_prompt])<br>    prompt = chat_prompt.format_prompt(query = message, context = context)<br>    response = chat(messages=prompt.to_messages()).content<br><br>    chat_history.append((message,response))<br>    return &quot;&quot;, chat_history</pre><p>With all done, let’s define a function that creates the UI where in we first upload a file. We also create a textbox wherein the user enters a query and finally the <em>answer_query()</em> function which returns the response and adds it to the chat history.</p><pre>with gr.Blocks() as demo:<br>    gr.HTML(&quot;&lt;h1 align = &#39;center&#39;&gt;Smart Assistant&lt;/h1&gt;&quot;)<br><br>    with gr.Row():<br><br>        upload_files = gr.File(label = &#39;Upload a PDF&#39;,file_types=[&#39;.pdf&#39;],file_count=&#39;single&#39;)<br><br>    chatbot = gr.Chatbot()<br>    msg = gr.Textbox(label = &quot;Enter your question here&quot;)<br>    upload_files.upload(add_docs,upload_files)<br>    msg.submit(answer_query,[msg,chatbot],[msg,chatbot])</pre><p>After testing this, we can deploy it to <a href="https://huggingface.co/spaces">Spaces</a> by HuggingFace. We first create a Space and next we need to upload two files: app.pyand requirements.txtin the files section of the space. Now, if the build gets successful, we will have our interface ready.</p><p>This is how the interface looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIARwwrSgh3iy0-uKLBHuQ.png" /></figure><p>You can try out the app here:</p><p><a href="https://huggingface.co/spaces/jayant012/Smart-Assistant">Smart Assistant - a Hugging Face Space by jayant012</a></p><p>I have added all the scripts here.</p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=536945dd92f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG using LangChain : Part 4-Retrievers]]></title>
            <link>https://jayant017.medium.com/rag-using-langchain-part-4-retrievers-79908145e28c?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/79908145e28c</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Tue, 16 Apr 2024 05:38:03 GMT</pubDate>
            <atom:updated>2024-04-16T05:38:03.466Z</atom:updated>
            <content:encoded><![CDATA[<h3>RAG using LangChain : Part 4-Retrievers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HbBHw-cU5L9CLJVes1j5A.png" /></figure><p>In the previous <a href="https://medium.com/@jayant017/rag-using-langchain-part-3-vector-stores-and-retrievers-a75f4d14cbf3">article</a>, we touched upon <strong><em>Vector Stores</em></strong> and <strong><em>Retrievers</em></strong>. In this article, we will briefly discuss about the different types of retrievers in LangChain.</p><p>A retriever is an interface that returns documents given an unstructured query. It does not have to store documents like <em>Vector Store</em>. Retrievers accept a string query as an input and return a list of Documents as an output. I talked about Vector-Store retriever and BM-25 Retriever in the previous article. Let’s explore few other retrievers.</p><blockquote><strong>Semantic Retrievers</strong></blockquote><p>Semantic Retrievers focus on understanding the underlying context of a query and documents in order to retrieve the relevant information from the database. Semantic Retrievers leverage word embeddings and sentence encoders to capture the semantic meaning of the text. Let’s look into few of these.</p><ol><li><strong>Multi Query Retriever</strong></li></ol><p>MultiQueryRetriever automates the process of prompt tuning. As the name suggests, it essentially uses an LLM to generate multiple queries for a given user input query. For each of the query, it retrieves a set of relevant documents and taken a union across all queries to get a larger set of relevant documents. Let’s look into it’s functionality.</p><p>Import all necessary dependencies.</p><pre>import chromadb<br>from dotenv import load_dotenv<br>from langchain.chat_models import ChatOpenAI<br>from langchain_community.document_loaders import WikipediaLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.embeddings import HuggingFaceBgeEmbeddings<br>from langchain.vectorstores import Chroma</pre><p>Let’s create a document using <em>WikipediaLoader</em>, split it, and embed using an embedding function and create a <em>Chroma DB</em>.</p><pre>chunk_size = 400<br>chunk_overlap = 100<br><br># loading environment variables<br>#load_dotenv()<br><br>import os<br>with open(&#39;../../openai_api_key.txt&#39;) as f:<br>    api_key = f.read()<br>os.environ[&#39;OPENAI_API_KEY&#39;] = api_key<br><br># loading chat model<br>chat = ChatOpenAI()<br><br># loading data<br>loader = WikipediaLoader(query=&quot;Steve Jobs&quot;, load_max_docs=5)<br>documents = loader.load()<br><br># text splitting<br>text_splitter = RecursiveCharacterTextSplitter(chunk_size = chunk_size, chunk_overlap = chunk_overlap)<br>docs = text_splitter.split_documents(documents=documents)<br><br># embedding function<br>embedding_function = HuggingFaceBgeEmbeddings(<br>    model_name = &quot;BAAI/bge-large-en-v1.5&quot;,<br>    model_kwargs = {&#39;device&#39;:&#39;cpu&#39;},<br>    encode_kwargs = {&#39;normalize_embeddings&#39;:True}<br>)<br><br># vector store<br>db = Chroma.from_documents(docs, embedding_function, persist_directory=&quot;output/steve_jobs.db&quot;)</pre><p>Creating an instance of the MultiQueryRetriever. We need to pass the vector database and the LLM that we are using for query generation.</p><pre>from langchain.retrievers.multi_query import MultiQueryRetriever<br>mq_retriever = MultiQueryRetriever.from_llm(retriever = db.as_retriever(), llm = chat)</pre><pre>query = &quot;When was Steve Jobs fired from Apple?&quot;<br>retrieved_docs = mq_retriever.get_relevant_documents(query=query)<br>retrieved_docs<br><br># [Document(page_content=&#39;On October 5, 2011, at the age of 56, Steve Jobs, the CEO of Apple, died due to complications from a relapse of islet cell neuroendocrine pancreatic cancer. Powell Jobs inherited the Steven P. Jobs Trust, which as of May 2013 had a 7.3% stake in The Walt Disney Company worth about $12.1 billion, and 38.5 million shares of Apple Inc.As of July 2020, Powell Jobs and her family were ranked 59th in&#39;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Laurene_Powell_Jobs&#39;, &#39;summary&#39;: &#39;Laurene Powell Jobs (née Powell; born November 6, 1963) is an American billionaire businesswoman and executive. She is the widow of Steve Jobs, co-founder and former CEO of Apple Inc., and she manages the Steve Jobs Trust. She is the founder and chair of Emerson Collective and XQ Institute. She is a major donor to Democratic Party politicians.&#39;, &#39;title&#39;: &#39;Laurene Powell Jobs&#39;}),<br># Document(page_content=&quot;Apple CEO John Sculley demands to know why the world believes he fired Jobs – Jobs was actually forced out by the Apple board, who were resolute on updating the Apple II following the Macintosh&#39;s lackluster sales. Despite Sculley&#39;s warnings, Jobs criticized the decision and dared them to cast a final vote on his tenure. After Hoffman and Jobs discuss NeXT&#39;s unclear direction, she realizes Jobs&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Steve_Jobs_(film)&#39;, &#39;summary&#39;: &quot;Steve Jobs is a 2015 biographical drama film directed by Danny Boyle and written by Aaron Sorkin. A British-American co-production, it was adapted from the 2011 biography by Walter Isaacson and interviews conducted by Sorkin. The film covers fourteen years in the life of Apple Inc. co-founder Steve Jobs, specifically ahead of three press conferences he gave during that time - the formal unveiling of the Macintosh 128K on January 24, 1984; the unveiling of the NeXT Computer on October 12, 1988; and the unveiling of the iMac G3 on May 6, 1998. Jobs is portrayed by Michael Fassbender, with Kate Winslet as Joanna Hoffman and Seth Rogen, Katherine Waterston, Michael Stuhlbarg, and Jeff Daniels in supporting roles.\nDevelopment began in 2011 after the rights to Isaacson&#39;s book were acquired. Filming began in January 2015. A variety of actors were considered and cast before Fassbender eventually took the role. Editing was extensive on the project, with editor Elliot Graham starting while the film was still shooting. Daniel Pemberton served as composer, with a focus on dividing the score into three distinguishable sections.\nSteve Jobs premiered at the 2015 Telluride Film Festival on September 5, 2015, and began a limited release in New York City and Los Angeles on October 9, 2015. It opened nationwide in the U.S. on October 23, 2015, to widespread critical acclaim, with Boyle&#39;s direction, visual style, Sorkin&#39;s screenplay, musical score, cinematography, editing and the acting of Fassbender and Winslet garnering unanimous acclaim. However, it was a financial disappointment, grossing only $34 million worldwide against a budget of $30 million. People close to Jobs such as Steve Wozniak and John Sculley praised the performances, but the film also received criticism for historical inaccuracy. Steve Jobs was nominated for Best Actor (Fassbender) and Best Supporting Actress (Winslet) at the 88th Academy Awards, and received numerous other accolades.&quot;, &#39;title&#39;: &#39;Steve Jobs (film)&#39;}),<br># Document(page_content=&quot;In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Steve_Jobs&#39;, &#39;summary&#39;: &#39;Steven Paul Jobs (February 24, 1955 – October 5, 2011) was an American businessman, inventor, and investor best known for co-founding the technology giant Apple Inc. Jobs was also the founder of NeXT and chairman and majority shareholder of Pixar. He was a pioneer of the personal computer revolution of the 1970s and 1980s, along with his early business partner and fellow Apple co-founder Steve Wozniak.\nJobs was born in San Francisco in 1955 and adopted shortly afterwards. He attended Reed College in 1972 before withdrawing that same year. In 1974, he traveled through India, seeking enlightenment before later studying Zen Buddhism. He and Wozniak co-founded Apple in 1976 to further develop and sell Wozniak\&#39;s Apple I personal computer. Together, the duo gained fame and wealth a year later with production and sale of the Apple II, one of the first highly successful mass-produced microcomputers. Jobs saw the commercial potential of the Xerox Alto in 1979, which was mouse-driven and had a graphical user interface (GUI). This led to the development of the unsuccessful Apple Lisa in 1983, followed by the breakthrough Macintosh in 1984, the first mass-produced computer with a GUI. The Macintosh launched the desktop publishing industry in 1985 with the addition of the Apple LaserWriter, the first laser printer to feature vector graphics and PostScript.\nIn 1985, Jobs departed Apple after a long power struggle with the company\&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer graphics division of Lucasfilm that eventually spun off independently as Pixar, which produced the first 3D computer-animated feature film Toy Story (1995) and became a leading animation studio, producing over 27 films since.\nIn 1997, Jobs returned to Apple as CEO after the company\&#39;s acquisition of NeXT. He was largely responsible for reviving Apple, which was on the verge of bankruptcy. He worked closely with British designer Jony Ive to develop a line of products and services that had larger cultural ramifications, beginning with the &quot;Think different&quot; advertising campaign, and leading to the iMac, iTunes, Mac OS X, Apple Store, iPod, iTunes Store, iPhone, App Store, and iPad. In 2003, Jobs was diagnosed with a pancreatic neuroendocrine tumor. He died of respiratory arrest related to the tumor in 2011, and in 2022, was posthumously awarded the Presidential Medal of Freedom.&#39;, &#39;title&#39;: &#39;Steve Jobs&#39;})]</pre><blockquote><strong>Contextual Compression</strong></blockquote><p>One major challenge with retrieval is that the information that is most relevant to the query may be buried in a document with lot of irrelevant text. Also passing the entire document can lead to more expensive LLM calls and poor results. This is where <strong><em>Contextual Compression</em></strong> comes into picture.</p><p>The main idea is to compress the documents based on the context of the query, so that only the relevant information is returned. To use a contextual compression retriever, we would need a base retriever and a document compressor to compress the documents. Let’s look into few document compressors.</p><blockquote><strong>LLMChainExtractor</strong></blockquote><p>It iterates over the initially returned documents and extract from each, only the content relevant to the query.</p><p>Let’s wrap the base retriever with a ContextualCompressionRetriever. We will also add an LLMChainExtractor. We can see that the result is much more accurate and specific.</p><pre>from langchain.retrievers.document_compressors import LLMChainExtractor<br>from langchain.retrievers import ContextualCompressionRetriever<br><br>retriever = db.as_retriever()<br>chat = ChatOpenAI(temperature=0)<br>compressor = LLMChainExtractor.from_llm(chat)<br>compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)<br>compressed_docs = compression_retriever.get_relevant_documents(query = query)<br>print(compressed_docs[0].page_content)<br><br># In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley.</pre><blockquote><strong>LLMChainFilter</strong></blockquote><p>It uses an LLM chain to decide which of the initially retrieved documents to filter out and which ones to return, without manipulating the document contents.</p><pre>from langchain.retrievers.document_compressors import LLMChainFilter<br><br>compressor = LLMChainFilter.from_llm(chat)<br>compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)<br>compressed_docs = compression_retriever.get_relevant_documents(query = query)<br>print(compressed_docs[0].page_content)<br><br># In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer</pre><blockquote><strong>EmbeddingsFilter</strong></blockquote><p>Making an extra LLM call over the retrieved documents can be slow and expensive. The EmbeddingsFilter provides a cheaper and faster option. It embeds the documents and query and only returns documents which have sufficiently similar embeddings to query.</p><pre>from langchain.retrievers.document_compressors import EmbeddingsFilter<br><br># using similarity threshold of 0.6<br>embeddings_filter  = EmbeddingsFilter(embeddings=embedding_function, similarity_threshold=0.6)<br>compression_retriever = ContextualCompressionRetriever(base_compressor=embeddings_filter, base_retriever=retriever)<br>compressed_docs = compression_retriever.get_relevant_documents(query = query)<br>print(compressed_docs[0].page_content)<br><br># In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer</pre><p><strong>2. Parent Document Retriever</strong></p><p>While splitting the documents for retrieval, there may be some conflicts.</p><ul><li>You may want to have small documents, so that their embeddings can most accurately reflect their meaning. If too long, then the embeddings can lose meaning.</li><li>You want to have long enough documents that the context of each chunk is retained.</li></ul><p>The ParentDocumentRetriever splits and stores small chunks of data. It first fetches smaller chunks, looks upto the parent ids for those chunks and then returns those larger documents.</p><p>Sometimes, the full documents can be too big to want to retrieve them as is. In that case, what we really want to do is to first split the raw documents into larger chunks, and then split it into smaller chunks. We then index the smaller chunks, but on retrieval we retrieve the larger chunks.</p><pre>from langchain.text_splitter import CharacterTextSplitter<br>from langchain.retrievers import ParentDocumentRetriever<br>from langchain.storage import InMemoryStore</pre><pre># This text splitter is used to create the parent documents<br>parent_splitter = CharacterTextSplitter(separator=&quot;\n\n&quot;, chunk_size=1000, chunk_overlap=100)<br># This text splitter is used to create the child documents<br>child_splitter = CharacterTextSplitter(separator=&quot;\n&quot;, chunk_size=200, chunk_overlap=50)<br><br>store = InMemoryStore() # parent documents<br><br>par_doc_retriever = ParentDocumentRetriever(vectorstore=db, docstore=store, child_splitter=child_splitter, parent_splitter=parent_splitter)<br>par_doc_retriever.add_documents(docs)<br>par_doc_retriever.get_relevant_documents(query=query)<br><br># [Document(page_content=&quot;In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer&quot;, metadata={&#39;title&#39;: &#39;Steve Jobs&#39;, &#39;summary&#39;: &#39;Steven Paul Jobs (February 24, 1955 – October 5, 2011) was an American businessman, inventor, and investor best known for co-founding the technology giant Apple Inc. Jobs was also the founder of NeXT and chairman and majority shareholder of Pixar. He was a pioneer of the personal computer revolution of the 1970s and 1980s, along with his early business partner and fellow Apple co-founder Steve Wozniak.\nJobs was born in San Francisco in 1955 and adopted shortly afterwards. He attended Reed College in 1972 before withdrawing that same year. In 1974, he traveled through India, seeking enlightenment before later studying Zen Buddhism. He and Wozniak co-founded Apple in 1976 to further develop and sell Wozniak\&#39;s Apple I personal computer. Together, the duo gained fame and wealth a year later with production and sale of the Apple II, one of the first highly successful mass-produced microcomputers. Jobs saw the commercial potential of the Xerox Alto in 1979, which was mouse-driven and had a graphical user interface (GUI). This led to the development of the unsuccessful Apple Lisa in 1983, followed by the breakthrough Macintosh in 1984, the first mass-produced computer with a GUI. The Macintosh launched the desktop publishing industry in 1985 with the addition of the Apple LaserWriter, the first laser printer to feature vector graphics and PostScript.\nIn 1985, Jobs departed Apple after a long power struggle with the company\&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer graphics division of Lucasfilm that eventually spun off independently as Pixar, which produced the first 3D computer-animated feature film Toy Story (1995) and became a leading animation studio, producing over 27 films since.\nIn 1997, Jobs returned to Apple as CEO after the company\&#39;s acquisition of NeXT. He was largely responsible for reviving Apple, which was on the verge of bankruptcy. He worked closely with British designer Jony Ive to develop a line of products and services that had larger cultural ramifications, beginning with the &quot;Think different&quot; advertising campaign, and leading to the iMac, iTunes, Mac OS X, Apple Store, iPod, iTunes Store, iPhone, App Store, and iPad. In 2003, Jobs was diagnosed with a pancreatic neuroendocrine tumor. He died of respiratory arrest related to the tumor in 2011, and in 2022, was posthumously awarded the Presidential Medal of Freedom.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Steve_Jobs&#39;})]</pre><p>3. <strong>Time Weighted Vector Store Retriever</strong></p><p>This uses a combination of semantic similarity and time decay. The algorithm used is:</p><pre>semantic_similarity + (1.0 - decay_rate) ^ hours_passed #hours passed = hours passed since object in the retriever was last accessed</pre><pre>import faiss<br>from langchain.vectorstores import FAISS<br>from langchain.docstore import InMemoryDocstore<br>from langchain.retrievers import TimeWeightedVectorStoreRetriever<br>from langchain_community.embeddings import FakeEmbeddings<br>from langchain_core.documents import Document</pre><blockquote><strong>Low Decay Rate</strong></blockquote><p>A low decay rate(close to 0) means that memories will be remembered for longer. A decay rate of 0 means memories will never be forgotten.</p><pre># define embedding model<br>embedding_function = FakeEmbeddings(size=300)<br>emb_size = 1024<br># initialize empty vector store<br>index = faiss.IndexFlatL2(emb_size)<br>vector_store = FAISS(embedding_function, index, docstore=InMemoryDocstore({}), index_to_docstore_id = {})<br><br>tw_retriever = TimeWeightedVectorStoreRetriever(vectorstore = vector_store, decay_rate = 0.0000000000000000000000001, k=1)</pre><pre>from datetime import datetime, timedelta<br>yesterday = datetime.now() - timedelta(days=1)<br>retriever.add_documents(<br>    [Document(page_content=&quot;hello world&quot;)]<br>)<br>retriever.add_documents([Document(page_content=&quot;hello foo&quot;)])</pre><pre># &quot;hello world&quot; is returned first because it is most salient, and the decay rate is close to 0., meaning it&#39;s still recent enough<br>retriever.get_relevant_documents(&quot;hello world&quot;)<br><br># [Document(page_content=&#39;hello world&#39;),<br># Document(page_content=&#39;hello foo&#39;)]</pre><blockquote><strong>High Decay Rate</strong></blockquote><p>With a high decay rate (several 9’s), the recency score quickly goes to 0. If you set this all the way to 1, recency is 0 for all objects, once again making this equivalent to a vector lookup.</p><pre>embedding_function = FakeEmbeddings(size=300)<br>emb_size = 1024<br>index = faiss.IndexFlatL2(emb_size)<br>temp_db = FAISS(embedding_function, index, InMemoryDocstore({}),{})<br><br>tw_retriever = TimeWeightedVectorStoreRetriever(vectorstore = temp_db, decay_rate = 0.999, k=1)</pre><pre>yesterday = datetime.now() - timedelta(days=1)<br>retriever.add_documents(<br>    [Document(page_content=&quot;hello world&quot;)]<br>)<br>retriever.add_documents([Document(page_content=&quot;hello foo&quot;)])<br><br>#&quot;hello foo&quot; is returned first because &quot;hello world&quot; is mostly forgotten<br>retriever.get_relevant_documents(&quot;hello world&quot;)</pre><p>That’s all I had in this article. We covered some of the most important Retrievers in LangChain that form an essential component in the RAG pipeline.</p><p>All the code have been added my in this <a href="https://github.com/jayantkp/LangChain/blob/main/05_RAG_Retrievers_2.ipynb">Github</a> link. Let me know if you have questions.</p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me</a></p><blockquote><strong>References</strong></blockquote><p><a href="https://python.langchain.com/docs/modules/data_connection/retrievers/">Retrievers | 🦜️🔗 LangChain</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79908145e28c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging RAG for Chatbot Development using Gemma-2b-it, FAISS VectorDB and Streamlit]]></title>
            <link>https://jayant017.medium.com/leveraging-rag-for-chatbot-development-using-gemma-2b-it-faiss-vectordb-and-streamlit-689ee7d060f9?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/689ee7d060f9</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[faiss]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 13:35:00 GMT</pubDate>
            <atom:updated>2024-04-05T13:35:00.683Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*4rlthCvu3ryQjjlR" /></figure><p>Large Language Models(LLMs) have significantly impacted the landscape of AI through several ways such as Enhanced NLP, Revolutionizing Content Generation and Democratization of AI among others.</p><p>While LLMs offer impressive capabilities, they’re not without limitations. Their knowledge base is limited to their recent training iteration. This means they might miss out on recent developments. Additionally, for tasks requiring specific domain expertise, LLMs may resort to “hallucination” — essentially making up information that sounds plausible but isn’t factually accurate.</p><p>While retraining the LLM could address these problems, the immense cost and time involved make it impractical. We need a more efficient solution. Let’s talk about the solution, then!</p><blockquote><strong>RAG(Retrieval Augmented Generation)</strong></blockquote><p>RAG emerges as a promising approach for the issues mentioned above. Using RAG, we can give the model access to specific information that be used by the model as context to generate responses.</p><p>In this article, we will build a LLM chatbot using Google’s Open Source model Gemma, along with LangChain, FAISS Vector Database and Streamlit.</p><p>Let’s first look into an overview of the RAG pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S-23qXgFImqb_ei1.png" /><figcaption>Image Source: <a href="https://developer.nvidia.com/blog/rag-101-demystifying-retrieval-augmented-generation-pipelines/">https://developer.nvidia.com/blog/rag-101-demystifying-retrieval-augmented-generation-pipelines/</a></figcaption></figure><p>The process begins with a knowledge base such as PDFs and these are stored as a documents. These documents are then preprocessed by being chunked into smaller pieces that fits into the model context length.</p><p>Then we transform the chunked documents into embedding vectors using an embedding model and store it in the vector database.</p><p>When the user submits a query, we first convert the query into an embedding vector using the same embedding model. The system then searches the vector database for similar vectors based on the query vector.</p><p>The <strong><em>retrieval </em></strong>step in RAG essentially identifies relevant information from the knowledge base.</p><p>Now, these information becomes the context for the LLM. Finally the query is <strong><em>augmented </em></strong>with this retrieved context, and then passed to the LLM to <strong><em>generate </em></strong>responses.</p><p>With all said, let’s start building the chatbot. I am using <a href="https://colab.research.google.com/">Google Colab</a> Free tier GPU for this entire pipeline.</p><p>We will use Gemma-2B model as the generator LLM.</p><blockquote><strong>Gemma-2B</strong></blockquote><p><a href="https://huggingface.co/google/gemma-2b-it">Gemma-2B</a> is a variant of the LLM <strong>Gemma</strong> that is developed by Google AI. It belongs to the family of decoder-only text-to-text LLMs, available in English, with open weights, pre-trained variants, and instruction-tuned variants. To use this model, we should accept the terms and conditions in HuggingFace and pass the HuggingFace token while logging in.</p><pre>from huggingface_hub import notebook_login<br>notebook_login()</pre><p>Let’s initialize the model. We also use 4-bit quantization(for which we need a GPU) to reduce the memory usage. We use <a href="https://huggingface.co/docs/transformers/main_classes/quantization#transformers.BitsAndBytesConfig">BitsAndBytesConfig</a> from Transformers for the same. We also cache the models in the drive for faster inference.</p><p>Note: We can also load the model in CPU without quantization using disk offloading.</p><pre>import os<br>import torch<br>from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig<br><br>CACHE_DIR = &quot;/content/drive/LLM_RAG_Bot/models&quot;<br><br>class ChatModel:<br>    def __init__(self, model_id: str = &quot;google/gemma-2b-it&quot;, device=&quot;cuda&quot;):<br><br><br>        self.tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=CACHE_DIR)<br><br>        quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)<br><br>        self.model = AutoModelForCausalLM.from_pretrained(model_id,device_map=&quot;auto&quot;,quantization_config=quantization_config,cache_dir=CACHE_DIR)<br><br>        self.model.eval()<br>        self.chat = []<br>        self.device = device</pre><p>Now, let us define the main inference function.</p><pre>    def inference(self, question: str, context: str = None, max_new_tokens: int = 250):<br><br>        if context == None or context == &quot;&quot;:<br>            prompt = f&quot;&quot;&quot;Give a detailed answer to the question. Question: {question}&quot;&quot;&quot;<br>        else:<br>            prompt = f&quot;&quot;&quot;Using the information contained from the context, give a detailed answer to the question. Do<br>            not add any extra information . Context: {context}. Question: {question}&quot;&quot;&quot;<br><br>        chat = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}]<br>        formatted_prompt = self.tokenizer.apply_chat_template(chat,tokenize=False,add_generation_prompt=True,)<br><br>        inputs = self.tokenizer.encode(formatted_prompt, add_special_tokens=False, return_tensors=&quot;pt&quot;).to(self.device)<br>        <br>with torch.no_grad():<br>            outputs = self.model.generate(input_ids=inputs,max_new_tokens=max_new_tokens,do_sample=False,)<br>        <br>        response = self.tokenizer.decode(outputs[0], skip_special_tokens=False)<br>        response = response[len(formatted_prompt) :]  <br>        response = response.replace(&quot;&lt;eos&gt;&quot;, &quot;&quot;) <br><br>        return response</pre><p>The inference function takes a question and context(if present) and generates response. We initialize the prompt based on the context.</p><p>Then we encode the prompt and feed it to the LLM along with other parameters. Next, we decode the output back into text by decoding the tokens.</p><blockquote><strong>Embedding model</strong></blockquote><p>Now, our model for inference is loaded. As I mentioned before, we need an embedding model to transform the documents(context) as well as the query into vectors. For this we use a sentence transformer encoder model “<em>all-MiniLM-L12-v2</em>”, which encodes text into a 384-dimensional vector.</p><p>You can use other embedding models such as <em>OpenAIEmbeddings</em>, <em>BGEEmbeddings</em>, etc. Let’s initialise the model.</p><pre>CACHE_DIR = &quot;/content/drive/LLM_RAG_Bot/models&quot;<br>class Encoder:<br>    def __init__(self, model_name: str = &quot;sentence-transformers/all-MiniLM-L12-v2&quot;, device=&quot;cpu&quot;):<br>        self.embedding_function = HuggingFaceEmbeddings(<br>            model_name=model_name,<br>            cache_folder=CACHE_DIR,<br>            model_kwargs={&quot;device&quot;: device})</pre><p>With the inference LLM and embedding model set, lets look into the ingestion part.</p><blockquote><strong>Document Loaders and Splitters</strong></blockquote><p>I mentioned that to provide the model some context around the query, we will need some information. We will essentially load a PDF document, that will act as context.</p><p>I have briefly covered about Document Loaders and Splitters in my <a href="https://jayant017.medium.com/rag-using-langchain-part-1-document-loaders-b4774c04fd85?source=user_profile---------3----------------------------">articles</a>. I would suggest you to have a read about it. Let’s define the loader and split functionality.</p><pre>def load_and_split_pdfs(file_paths: list, chunk_size: int = 256):<br>    loaders = [PyPDFLoader(file_path) for file_path in file_paths]<br>    pages = []<br>    for loader in loaders:<br>        pages.extend(loader.load())<br><br>    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(<br>        tokenizer=AutoTokenizer.from_pretrained(&quot;sentence-transformers/all-MiniLM-L12-v2&quot;),<br>        chunk_size=chunk_size,<br>        chunk_overlap=int(chunk_size / 10),<br>        strip_whitespace=True)<br>        <br>    docs = text_splitter.split_documents(pages)<br>    return docs</pre><blockquote><strong>Vector Database</strong></blockquote><p>After loading and chunking the documents, we need to store it in a vector database. I have used <a href="https://python.langchain.com/docs/integrations/vectorstores/faiss">FAISS</a> from Meta AI, which is very efficient for similarity search and indexing.</p><pre>class FaissDb:<br>    def __init__(self, docs, embedding_function):<br>        self.db = FAISS.from_documents(docs, embedding_function, distance_strategy=DistanceStrategy.COSINE)<br><br>    def similarity_search(self, question: str, k: int = 3):<br>        retrieved_docs = self.db.similarity_search(question, k=k)<br>        context = &quot;&quot;.join(doc.page_content + &quot;\n&quot; for doc in retrieved_docs)<br>        return context</pre><p>To compute the similarity, I have used cosine similarity.</p><blockquote><strong>Streamlit UI</strong></blockquote><p>I have used Streamlit to create a user interface for the chatbot. We need to create an <em>app.py</em> file that contains all the required classes and functionalities.</p><p>First, we are loading the LLM and embedding model using a decorator and initializing them. Then we define a helper function to save all documents to disk.</p><pre>import os<br>import streamlit as st<br>from model import ChatModel<br>import rag_util<br><br>FILES_DIR = &quot;/content/drive/LLM_RAG_Bot/files&quot;<br><br>st.title(&quot;Gemma 2B Chatbot&quot;)<br><br>@st.cache_resource<br>def load_model():<br>    model = ChatModel(model_id=&quot;google/gemma-2b-it&quot;, device=&quot;cuda&quot;)<br>    return model<br><br>@st.cache_resource<br>def load_encoder():<br>    encoder = rag_util.Encoder(model_name=&quot;sentence-transformers/all-MiniLM-L12-v2&quot;, device=&quot;cpu&quot;)<br>    return encoder<br><br>model = load_model()<br><br>encoder = load_encoder()<br><br>def save_file(uploaded_file):<br>    &quot;&quot;&quot;helper function to save documents to disk&quot;&quot;&quot;<br>    file_path = os.path.join(FILES_DIR, uploaded_file.name)<br>    with open(file_path, &quot;wb&quot;) as f:<br>        f.write(uploaded_file.getbuffer())<br>    return file_path</pre><p>Next, we create a sidebar where we specify the user can upload any PDF and mention <em>max_tokens</em> and top <em>k </em>responses. After the document is uploaded, we load and split it, and store it in the <em>FAISS DB</em>.</p><pre>with st.sidebar:<br>  max_new_tokens = st.number_input(&quot;max_new_tokens&quot;, 128, 4096, 512)<br>  k = st.number_input(&quot;k&quot;, 1, 10, 3)<br>  uploaded_files = st.file_uploader(&quot;Upload PDFs for model context&quot;, type = [&#39;pdf&#39;,&#39;PDF&#39;], accept_multiple_files = True)<br><br>  file_paths = []<br>  for file in uploaded_files:<br>    file_paths.append(save_file(file))<br>  <br>  if uploaded_files != []:<br>    # create DB <br>    docs = rag_util.load_and_split_pdfs(file_paths)<br>    DB = rag_util.FaissDb(docs = docs, embedding_function = encoder.embedding_function)</pre><p>We then take in the user prompt, find the context using similarity search. This information is passed onto the model to generate response to the query.</p><pre>with st.chat_message(&#39;assistant&#39;):<br>    user_prompt = st.session_state.messages[-1][&#39;content&#39;]<br><br>    context = (None if uploaded_files == [] else DB.similarity_search(user_prompt, k = k))<br><br>    answer = model.generate(user_prompt, context = context, max_new_tokens = max_new_tokens)<br><br>    response = st.write(answer)<br>  <br>  st.session_state.messages.append({&#39;role&#39;:&#39;assistant&#39;,&#39;content&#39;:answer})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJO4xmz0Pmje4fm8gCSYtA.png" /><figcaption>The Chatbot we build in this article using Gemma &amp; LangChain that answers questions related to a refrigerator user manual.</figcaption></figure><p>When asked about a specific question, this is how the LLM responded by adding the context.</p><p>So, this is how we can create a custom chatbot that can read in PDF document and answer domain specific questions. The entire code is available on my <a href="https://github.com/jayantkp/rag-chatbot-llm">Github</a>.</p><p>Let me know if you any questions. Keep learning!</p><blockquote><strong>References</strong></blockquote><p>[1] <a href="https://huggingface.co/google/gemma-2b-it">https://huggingface.co/google/gemma-2b-it</a></p><p>[2] <a href="https://streamlit.io/generative-ai">https://streamlit.io/generative-ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=689ee7d060f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG using LangChain : Part 3- Vector Stores and Retrievers]]></title>
            <link>https://jayant017.medium.com/rag-using-langchain-part-3-vector-stores-and-retrievers-a75f4d14cbf3?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/a75f4d14cbf3</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[vector-store]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sun, 31 Mar 2024 08:25:13 GMT</pubDate>
            <atom:updated>2024-03-31T08:25:16.172Z</atom:updated>
            <content:encoded><![CDATA[<h3>RAG using LangChain : Part 3- Vector Stores and Retrievers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMl7AspiLlQtVQC1YnMc9w.png" /></figure><p>Welcome to the third article in this series of RAG where we explore the different components in the Retrieval process in RAG. So far we have looked into how to load unstructured data, transform it by means of text splitters and embed it using various text embedding models.</p><p>In this article, we will understand the functionalities of Vector Stores and Retrievers which are very important components in the Retrieval process.</p><blockquote><strong>So, what are Vector Stores?</strong></blockquote><p>Vector stores are essentially databases that are designed for a specific kind of data format — vector embeddings. They specialize in storing these embeddings very efficiently. Vector stores enable fast retrieval based on similarity searches.</p><p>Essentially the unstructured data is loaded, transformed and embedded into vectors and stored in a vector store. During query time, the unstructured query is embedded and the most similar embedding vectors with respect to the embedded query is retrieved. Vector stores takes care of storing the embedded data and performing vector search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lN7sEOHJEjMpyNH1ahAvbQ.png" /></figure><p>There are a lot of free, open-source vector stores namely, <strong><em>ChromaDB</em></strong>, <strong><em>FAISS</em></strong>, <strong><em>Qdrant</em></strong>, <strong><em>Lance</em></strong>, etc. Let’s look into the functionality of some of these.</p><p>Import all necessary dependencies.</p><pre>from langchain_community.document_loaders import WikipediaLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.embeddings import HuggingFaceBgeEmbeddings</pre><p>Step 1: <em>Document Loading- </em>Let’s use <em>WikipediaLoader </em>as the Document Loader.</p><pre>loader = WikipediaLoader(query = &#39;Elon Musk&#39;, load_max_docs=5)<br>documents = loader.load()<br>documents<br><br>#[Document(page_content=&#39;Elon Reeve Musk (; EE-lon; born June 28, 1971) is a businessman and investor. He is the founder, chairman, CEO, and CTO of SpaceX; angel investor, CEO, product architect, and former chairman of Tesla, Inc.; owner, executive chairman, and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is one of the wealthiest people in the world, with an estimated net worth of US$190 billion as of March 2024, according to the Bloomberg Billionaires Index, and $195 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX.A member of the wealthy South African Musk family, Elon was born in Pretoria and briefly attended the University of Pretoria before immigrating to Canada at age 18, acquiring citizenship through his Canadian-born mother. Two years later, he matriculated at Queen\&#39;s University at Kingston in Canada. Musk later transferred to the University of Pennsylvania, and received bachelor\&#39;s degrees in economics and physics. He moved to California in 1995 to attend Stanford University, but dropped out after two days and, with his brother Kimbal, co-founded online city guide software company Zip2. The startup was acquired by Compaq for $307 million in 1999, and that same year Musk co-founded X.com, a direct bank. X.com merged with Confinity in 2000 to form PayPal.\nIn October 2002, eBay acquired PayPal for $1.5 billion, and that same year, with $100 million of the money he made, Musk founded SpaceX, a spaceflight services company. In 2004, he became an early investor in electric vehicle manufacturer Tesla Motors, Inc. (now Tesla, Inc.). He became its chairman and product architect, assuming the position of CEO in 2008. In 2006, Musk helped create SolarCity, a solar-energy company that was acquired by Tesla in 2016 and became Tesla Energy. In 2013, he proposed a hyperloop high-speed vactrain transportation system. In 2015, he co-founded OpenAI, a nonprofit artificial intelligence research company. The following year, Musk co-founded Neuralink—a neurotechnology company developing brain–computer interfaces—and the Boring Company, a tunnel construction company. In 2022, he acquired Twitter for $44 billion. He subsequently merged the company into newly created X Corp. and rebranded the service as X the following year. In March 2023, he founded xAI, an artificial intelligence company.\nMusk has expressed views that have made him a polarizing figure. He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and antisemitic conspiracy theories. His ownership of Twitter has been similarly controversial, being marked by the laying off of a large number of employees, an increase in hate speech and misinformation and disinformation on the website, as well as changes to Twitter Blue verification. In 2018, the U.S. Securities and Exchange Commission (SEC) sued him, alleging that he had falsely announced that he had secured funding for a private takeover of Tesla. To settle the case, Musk stepped down as the chairman of Tesla and paid a $20 million fine.\n\n\n== Early life and education ==\n\n\n=== Childhood and family ===\n\nElon Reeve Musk was born on June 28, 1971, in Pretoria, South Africa\&#39;s administrative capital. He is of British and Pennsylvania Dutch ancestry. His mother, Maye Musk (née Haldeman), is a model and dietitian born in Saskatchewan, Canada, and raised in South Africa. His father, Errol Musk, is a South African electromechanical engineer, pilot, sailor, consultant, emerald dealer, and property developer, who partly owned a rental lodge at the Timbavati Private Nature Reserve. Elon has a younger brother, Kimbal, and a younger sister, Tosca.The family was wealthy during Elon\&#39;s youth. Despite both Musk and Errol previously stating that Errol was a part owner of a Zambian emerald mine, in 2023, Errol recounted that the deal he made was to receive &quot;a portion of the emeralds produced at three sma&#39;, metadata={&#39;title&#39;: &#39;Elon Musk&#39;, &#39;summary&#39;: &quot;Elon Reeve Musk (; EE-lon; born June 28, 1971) is a businessman and investor. He is the founder, chairman, CEO, and CTO of SpaceX; angel investor, CEO, product architect, and former chairman of Tesla, Inc.; owner, executive chairman, and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is one of the wealthiest people in the world, with an estimated net worth of US$190 billion as of March 2024, according to the Bloomberg Billionaires Index, and $195 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX.A member of the wealthy South African Musk family, Elon was born in Pretoria and briefly attended the University of Pretoria before immigrating to Canada at age 18, acquiring citizenship through his Canadian-born mother. Two years later, he matriculated at Queen&#39;s University at Kingston in Canada. Musk later transferred to the University of Pennsylvania, and received bachelor&#39;s degrees in economics and physics. He moved to California in 1995 to attend Stanford University, but dropped out after two days and, with his brother Kimbal, co-founded online city guide software company Zip2. The startup was acquired by Compaq for $307 million in 1999, and that same year Musk co-founded X.com, a direct bank. X.com merged with Confinity in 2000 to form PayPal.\nIn October 2002, eBay acquired PayPal for $1.5 billion, and that same year, with $100 million of the money he made, Musk founded SpaceX, a spaceflight services company. In 2004, he became an early investor in electric vehicle manufacturer Tesla Motors, Inc. (now Tesla, Inc.). He became its chairman and product architect, assuming the position of CEO in 2008. In 2006, Musk helped create SolarCity, a solar-energy company that was acquired by Tesla in 2016 and became Tesla Energy. In 2013, he proposed a hyperloop high-speed vactrain transportation system. In 2015, he co-founded OpenAI, a nonprofit artificial intelligence research company. The following year, Musk co-founded Neuralink—a neurotechnology company developing brain–computer interfaces—and the Boring Company, a tunnel construction company. In 2022, he acquired Twitter for $44 billion. He subsequently merged the company into newly created X Corp. and rebranded the service as X the following year. In March 2023, he founded xAI, an artificial intelligence company.\nMusk has expressed views that have made him a polarizing figure. He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and antisemitic conspiracy theories. His ownership of Twitter has been similarly controversial, being marked by the laying off of a large number of employees, an increase in hate speech and misinformation and disinformation on the website, as well as changes to Twitter Blue verification. In 2018, the U.S. Securities and Exchange Commission (SEC) sued him, alleging that he had falsely announced that he had secured funding for a private takeover of Tesla. To settle the case, Musk stepped down as the chairman of Tesla and paid a $20 million fine.&quot;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Elon_Musk&#39;}),<br>.....]</pre><p>Step 2: <em>Text Splitting- </em>We will use <em>RecursiveCharacterTextSplitter </em>as the text splitter.</p><pre>text_splitter = RecursiveCharacterTextSplitter(chunk_size = 400, chunk_overlap = 100)<br>docs = text_splitter.split_documents(documents=documents)<br>print(docs)<br><br>#[Document(page_content=&#39;Elon Reeve Musk (; EE-lon; born June 28, 1971) is a businessman and investor. He is the founder, chairman, CEO, and CTO of SpaceX; angel investor, CEO, product architect, and former chairman of Tesla, Inc.; owner, executive chairman, and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is one of the wealthiest&#39;, metadata={&#39;title&#39;: &#39;Elon Musk&#39;, &#39;summary&#39;: &quot;Elon Reeve Musk (; EE-lon; born June 28, 1971) is a businessman and investor. He is the founder, chairman, CEO, and CTO of SpaceX; angel investor, CEO, product architect, and former chairman of Tesla, Inc.; owner, executive chairman, and CTO of X Corp.; founder of the Boring Company and xAI; co-founder of Neuralink and OpenAI; and president of the Musk Foundation. He is one of the wealthiest people in the world, with an estimated net worth of US$190 billion as of March 2024, according to the Bloomberg Billionaires Index, and $195 billion according to Forbes, primarily from his ownership stakes in Tesla and SpaceX.A member of the wealthy South African Musk family, Elon was born in Pretoria and briefly attended the University of Pretoria before immigrating to Canada at age 18, acquiring citizenship through his Canadian-born mother. Two years later, he matriculated at Queen&#39;s University at Kingston in Canada. Musk later transferred to the University of Pennsylvania, and received bachelor&#39;s degrees in economics and physics. He moved to California in 1995 to attend Stanford University, but dropped out after two days and, with his brother Kimbal, co-founded online city guide software company Zip2. The startup was acquired by Compaq for $307 million in 1999, and that same year Musk co-founded X.com, a direct bank. X.com merged with Confinity in 2000 to form PayPal.\nIn October 2002, eBay acquired PayPal for $1.5 billion, and that same year, with $100 million of the money he made, Musk founded SpaceX, a spaceflight services company. In 2004, he became an early investor in electric vehicle manufacturer Tesla Motors, Inc. (now Tesla, Inc.). He became its chairman and product architect, assuming the position of CEO in 2008. In 2006, Musk helped create SolarCity, a solar-energy company that was acquired by Tesla in 2016 and became Tesla Energy. In 2013, he proposed a hyperloop high-speed vactrain transportation system. In 2015, he co-founded OpenAI, a nonprofit artificial intelligence research company. The following year, Musk co-founded Neuralink—a neurotechnology company developing brain–computer interfaces—and the Boring Company, a tunnel construction company. In 2022, he acquired Twitter for $44 billion. He subsequently merged the company into newly created X Corp. and rebranded the service as X the following year. In March 2023, he founded xAI, an artificial intelligence company.\nMusk has expressed views that have made him a polarizing figure. He has been criticized for making unscientific and misleading statements, including COVID-19 misinformation and antisemitic conspiracy theories. His ownership of Twitter has been similarly controversial, being marked by the laying off of a large number of employees, an increase in hate speech and misinformation and disinformation on the website, as well as changes to Twitter Blue verification. In 2018, the U.S. Securities and Exchange Commission (SEC) sued him, alleging that he had falsely announced that he had secured funding for a private takeover of Tesla. To settle the case, Musk stepped down as the chairman of Tesla and paid a $20 million fine.&quot;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Elon_Musk&#39;}),<br>....]</pre><p>Step 3: <em>Embedding Function</em>- For the embedding function, we will use <em>HuggingFace</em> BGE embedding model and let’s create a query.</p><pre>model_name = &quot;BAAI/bge-large-en-v1.5&quot;<br>model_kwargs = {&#39;device&#39;:&#39;cpu&#39;}<br>encode_kwargs = {&#39;normalize_embeddings&#39;:True}<br><br>embedding_function = HuggingFaceBgeEmbeddings(<br>    model_name = model_name,<br>    model_kwargs = model_kwargs,<br>    encode_kwargs = encode_kwargs<br>)<br><br>query = &quot;Who is elon musk&#39;s father?&quot;</pre><blockquote><strong>FAISS</strong></blockquote><p>Let’s look into the functionalities of FAISS vector database, which makes use of the Facebook AI Similarity Search (FAISS) library.</p><pre>pip install faiss-cpu</pre><pre>from langchain_community.vectorstores import FAISS<br>db = FAISS.from_documents(<br>    docs,<br>    embedding_function<br>)</pre><p>Next, we can query the vector database. We can either do <strong><em>similarity search</em></strong> or <strong><em>similarity search with vector</em>.</strong></p><pre># Similarity search with query<br>matched_docs = db.similarity_search(query = query, k = 5)<br>matched_docs<br><br>#[Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;title&#39;: &#39;Musk family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;}),<br>....]</pre><pre># Similarity search with query vector<br>embedding_vector = embedding_function.embed_query(query)<br>matched_docs = db.similarity_search_by_vector(embedding_vector)<br>matched_docs<br><br>#[Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;title&#39;: &#39;Musk family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;}),<br>....]</pre><p>Since the embedding function is same for both, the result will be same.</p><p>Most of the times, we won’t necessarily use an in-memory database. Let’s create another open-source database.</p><blockquote><strong>ChromaDB</strong></blockquote><p>ChromaDB can be used as both an in-memory database and a backend with persistent storage options.</p><pre>!pip install chromadb</pre><pre>from langchain.vectorstores import Chroma<br>db = Chroma.from_documents(docs, embedding_function, persist_directory=&quot;output/elon_muskdb&quot;)<br><br># the persist_directory parameter saves the database in the specified path</pre><p><em>Loading the database</em></p><pre>loaded_db = Chroma(persist_directory = &quot;output/elon_muskdb&quot;, embedding_function = embedding_function)</pre><p>Similarly, here also we can do <em>similarity search</em> as well as <strong><em>similarity search with vector</em>.</strong></p><pre># Similarity search with query<br>matched_docs = loaded_db.similarity_search(query = query, k = 5)<br>matched_docs<br><br>#[Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;title&#39;: &#39;Musk family&#39;}),<br>....]</pre><pre># Similarity search with query vector<br>embedding_vector = embedding_function.embed_query(query)<br>matched_docs = loaded_db.similarity_search_by_vector(embedding_vector)<br>matched_docs<br><br>#[Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;title&#39;: &#39;Musk family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;}),<br>....]</pre><blockquote><strong>Retrievers</strong></blockquote><p>A retriever is an interface that returns documents given an unstructured query. It does not have to store documents like Vector store. Retrievers accept a string query as an input and return a list of Documents as an output. Let look into some basic retrievers in this article. We will cover more of Retrievers in the next one!</p><blockquote><strong>Vector Store-backed retriever</strong></blockquote><p>It uses a Vector store to retrieve documents. Let’s construct a retriever using the existing <strong><em>ChromaDB </em></strong><em>V</em>ector store that we have.</p><pre>retriever = db.as_retriever()<br>retriever<br><br>#VectorStoreRetriever(tags=[&#39;Chroma&#39;, &#39;HuggingFaceBgeEmbeddings&#39;], vectorstore=&lt;langchain_community.vectorstores.chroma.Chroma object at 0x000001EF7F17E080&gt;)</pre><p><em>Querying the retriever</em></p><pre>query = &quot;Who is elon musk&#39;s father?&quot;<br>matched_docs = retriever.get_relevant_documents(query = query)<br>matched_docs<br><br># [Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;title&#39;: &#39;Musk family&#39;}),<br>....]</pre><p>While creating the retriever, we can also mention how the retriever should retrieve the document( like <em>Maximum Marginal Relevance Search</em> or <em>Similarity Search</em>)and how many documents to retrieve.</p><pre># Using MMR and limiting the number of retrieved documents to 1<br><br>retriever = db.as_retriever(search_type=&#39;mmr&#39;, search_kwargs={&quot;k&quot;: 1})<br>matched_docs = retriever.get_relevant_documents(query=query)<br>matched_docs<br><br># [Document(page_content=&quot;In 1985, Jobs departed Apple after a long power struggle with the company&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Steve_Jobs&#39;, &#39;summary&#39;: &#39;Steven Paul Jobs (February 24, 1955 – October 5, 2011) was an American businessman, inventor, and investor best known for co-founding the technology giant Apple Inc. Jobs was also the founder of NeXT and chairman and majority shareholder of Pixar. He was a pioneer of the personal computer revolution of the 1970s and 1980s, along with his early business partner and fellow Apple co-founder Steve Wozniak.\nJobs was born in San Francisco in 1955 and adopted shortly afterwards. He attended Reed College in 1972 before withdrawing that same year. In 1974, he traveled through India, seeking enlightenment before later studying Zen Buddhism. He and Wozniak co-founded Apple in 1976 to further develop and sell Wozniak\&#39;s Apple I personal computer. Together, the duo gained fame and wealth a year later with production and sale of the Apple II, one of the first highly successful mass-produced microcomputers. Jobs saw the commercial potential of the Xerox Alto in 1979, which was mouse-driven and had a graphical user interface (GUI). This led to the development of the unsuccessful Apple Lisa in 1983, followed by the breakthrough Macintosh in 1984, the first mass-produced computer with a GUI. The Macintosh launched the desktop publishing industry in 1985 with the addition of the Apple LaserWriter, the first laser printer to feature vector graphics and PostScript.\nIn 1985, Jobs departed Apple after a long power struggle with the company\&#39;s board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer graphics division of Lucasfilm that eventually spun off independently as Pixar, which produced the first 3D computer-animated feature film Toy Story (1995) and became a leading animation studio, producing over 27 films since.\nIn 1997, Jobs returned to Apple as CEO after the company\&#39;s acquisition of NeXT. He was largely responsible for reviving Apple, which was on the verge of bankruptcy. He worked closely with British designer Jony Ive to develop a line of products and services that had larger cultural ramifications, beginning with the &quot;Think different&quot; advertising campaign, and leading to the iMac, iTunes, Mac OS X, Apple Store, iPod, iTunes Store, iPhone, App Store, and iPad. In 2003, Jobs was diagnosed with a pancreatic neuroendocrine tumor. He died of respiratory arrest related to the tumor in 2011, and in 2022, was posthumously awarded the Presidential Medal of Freedom.&#39;, &#39;title&#39;: &#39;Steve Jobs&#39;})]</pre><pre># Using Similarity Search.<br># Also keeping a minimum similarity threshold of 0.5 and retrieved documents = 2<br>retriever = db.as_retriever(search_type=&quot;similarity_score_threshold&quot;, search_kwargs={&quot;score_threshold&quot;: 0.5, &quot;k&quot; : 2})<br>matched_docs = retriever.get_relevant_documents(query=query)<br>matched_docs<br><br>#[Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;title&#39;: &#39;Musk family&#39;}),<br># Document(page_content=&quot;Elon Musk&#39;s paternal great-grandmother was a Dutchwoman descended from the Dutch Free Burghers, while one of his maternal great-grandparents came from Switzerland. His paternal grandmother was English from Liverpool and his paternal grandfather Walter Henry J. Musk was South African. Elon Musk&#39;s father, Errol Musk, is a South African former electrical and mechanical engineer consultant and&quot;, metadata={&#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Musk_family&#39;, &#39;summary&#39;: &#39;The Musk family is a wealthy family of South African origin that is largely active in the United States and Canada. The Musks are of English, Anglo-Canadian, Pennsylvania Dutch, and Swiss descent. The family is known for its entrepreneurial endeavours. Elon Musk was formerly the wealthiest person in the world, with an estimated net worth of US$232 billion as of December 2023, according to the Bloomberg Billionaires Index.&#39;, &#39;title&#39;: &#39;Musk family&#39;})]</pre><blockquote><strong>BM25 Retriever</strong></blockquote><p>It uses the BM25(Best Matching 25) ranking function ranking function to retrieve documents based on a query. It is similar to a bag-of-words approach.</p><pre>!pip install rank_bm25</pre><pre>from langchain.retrievers import BM25Retriever<br>bm25_retriever = BM25Retriever.from_documents(docs)</pre><p><em>Querying the retriever</em></p><pre>matched_docs = bm25_retriever.get_relevant_documents(&#39;Musk&#39;)<br>matched_docs</pre><pre>[Document(page_content=&quot;October 17. Weeks before the trial was set to begin, Musk reversed course, announcing that he would move forward with the acquisition. The deal was closed on October 27, with Musk immediately becoming Twitter&#39;s new owner and CEO. Twitter was taken private and merged into a new parent company named X Corp. Musk promptly fired several top executives, including previous CEO Parag Agrawal. Musk has&quot;, metadata={&#39;title&#39;: &#39;Acquisition of Twitter by Elon Musk&#39;, &#39;summary&#39;: &#39;Business magnate Elon Musk initiated an acquisition of American social media company Twitter, Inc. on April 14, 2022, and concluded it on October 27, 2022. Musk had begun buying shares of the company in January 2022, becoming its largest shareholder by April with a 9.1 percent ownership stake. Twitter invited Musk to join its board of directors, an offer he initially accepted before declining. On April 14, Musk made an unsolicited offer to purchase the company, to which Twitter\&#39;s board responded with a &quot;poison pill&quot; strategy to resist a hostile takeover before unanimously accepting Musk\&#39;s buyout offer of $44 billion on April 25. Musk stated that he planned to introduce new features to the platform, make its algorithms open-source, combat spambot accounts, and promote free speech.\nIn July, Musk announced his intention to terminate the agreement, asserting that Twitter had breached their agreement by refusing to crack down on spambot accounts. The company filed a lawsuit against Musk in the Delaware Court of Chancery shortly thereafter, with a trial scheduled for the week of October 17. Weeks before the trial was set to begin, Musk reversed course, announcing that he would move forward with the acquisition. The deal was closed on October 27, with Musk immediately becoming Twitter\&#39;s new owner and CEO. Twitter was taken private and merged into a new parent company named X Corp. Musk promptly fired several top executives, including previous CEO Parag Agrawal. Musk has since proposed several reforms to Twitter and laid off half of the company\&#39;s workforce. Hundreds of employees then resigned from the company after Musk issued an ultimatum demanding they commit to &quot;extremely hardcore&quot; work. Linda Yaccarino was appointed CEO of X Corp. In June 2023, the Twitter service was rebranded as X.\nReactions to the buyout were mixed, with praise for Musk\&#39;s planned reforms and vision for the company, particularly his calls for greater free speech, but criticism over fears of a potential rise in misinformation and disinformation, harassment, and hate speech on the platform. Within the United States, conservatives have largely supported the acquisition, while many liberals and former Twitter employees have voiced concerns about Musk\&#39;s intentions. Since becoming owner, Musk has faced backlash for his handling of the company and account suspensions, including the December 2022 suspensions of ten journalists.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk&#39;}),<br> Document(page_content=&#39;just direct transmitters of messages from Vladimir Putin&quot;.In an email newsletter, Ian Bremmer, head of Eurasia Group, a political-risk consultancy, said that Musk told him that he had spoken directly with Putin, which Musk denied in a reply on Twitter. Musk claimed he had spoken to Putin only once 18 months prior, on a space-related subject matter. Bremmer later took to Twitter to say Musk had&#39;, metadata={&#39;title&#39;: &#39;Views of Elon Musk&#39;, &#39;summary&#39;: &#39;Elon Musk is the CEO or owner of multiple companies including Tesla, SpaceX, and X Corp, and has expressed many views on a wide variety of subjects, ranging from politics to science.\nWithin the context of American politics, Musk has said he supported Barack Obama in 2008 and 2012, Hillary Clinton in 2016, and Joe Biden in 2020. In 2022, Musk said he would start voting for Republican Party candidates, with him showing support for Vivek Ramaswamy in the 2024 Republican presidential primaries. He has stated support for universal basic income, gun rights, freedom of speech, a tax on carbon emissions and opposes government subsidies. He is also a critic of short-selling.\nMusk has expressed concern about issues such as artificial intelligence (AI), climate change and population decline. He has also criticized COVID-19 lockdowns, public transportation, and labor unions. He has promoted conspiracy theories, and made controversial statements that have led to accusations of antisemitism, support of white pride and transphobia. His views on international relations, including on the China-Taiwan and Russia-Ukraine conflicts, have received mixed reactions.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Views_of_Elon_Musk&#39;}),<br> Document(page_content=&#39;X Corp. Musk promptly fired several top executives, including previous CEO Parag Agrawal. Musk has since proposed several reforms to Twitter and laid off half of the company\&#39;s workforce. Hundreds of employees then resigned from the company after Musk issued an ultimatum demanding they commit to &quot;extremely hardcore&quot; work. Linda Yaccarino was appointed CEO of X Corp. In June 2023, the Twitter&#39;, metadata={&#39;title&#39;: &#39;Acquisition of Twitter by Elon Musk&#39;, &#39;summary&#39;: &#39;Business magnate Elon Musk initiated an acquisition of American social media company Twitter, Inc. on April 14, 2022, and concluded it on October 27, 2022. Musk had begun buying shares of the company in January 2022, becoming its largest shareholder by April with a 9.1 percent ownership stake. Twitter invited Musk to join its board of directors, an offer he initially accepted before declining. On April 14, Musk made an unsolicited offer to purchase the company, to which Twitter\&#39;s board responded with a &quot;poison pill&quot; strategy to resist a hostile takeover before unanimously accepting Musk\&#39;s buyout offer of $44 billion on April 25. Musk stated that he planned to introduce new features to the platform, make its algorithms open-source, combat spambot accounts, and promote free speech.\nIn July, Musk announced his intention to terminate the agreement, asserting that Twitter had breached their agreement by refusing to crack down on spambot accounts. The company filed a lawsuit against Musk in the Delaware Court of Chancery shortly thereafter, with a trial scheduled for the week of October 17. Weeks before the trial was set to begin, Musk reversed course, announcing that he would move forward with the acquisition. The deal was closed on October 27, with Musk immediately becoming Twitter\&#39;s new owner and CEO. Twitter was taken private and merged into a new parent company named X Corp. Musk promptly fired several top executives, including previous CEO Parag Agrawal. Musk has since proposed several reforms to Twitter and laid off half of the company\&#39;s workforce. Hundreds of employees then resigned from the company after Musk issued an ultimatum demanding they commit to &quot;extremely hardcore&quot; work. Linda Yaccarino was appointed CEO of X Corp. In June 2023, the Twitter service was rebranded as X.\nReactions to the buyout were mixed, with praise for Musk\&#39;s planned reforms and vision for the company, particularly his calls for greater free speech, but criticism over fears of a potential rise in misinformation and disinformation, harassment, and hate speech on the platform. Within the United States, conservatives have largely supported the acquisition, while many liberals and former Twitter employees have voiced concerns about Musk\&#39;s intentions. Since becoming owner, Musk has faced backlash for his handling of the company and account suspensions, including the December 2022 suspensions of ten journalists.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk&#39;}),<br> Document(page_content=&#39;Business magnate Elon Musk initiated an acquisition of American social media company Twitter, Inc. on April 14, 2022, and concluded it on October 27, 2022. Musk had begun buying shares of the company in January 2022, becoming its largest shareholder by April with a 9.1 percent ownership stake. Twitter invited Musk to join its board of directors, an offer he initially accepted before declining. On&#39;, metadata={&#39;title&#39;: &#39;Acquisition of Twitter by Elon Musk&#39;, &#39;summary&#39;: &#39;Business magnate Elon Musk initiated an acquisition of American social media company Twitter, Inc. on April 14, 2022, and concluded it on October 27, 2022. Musk had begun buying shares of the company in January 2022, becoming its largest shareholder by April with a 9.1 percent ownership stake. Twitter invited Musk to join its board of directors, an offer he initially accepted before declining. On April 14, Musk made an unsolicited offer to purchase the company, to which Twitter\&#39;s board responded with a &quot;poison pill&quot; strategy to resist a hostile takeover before unanimously accepting Musk\&#39;s buyout offer of $44 billion on April 25. Musk stated that he planned to introduce new features to the platform, make its algorithms open-source, combat spambot accounts, and promote free speech.\nIn July, Musk announced his intention to terminate the agreement, asserting that Twitter had breached their agreement by refusing to crack down on spambot accounts. The company filed a lawsuit against Musk in the Delaware Court of Chancery shortly thereafter, with a trial scheduled for the week of October 17. Weeks before the trial was set to begin, Musk reversed course, announcing that he would move forward with the acquisition. The deal was closed on October 27, with Musk immediately becoming Twitter\&#39;s new owner and CEO. Twitter was taken private and merged into a new parent company named X Corp. Musk promptly fired several top executives, including previous CEO Parag Agrawal. Musk has since proposed several reforms to Twitter and laid off half of the company\&#39;s workforce. Hundreds of employees then resigned from the company after Musk issued an ultimatum demanding they commit to &quot;extremely hardcore&quot; work. Linda Yaccarino was appointed CEO of X Corp. In June 2023, the Twitter service was rebranded as X.\nReactions to the buyout were mixed, with praise for Musk\&#39;s planned reforms and vision for the company, particularly his calls for greater free speech, but criticism over fears of a potential rise in misinformation and disinformation, harassment, and hate speech on the platform. Within the United States, conservatives have largely supported the acquisition, while many liberals and former Twitter employees have voiced concerns about Musk\&#39;s intentions. Since becoming owner, Musk has faced backlash for his handling of the company and account suspensions, including the December 2022 suspensions of ten journalists.&#39;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk&#39;})]</pre><p>That’s all from this article. We have looked into some open source Vector stores and how to query them. We also looked into few Retrievers which retrieves the documents based on the user’s query. We will learn more about Retrievers in detail in the upcoming articles.</p><p>All the code have been added in the <a href="https://github.com/jayantkp/LangChain/tree/main/RAG">Github</a> link. Let me know if you have any questions.</p><p><a href="https://www.linkedin.com/in/jayantkp/">Connect with me.</a></p><blockquote><strong>References</strong></blockquote><p><a href="https://python.langchain.com/docs/modules/data_connection/vectorstores/">Vector stores | 🦜️🔗 Langchain</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a75f4d14cbf3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG using LangChain : Part 2- Text Splitters and Embeddings]]></title>
            <link>https://jayant017.medium.com/rag-using-langchain-part-2-text-splitters-and-embeddings-3225af44e7ea?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/3225af44e7ea</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sun, 24 Mar 2024 12:33:46 GMT</pubDate>
            <atom:updated>2024-03-24T12:33:46.192Z</atom:updated>
            <content:encoded><![CDATA[<h3>RAG using LangChain : Part 2- Text Splitters and Embeddings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyXS4oRtrW2NhhMRBxsdQQ.png" /></figure><p>The next step in the Retrieval process in RAG is to <strong><em>transform </em></strong>and <strong><em>embed </em></strong>the loaded Documents. If you are not familiar with how to load raw text as documents using Document Loaders, I would encourage you to look into the first <a href="https://jayant017.medium.com/rag-using-langchain-part-1-document-loaders-b4774c04fd85"><em>article</em></a>.</p><h3><strong>How to do Transformation? — Text Splitters</strong></h3><p>Once we have loaded the documents, we may want to split the entire document into smaller chunks(or parts), that can fit into our model’s context window. This is where LangChain’s text splitters comes into picture!</p><p>But this is not as easy as it sounds, there may be a lot of complexity involved here. Ideally, we would want to keep the semantically related pieces of text together. By semantically, I mean texts have similar contextual meaning.</p><p>Let’s hop onto the different types of text splitters in LangChain.</p><blockquote><strong>Character Text Splitter</strong></blockquote><p>This is one of the simplest method. It splits the text based on a specific character only if the chunk exceeds the given chunk size.</p><pre>#Let&#39;s read a text file<br>filepath = &quot;../../Sessions_Part2\datasets\Harry Potter 1 - Sorcerer_s Stone.txt&quot;<br>with open(filepath,&#39;r&#39;) as f:<br>    hp_book = f.read()</pre><pre>from langchain.text_splitter import CharacterTextSplitter<br><br>def len_func(text):<br>    return len(text)<br><br>text_splitter = CharacterTextSplitter(<br>    separator=&quot;\n\n&quot;,<br>    chunk_size = 1200,<br>    chunk_overlap = 100,<br>    length_function = len_func,<br>    is_separator_regex= False)<br><br><br>para_list = text_splitter.create_documents(texts = [hp_book])<br>para_list[:2]</pre><pre>[Document(page_content=&quot;Harry Potter and the Sorcerer&#39;s Stone\n\n\nCHAPTER ONE\n\nTHE BOY WHO LIVED\n\nMr. and Mrs. Dursley, of number four, Privet Drive, were proud to say\nthat they were perfectly normal, thank you very much. They were the last\npeople you&#39;d expect to be involved in anything strange or mysterious,\nbecause they just didn&#39;t hold with such nonsense.\n\nMr. Dursley was the director of a firm called Grunnings, which made\ndrills. He was a big, beefy man with hardly any neck, although he did\nhave a very large mustache. Mrs. Dursley was thin and blonde and had\nnearly twice the usual amount of neck, which came in very useful as she\nspent so much of her time craning over garden fences, spying on the\nneighbors. The Dursleys had a small son called Dudley and in their\nopinion there was no finer boy anywhere.&quot;),<br> Document(page_content=&quot;The Dursleys had everything they wanted, but they also had a secret, and\ntheir greatest fear was that somebody would discover it. They didn&#39;t\nthink they could bear it if anyone found out about the Potters. Mrs.\nPotter was Mrs. Dursley&#39;s sister, but they hadn&#39;t met for several years;\nin fact, Mrs. Dursley pretended she didn&#39;t have a sister, because her\nsister and her good-for-nothing husband were as unDursleyish as it was\npossible to be. The Dursleys shuddered to think what the neighbors would\nsay if the Potters arrived in the street. The Dursleys knew that the\nPotters had a small son, too, but they had never even seen him. This boy\nwas another good reason for keeping the Potters away; they didn&#39;t want\nDudley mixing with a child like that.\n\nWhen Mr. and Mrs. Dursley woke up on the dull, gray Tuesday our story\nstarts, there was nothing about the cloudy sky outside to suggest that\nstrange and mysterious things would soon be happening all over the\ncountry. Mr. Dursley hummed as he picked out his most boring tie for\nwork, and Mrs. Dursley gossiped away happily as she wrestled a screaming\nDudley into his high chair.\n\nNone of them noticed a large, tawny owl flutter past the window.&quot;)]</pre><p>There is no metadata associated with the result,so we can add the same.</p><pre>first_chunk = para_list[0]<br><br>first_chunk.metadata = {&quot;source&quot;:filepath}<br>first_chunk.metadata<br><br>{&#39;source&#39;: &#39;../../Sessions_Part2\\datasets\\Harry Potter 1 - Sorcerer_s Stone.txt&#39;}</pre><p>One of the major issues with this text splitter is that it can be an issue if the text exceeds the chunk size and there is no separator to chunk the text. This is where the next separator is used.</p><blockquote><strong>Recursive Character Splitter</strong></blockquote><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br><br>text_splitter = RecursiveCharacterTextSplitter(<br>    separators=[&quot;\n\n&quot;,&quot;\n&quot;, &quot; &quot;],<br>    chunk_size = 200,<br>    chunk_overlap = 100,<br>    length_function = len_func,<br>    is_separator_regex=False<br>)<br><br>chunk_list = text_splitter.create_documents(texts = [hp_book]</pre><p>Here, we first split at paragraph level, if the chunk size exceeds, it will move onto the next separator, at sentence level, if it still exceeds, it will move onto the next separator which is at word level.</p><blockquote><strong>Split by tokens</strong></blockquote><p>There is always a token limit associated with language models which we cannot exceed. When we split the text into chunks, it is a good idea to count the number of tokens.</p><blockquote><strong>tiktoken</strong></blockquote><p>tiktoken is a fast BPE(Byte-Pair encoding) tokenizer created by OpenAI to count the number of tokens.</p><pre># Splitting based on the token limit<br>from langchain.text_splitter import CharacterTextSplitter<br><br>text_splitter = CharacterTextSplitter.from_tiktoken_encoder(<br>    separator = &quot;\n\n&quot;,<br>    chunk_size = 1200,<br>    chunk_overlap = 100,<br>    is_separator_regex = False,<br>    model_name=&#39;text-embedding-3-small&#39;, #used to calculate tokens<br>    encoding_name=&#39;text-embedding-3-small&#39;<br>)<br><br>doc_list = text_splitter.create_documents([hp_book])<br>doc_list # returns list of document objects</pre><p>We can split the text, and return text chunks.</p><pre>line_list = text_splitter.split_text(hp_book)</pre><p>To convert the split text back to list of document objects</p><pre>from langchain.docstore.document import Document<br><br>doc_list = []<br>for line in line_list:<br>    curr_doc = Document(page_content = line, metadata = {&quot;source&quot;:filepath})<br>     doc_list.append(curr_doc)</pre><blockquote><strong>Splitting by code</strong></blockquote><p>We can split codes written in any programming language. Here is an example using <em>PythonTextSplitter</em>.</p><pre>python_code = &quot;&quot;&quot;def peer_review(article_id):<br>    chat = ChatOpenAI()<br>    loader = ArxivLoader(query=article_id, load_max_docs=2)<br>    data = loader.load()<br>    first_record = data[0]<br>    page_content = first_record.page_content<br>    title = first_record.metadata[&#39;Title&#39;]<br>    summary = first_record.metadata[&#39;Summary&#39;]<br>    &#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;<br>    &#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;&#39;<br>    return response.content&quot;&quot;&quot;</pre><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter, Language<br><br>text_splitter = RecursiveCharacterTextSplitter.from_language(<br>    language=Language.PYTHON,<br>    chunk_size = 50,<br>    chunk_overlap = 10<br>)<br><br>text_splitter.create_documents(texts = [python_code])</pre><pre>[Document(page_content=&#39;def peer_review(article_id):&#39;),<br> Document(page_content=&#39;chat = ChatOpenAI()&#39;),<br> Document(page_content=&#39;loader = ArxivLoader(query=article_id,&#39;),<br> Document(page_content=&#39;load_max_docs=2)&#39;),<br> .....................................................................)]</pre><p>After from all these text splitters, we also have splitters using NLTK,Spacy, Sentence Transformers, etc.</p><h3>Text Embeddings</h3><p>Embeddings are used to create a vector representation of the text. These are stored along with their corresponding text in the vector database. We use an embedding function to create embeddings of the documents.</p><p>The base Embeddings class in LangChain provides two methods: one for embedding documents(to be searched over) and one for embedding a query(the search query). The former takes as input multiple texts, while the latter takes a single text.</p><blockquote><strong>OpenAI Embeddings</strong></blockquote><pre>import os<br>from langchain.embeddings import OpenAIEmbeddings<br><br># setting up OPENAI API key as environment variable<br>with open(&quot;../../openai_api_key.txt&quot;) as f:<br>    api_key = f.read()<br>os.environ[&#39;OPENAI_API_KEY&#39;] = api_key<br><br># creating an instance of OpenAIEmbeddings model<br>embeddings = OpenAIEmbeddings()</pre><pre>text = &quot;The scar had not pained Harry for nineteen years. All was well.&quot;<br>embedded_text = embeddings.embed_query(text)<br>print(embedded_text[:5]), print(len(embedded_text))<br><br>#[-0.006067691294778975, -0.006654051049083575, 0.03343223365953213, -0.02039625470103048, -0.008338620781671906]<br>#1536</pre><p>If we have multiple lines in the document.</p><pre>from langchain.docstore.document import Document<br><br># convert into document<br>doc_lines = [<br>    Document(page_content=text, metadata = {&quot;source&quot;:&quot;Harry Potter&quot;}),<br>    Document(page_content=&quot;It is our choices, Harry, that show what we truly are, far more than our abilities&quot;, metadata = {&quot;source&quot;:&quot;Harry Potter&quot;})<br>]<br><br># to extract the page content<br>line_List = [doc.page_content for doc in doc_lines]<br><br># embed the documents<br>embedded_docs = [embeddings.embed_query(text) for text in line_List]<br><br>np.array(embedded_docs).shape<br># (2, 1536)</pre><p>Now, OpenAI Embeddings are expensive. Let’s explore some <a href="https://huggingface.co/spaces/mteb/leaderboard"><em>best performing</em></a> open source embedding models.</p><blockquote><strong>BGE Model( BAAI(Beijing Academy of Artificial Intelligence) General Embeddings) Model</strong></blockquote><p>BGE models on HuggingFaceare one of the best open source embedding models.</p><pre>import numpy as np<br>from langchain.embeddings import HuggingFaceEmbeddings<br><br>model_name = &quot;BAAI/bge-base-en-v1.5&quot;<br>model_kwargs = {&quot;device&quot;:&#39;cpu&#39;}<br>encode_kwargs = {&#39;normalize_embeddings&#39;:True}<br><br>hf = HuggingFaceEmbeddings(<br>    model_name = model_name,<br>    model_kwargs = model_kwargs,<br>    encode_kwargs = encode_kwargs<br>)</pre><pre>import numpy as np<br>embedded_docs = [hf.embed_query(text) for text in line_List]<br><br>np.array(embedded_docs).shape<br># (2,768)</pre><blockquote><strong>Fake Embeddings</strong></blockquote><p>If we have some hardware constraints, we can also use something <em>Fake Embeddings</em> from LangChain.</p><pre>from langchain_community.embeddings import FakeEmbeddings<br><br>fake_embeddings = FakeEmbeddings(size = 300) # embedding size<br><br>fake_embedding_record = fake_embeddings.embed_query(&quot;This is a random text&quot;)<br>fake_embedding_records = fake_embeddings.embed_documents([&quot;This is a random text&quot;])</pre><p>So,this is all I had regarding Text Splitters and Text Embeddings in Retrievers. There are a lot more information available in the official LangChain documentation.</p><p>I have added all the code in my <a href="https://github.com/jayantkp/LangChain/tree/main/RAG">Github</a> repo. Let me know if you have questions. I will see you in the next one.</p><p>Keep Learning!</p><blockquote><strong>References</strong></blockquote><ul><li><a href="https://python.langchain.com/docs/modules/data_connection/document_transformers/">Text Splitters | 🦜️🔗 Langchain</a></li><li><a href="https://python.langchain.com/docs/modules/data_connection/text_embedding/">Text embedding models | 🦜️🔗 LangChain</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3225af44e7ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG using LangChain : Part 1-Document Loaders]]></title>
            <link>https://jayant017.medium.com/rag-using-langchain-part-1-document-loaders-b4774c04fd85?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4774c04fd85</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 12:08:47 GMT</pubDate>
            <atom:updated>2024-03-17T14:36:23.276Z</atom:updated>
            <content:encoded><![CDATA[<h3>RAG using LangChain : Part 1-Document Loaders</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CP7Tvbmj9cJ825eV" /><figcaption>Photo by <a href="https://unsplash.com/@paulfrenzel?utm_source=medium&amp;utm_medium=referral">Paul Frenzel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome to a new series of articles on LangChain and LLMs. In this series, we will be learning about RAG in LLMs. Specifically in this article, we will be looking into Document Loaders in RAG. Let’s begin!</p><blockquote><strong>Beyond the Hype: Addressing the Challenges of LLMs</strong></blockquote><p>We know that Large Language Models are often trained on huge corpus of data. This type of training makes them very efficient in terms of generating content very effectively. But there are also some challenges that exists with these models.</p><ul><li><strong><em>Limited Knowledge Base</em></strong>- The data that the model is trained upon might not be the latest data. It can contain outdated information, factual errors or lack of context. There can also be some potential biases in the data.</li></ul><p>The trained model itself may have some potential issues like:</p><ul><li><strong><em>Black Box Nature</em></strong><em> </em>-The output generated by a LLM is sometimes difficult to understand which makes it difficult to verify the source or accuracy of the generated information.</li><li><strong><em>Hallucinations/Generation Bias </em></strong>-LLMs can sometimes fabricate information i.e. generate information which is incorrect or misleading. This can be due to their limited understanding or incomplete or biased data.</li></ul><p>Now, for a domain specific task, we can fine tune any pre-trained LLM and make it suitable for our task. But that itself is a computationally expensive task. Imagine the time and resource needed to retrain a massive LLM!</p><p>This is where <strong><em>RAG</em></strong> comes into picture!</p><blockquote><strong>RAG(Retrieval Augmented Generation)</strong></blockquote><p>RAG is a technique that improves the capabilities of LLMs by combining them with external data sources. There are three key functionalities of RAG.</p><ol><li><strong><em>Retrieval</em></strong>: This is where the most relevant information is identified(from the external data). In this article, we will mostly be covering this series.</li><li><strong><em>Augmentation</em></strong>: This step involves preprocessing and analyzing the retrieved information to make it suitable for the LLM using Summarization, Fact Checking and Formatting.</li><li><strong><em>Generation</em></strong>: With the retrieved and augmented information at end, the LLM generates better in context responses.</li></ol><p>The Retrieval part has multiple components. We will be looking into the load part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*UBX8w3Ef-89P-Ezv57L1hw.png" /></figure><h3><strong><em>Document Loaders</em></strong></h3><p>Document loaders are tools that play a crucial role in data ingestion. They take in raw data from different sources and convert them into a structured format called <strong><em>“Documents”. </em></strong>These documents contain the document content as well as the associated metadata like source and timestamps. Let’s look into the different types of document loaders.</p><blockquote><strong>CSV Loader</strong></blockquote><p>The CSV loader loads a CSV file with a single row per document. The output contains both page content as well as the metadata associated.</p><pre>from langchain.document_loaders import CSVLoader<br>loader = CSVLoader(file_path=&quot;../datasets/sns_datasets/titanic.csv&quot;)<br>data = loader.load()<br>print(data[0])</pre><pre>page_content=&#39;survived: 0\npclass: 3\nsex: male\nage: 22.0\nsibsp: 1\nparch: 0\nfare: 7.25\nembarked: S\nclass: Third\nwho: man\nadult_male: True\ndeck: \nembark_town: Southampton\nalive: no\nalone: False&#39; metadata={&#39;source&#39;: &#39;../datasets/sns_datasets/titanic.csv&#39;, &#39;row&#39;: 0}</pre><blockquote><strong>HTML Loader</strong></blockquote><p>We can load HTML documents in a document format that we can use for further downstream tasks. We have similar syntax.</p><pre>from langchain.document_loaders import UnstructuredHTMLLoader<br>loader = UnstructuredHTMLLoader(&#39;../datasets/harry_potter_html/001.htm&#39;)<br>data = loader.load()<br>data</pre><pre>[Document(page_content=&#39;A Day of Very Low Probability\n\nBeneath the moonlight glints a tiny fragment of silver, a fraction of a line…\n\n(black robes, falling)\n\n…blood spills out in litres, and someone screams a word.\n\nEvery inch of wall space is covered by a bookcase. Each bookcase has six shelves, going almost to the ceiling. Some bookshelves are stacked to the brim with hardback books: science, maths, history, and everything else. Other shelves have two layers of paperback science fiction, with the back layer of books propped up on old tissue boxes or lengths of wood, so that you can see the back layer of books above the books in front. And it still isn’t enough. Books are overflowing onto the tables and the sofas and making little heaps under the windows.\n\nThis is the living-room of the house occupied by the eminent Professor Michael Verres-Evans, and his wife, Mrs. Petunia Evans-Verres, and their adopted son, Harry James Potter-Evans-Verres.\n\nThere is a letter lying on the living-room table, and an unstamped envelope of yellowish parchment, addressed to Mr. H. Potter in emerald-green ink.\n\nThe Professor and his wife are speaking sharply at each other, but they are not shouting. The Professor considers shouting to be uncivilised.\n\n“You’re joking,” Michael said to Petunia. His tone indicated that he was very much afraid that she was serious.\n\n“My sister was a witch,” Petunia repeated. She looked frightened, but stood her ground. “Her husband was a wizard.”\n\n“This is absurd!” Michael said sharply. “They were at our wedding—they visited for Christmas—”\n\n“I told them you weren’t to know,” Petunia whispered. “But it’s true. I’ve seen things—”\n\nThe Professor rolled his eyes. “Dear, I understand that you’re not familiar with the sceptical literature. You may not realise how easy it is for a trained magician to fake the seemingly impossible. Remember how I taught Harry to bend spoons? If it seemed like they could always guess what you were thinking, that’s called cold reading—”\n\n“It wasn’t bending spoons—”\n\n“What was it, then?”\n\nPetunia bit her lip. “I can’t just tell you. You’ll think I’m—” She swallowed. “Listen. Michael. I wasn’t—always like this—” She gestured at herself, as though to indicate her lithe form. “Lily did this. Because I—because I begged her. For years, I begged her. Lily had always been prettier than me, and I’d… been mean to her, because of that, and then she got magic, can you imagine how I felt? And I begged her to use some of that magic on me so that I could be pretty too, even if I couldn’t have her magic, at least I could be pretty.”\n\nTears were gathering in Petunia’s eyes.\n\n“And Lily would tell me no, and make up the most ridiculous excuses, like the world would end if she were nice to her sister, or a centaur told her not to—the most ridiculous things, and I hated her for it. And when I had just graduated from university, I was going out with this boy, Vernon Dursley, he was fat and he was the only boy who would talk to me. And he said he wanted children, and that his first son would be named Dudley. And I thought to myself, what kind of parent names their child Dudley Dursley? It was like I saw my whole future life stretching out in front of me, and I couldn’t stand it. And I wrote to my sister and told her that if she didn’t help me I’d rather just—”\n\nPetunia stopped.\n\n“Anyway,” Petunia said, her voice small, “she gave in. She told me it was dangerous, and I said I didn’t care any more, and I drank this potion and I was sick for weeks, but when I got better my skin cleared up and I finally filled out and… I was beautiful, people were nice to me,” her voice broke, “and after that I couldn’t hate my sister any more, especially when I learned what her magic brought her in the end—”\n\n“Darling,” Michael said gently, “you got sick, you gained some weight while resting in bed, and your skin cleared up on its own. Or being sick made you change your diet—”\n\n“She was a witch,” Petunia repeated. “I saw it.”\n\n“Petunia,” Michael said. The annoyance was creeping into his voice. “You know that can’t be true. Do I really have to explain why?”\n\nPetunia wrung her hands. She seemed to be on the verge of tears. “My love, I know I can’t win arguments with you, but please, you have to trust me on this—”\n\n“Dad! Mum!”\n\nThe two of them stopped and looked at Harry as though they’d forgotten there was a third person in the room.\n\nHarry took a deep breath. “Mum, your parents didn’t have magic, did they?”\n\n“No,” Petunia said, looking puzzled.\n\n“Then no one in your family knew about magic when Lily got her letter. How did they get convinced?”\n\n“Ah…” Petunia said. “They didn’t just send a letter. They sent a professor from Hogwarts. He—” Petunia’s eyes flicked to Michael. “He showed us some magic.”\n\n“Then you don’t have to fight over this,” Harry said firmly. Hoping against hope that this time, just this once, they would listen to him. “If it’s true, we can just get a Hogwarts professor here and see the magic for ourselves, and Dad will admit that it’s true. And if not, then Mum will admit that it’s false. That’s what the experimental method is for, so that we don’t have to resolve things just by arguing.”\n\nThe Professor turned and looked down at him, dismissive as usual. “Oh, come now, Harry. Really, magic? I thought you’d know better than to take this seriously, son, even if you’re only ten. Magic is just about the most unscientific thing there is!”\n\nHarry’s mouth twisted bitterly. He was treated well, probably better than most genetic fathers treated their own children. Harry had been sent to the best primary schools—and when that didn’t work out, he was provided with tutors from the endless pool of starving students. Always Harry had been encouraged to study whatever caught his attention, bought all the books that caught his fancy, sponsored in whatever maths or science competitions he entered. He was given anything reasonable that he wanted, except, maybe, the slightest shred of respect. A Doctor teaching biochemistry at Oxford could hardly be expected to listen to the advice of a little boy. You would listen to Show Interest, of course; that’s what a Good Parent would do, and so, if you conceived of yourself as a Good Parent, you would do it. But take a ten-year-old seriously? Hardly.\n\nSometimes Harry wanted to scream at his father.\n\n“Mum,” Harry said. “If you want to win this argument with Dad, look in chapter two of the first book of the Feynman Lectures on Physics. There’s a quote there about how philosophers say a great deal about what science absolutely requires, and it is all wrong, because the only rule in science is that the final arbiter is observation—that you just have to look at the world and report what you see. Um… off the top of my head I can’t think of where to find something about how it’s an ideal of science to settle things by experiment instead of arguments—”\n\nHis mother looked down at him and smiled. “Thank you, Harry. But—” her head rose back up to stare at her husband. “I don’t want to win an argument with your father. I want my husband to, to listen to his wife who loves him, and trust her just this once—”\n\nHarry closed his eyes briefly. Hopeless. Both of his parents were just hopeless.\n\nNow his parents were getting into one of those arguments again, one where his mother tried to make his father feel guilty, and his father tried to make his mother feel stupid.\n\n“I’m going to go to my room,” Harry announced. His voice trembled a little. “Please try not to fight too much about this, Mum, Dad, we’ll know soon enough how it comes out, right?”\n\n“Of course, Harry,” said his father, and his mother gave him a reassuring kiss, and then they went on fighting while Harry climbed the stairs to his bedroom.\n\nHe shut the door behind him and tried to think.\n\nThe funny thing was, he should have agreed with Dad. No one had ever seen any evidence of magic, and according to Mum, there was a whole magical world out there. How could anyone keep something like that a secret? More magic? That seemed like a rather suspicious sort of excuse.\n\nIt should have been a clean case for Mum joking, lying or being insane, in ascending order of awfulness. If Mum had sent the letter herself, that would explain how it arrived at the letterbox without a stamp. A little insanity was far, far less improbable than the universe really working like that.\n\nExcept that some part of Harry was utterly convinced that magic was real, and had been since the instant he saw the putative letter from the Hogwarts School of Witchcraft and Wizardry.\n\nHarry rubbed his forehead, grimacing. Don’t believe everything you think, one of his books had said.\n\nBut this bizarre certainty… Harry was finding himself just expecting that, yes, a Hogwarts professor would show up and wave a wand and magic would come out. The strange certainty was making no effort to guard itself against falsification—wasn’t making excuses in advance for why there wouldn’t be a professor, or the professor would only be able to bend spoons.\n\nWhere do you come from, strange little prediction? Harry directed the thought at his brain. Why do I believe what I believe?\n\nUsually Harry was pretty good at answering that question, but in this particular case, he had no clue what his brain was thinking.\n\nHarry mentally shrugged. A flat metal plate on a door affords pushing, and a handle on a door affords pulling, and the thing to do with a testable hypothesis is to go and test it.\n\nHe took a piece of lined paper from his desk, and started writing.\n\nDear Deputy Headmistress\n\nHarry paused, reflecting; then discarded the paper for another, tapping another millimetre of graphite from his mechanical pencil. This called for careful calligraphy.\n\nDear Deputy Headmistress Minerva McGonagall,\n\nOr Whomsoever It May Concern:\n\nI recently received your letter of acceptance to Hogwarts, addressed to Mr. H. Potter. You may not be aware that my genetic parents, James Potter and Lily Potter (formerly Lily Evans) are dead. I was adopted by Lily’s sister, Petunia Evans-Verres, and her husband, Michael Verres-Evans.\n\nI am extremely interested in attending Hogwarts, conditional on such a place actually existing. Only my mother Petunia says she knows about magic, and she can’t use it herself. My father is highly sceptical. I myself am uncertain. I also don’t know where to obtain any of the books or equipment listed in your acceptance letter.\n\nMother mentioned that you sent a Hogwarts representative to Lily Potter (then Lily Evans) in order to demonstrate to her family that magic was real, and, I presume, help Lily obtain her school materials. If you could do this for my own family it would be extremely helpful.\n\nSincerely,\n\nHarry James Potter-Evans-Verres.\n\nHarry added their current address, then folded up the letter and put it in an envelope, which he addressed to Hogwarts. Further consideration led him to obtain a candle and drip wax onto the flap of the envelope, into which, using a penknife’s tip, he impressed the initials H.J.P.E.V. If he was going to descend into this madness, he was going to do it with style.\n\nThen he opened his door and went back downstairs. His father was sitting in the living-room and reading a book of higher maths to show how smart he was; and his mother was in the kitchen preparing one of his father’s favourite meals to show how loving she was. It didn’t look like they were talking to one another at all. As scary as arguments could be, not arguing was somehow much worse.\n\n“Mum,” Harry said into the unnerving silence, “I’m going to test the hypothesis. According to your theory, how do I send an owl to Hogwarts?”\n\nHis mother turned from the kitchen sink to stare at him, looking shocked. “I—I don’t know, I think you just have to own a magic owl.”\n\nThat should’ve sounded highly suspicious, oh, so there’s no way to test your theory then, but the peculiar certainty in Harry seemed willing to stick its neck out even further.\n\n“Well, the letter got here somehow,” Harry said, “so I’ll just wave it around outside and call ‘letter for Hogwarts!’ and see if an owl picks it up. Dad, do you want to come and watch?”\n\nHis father shook his head minutely and kept on reading. Of course, Harry thought to himself. Magic was a disgraceful thing that only stupid people believed in; if his father went so far as to test the hypothesis, or even watch it being tested, that would feel like associating himself with that…\n\nOnly as Harry stumped out the back door, into the back garden, did it occur to him that if an owl did come down and snatch the letter, he was going to have some trouble telling Dad about it.\n\nBut—well—that can’t really happen, can it? No matter what my brain seems to believe. If an owl really comes down and grabs this envelope, I’m going to have worries a lot more important than what Dad thinks.\n\nHarry took a deep breath, and raised the envelope into the air.\n\nHe swallowed.\n\nCalling out Letter for Hogwarts! while holding an envelope high in the air in the middle of your own back garden was… actually pretty embarrassing, now that he thought about it.\n\nNo. I’m better than Dad. I will use the scientific method even if it makes me feel stupid.\n\n“Letter—” Harry said, but it actually came out as more of a whispered croak.\n\nHarry steeled his will, and shouted into the empty sky, “Letter for Hogwarts! Can I get an owl?”\n\n“Harry?” asked a bemused woman’s voice, one of the neighbours.\n\nHarry pulled down his hand like it was on fire and hid the envelope behind his back like it was drug money. His whole face was hot with shame.\n\nAn old woman’s face peered out from above the neighbouring fence, grizzled grey hair escaping from her hairnet. Mrs. Figg, the occasional babysitter. “What are you doing, Harry?”\n\n“Nothing,” Harry said in a strangled voice. “Just—testing a really silly theory—”\n\n“Did you get your acceptance letter from Hogwarts?”\n\nHarry froze in place.\n\n“Yes,” Harry’s lips said a little while later. “I got a letter from Hogwarts. They say they want my owl by the 31st of July, but—”\n\n“But you don’t have an owl. Poor dear! I can’t imagine what someone must have been thinking, sending you just the standard letter.”\n\nA wrinkled arm stretched out over the fence, and opened an expectant hand. Hardly even thinking at this point, Harry gave over his envelope.\n\n“Just leave it to me, dear,” said Mrs. Figg, “and in a jiffy or two I’ll have someone over.”\n\nAnd her face disappeared from over the fence.\n\nThere was a long silence in the garden.\n\nThen a boy’s voice said, calmly and quietly, “What.”&#39;, metadata={&#39;source&#39;: &#39;../datasets/harry_potter_html/001.htm&#39;})]</pre><p>We can also load HTML documents with BeautifulSoup using theBSHTMLLoader.</p><pre>from langchain.document_loaders import BSHTMLLoader<br>loader = BSHTMLLoader(&#39;../datasets/harry_potter_html/001.htm&#39;)<br>data = loader.load()<br>data</pre><blockquote><strong>Markdown Loader</strong></blockquote><pre>from langchain.document_loaders import UnstructuredMarkdownLoader<br>loader = UnstructuredMarkdownLoader(file_path=&#39;../datasets/harry_potter_md/001.md&#39;)<br>data = loader.load()<br>print(data[0].page_content)</pre><pre>A Day of Very Low Probability<br><br>Beneath the moonlight glints a tiny fragment of silver, a fraction of a line…<br><br>(black robes, falling)<br><br>…blood spills out in litres, and someone screams a word.<br><br>Every inch of wall space is covered by a bookcase. Each bookcase has six shelves, going almost to the ceiling. Some bookshelves are stacked to the brim with hardback books: science, maths, history, and everything else. Other shelves have two layers of paperback science fiction, with the back layer of books propped up on old tissue boxes or lengths of wood, so that you can see the back layer of books above the books in front. And it still isn’t enough. Books are overflowing onto the tables and the sofas and making little heaps under the windows....</pre><blockquote><strong>PDF Loader</strong></blockquote><p>Here we use PyPDF load the PDF documents. Each document contains the page content and metadata with page numbers.</p><pre>from langchain.document_loaders import PyPDFLoader<br>loader = PyPDFLoader(&#39;../datasets/harry_potter_pdf/hpmor-trade-classic.pdf&#39;)<br><br>data = loader.load()<br><br>print(data[0].page_content)</pre><pre>Harry Potter and the Methods of Rationality</pre><blockquote><strong>Wikipedia Loader</strong></blockquote><p>We use WikipediaLoader to fetch the documents. It mainly has 3 arguments: query: which is used to find the topic in Wikipedia, optional lang: default=“en”, and load_max_docs: default=100, which is used to limit the number of documents downloaded.</p><pre>from langchain.document_loaders import WikipediaLoader<br>loader = WikipediaLoader(query=&#39;LangChain&#39;, load_max_docs=1)<br>data = loader.load()<br><br>data</pre><pre>[Document(page_content=&#39;LangChain is a framework designed to simplify the creation of applications using large language models (LLMs). As a language model integration framework, LangChain\&#39;s use-cases largely overlap with those of language models in general, including document analysis and summarization, chatbots, and code analysis.\n\n\n== History ==\nLangChain was launched in October 2022 as an open source project by Harrison Chase, while working at machine learning startup Robust Intelligence. The project quickly garnered popularity, with improvements from hundreds of contributors on GitHub, trending discussions on Twitter, lively activity on the project\&#39;s Discord server, many YouTube tutorials, and meetups in San Francisco and London. In April 2023, LangChain had incorporated and the new startup raised over $20 million in funding at a valuation of at least $200 million from venture firm Sequoia Capital, a week after announcing a $10 million seed investment from Benchmark.In October 2023 LangChain introduced LangServe, a deployment tool designed to facilitate the transition from LCEL (LangChain Expression Language) prototypes to production-ready applications.\n\n\n== Capabilities ==\nLangChain\&#39;s developers highlight the framework\&#39;s applicability to use-cases including chatbots, retrieval-augmented generation,  document summarization, and synthetic data generation.As of March 2023, LangChain included integrations with systems including Amazon, Google, and Microsoft Azure cloud storage; API wrappers for news, movie information, and weather; Bash for summarization, syntax and semantics checking, and execution of shell scripts; multiple web scraping subsystems and templates; few-shot learning prompt generation support; finding and summarizing &quot;todo&quot; tasks in code; Google Drive documents, spreadsheets, and presentations summarization, extraction, and creation; Google Search and Microsoft Bing web search; OpenAI, Anthropic, and Hugging Face language models; iFixit repair guides and wikis search and summarization; MapReduce for question answering, combining documents, and question generation; N-gram overlap scoring; PyPDF, pdfminer, fitz, and pymupdf for PDF file text extraction and manipulation; Python and JavaScript code generation, analysis, and debugging; Milvus vector database to store and retrieve vector embeddings; Weaviate vector database to cache embedding and data objects; Redis cache database storage; Python RequestsWrapper and other methods for API requests; SQL and NoSQL databases including JSON support; Streamlit, including for logging; text mapping for k-nearest neighbors search; time zone conversion and calendar operations; tracing and recording stack symbols in threaded and asynchronous subprocess runs; and the Wolfram Alpha website and SDK. As of April 2023, it can read from more than 50 document types and data sources.\n\n\n== References ==\n\n\n== External links ==\n\nOfficial website\nDiscord server support hub\nLangchain-ai on GitHub&#39;, metadata={&#39;title&#39;: &#39;LangChain&#39;, &#39;summary&#39;: &quot;LangChain is a framework designed to simplify the creation of applications using large language models (LLMs). As a language model integration framework, LangChain&#39;s use-cases largely overlap with those of language models in general, including document analysis and summarization, chatbots, and code analysis.\n\n&quot;, &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/LangChain&#39;})]</pre><p>To extract the metadata.</p><pre>data[0].metadata<br><br>{&#39;title&#39;: &#39;LangChain&#39;,<br> &#39;summary&#39;: &quot;LangChain is a framework designed to simplify the creation of applications using large language models (LLMs). As a language model integration framework, LangChain&#39;s use-cases largely overlap with those of language models in general, including document analysis and summarization, chatbots, and code analysis.\n\n&quot;,<br> &#39;source&#39;: &#39;https://en.wikipedia.org/wiki/LangChain&#39;}</pre><p>We can use this dictionary to extract specific information from the data.</p><blockquote><strong>ArXiv Loader</strong></blockquote><p><a href="https://arxiv.org/">arXiv</a> is an open-access archive for 2 million scholarly articles. We can use ArxivLoader to extract information of any paper. We need the article id that would be available in the URL of the paper to use the loader.</p><pre>from langchain_community.document_loaders import ArxivLoader<br>loader = ArxivLoader(query=&#39;1706.03762&#39;, load_max_docs=1) <br>data = loader.load()<br>data</pre><pre>[Document(page_content=&#39;Provided proper attribution is provided, Google hereby grants permission to\nreproduce the tables and figures in this paper solely for use in journalistic or\nscholarly works.\nAttention Is All You Need\nAshish Vaswani∗\nGoogle Brain\navaswani@google.com\nNoam Shazeer∗\nGoogle Brain\nnoam@google.com\nNiki Parmar∗\nGoogle Research\nnikip@google.com\nJakob Uszkoreit∗\nGoogle Research\nusz@google.com\nLlion Jones∗\nGoogle Research\nllion@google.com\nAidan N. Gomez∗ †\nUniversity of Toronto\naidan@cs.toronto.edu\nŁukasz Kaiser∗\nGoogle Brain\nlukaszkaiser@google.com\nIllia Polosukhin∗ ‡\nillia.polosukhin@gmail.com\nAbstract\nThe dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks that include an encoder and a decoder. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer,\nbased solely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to\nbe superior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-\nto-German translation task, improving over the existing best results, including\nensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,\nour model establishes a new single-model state-of-the-art BLEU score of 41.8 after\ntraining for 3.5 days on eight GPUs, a small fraction of the training costs of the\nbest models from the literature. We show that the Transformer generalizes well to\nother tasks by applying it successfully to English constituency parsing both with\nlarge and limited training data.\n∗Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started\nthe effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and\nhas been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head\nattention and the parameter-free position representation and became the other person involved in nearly every\ndetail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and\ntensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and\nefficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and\nimplementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating\nour research.\n†Work performed while at Google Brain.\n‡Work performed while at Google Research.\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\narXiv:1706.03762v7  [cs.CL]  2 Aug 2023\n1\nIntroduction\nRecurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks\nin particular, have been firmly established as state of the art approaches in sequence modeling and\ntransduction problems such as language modeling and machine translation [35, 2, 5]. Numerous\nefforts have since continued to push the boundaries of recurrent language models and encoder-decoder\narchitectures [38, 24, 15].\nRecurrent models typically factor computation along the symbol positions of the input and output\nsequences. Aligning the positions to steps in computation time, they generate a sequence of hidden\nstates ht, as a function of the previous hidden state ht−1 and the input for position t. This inherently\nsequential nature precludes parallelization within training examples, which becomes critical at longer\nsequence lengths, as memory constraints limit batching across examples. Recent work has achieved\nsignificant improvements in computational efficiency through factorization tricks [21] and conditional\ncomputation [32], while also improving model performance in case of the latter. The fundamental\nconstraint of sequential computation, however, remains.\nAttention mechanisms have become an integral part of compelling sequence modeling and transduc-\ntion models in various tasks, allowing modeling of dependencies without regard to their distance in\nthe input or output sequences [2, 19]. In all but a few cases [27], however, such attention mechanisms\nare used in conjunction with a recurrent network.\nIn this work we propose the Transformer, a model architecture eschewing recurrence and instead\nrelying entirely on an attention mechanism to draw global dependencies between input and output.\nThe Transformer allows for significantly more parallelization and can reach a new state of the art in\ntranslation quality after being trained for as little as twelve hours on eight P100 GPUs.\n2\nBackground\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU\n[16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building\nblock, computing hidden representations in parallel for all input and output positions. In these models,\nthe number of operations required to relate signals from two arbitrary input or output positions grows\nin the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes\nit more difficult to learn dependencies between distant positions [12]. In the Transformer this is\nreduced to a constant number of operations, albeit at the cost of reduced effective resolution due\nto averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as\ndescribed in section 3.2.\nSelf-attention, sometimes called intra-attention is an attention mechanism relating different positions\nof a single sequence in order to compute a representation of the sequence. Self-attention has been\nused successfully in a variety of tasks including reading comprehension, abstractive summarization,\ntextual entailment and learning task-independent sentence representations [4, 27, 28, 22].\nEnd-to-end memory networks are based on a recurrent attention mechanism instead of sequence-\naligned recurrence and have been shown to perform well on simple-language question answering and\nlanguage modeling tasks [34].\nTo the best of our knowledge, however, the Transformer is the first transduction model relying\nentirely on self-attention to compute representations of its input and output without using sequence-\naligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate\nself-attention and discuss its advantages over models such as [17, 18] and [9].\n3\nModel Architecture\nMost competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].\nHere, the encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence\nof continuous representations z = (z1, ..., zn). Given z, the decoder then generates an output\nsequence (y1, ..., ym) of symbols one element at a time. At each step the model is auto-regressive\n[10], consuming the previously generated symbols as additional input when generating the next.\n2\nFigure 1: The Transformer - model architecture.\nThe Transformer follows this overall architecture using stacked self-attention and point-wise, fully\nconnected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\nrespectively.\n3.1\nEncoder and Decoder Stacks\nEncoder:\nThe encoder is composed of a stack of N = 6 identical layers. Each layer has two\nsub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\nwise fully connected feed-forward network. We employ a residual connection [11] around each of\nthe two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is\nLayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer\nitself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\nlayers, produce outputs of dimension dmodel = 512.\nDecoder:\nThe decoder is also composed of a stack of N = 6 identical layers. In addition to the two\nsub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\nattention over the output of the encoder stack. Similar to the encoder, we employ residual connections\naround each of the sub-layers, followed by layer normalization. We also modify the self-attention\nsub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\nmasking, combined with fact that the output embeddings are offset by one position, ensures that the\npredictions for position i can depend only on the known outputs at positions less than i.\n3.2\nAttention\nAn attention function can be described as mapping a query and a set of key-value pairs to an output,\nwhere the query, keys, values, and output are all vectors. The output is computed as a weighted sum\n3\nScaled Dot-Product Attention\nMulti-Head Attention\nFigure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several\nattention layers running in parallel.\nof the values, where the weight assigned to each value is computed by a compatibility function of the\nquery with the corresponding key.\n3.2.1\nScaled Dot-Product Attention\nWe call our particular attention &quot;Scaled Dot-Product Attention&quot; (Figure 2). The input consists of\nqueries and keys of dimension dk, and values of dimension dv. We compute the dot products of the\nquery with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the\nvalues.\nIn practice, we compute the attention function on a set of queries simultaneously, packed together\ninto a matrix Q. The keys and values are also packed together into matrices K and V . We compute\nthe matrix of outputs as:\nAttention(Q, K, V ) = softmax(QKT\n√dk\n)V\n(1)\nThe two most commonly used attention functions are additive attention [2], and dot-product (multi-\nplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor\nof\n1\n√dk . Additive attention computes the compatibility function using a feed-forward network with\na single hidden layer. While the two are similar in theoretical complexity, dot-product attention is\nmuch faster and more space-efficient in practice, since it can be implemented using highly optimized\nmatrix multiplication code.\nWhile for small values of dk the two mechanisms perform similarly, additive attention outperforms\ndot product attention without scaling for larger values of dk [3]. We suspect that for large values of\ndk, the dot products grow large in magnitude, pushing the softmax function into regions where it has\nextremely small gradients 4. To counteract this effect, we scale the dot products by\n1\n√dk .\n3.2.2\nMulti-Head Attention\nInstead of performing a single attention function with dmodel-dimensional keys, values and queries,\nwe found it beneficial to linearly project the queries, keys and values h times with different, learned\nlinear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of\nqueries, keys and values we then perform the attention function in parallel, yielding dv-dimensional\n4To illustrate why the dot products get large, assume that the components of q and k are independent random\nvariables with mean 0 and variance 1. Then their dot product, q · k = Pdk\ni=1 qiki, has mean 0 and variance dk.\n4\noutput values. These are concatenated and once again projected, resulting in the final values, as\ndepicted in Figure 2.\nMulti-head attention allows the model to jointly attend to information from different representation\nsubspaces at different positions. With a single attention head, averaging inhibits this.\nMultiHead(Q, K, V ) = Concat(head1, ..., headh)W O\nwhere headi = Attention(QW Q\ni , KW K\ni , V W V\ni )\nWhere the projections are parameter matrices W Q\ni\n∈ Rdmodel×dk, W K\ni\n∈ Rdmodel×dk, W V\ni\n∈ Rdmodel×dv\nand W O ∈ Rhdv×dmodel.\nIn this work we employ h = 8 parallel attention layers, or heads. For each of these we use\ndk = dv = dmodel/h = 64. Due to the reduced dimension of each head, the total computational cost\nis similar to that of single-head attention with full dimensionality.\n3.2.3\nApplications of Attention in our Model\nThe Transformer uses multi-head attention in three different ways:\n• In &quot;encoder-decoder attention&quot; layers, the queries come from the previous decoder layer,\nand the memory keys and values come from the output of the encoder. This allows every\nposition in the decoder to attend over all positions in the input sequence. This mimics the\ntypical encoder-decoder attention mechanisms in sequence-to-sequence models such as\n[38, 2, 9].\n• The encoder contains self-attention layers. In a self-attention layer all of the keys, values\nand queries come from the same place, in this case, the output of the previous layer in the\nencoder. Each position in the encoder can attend to all positions in the previous layer of the\nencoder.\n• Similarly, self-attention layers in the decoder allow each position in the decoder to attend to\nall positions in the decoder up to and including that position. We need to prevent leftward\ninformation flow in the decoder to preserve the auto-regressive property. We implement this\ninside of scaled dot-product attention by masking out (setting to −∞) all values in the input\nof the softmax which correspond to illegal connections. See Figure 2.\n3.3\nPosition-wise Feed-Forward Networks\nIn addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully\nconnected feed-forward network, which is applied to each position separately and identically. This\nconsists of two linear transformations with a ReLU activation in between.\nFFN(x) = max(0, xW1 + b1)W2 + b2\n(2)\nWhile the linear transformations are the same across different positions, they use different parameters\nfrom layer to layer. Another way of describing this is as two convolutions with kernel size 1.\nThe dimensionality of input and output is dmodel = 512, and the inner-layer has dimensionality\ndff = 2048.\n3.4\nEmbeddings and Softmax\nSimilarly to other sequence transduction models, we use learned embeddings to convert the input\ntokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transfor-\nmation and softmax function to convert the decoder output to predicted next-token probabilities. In\nour model, we share the same weight matrix between the two embedding layers and the pre-softmax\nlinear transformation, similar to [30]. In the embedding layers, we multiply those weights by √dmodel.\n5\nTable 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations\nfor different layer types. n is the sequence length, d is the representation dimension, k is the kernel\nsize of convolutions and r the size of the neighborhood in restricted self-attention.\nLayer Type\nComplexity per Layer\nSequential\nMaximum Path Length\nOperations\nSelf-Attention\nO(n2 · d)\nO(1)\nO(1)\nRecurrent\nO(n · d2)\nO(n)\nO(n)\nConvolutional\nO(k · n · d2)\nO(1)\nO(logk(n))\nSelf-Attention (restricted)\nO(r · n · d)\nO(1)\nO(n/r)\n3.5\nPositional Encoding\nSince our model contains no recurrence and no convolution, in order for the model to make use of the\norder of the sequence, we must inject some information about the relative or absolute position of the\ntokens in the sequence. To this end, we add &quot;positional encodings&quot; to the input embeddings at the\nbottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel\nas the embeddings, so that the two can be summed. There are many choices of positional encodings,\nlearned and fixed [9].\nIn this work, we use sine and cosine functions of different frequencies:\nPE(pos,2i) = sin(pos/100002i/dmodel)\nPE(pos,2i+1) = cos(pos/100002i/dmodel)\nwhere pos is the position and i is the dimension. That is, each dimension of the positional encoding\ncorresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We\nchose this function because we hypothesized it would allow the model to easily learn to attend by\nrelative positions, since for any fixed offset k, PEpos+k can be represented as a linear function of\nPEpos.\nWe also experimented with using learned positional embeddings [9] instead, and found that the two\nversions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version\nbecause it may allow the model to extrapolate to sequence lengths longer than the ones encountered\nduring training.\n4\nWhy Self-Attention\nIn this section we compare various aspects of self-attention layers to the recurrent and convolu-\ntional layers commonly used for mapping one variable-length sequence of symbol representations\n(x1, ..., xn) to another sequence of equal length (z1, ..., zn), with xi, zi ∈ Rd, such as a hidden\nlayer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we\nconsider three desiderata.\nOne is the total computational complexity per layer. Another is the amount of computation that can\nbe parallelized, as measured by the minimum number of sequential operations required.\nThe third is the path length between long-range dependencies in the network. Learning long-range\ndependencies is a key challenge in many sequence transduction tasks. One key factor affecting the\nability to learn such dependencies is the length of the paths forward and backward signals have to\ntraverse in the network. The shorter these paths between any combination of positions in the input\nand output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare\nthe maximum path length between any two input and output positions in networks composed of the\ndifferent layer types.\nAs noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially\nexecuted operations, whereas a recurrent layer requires O(n) sequential operations. In terms of\ncomputational complexity, self-attention layers are faster than recurrent layers when the sequence\n6\nlength n is smaller than the representation dimensionality d, which is most often the case with\nsentence representations used by state-of-the-art models in machine translations, such as word-piece\n[38] and byte-pair [31] representations. To improve computational performance for tasks involving\nvery long sequences, self-attention could be restricted to considering only a neighborhood of size r in\nthe input sequence centered around the respective output position. This would increase the maximum\npath length to O(n/r). We plan to investigate this approach further in future work.\nA single convolutional layer with kernel width k &lt; n does not connect all pairs of input and output\npositions. Doing so requires a stack of O(n/k) convolutional layers in the case of contiguous kernels,\nor O(logk(n)) in the case of dilated convolutions [18], increasing the length of the longest paths\nbetween any two positions in the network. Convolutional layers are generally more expensive than\nrecurrent layers, by a factor of k. Separable convolutions [6], however, decrease the complexity\nconsiderably, to O(k · n · d + n · d2). Even with k = n, however, the complexity of a separable\nconvolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer,\nthe approach we take in our model.\nAs side benefit, self-attention could yield more interpretable models. We inspect attention distributions\nfrom our models and present and discuss examples in the appendix. Not only do individual attention\nheads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic\nand semantic structure of the sentences.\n5\nTraining\nThis section describes the training regime for our models.\n5.1\nTraining Data and Batching\nWe trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million\nsentence pairs. Sentences were encoded using byte-pair encoding [3], which has a shared source-\ntarget vocabulary of about 37000 tokens. For English-French, we used the significantly larger WMT\n2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece\nvocabulary [38]. Sentence pairs were batched together by approximate sequence length. Each training\nbatch contained a set of sentence pairs containing approximately 25000 source tokens and 25000\ntarget tokens.\n5.2\nHardware and Schedule\nWe trained our models on one machine with 8 NVIDIA P100 GPUs. For our base models using\nthe hyperparameters described throughout the paper, each training step took about 0.4 seconds. We\ntrained the base models for a total of 100,000 steps or 12 hours. For our big models,(described on the\nbottom line of table 3), step time was 1.0 seconds. The big models were trained for 300,000 steps\n(3.5 days).\n5.3\nOptimizer\nWe used the Adam optimizer [20] with β1 = 0.9, β2 = 0.98 and ϵ = 10−9. We varied the learning\nrate over the course of training, according to the formula:\nlrate = d−0.5\nmodel · min(step_num−0.5, step_num · warmup_steps−1.5)\n(3)\nThis corresponds to increasing the learning rate linearly for the first warmup_steps training steps,\nand decreasing it thereafter proportionally to the inverse square root of the step number. We used\nwarmup_steps = 4000.\n5.4\nRegularization\nWe employ three types of regularization during training:\n7\nTable 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the\nEnglish-to-German and English-to-French newstest2014 tests at a fraction of the training cost.\nModel\nBLEU\nTraining Cost (FLOPs)\nEN-DE\nEN-FR\nEN-DE\nEN-FR\nByteNet [18]\n23.75\nDeep-Att + PosUnk [39]\n39.2\n1.0 · 1020\nGNMT + RL [38]\n24.6\n39.92\n2.3 · 1019\n1.4 · 1020\nConvS2S [9]\n25.16\n40.46\n9.6 · 1018\n1.5 · 1020\nMoE [32]\n26.03\n40.56\n2.0 · 1019\n1.2 · 1020\nDeep-Att + PosUnk Ensemble [39]\n40.4\n8.0 · 1020\nGNMT + RL Ensemble [38]\n26.30\n41.16\n1.8 · 1020\n1.1 · 1021\nConvS2S Ensemble [9]\n26.36\n41.29\n7.7 · 1019\n1.2 · 1021\nTransformer (base model)\n27.3\n38.1\n3.3 · 1018\nTransformer (big)\n28.4\n41.8\n2.3 · 1019\nResidual Dropout\nWe apply dropout [33] to the output of each sub-layer, before it is added to the\nsub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the\npositional encodings in both the encoder and decoder stacks. For the base model, we use a rate of\nPdrop = 0.1.\nLabel Smoothing\nDuring training, we employed label smoothing of value ϵls = 0.1 [36]. This\nhurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\n6\nResults\n6.1\nMachine Translation\nOn the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big)\nin Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0\nBLEU, establishing a new state-of-the-art BLEU score of 28.4. The configuration of this model is\nlisted in the bottom line of Table 3. Training took 3.5 days on 8 P100 GPUs. Even our base model\nsurpasses all previously published models and ensembles, at a fraction of the training cost of any of\nthe competitive models.\nOn the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0,\noutperforming all of the previously published single models, at less than 1/4 the training cost of the\nprevious state-of-the-art model. The Transformer (big) model trained for English-to-French used\ndropout rate Pdrop = 0.1, instead of 0.3.\nFor the base models, we used a single model obtained by averaging the last 5 checkpoints, which\nwere written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints. We\nused beam search with a beam size of 4 and length penalty α = 0.6 [38]. These hyperparameters\nwere chosen after experimentation on the development set. We set the maximum output length during\ninference to input length + 50, but terminate early when possible [38].\nTable 2 summarizes our results and compares our translation quality and training costs to other model\narchitectures from the literature. We estimate the number of floating point operations used to train a\nmodel by multiplying the training time, the number of GPUs used, and an estimate of the sustained\nsingle-precision floating-point capacity of each GPU 5.\n6.2\nModel Variations\nTo evaluate the importance of different components of the Transformer, we varied our base model\nin different ways, measuring the change in performance on English-to-German translation on the\n5We used values of 2.8, 3.7, 6.0 and 9.5 TFLOPS for K80, K40, M40 and P100, respectively.\n8\nTable 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base\nmodel. All metrics are on the English-to-German translation development set, newstest2013. Listed\nperplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to\nper-word perplexities.\nN\ndmodel\ndff\nh\ndk\ndv\nPdrop\nϵls\ntrain\nPPL\nBLEU\nparams\nsteps\n(dev)\n(dev)\n×106\nbase\n6\n512\n2048\n8\n64\n64\n0.1\n0.1\n100K\n4.92\n25.8\n65\n(A)\n1\n512\n512\n5.29\n24.9\n4\n128\n128\n5.00\n25.5\n16\n32\n32\n4.91\n25.8\n32\n16\n16\n5.01\n25.4\n(B)\n16\n5.16\n25.1\n58\n32\n5.01\n25.4\n60\n(C)\n2\n6.11\n23.7\n36\n4\n5.19\n25.3\n50\n8\n4.88\n25.5\n80\n256\n32\n32\n5.75\n24.5\n28\n1024\n128\n128\n4.66\n26.0\n168\n1024\n5.12\n25.4\n53\n4096\n4.75\n26.2\n90\n(D)\n0.0\n5.77\n24.6\n0.2\n4.95\n25.5\n0.0\n4.67\n25.3\n0.2\n5.47\n25.7\n(E)\npositional embedding instead of sinusoids\n4.92\n25.7\nbig\n6\n1024\n4096\n16\n0.3\n300K\n4.33\n26.4\n213\ndevelopment set, newstest2013. We used beam search as described in the previous section, but no\ncheckpoint averaging. We present these results in Table 3.\nIn Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions,\nkeeping the amount of computation constant, as described in Section 3.2.2. While single-head\nattention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.\nIn Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality. This\nsuggests that determining compatibility is not easy and that a more sophisticated compatibility\nfunction than dot product may be beneficial. We further observe in rows (C) and (D) that, as expected,\nbigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our\nsinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical\nresults to the base model.\n6.3\nEnglish Constituency Parsing\nTo evaluate if the Transformer can generalize to other tasks we performed experiments on English\nconstituency parsing. This task presents specific challenges: the output is subject to strong structural\nconstraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence\nmodels have not been able to attain state-of-the-art results in small-data regimes [37].\nWe trained a 4-layer transformer with dmodel = 1024 on the Wall Street Journal (WSJ) portion of the\nPenn Treebank [25], about 40K training sentences. We also trained it in a semi-supervised setting,\nusing the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences\n[37]. We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens\nfor the semi-supervised setting.\nWe performed only a small number of experiments to select the dropout, both attention and residual\n(section 5.4), learning rates and beam size on the Section 22 development set, all other parameters\nremained unchanged from the English-to-German base translation model. During inference, we\n9\nTable 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23\nof WSJ)\nParser\nTraining\nWSJ 23 F1\nVinyals &amp; Kaiser el al. (2014) [37]\nWSJ only, discriminative\n88.3\nPetrov et al. (2006) [29]\nWSJ only, discriminative\n90.4\nZhu et al. (2013) [40]\nWSJ only, discriminative\n90.4\nDyer et al. (2016) [8]\nWSJ only, discriminative\n91.7\nTransformer (4 layers)\nWSJ only, discriminative\n91.3\nZhu et al. (2013) [40]\nsemi-supervised\n91.3\nHuang &amp; Harper (2009) [14]\nsemi-supervised\n91.3\nMcClosky et al. (2006) [26]\nsemi-supervised\n92.1\nVinyals &amp; Kaiser el al. (2014) [37]\nsemi-supervised\n92.1\nTransformer (4 layers)\nsemi-supervised\n92.7\nLuong et al. (2015) [23]\nmulti-task\n93.0\nDyer et al. (2016) [8]\ngenerative\n93.3\nincreased the maximum output length to input length + 300. We used a beam size of 21 and α = 0.3\nfor both WSJ only and the semi-supervised setting.\nOur results in Table 4 show that despite the lack of task-specific tuning our model performs sur-\nprisingly well, yielding better results than all previously reported models with the exception of the\nRecurrent Neural Network Grammar [8].\nIn contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the Berkeley-\nParser [29] even when training only on the WSJ training set of 40K sentences.\n7\nConclusion\nIn this work, we presented the Transformer, the first sequence transduction model based entirely on\nattention, replacing the recurrent layers most commonly used in encoder-decoder architectures with\nmulti-headed self-attention.\nFor translation tasks, the Transformer can be trained significantly faster than architectures based\non recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014\nEnglish-to-French translation tasks, we achieve a new state of the art. In the former task our best\nmodel outperforms even all previously reported ensembles.\nWe are excited about the future of attention-based models and plan to apply them to other tasks. We\nplan to extend the Transformer to problems involving input and output modalities other than text and\nto investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs\nsuch as images, audio and video. Making generation less sequential is another research goals of ours.\nThe code we used to train and evaluate our models is available at https://github.com/\ntensorflow/tensor2tensor.\nAcknowledgements\nWe are grateful to Nal Kalchbrenner and Stephan Gouws for their fruitful\ncomments, corrections and inspiration.\nReferences\n[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint\narXiv:1607.06450, 2016.\n[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\nlearning to align and translate. CoRR, abs/1409.0473, 2014.\n[3] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. Massive exploration of neural\nmachine translation architectures. CoRR, abs/1703.03906, 2017.\n[4] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine\nreading. arXiv preprint arXiv:1601.06733, 2016.\n10\n[5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk,\nand Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical\nmachine translation. CoRR, abs/1406.1078, 2014.\n[6] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv\npreprint arXiv:1610.02357, 2016.\n[7] Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation\nof gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.\n[8] Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural\nnetwork grammars. In Proc. of NAACL, 2016.\n[9] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu-\ntional sequence to sequence learning. arXiv preprint arXiv:1705.03122v2, 2017.\n[10] Alex Graves.\nGenerating sequences with recurrent neural networks.\narXiv preprint\narXiv:1308.0850, 2013.\n[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im-\nage recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 770–778, 2016.\n[12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in\nrecurrent nets: the difficulty of learning long-term dependencies, 2001.\n[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation,\n9(8):1735–1780, 1997.\n[14] Zhongqiang Huang and Mary Harper. Self-training PCFG grammars with latent annotations\nacross languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural\nLanguage Processing, pages 832–841. ACL, August 2009.\n[15] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring\nthe limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n[16] Łukasz Kaiser and Samy Bengio. Can active memory replace attention? In Advances in Neural\nInformation Processing Systems, (NIPS), 2016.\n[17] Łukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. In International Conference\non Learning Representations (ICLR), 2016.\n[18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Ko-\nray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099v2,\n2017.\n[19] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. Structured attention networks.\nIn International Conference on Learning Representations, 2017.\n[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n[21] Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for LSTM networks. arXiv preprint\narXiv:1703.10722, 2017.\n[22] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen\nZhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint\narXiv:1703.03130, 2017.\n[23] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task\nsequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015.\n[24] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-\nbased neural machine translation. arXiv preprint arXiv:1508.04025, 2015.\n11\n[25] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated\ncorpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993.\n[26] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In\nProceedings of the Human Language Technology Conference of the NAACL, Main Conference,\npages 152–159. ACL, June 2006.\n[27] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention\nmodel. In Empirical Methods in Natural Language Processing, 2016.\n[28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive\nsummarization. arXiv preprint arXiv:1705.04304, 2017.\n[29] Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. Learning accurate, compact,\nand interpretable tree annotation. In Proceedings of the 21st International Conference on\nComputational Linguistics and 44th Annual Meeting of the ACL, pages 433–440. ACL, July\n2006.\n[30] Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv\npreprint arXiv:1608.05859, 2016.\n[31] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words\nwith subword units. arXiv preprint arXiv:1508.07909, 2015.\n[32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,\nand Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts\nlayer. arXiv preprint arXiv:1701.06538, 2017.\n[33] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi-\nnov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine\nLearning Research, 15(1):1929–1958, 2014.\n[34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory\nnetworks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors,\nAdvances in Neural Information Processing Systems 28, pages 2440–2448. Curran Associates,\nInc., 2015.\n[35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural\nnetworks. In Advances in Neural Information Processing Systems, pages 3104–3112, 2014.\n[36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.\nRethinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015.\n[37] Vinyals &amp; Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In\nAdvances in Neural Information Processing Systems, 2015.\n[38] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang\nMacherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine\ntranslation system: Bridging the gap between human and machine translation. arXiv preprint\narXiv:1609.08144, 2016.\n[39] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with\nfast-forward connections for neural machine translation. CoRR, abs/1606.04199, 2016.\n[40] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. Fast and accurate\nshift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the ACL (Volume\n1: Long Papers), pages 434–443. ACL, August 2013.\n12\nAttention Visualizations\nInput-Input Layer5\nIt\nis\nin\nthis\nspirit\nthat\na\nmajority\nof\nAmerican\ngovernments\nhave\npassed\nnew\nlaws\nsince\n2009\nmaking\nthe\nregistration\nor\nvoting\nprocess\nmore\ndifficult\n.\n&lt;EOS&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\nIt\nis\nin\nthis\nspirit\nthat\na\nmajority\nof\nAmerican\ngovernments\nhave\npassed\nnew\nlaws\nsince\n2009\nmaking\nthe\nregistration\nor\nvoting\nprocess\nmore\ndifficult\n.\n&lt;EOS&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\nFigure 3: An example of the attention mechanism following long-distance dependencies in the\nencoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of\nthe verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for\nthe word ‘making’. Different colors represent different heads. Best viewed in color.\n13\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nFigure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top:\nFull attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5\nand 6. Note that the attentions are very sharp for this word.\n14\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nFigure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the\nsentence. We give two such examples above, from two different heads from the encoder self-attention\nat layer 5 of 6. The heads clearly learned to perform different tasks.\n15\n&#39;, metadata={&#39;Published&#39;: &#39;2023-08-02&#39;, &#39;Title&#39;: &#39;Attention Is All You Need&#39;, &#39;Authors&#39;: &#39;Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin&#39;, &#39;Summary&#39;: &#39;The dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks in an encoder-decoder configuration. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer, based\nsolely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to be\nsuperior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014\nEnglish-to-German translation task, improving over the existing best results,\nincluding ensembles by over 2 BLEU. On the WMT 2014 English-to-French\ntranslation task, our model establishes a new single-model state-of-the-art\nBLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction\nof the training costs of the best models from the literature. We show that the\nTransformer generalizes well to other tasks by applying it successfully to\nEnglish constituency parsing both with large and limited training data.&#39;})][Document(page_content=&#39;Provided proper attribution is provided, Google hereby grants permission to\nreproduce the tables and figures in this paper solely for use in journalistic or\nscholarly works.\nAttention Is All You Need\nAshish Vaswani∗\nGoogle Brain\navaswani@google.com\nNoam Shazeer∗\nGoogle Brain\nnoam@google.com\nNiki Parmar∗\nGoogle Research\nnikip@google.com\nJakob Uszkoreit∗\nGoogle Research\nusz@google.com\nLlion Jones∗\nGoogle Research\nllion@google.com\nAidan N. Gomez∗ †\nUniversity of Toronto\naidan@cs.toronto.edu\nŁukasz Kaiser∗\nGoogle Brain\nlukaszkaiser@google.com\nIllia Polosukhin∗ ‡\nillia.polosukhin@gmail.com\nAbstract\nThe dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks that include an encoder and a decoder. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer,\nbased solely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to\nbe superior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-\nto-German translation task, improving over the existing best results, including\nensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,\nour model establishes a new single-model state-of-the-art BLEU score of 41.8 after\ntraining for 3.5 days on eight GPUs, a small fraction of the training costs of the\nbest models from the literature. We show that the Transformer generalizes well to\nother tasks by applying it successfully to English constituency parsing both with\nlarge and limited training data.\n∗Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started\nthe effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and\nhas been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head\nattention and the parameter-free position representation and became the other person involved in nearly every\ndetail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and\ntensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and\nefficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and\nimplementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating\nour research.\n†Work performed while at Google Brain.\n‡Work performed while at Google Research.\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\narXiv:1706.03762v7  [cs.CL]  2 Aug 2023\n1\nIntroduction\nRecurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks\nin particular, have been firmly established as state of the art approaches in sequence modeling and\ntransduction problems such as language modeling and machine translation [35, 2, 5]. Numerous\nefforts have since continued to push the boundaries of recurrent language models and encoder-decoder\narchitectures [38, 24, 15].\nRecurrent models typically factor computation along the symbol positions of the input and output\nsequences. Aligning the positions to steps in computation time, they generate a sequence of hidden\nstates ht, as a function of the previous hidden state ht−1 and the input for position t. This inherently\nsequential nature precludes parallelization within training examples, which becomes critical at longer\nsequence lengths, as memory constraints limit batching across examples. Recent work has achieved\nsignificant improvements in computational efficiency through factorization tricks [21] and conditional\ncomputation [32], while also improving model performance in case of the latter. The fundamental\nconstraint of sequential computation, however, remains.\nAttention mechanisms have become an integral part of compelling sequence modeling and transduc-\ntion models in various tasks, allowing modeling of dependencies without regard to their distance in\nthe input or output sequences [2, 19]. In all but a few cases [27], however, such attention mechanisms\nare used in conjunction with a recurrent network.\nIn this work we propose the Transformer, a model architecture eschewing recurrence and instead\nrelying entirely on an attention mechanism to draw global dependencies between input and output.\nThe Transformer allows for significantly more parallelization and can reach a new state of the art in\ntranslation quality after being trained for as little as twelve hours on eight P100 GPUs.\n2\nBackground\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU\n[16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building\nblock, computing hidden representations in parallel for all input and output positions. In these models,\nthe number of operations required to relate signals from two arbitrary input or output positions grows\nin the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes\nit more difficult to learn dependencies between distant positions [12]. In the Transformer this is\nreduced to a constant number of operations, albeit at the cost of reduced effective resolution due\nto averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as\ndescribed in section 3.2.\nSelf-attention, sometimes called intra-attention is an attention mechanism relating different positions\nof a single sequence in order to compute a representation of the sequence. Self-attention has been\nused successfully in a variety of tasks including reading comprehension, abstractive summarization,\ntextual entailment and learning task-independent sentence representations [4, 27, 28, 22].\nEnd-to-end memory networks are based on a recurrent attention mechanism instead of sequence-\naligned recurrence and have been shown to perform well on simple-language question answering and\nlanguage modeling tasks [34].\nTo the best of our knowledge, however, the Transformer is the first transduction model relying\nentirely on self-attention to compute representations of its input and output without using sequence-\naligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate\nself-attention and discuss its advantages over models such as [17, 18] and [9].\n3\nModel Architecture\nMost competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].\nHere, the encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence\nof continuous representations z = (z1, ..., zn). Given z, the decoder then generates an output\nsequence (y1, ..., ym) of symbols one element at a time. At each step the model is auto-regressive\n[10], consuming the previously generated symbols as additional input when generating the next.\n2\nFigure 1: The Transformer - model architecture.\nThe Transformer follows this overall architecture using stacked self-attention and point-wise, fully\nconnected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\nrespectively.\n3.1\nEncoder and Decoder Stacks\nEncoder:\nThe encoder is composed of a stack of N = 6 identical layers. Each layer has two\nsub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\nwise fully connected feed-forward network. We employ a residual connection [11] around each of\nthe two sub-layers, followed by layer normalization [1]. That is, the output of each sub-layer is\nLayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer\nitself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\nlayers, produce outputs of dimension dmodel = 512.\nDecoder:\nThe decoder is also composed of a stack of N = 6 identical layers. In addition to the two\nsub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\nattention over the output of the encoder stack. Similar to the encoder, we employ residual connections\naround each of the sub-layers, followed by layer normalization. We also modify the self-attention\nsub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\nmasking, combined with fact that the output embeddings are offset by one position, ensures that the\npredictions for position i can depend only on the known outputs at positions less than i.\n3.2\nAttention\nAn attention function can be described as mapping a query and a set of key-value pairs to an output,\nwhere the query, keys, values, and output are all vectors. The output is computed as a weighted sum\n3\nScaled Dot-Product Attention\nMulti-Head Attention\nFigure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several\nattention layers running in parallel.\nof the values, where the weight assigned to each value is computed by a compatibility function of the\nquery with the corresponding key.\n3.2.1\nScaled Dot-Product Attention\nWe call our particular attention &quot;Scaled Dot-Product Attention&quot; (Figure 2). The input consists of\nqueries and keys of dimension dk, and values of dimension dv. We compute the dot products of the\nquery with all keys, divide each by √dk, and apply a softmax function to obtain the weights on the\nvalues.\nIn practice, we compute the attention function on a set of queries simultaneously, packed together\ninto a matrix Q. The keys and values are also packed together into matrices K and V . We compute\nthe matrix of outputs as:\nAttention(Q, K, V ) = softmax(QKT\n√dk\n)V\n(1)\nThe two most commonly used attention functions are additive attention [2], and dot-product (multi-\nplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor\nof\n1\n√dk . Additive attention computes the compatibility function using a feed-forward network with\na single hidden layer. While the two are similar in theoretical complexity, dot-product attention is\nmuch faster and more space-efficient in practice, since it can be implemented using highly optimized\nmatrix multiplication code.\nWhile for small values of dk the two mechanisms perform similarly, additive attention outperforms\ndot product attention without scaling for larger values of dk [3]. We suspect that for large values of\ndk, the dot products grow large in magnitude, pushing the softmax function into regions where it has\nextremely small gradients 4. To counteract this effect, we scale the dot products by\n1\n√dk .\n3.2.2\nMulti-Head Attention\nInstead of performing a single attention function with dmodel-dimensional keys, values and queries,\nwe found it beneficial to linearly project the queries, keys and values h times with different, learned\nlinear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of\nqueries, keys and values we then perform the attention function in parallel, yielding dv-dimensional\n4To illustrate why the dot products get large, assume that the components of q and k are independent random\nvariables with mean 0 and variance 1. Then their dot product, q · k = Pdk\ni=1 qiki, has mean 0 and variance dk.\n4\noutput values. These are concatenated and once again projected, resulting in the final values, as\ndepicted in Figure 2.\nMulti-head attention allows the model to jointly attend to information from different representation\nsubspaces at different positions. With a single attention head, averaging inhibits this.\nMultiHead(Q, K, V ) = Concat(head1, ..., headh)W O\nwhere headi = Attention(QW Q\ni , KW K\ni , V W V\ni )\nWhere the projections are parameter matrices W Q\ni\n∈ Rdmodel×dk, W K\ni\n∈ Rdmodel×dk, W V\ni\n∈ Rdmodel×dv\nand W O ∈ Rhdv×dmodel.\nIn this work we employ h = 8 parallel attention layers, or heads. For each of these we use\ndk = dv = dmodel/h = 64. Due to the reduced dimension of each head, the total computational cost\nis similar to that of single-head attention with full dimensionality.\n3.2.3\nApplications of Attention in our Model\nThe Transformer uses multi-head attention in three different ways:\n• In &quot;encoder-decoder attention&quot; layers, the queries come from the previous decoder layer,\nand the memory keys and values come from the output of the encoder. This allows every\nposition in the decoder to attend over all positions in the input sequence. This mimics the\ntypical encoder-decoder attention mechanisms in sequence-to-sequence models such as\n[38, 2, 9].\n• The encoder contains self-attention layers. In a self-attention layer all of the keys, values\nand queries come from the same place, in this case, the output of the previous layer in the\nencoder. Each position in the encoder can attend to all positions in the previous layer of the\nencoder.\n• Similarly, self-attention layers in the decoder allow each position in the decoder to attend to\nall positions in the decoder up to and including that position. We need to prevent leftward\ninformation flow in the decoder to preserve the auto-regressive property. We implement this\ninside of scaled dot-product attention by masking out (setting to −∞) all values in the input\nof the softmax which correspond to illegal connections. See Figure 2.\n3.3\nPosition-wise Feed-Forward Networks\nIn addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully\nconnected feed-forward network, which is applied to each position separately and identically. This\nconsists of two linear transformations with a ReLU activation in between.\nFFN(x) = max(0, xW1 + b1)W2 + b2\n(2)\nWhile the linear transformations are the same across different positions, they use different parameters\nfrom layer to layer. Another way of describing this is as two convolutions with kernel size 1.\nThe dimensionality of input and output is dmodel = 512, and the inner-layer has dimensionality\ndff = 2048.\n3.4\nEmbeddings and Softmax\nSimilarly to other sequence transduction models, we use learned embeddings to convert the input\ntokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transfor-\nmation and softmax function to convert the decoder output to predicted next-token probabilities. In\nour model, we share the same weight matrix between the two embedding layers and the pre-softmax\nlinear transformation, similar to [30]. In the embedding layers, we multiply those weights by √dmodel.\n5\nTable 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations\nfor different layer types. n is the sequence length, d is the representation dimension, k is the kernel\nsize of convolutions and r the size of the neighborhood in restricted self-attention.\nLayer Type\nComplexity per Layer\nSequential\nMaximum Path Length\nOperations\nSelf-Attention\nO(n2 · d)\nO(1)\nO(1)\nRecurrent\nO(n · d2)\nO(n)\nO(n)\nConvolutional\nO(k · n · d2)\nO(1)\nO(logk(n))\nSelf-Attention (restricted)\nO(r · n · d)\nO(1)\nO(n/r)\n3.5\nPositional Encoding\nSince our model contains no recurrence and no convolution, in order for the model to make use of the\norder of the sequence, we must inject some information about the relative or absolute position of the\ntokens in the sequence. To this end, we add &quot;positional encodings&quot; to the input embeddings at the\nbottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel\nas the embeddings, so that the two can be summed. There are many choices of positional encodings,\nlearned and fixed [9].\nIn this work, we use sine and cosine functions of different frequencies:\nPE(pos,2i) = sin(pos/100002i/dmodel)\nPE(pos,2i+1) = cos(pos/100002i/dmodel)\nwhere pos is the position and i is the dimension. That is, each dimension of the positional encoding\ncorresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We\nchose this function because we hypothesized it would allow the model to easily learn to attend by\nrelative positions, since for any fixed offset k, PEpos+k can be represented as a linear function of\nPEpos.\nWe also experimented with using learned positional embeddings [9] instead, and found that the two\nversions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version\nbecause it may allow the model to extrapolate to sequence lengths longer than the ones encountered\nduring training.\n4\nWhy Self-Attention\nIn this section we compare various aspects of self-attention layers to the recurrent and convolu-\ntional layers commonly used for mapping one variable-length sequence of symbol representations\n(x1, ..., xn) to another sequence of equal length (z1, ..., zn), with xi, zi ∈ Rd, such as a hidden\nlayer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we\nconsider three desiderata.\nOne is the total computational complexity per layer. Another is the amount of computation that can\nbe parallelized, as measured by the minimum number of sequential operations required.\nThe third is the path length between long-range dependencies in the network. Learning long-range\ndependencies is a key challenge in many sequence transduction tasks. One key factor affecting the\nability to learn such dependencies is the length of the paths forward and backward signals have to\ntraverse in the network. The shorter these paths between any combination of positions in the input\nand output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare\nthe maximum path length between any two input and output positions in networks composed of the\ndifferent layer types.\nAs noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially\nexecuted operations, whereas a recurrent layer requires O(n) sequential operations. In terms of\ncomputational complexity, self-attention layers are faster than recurrent layers when the sequence\n6\nlength n is smaller than the representation dimensionality d, which is most often the case with\nsentence representations used by state-of-the-art models in machine translations, such as word-piece\n[38] and byte-pair [31] representations. To improve computational performance for tasks involving\nvery long sequences, self-attention could be restricted to considering only a neighborhood of size r in\nthe input sequence centered around the respective output position. This would increase the maximum\npath length to O(n/r). We plan to investigate this approach further in future work.\nA single convolutional layer with kernel width k &lt; n does not connect all pairs of input and output\npositions. Doing so requires a stack of O(n/k) convolutional layers in the case of contiguous kernels,\nor O(logk(n)) in the case of dilated convolutions [18], increasing the length of the longest paths\nbetween any two positions in the network. Convolutional layers are generally more expensive than\nrecurrent layers, by a factor of k. Separable convolutions [6], however, decrease the complexity\nconsiderably, to O(k · n · d + n · d2). Even with k = n, however, the complexity of a separable\nconvolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer,\nthe approach we take in our model.\nAs side benefit, self-attention could yield more interpretable models. We inspect attention distributions\nfrom our models and present and discuss examples in the appendix. Not only do individual attention\nheads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic\nand semantic structure of the sentences.\n5\nTraining\nThis section describes the training regime for our models.\n5.1\nTraining Data and Batching\nWe trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million\nsentence pairs. Sentences were encoded using byte-pair encoding [3], which has a shared source-\ntarget vocabulary of about 37000 tokens. For English-French, we used the significantly larger WMT\n2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece\nvocabulary [38]. Sentence pairs were batched together by approximate sequence length. Each training\nbatch contained a set of sentence pairs containing approximately 25000 source tokens and 25000\ntarget tokens.\n5.2\nHardware and Schedule\nWe trained our models on one machine with 8 NVIDIA P100 GPUs. For our base models using\nthe hyperparameters described throughout the paper, each training step took about 0.4 seconds. We\ntrained the base models for a total of 100,000 steps or 12 hours. For our big models,(described on the\nbottom line of table 3), step time was 1.0 seconds. The big models were trained for 300,000 steps\n(3.5 days).\n5.3\nOptimizer\nWe used the Adam optimizer [20] with β1 = 0.9, β2 = 0.98 and ϵ = 10−9. We varied the learning\nrate over the course of training, according to the formula:\nlrate = d−0.5\nmodel · min(step_num−0.5, step_num · warmup_steps−1.5)\n(3)\nThis corresponds to increasing the learning rate linearly for the first warmup_steps training steps,\nand decreasing it thereafter proportionally to the inverse square root of the step number. We used\nwarmup_steps = 4000.\n5.4\nRegularization\nWe employ three types of regularization during training:\n7\nTable 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the\nEnglish-to-German and English-to-French newstest2014 tests at a fraction of the training cost.\nModel\nBLEU\nTraining Cost (FLOPs)\nEN-DE\nEN-FR\nEN-DE\nEN-FR\nByteNet [18]\n23.75\nDeep-Att + PosUnk [39]\n39.2\n1.0 · 1020\nGNMT + RL [38]\n24.6\n39.92\n2.3 · 1019\n1.4 · 1020\nConvS2S [9]\n25.16\n40.46\n9.6 · 1018\n1.5 · 1020\nMoE [32]\n26.03\n40.56\n2.0 · 1019\n1.2 · 1020\nDeep-Att + PosUnk Ensemble [39]\n40.4\n8.0 · 1020\nGNMT + RL Ensemble [38]\n26.30\n41.16\n1.8 · 1020\n1.1 · 1021\nConvS2S Ensemble [9]\n26.36\n41.29\n7.7 · 1019\n1.2 · 1021\nTransformer (base model)\n27.3\n38.1\n3.3 · 1018\nTransformer (big)\n28.4\n41.8\n2.3 · 1019\nResidual Dropout\nWe apply dropout [33] to the output of each sub-layer, before it is added to the\nsub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the\npositional encodings in both the encoder and decoder stacks. For the base model, we use a rate of\nPdrop = 0.1.\nLabel Smoothing\nDuring training, we employed label smoothing of value ϵls = 0.1 [36]. This\nhurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\n6\nResults\n6.1\nMachine Translation\nOn the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big)\nin Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0\nBLEU, establishing a new state-of-the-art BLEU score of 28.4. The configuration of this model is\nlisted in the bottom line of Table 3. Training took 3.5 days on 8 P100 GPUs. Even our base model\nsurpasses all previously published models and ensembles, at a fraction of the training cost of any of\nthe competitive models.\nOn the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0,\noutperforming all of the previously published single models, at less than 1/4 the training cost of the\nprevious state-of-the-art model. The Transformer (big) model trained for English-to-French used\ndropout rate Pdrop = 0.1, instead of 0.3.\nFor the base models, we used a single model obtained by averaging the last 5 checkpoints, which\nwere written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints. We\nused beam search with a beam size of 4 and length penalty α = 0.6 [38]. These hyperparameters\nwere chosen after experimentation on the development set. We set the maximum output length during\ninference to input length + 50, but terminate early when possible [38].\nTable 2 summarizes our results and compares our translation quality and training costs to other model\narchitectures from the literature. We estimate the number of floating point operations used to train a\nmodel by multiplying the training time, the number of GPUs used, and an estimate of the sustained\nsingle-precision floating-point capacity of each GPU 5.\n6.2\nModel Variations\nTo evaluate the importance of different components of the Transformer, we varied our base model\nin different ways, measuring the change in performance on English-to-German translation on the\n5We used values of 2.8, 3.7, 6.0 and 9.5 TFLOPS for K80, K40, M40 and P100, respectively.\n8\nTable 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base\nmodel. All metrics are on the English-to-German translation development set, newstest2013. Listed\nperplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to\nper-word perplexities.\nN\ndmodel\ndff\nh\ndk\ndv\nPdrop\nϵls\ntrain\nPPL\nBLEU\nparams\nsteps\n(dev)\n(dev)\n×106\nbase\n6\n512\n2048\n8\n64\n64\n0.1\n0.1\n100K\n4.92\n25.8\n65\n(A)\n1\n512\n512\n5.29\n24.9\n4\n128\n128\n5.00\n25.5\n16\n32\n32\n4.91\n25.8\n32\n16\n16\n5.01\n25.4\n(B)\n16\n5.16\n25.1\n58\n32\n5.01\n25.4\n60\n(C)\n2\n6.11\n23.7\n36\n4\n5.19\n25.3\n50\n8\n4.88\n25.5\n80\n256\n32\n32\n5.75\n24.5\n28\n1024\n128\n128\n4.66\n26.0\n168\n1024\n5.12\n25.4\n53\n4096\n4.75\n26.2\n90\n(D)\n0.0\n5.77\n24.6\n0.2\n4.95\n25.5\n0.0\n4.67\n25.3\n0.2\n5.47\n25.7\n(E)\npositional embedding instead of sinusoids\n4.92\n25.7\nbig\n6\n1024\n4096\n16\n0.3\n300K\n4.33\n26.4\n213\ndevelopment set, newstest2013. We used beam search as described in the previous section, but no\ncheckpoint averaging. We present these results in Table 3.\nIn Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions,\nkeeping the amount of computation constant, as described in Section 3.2.2. While single-head\nattention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.\nIn Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality. This\nsuggests that determining compatibility is not easy and that a more sophisticated compatibility\nfunction than dot product may be beneficial. We further observe in rows (C) and (D) that, as expected,\nbigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our\nsinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical\nresults to the base model.\n6.3\nEnglish Constituency Parsing\nTo evaluate if the Transformer can generalize to other tasks we performed experiments on English\nconstituency parsing. This task presents specific challenges: the output is subject to strong structural\nconstraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence\nmodels have not been able to attain state-of-the-art results in small-data regimes [37].\nWe trained a 4-layer transformer with dmodel = 1024 on the Wall Street Journal (WSJ) portion of the\nPenn Treebank [25], about 40K training sentences. We also trained it in a semi-supervised setting,\nusing the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences\n[37]. We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens\nfor the semi-supervised setting.\nWe performed only a small number of experiments to select the dropout, both attention and residual\n(section 5.4), learning rates and beam size on the Section 22 development set, all other parameters\nremained unchanged from the English-to-German base translation model. During inference, we\n9\nTable 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23\nof WSJ)\nParser\nTraining\nWSJ 23 F1\nVinyals &amp; Kaiser el al. (2014) [37]\nWSJ only, discriminative\n88.3\nPetrov et al. (2006) [29]\nWSJ only, discriminative\n90.4\nZhu et al. (2013) [40]\nWSJ only, discriminative\n90.4\nDyer et al. (2016) [8]\nWSJ only, discriminative\n91.7\nTransformer (4 layers)\nWSJ only, discriminative\n91.3\nZhu et al. (2013) [40]\nsemi-supervised\n91.3\nHuang &amp; Harper (2009) [14]\nsemi-supervised\n91.3\nMcClosky et al. (2006) [26]\nsemi-supervised\n92.1\nVinyals &amp; Kaiser el al. (2014) [37]\nsemi-supervised\n92.1\nTransformer (4 layers)\nsemi-supervised\n92.7\nLuong et al. (2015) [23]\nmulti-task\n93.0\nDyer et al. (2016) [8]\ngenerative\n93.3\nincreased the maximum output length to input length + 300. We used a beam size of 21 and α = 0.3\nfor both WSJ only and the semi-supervised setting.\nOur results in Table 4 show that despite the lack of task-specific tuning our model performs sur-\nprisingly well, yielding better results than all previously reported models with the exception of the\nRecurrent Neural Network Grammar [8].\nIn contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the Berkeley-\nParser [29] even when training only on the WSJ training set of 40K sentences.\n7\nConclusion\nIn this work, we presented the Transformer, the first sequence transduction model based entirely on\nattention, replacing the recurrent layers most commonly used in encoder-decoder architectures with\nmulti-headed self-attention.\nFor translation tasks, the Transformer can be trained significantly faster than architectures based\non recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014\nEnglish-to-French translation tasks, we achieve a new state of the art. In the former task our best\nmodel outperforms even all previously reported ensembles.\nWe are excited about the future of attention-based models and plan to apply them to other tasks. We\nplan to extend the Transformer to problems involving input and output modalities other than text and\nto investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs\nsuch as images, audio and video. Making generation less sequential is another research goals of ours.\nThe code we used to train and evaluate our models is available at https://github.com/\ntensorflow/tensor2tensor.\nAcknowledgements\nWe are grateful to Nal Kalchbrenner and Stephan Gouws for their fruitful\ncomments, corrections and inspiration.\nReferences\n[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint\narXiv:1607.06450, 2016.\n[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\nlearning to align and translate. CoRR, abs/1409.0473, 2014.\n[3] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. Massive exploration of neural\nmachine translation architectures. CoRR, abs/1703.03906, 2017.\n[4] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine\nreading. arXiv preprint arXiv:1601.06733, 2016.\n10\n[5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk,\nand Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical\nmachine translation. CoRR, abs/1406.1078, 2014.\n[6] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv\npreprint arXiv:1610.02357, 2016.\n[7] Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation\nof gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.\n[8] Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural\nnetwork grammars. In Proc. of NAACL, 2016.\n[9] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu-\ntional sequence to sequence learning. arXiv preprint arXiv:1705.03122v2, 2017.\n[10] Alex Graves.\nGenerating sequences with recurrent neural networks.\narXiv preprint\narXiv:1308.0850, 2013.\n[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im-\nage recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 770–778, 2016.\n[12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in\nrecurrent nets: the difficulty of learning long-term dependencies, 2001.\n[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation,\n9(8):1735–1780, 1997.\n[14] Zhongqiang Huang and Mary Harper. Self-training PCFG grammars with latent annotations\nacross languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural\nLanguage Processing, pages 832–841. ACL, August 2009.\n[15] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring\nthe limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n[16] Łukasz Kaiser and Samy Bengio. Can active memory replace attention? In Advances in Neural\nInformation Processing Systems, (NIPS), 2016.\n[17] Łukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. In International Conference\non Learning Representations (ICLR), 2016.\n[18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Ko-\nray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099v2,\n2017.\n[19] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. Structured attention networks.\nIn International Conference on Learning Representations, 2017.\n[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n[21] Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for LSTM networks. arXiv preprint\narXiv:1703.10722, 2017.\n[22] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen\nZhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint\narXiv:1703.03130, 2017.\n[23] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task\nsequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015.\n[24] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-\nbased neural machine translation. arXiv preprint arXiv:1508.04025, 2015.\n11\n[25] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated\ncorpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993.\n[26] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In\nProceedings of the Human Language Technology Conference of the NAACL, Main Conference,\npages 152–159. ACL, June 2006.\n[27] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention\nmodel. In Empirical Methods in Natural Language Processing, 2016.\n[28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive\nsummarization. arXiv preprint arXiv:1705.04304, 2017.\n[29] Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. Learning accurate, compact,\nand interpretable tree annotation. In Proceedings of the 21st International Conference on\nComputational Linguistics and 44th Annual Meeting of the ACL, pages 433–440. ACL, July\n2006.\n[30] Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv\npreprint arXiv:1608.05859, 2016.\n[31] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words\nwith subword units. arXiv preprint arXiv:1508.07909, 2015.\n[32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,\nand Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts\nlayer. arXiv preprint arXiv:1701.06538, 2017.\n[33] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi-\nnov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine\nLearning Research, 15(1):1929–1958, 2014.\n[34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory\nnetworks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors,\nAdvances in Neural Information Processing Systems 28, pages 2440–2448. Curran Associates,\nInc., 2015.\n[35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural\nnetworks. In Advances in Neural Information Processing Systems, pages 3104–3112, 2014.\n[36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.\nRethinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015.\n[37] Vinyals &amp; Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In\nAdvances in Neural Information Processing Systems, 2015.\n[38] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang\nMacherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine\ntranslation system: Bridging the gap between human and machine translation. arXiv preprint\narXiv:1609.08144, 2016.\n[39] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with\nfast-forward connections for neural machine translation. CoRR, abs/1606.04199, 2016.\n[40] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. Fast and accurate\nshift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the ACL (Volume\n1: Long Papers), pages 434–443. ACL, August 2013.\n12\nAttention Visualizations\nInput-Input Layer5\nIt\nis\nin\nthis\nspirit\nthat\na\nmajority\nof\nAmerican\ngovernments\nhave\npassed\nnew\nlaws\nsince\n2009\nmaking\nthe\nregistration\nor\nvoting\nprocess\nmore\ndifficult\n.\n&lt;EOS&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\nIt\nis\nin\nthis\nspirit\nthat\na\nmajority\nof\nAmerican\ngovernments\nhave\npassed\nnew\nlaws\nsince\n2009\nmaking\nthe\nregistration\nor\nvoting\nprocess\nmore\ndifficult\n.\n&lt;EOS&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\n&lt;pad&gt;\nFigure 3: An example of the attention mechanism following long-distance dependencies in the\nencoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of\nthe verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for\nthe word ‘making’. Different colors represent different heads. Best viewed in color.\n13\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nFigure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top:\nFull attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5\nand 6. Note that the attentions are very sharp for this word.\n14\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nInput-Input Layer5\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nThe\nLaw\nwill\nnever\nbe\nperfect\n,\nbut\nits\napplication\nshould\nbe\njust\n-\nthis\nis\nwhat\nwe\nare\nmissing\n,\nin\nmy\nopinion\n.\n&lt;EOS&gt;\n&lt;pad&gt;\nFigure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the\nsentence. We give two such examples above, from two different heads from the encoder self-attention\nat layer 5 of 6. The heads clearly learned to perform different tasks.\n15\n&#39;, metadata={&#39;Published&#39;: &#39;2023-08-02&#39;, &#39;Title&#39;: &#39;Attention Is All You Need&#39;, &#39;Authors&#39;: &#39;Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin&#39;, &#39;Summary&#39;: &#39;The dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks in an encoder-decoder configuration. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer, based\nsolely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to be\nsuperior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014\nEnglish-to-German translation task, improving over the existing best results,\nincluding ensembles by over 2 BLEU. On the WMT 2014 English-to-French\ntranslation task, our model establishes a new single-model state-of-the-art\nBLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction\nof the training costs of the best models from the literature. We show that the\nTransformer generalizes well to other tasks by applying it successfully to\nEnglish constituency parsing both with large and limited training data.&#39;})]</pre><p>Let’s check the metadata.</p><pre>data[0].metadata<br><br>{&#39;Published&#39;: &#39;2023-08-02&#39;,<br> &#39;Title&#39;: &#39;Attention Is All You Need&#39;,<br> &#39;Authors&#39;: &#39;Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin&#39;,<br> &#39;Summary&#39;: &#39;The dominant sequence transduction models are based on complex recurrent or\nconvolutional neural networks in an encoder-decoder configuration. The best\nperforming models also connect the encoder and decoder through an attention\nmechanism. We propose a new simple network architecture, the Transformer, based\nsolely on attention mechanisms, dispensing with recurrence and convolutions\nentirely. Experiments on two machine translation tasks show these models to be\nsuperior in quality while being more parallelizable and requiring significantly\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014\nEnglish-to-German translation task, improving over the existing best results,\nincluding ensembles by over 2 BLEU. On the WMT 2014 English-to-French\ntranslation task, our model establishes a new single-model state-of-the-art\nBLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction\nof the training costs of the best models from the literature. We show that the\nTransformer generalizes well to other tasks by applying it successfully to\nEnglish constituency parsing both with large and limited training data.&#39;}</pre><p>So, we have covered some document loaders in LangChain. There are more loaders which you can read about in <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/">LangChain</a> documentation.</p><p>Right, now let’s use this knowledge that we have learned and create a bot that can answer questions based on Wikipedia articles. Sounds interesting? Let’s get our hands dirty!</p><p>First, let’s set up the environment and import all necessary libraries.</p><pre>import os<br>from langchain_openai import ChatOpenAI #model<br>from langchain.globals import set_llm_cache #response caching<br>from langchain.cache import InMemoryCache<br>from langchain.prompts import HumanMessagePromptTemplate, ChatPromptTemplate #prompt templating<br>from langchain.document_loaders import WikipediaLoader #document loader<br><br>import os<br>with open(&#39;../../openai_api_key.txt&#39;) as f:<br>    api_key = f.read()<br>os.environ[&#39;OPENAI_API_KEY&#39;] = api_key<br><br>chat = ChatOpenAI()<br>set_llm_cache(InMemoryCache())</pre><p>Let’s create a function that takes in a topic and a question and returns a LLM response.</p><pre>def qa_bot(topic, question):<br>    chat = ChatOpenAI()<br>    loader = WikipediaLoader(query = topic, load_max_docs=1)<br>    data = loader.load()<br>    <br>    document = data[0]<br>    title = document.metadata[&#39;title&#39;]<br>    summary = document.metadata[&#39;summary&#39;]<br>    <br>    question = question<br><br>    human_template = &quot;Read about the wikipedia article on {title} having content {content} and answer the {question}&quot;<br>    <br>    human_message_prompt= HumanMessagePromptTemplate.from_template(human_template)<br>    <br>    chat_prompt = ChatPromptTemplate.from_messages([human_message_prompt])<br>    <br>    prompt = chat_prompt.format_prompt(title = title, content = summary, question = question)<br>    <br>    response = chat(messages = prompt.to_messages())<br>    <br>    return response.content</pre><p>Here, we are first passing the wikipedia article to the loader and extracting metadata: <em>title </em>and <em>summary</em>. Instead of summary we can also use <em>page_content. </em>I have used summary because of the memory constraints my system<em>. </em>These will be passed onto the LLM. Next we create a prompt template where we pass some instructions to the model as well the input. I have covered prompt templates in my first series of Langchain <a href="https://medium.com/@jayant017/efficient-few-shot-prompting-in-langchain-part-2-265b0bc013c5"><strong><em>articles</em></strong></a>. You can go through it in case you some doubts. Let’s test this.</p><pre>qa_bot(&#39;Artificial Intelligence&#39;, &#39;When was AI born?&#39;)<br><br>&#39;AI was founded as an academic discipline in 1956.&#39;</pre><p>So, this is how we just built a question answering bot that can answer questions based on the Wikipedia article.</p><p>That brings this article to an end. Hope this read was refreshing and worth it. I’ll see you in the next one.</p><p>Thanks for your time. I have added all the code in my <a href="https://github.com/jayantkp/LangChain/tree/main/RAG">Github</a> repo. Keep learning!</p><blockquote><strong>References:</strong></blockquote><ul><li><a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/">Document loaders | 🦜️🔗 Langchain</a></li><li><a href="https://jayant017.medium.com/efficient-few-shot-prompting-in-langchain-part-2-265b0bc013c5">Efficient Few-Shot Prompting in LangChain: Response Caching, Prompt Templating and Prompt…</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4774c04fd85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy a Live Text Translation App using Gradio and Hugging Face Spaces]]></title>
            <link>https://jayant017.medium.com/deploy-a-live-text-translation-app-using-gradio-and-hugging-face-spaces-be2c99beccb6?source=rss-782051e88a53------2</link>
            <guid isPermaLink="false">https://medium.com/p/be2c99beccb6</guid>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Jayant Pal]]></dc:creator>
            <pubDate>Sat, 09 Mar 2024 13:08:23 GMT</pubDate>
            <atom:updated>2024-03-09T13:25:44.425Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mi4NWwA4c5_KIx1i" /><figcaption>Photo by <a href="https://unsplash.com/@balazsketyi?utm_source=medium&amp;utm_medium=referral">Balázs Kétyi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Many a times, making a model is just not enough. To make it really usable, we need to deploy a model, so that it can be tested by the end user. Deploying a model can be difficult as there are several bottlenecks that we can come across both in front-end as well as in back-end.</p><p><em>Gradio</em>, a Python library, simplifies UI creation for your machine learning models. Hugging Face Spaces provides a seamless deployment platform. This efficient combination empowers rapid iteration and streamlines your development workflow.</p><p>Let’s build a real time text translation app using <em>LangChain </em>and deploy it using Gradio and HuggingFace spaces!</p><p>For this, first we need to create a script app.py</p><p>Before that make sure you have all the packages installed. We will use the following packages</p><pre>import gradio as gr<br>from langchain_core.pydantic_v1 import BaseModel, Field<br>from langchain.prompts import HumanMessagePromptTemplate, ChatPromptTemplate<br>from langchain.output_parsers import PydanticOutputParser<br>from langchain_openai import ChatOpenAI</pre><p>We would need to use <em>Pydantic Output Parser</em>, and some <em>Prompt Templates</em> from LangChain. I have covered all the LangChain components that we need briefly in my previous <a href="https://jayant017.medium.com/efficient-few-shot-prompting-in-langchain-output-parsers-part-3-56c036a01321">articles</a>. I would highly recommend you to have a read.</p><p>First, let’s create an instance of the chat model, that we will be using. Next, we define our <em>pydantic </em>model that would will specify the output format. Finally, we will use the output parser to parse the output in the required format.</p><pre>chat = ChatOpenAI()<br><br>class TextTranslator(BaseModel):<br>    output: str = Field(description=&quot;Python string containing the output text translated in the desired language&quot;)<br>    <br>output_parser = PydanticOutputParser(pydantic_object=TextTranslator)<br>format_instructions = output_parser.get_format_instructions()</pre><p>Now, we define the <em>text_translator </em>method that takes in two inputs: the <em>input text</em> and <em>target language</em>. We first template the prompts, add the instructions. This would be the input to the chat model(LLM). Finally, we return the parsed output.</p><pre>def text_translator(input_text : str, language : str) -&gt; str:<br>    human_template = &quot;&quot;&quot;Enter the text that you want to translate: <br>                {input_text}, and enter the language that you want it to translate to {language}. {format_instructions}&quot;&quot;&quot;<br>    human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)<br>    <br>    chat_prompt = ChatPromptTemplate.from_messages([human_message_prompt])<br>    <br>    prompt = chat_prompt.format_prompt(input_text = input_text, language = language, format_instructions = format_instructions)<br>    <br>    messages = prompt.to_messages()<br>    <br>    response = chat(messages = messages)<br>    <br>    output = output_parser.parse(response.content)<br>    <br>    output_text = output.output<br>    <br>    return output_text</pre><p>We are almost done with our back-end code. Now let’s create an interface for our app. Here we add two text boxes that will have the input and one text box for the translated text. We also add a button to generate the output.</p><pre>with gr.Blocks() as demo:<br>    gr.HTML(&quot;&lt;h1 align = &#39;center&#39;&gt; Text Translator &lt;/h1&gt;&quot;)<br>    gr.HTML(&quot;&lt;h4 align = &#39;center&#39;&gt; Translate to any language &lt;/h4&gt;&quot;)<br>    <br>    inputs = [gr.Textbox(label = &quot;Enter the text that you want to translate&quot;), gr.Textbox(label = &quot;Enter the language that you want it to translate to&quot;, placeholder = &quot;Example : Hindi,French,Bengali,etc&quot;)]<br>    generate_btn = gr.Button(value = &#39;Generate&#39;)<br>    outputs  = [gr.Textbox(label = &quot;Translated text&quot;)]<br>    generate_btn.click(fn = text_translator, inputs= inputs, outputs = outputs)<br><br>if __name__ == &#39;__main__&#39;:<br>    demo.launch() </pre><p>After testing this, we can deploy it to <a href="https://huggingface.co/spaces">Spaces</a> by HuggingFace. We first create a Space and next we need to upload two files: app.pyand requirements.txtin the files section of the space. Now, if the build gets successful, we will have our interface ready.</p><p>You can try out the app here:</p><p><a href="https://huggingface.co/spaces/jayant012/Text_Translator">Text Translator - a Hugging Face Space by jayant012</a></p><p>So, this is how quickly we can build and deploy any application. I have added all the scripts here.</p><p><a href="https://github.com/jayantkp/LangChain/tree/main/Text_Translator">LangChain/Text_Translator at main · jayantkp/LangChain</a></p><p>Let me know if you have any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be2c99beccb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>