<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhijeet Pandey on Medium]]></title>
        <description><![CDATA[Stories by Abhijeet Pandey on Medium]]></description>
        <link>https://medium.com/@itsmeabhijeetpandey?source=rss-b34699dfb8e1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wow8FXYhiDZxse30B-GVWw.png</url>
            <title>Stories by Abhijeet Pandey on Medium</title>
            <link>https://medium.com/@itsmeabhijeetpandey?source=rss-b34699dfb8e1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:51:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itsmeabhijeetpandey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decoding Multimodal RAG: Advanced Techniques for Seamless Document Interaction (Part 2)]]></title>
            <link>https://medium.com/walmartglobaltech/decoding-multimodal-rag-advanced-techniques-for-seamless-document-interaction-part-2-2200cb013a86?source=rss-b34699dfb8e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2200cb013a86</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Abhijeet Pandey]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 04:14:31 GMT</pubDate>
            <atom:updated>2025-08-22T04:14:31.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5F5ZDMZZVUH0wWRjK5MQNw.jpeg" /><figcaption>Image generated using Stable Diffusion 2</figcaption></figure><h3>Introduction</h3><p>Multimodal RAG (Retrieval-Augmented Generation) is an advanced version of the RAG pipeline that incorporates multiple types of data — such as <strong>text, images, audio, and video</strong> — into its retrieval and generation process. In the first <strong>part</strong> of this series, we discussed various techniques to enhance a multimodal RAG model. In this part, we will implement some solutions (see figure below)we covered earlier. We will use only open-source tools to ensure that you can easily replicate these solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9c2zZFF5F2BjpYl8eB5-g.png" /><figcaption><strong>Multimodel RAG: Indexing the summaries of chunks</strong></figcaption></figure><h3>Prerequisite</h3><ul><li><strong>Large Language Model (LLM)</strong> is a type of AI designed to comprehend and generate text similar to human language. These models rely on deep learning architectures, such as transformers, and are trained on extensive text datasets. LLMs can handle various natural language processing tasks, including generating text, translating languages, summarizing information, and answering questions. Notable examples of LLMs include the GPT series (like GPT-3 and GPT-4), BERT and its variants, T5 (Text-To-Text Transfer Transformer), PaLM2, and Llama2, among others. In this article, we will focus on using Llama2.</li><li><strong>Multimodal Large Language Model (Multimodal LLM)</strong> extends the capabilities of standard LLMs by processing and integrating diverse data types, such as text, images, audio, and video. Unlike traditional models that work exclusively with text, multimodal LLMs are designed to interpret and generate content from various input formats. This versatility enables them to perform tasks that require cross-modal understanding, including visual question answering, image captioning, and audio-to-text transcription. Examples of popular multimodal LLMs include GPT-4V, ImageBind, LLaVA, and Gemini. In this article, we will focus on using LLaVA.</li><li><strong>Vector Embedings:</strong> Vector embeddings in the context of large language models (LLMs) are numerical representations of text data. They capture the semantic meaning of words, phrases, sentences, or entire documents in a continuous vector space. There are various models available to convert a textual document into vectors, in this article we will be using the General Text Embeddings (<a href="https://huggingface.co/thenlper/gte-large"><strong>gte-large</strong></a>) model which generates <strong>1024</strong> dimension vector embeddings.</li><li><strong>Vector Database</strong>: A vector database stores data as high-dimensional vectors and their associated metadata, allowing for fast and accurate similarity searches. In this article, we will use Chroma DB to store vector embeddings of the documents, although other options like FAISS, Weaviate, Milvus, and Pinecone are available.</li><li><strong>LangChain</strong>: LangChain is an open-source framework that simplifies creating applications using large language models (LLMs).</li><li><strong>Unstructured</strong>: Unstructured provides capabilities to read and extract components from different types of files.</li></ul><h3>Setup</h3><p>To install Unstructured, you must first install Poppler and Tesseract. We will also install <strong>Chroma DB</strong> as our vector database, <strong>sentence-transformer</strong> for generating embeddings, and <strong>LangChain</strong> to connect all the components of the LLM.</p><blockquote><strong>Langchain</strong> is a powerful open-source framework that simplifies the development of language-based applications by providing a modular and flexible interface for integrating various language models and tools.</blockquote><blockquote><strong>Hugging Face</strong> offers a vast library of pre-trained machine learning models, covering a wide range of tasks such as natural language processing, computer vision, and audio processing, making it a valuable resource for developers and researchers.</blockquote><blockquote><strong>Chroma DB </strong>is a versatile open-source vector database that enables efficient similarity search and retrieval of large-scale datasets, making it a valuable tool for applications in natural language processing, machine learning, and recommendation systems.</blockquote><pre>brew install poppler<br>brew install tesseract<br><br>pip install langchain &#39;unstructured[all-docs]&#39; pydantic lxml<br>pip install -U langchain_openai chromadb langchain-experimental <br>pip install sentence-transformers</pre><p>Now that our setup is done let&#39;s start coding…</p><p><strong>STEP 1</strong>: Import all the libraries</p><pre>import os<br>import base64<br>import ollama<br>import uuid<br>from PIL import Image<br><br>from langchain.text_splitter import CharacterTextSplitter<br>from unstructured.partition.pdf import partition_pdf<br>from langchain_community.chat_models import ChatOllama<br><br>from langchain_core.messages import HumanMessage<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.documents import Document<br><br>from langchain_openai import ChatOpenAI<br>from langchain.retrievers.multi_vector import MultiVectorRetriever<br>from langchain.storage import InMemoryStore<br>from langchain_community.vectorstores import Chroma<br><br>from langchain.embeddings import HuggingFaceEmbeddings<br>from langchain.chains import LLMChain<br>from langchain.prompts import PromptTemplate<br><br>from IPython.display import HTML, display<br>from langchain_core.runnables import RunnableLambda, RunnablePassthroughfrom langchain.text_splitter import CharacterTextSplitter<br>from unstructured.partition.pdf import partition_pdf<br>from langchain_community.chat_models import ChatOllama</pre><p><strong>Step 2: pdf_component_extractor()</strong></p><p>This function utilizes partition_pdf from the Unstructured library to extract various components from PDF files. Although this example focuses on PDF files, the code can easily be adapted to handle other file types (refer to the <a href="https://docs.unstructured.io/api-reference/api-services/document-elements"><strong>documentation</strong></a><strong> </strong>for details). The function takes the location of the PDF file (we are using <a href="https://corporate.walmart.com/content/dam/corporate/documents/newsroom/2024/02/20/walmart-releases-q4-and-fy24-earnings/walmart-earnings-release-fy24-q4.pdf"><strong>this</strong></a> PDF file in our code) and the destination folder, where all extracted images will be saved for further processing. It returns all the text and table chunks, while the extracted images are stored in the specified folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*igoMfGJZ1b5qfRfAl5iD8A.png" /><figcaption><strong>Component extractor</strong></figcaption></figure><pre>def pdf_component_extractor(filePath,saveImagesInFilePath):<br>    &quot;&quot;&quot;<br>    Extract different component from the pdf<br>    Input:<br>    filePath: Location of the pdf file <br>    saveImagesInFilePath: location where all the extracted images from the pdf is saved<br>    Output:<br>    table_chunks: all the extracted table chunks<br>    text_chunks: all the extracted text chunks<br>    &quot;&quot;&quot;<br><br>    pdf_components= partition_pdf(<br>        filename=filePath,<br>        strategy=&quot;hi_res&quot;,<br>        extract_images_in_pdf=True,<br>        infer_table_structure=True,<br>        chunking_strategy=&quot;by_title&quot;,<br>        max_characters=3000,<br>        new_after_n_chars=2800,<br>        combine_text_under_n_chars=2000,<br>        extract_image_block_types=[&quot;Image&quot;, &quot;Table&quot;],   <br>        extract_image_block_output_dir=saveImagesInFilePath,<br>    )<br><br>    table_chunks = []<br>    text_chunks = []<br>    ### separating different type of extracted component from chunk<br>    for element in pdf_components:<br>        if &quot;unstructured.documents.elements.Table&quot; in str(type(element)):<br>            table_chunks.append(str(element))<br>        elif &quot;unstructured.documents.elements.CompositeElement&quot; in str(type(element)):<br>            text_chunks.append(str(element))<br>    <br>    return text_chunks,table_chunks <br><br>filePath=&quot;../pdf_filename.pdf&quot;<br>saveImagesInFilePath=&quot;../all_extracted_images&quot;<br>text_chunks,table_chunks=pdf_component_extractor(filePath,saveImagesInFilePath)</pre><p><strong>Step 3:</strong> <strong>text_summarizer_llm()</strong></p><p>This function takes all the extracted text and table components as input and summarizes them using LLaMA 2. We have defined a custom prompt to guide LLaMA 2 in performing the summarization. The function then returns the summarized text (text_chunk_summaries) and the summarized tables (table_chunk_summaries).</p><blockquote><a href="https://ollama.com/"><strong>Ollama</strong></a> is a free and open-source tool that enables users to operate open LLMs locally on their systems. It is compatible with Linux (specifically, Systemd-powered distributions), Windows, and macOS (including Apple Silicon).</blockquote><blockquote>We are using <strong>LLaMA 2</strong> as our summarization model. You can install LLaMA 2 by following the steps outlined <a href="https://www.codingforentrepreneurs.com/shorts/ollama-and-llama-2-as-drop-in-replacement-for-openai-with-python/"><strong>here</strong></a>. Additionally, ensure that <strong>Ollama</strong> is running on port <em>11434</em>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HVgj7P3mQm-_g7eX5CEmxQ.png" /><figcaption><strong>Summaring table and text chunks</strong></figcaption></figure><pre>def text_summarizer_llm(text_chunks,table_chunks):<br>    &quot;&quot;&quot;<br>    Generate summaries for text and table elements<br>    Input:<br>    table_chunks: table chunks created above<br>    text_chunks: text chunks created above<br>    Output<br>    table_chunks_summaries: List of summaries of table chunks<br>    text_chunks_summaries: List of summaries of text chunks<br>    &quot;&quot;&quot;<br><br>    ### Create rag pipleine to get summaries<br><br>    # Prompt<br>    prompt_text = &quot;&quot;&quot;You are an assistant tasked with summarizing tables and text for retrieval. \<br>    These summaries will be embedded and used to retrieve the raw text or table elements. \<br>    Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} &quot;&quot;&quot;<br>    prompt = ChatPromptTemplate.from_template(prompt_text)<br><br>    ### Initialize model<br>    model= ChatOllama(model=&quot;llama2&quot;)<br><br>    ### Initialize rag chain<br>    rag_summary_chain = {&quot;element&quot;: lambda x: x} | prompt | model | StrOutputParser()<br><br>    # Initialize empty summaries<br>    text_chunks_summaries = []<br>    table_chunks_summaries = []<br><br>    <br>    text_chunks_summaries = rag_summary_chain.batch(text_chunks, {&quot;max_concurrency&quot;: 10})<br>    table_chunks_summaries = rag_summary_chain.batch(table_chunks, {&quot;max_concurrency&quot;: 10})<br>    <br>    return text_chunks_summaries,table_chunks_summaries<br><br># Get text, table summaries<br>text_chunks_summaries, table_chunks_summaries = text_summarizer_llm(text_chunks, table_chunks)</pre><p><strong>Step 4:</strong> <strong>image_summarizer_llm</strong>()</p><p>This function processes the images saved in the folder (saveImageInFilePath) by iterating over each image. It utilizes a utility function base64_image_encoder()to convert each image into a base64 string. The generate_image_summaries() function, which employs the <strong>LLaVA</strong> 3 model, then creates summaries for these images. Finally the function returns both the base64-encoded images (encoded_image_base64_all) and their corresponding summaries (image_summaries).</p><blockquote><strong>LLaVA</strong> is an end-to-end multimodal model designed to understand and generate content from both visual inputs (images) and textual instructions. It integrates a visual encoder and a language model to effectively process and respond to multimodal inputs.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4350Ft98KhvYWVA6gXKWtw.png" /><figcaption><strong>Generating extracted Image Summaries</strong></figcaption></figure><pre>def base64_image_encoder(encoded_image_base64):<br>    &quot;&quot;&quot;Getting the base64 string&quot;&quot;&quot;<br>    with open(encoded_image_base64, &quot;rb&quot;) as image_file:<br>        return base64.b64encode(image_file.read()).decode(&quot;utf-8&quot;)<br><br>def generate_image_summaries(img_base64, prompt):<br>    &quot;&quot;&quot;<br>    generating summary for the image passed in<br>    input:<br>    img_base64: base64 string of the image<br>    prompt: passed to llm to generate image summary<br>    &quot;&quot;&quot;<br>    res = ollama.chat(<br>    model=&quot;llava-llama3&quot;,<br>    messages=[<br>    {<br>     &#39;role&#39;: &#39;user&#39;,<br>     &#39;content&#39;: prompt,<br>      &quot;images&quot;: [img_base64],<br>    }<br>       ]<br>     )<br>    <br>    return res[&#39;message&#39;][&#39;content&#39;]<br><br>def image_summarizer_llm(saveImagesInFilePath):<br>    &quot;&quot;&quot;<br>    Generate summaries for images<br>    Input:<br>    saveImagesInFilePath: location of the folder that contains all the extracted images<br>    text_chunks: text chunks created above<br>    Output<br>    encoded_image_base64_all: List of summaries of text chunks<br>    image_summaries: List of summaries of text chunks<br>    &quot;&quot;&quot;<br><br>    # Store base64 encoded images<br>    encoded_image_base64_all = []<br><br>    # Store image summaries<br>    image_summaries = []<br><br>    # Prompt<br>    prompt = &quot;&quot;&quot;You are an assistant tasked with summarizing images for retrieval. \<br>    These summaries will be embedded and used to retrieve the raw image. \<br>    Give a concise summary of the image that is well optimized for retrieval.&quot;&quot;&quot;<br><br>    # Apply to images<br>    for image_filename in sorted(os.listdir(saveImagesInFilePath)):<br>        if image_filename.endswith(&#39;.jpg&#39;):<br>            image_path=os.path.join(saveImagesInFilePath, image_filename)<br>            encoded_image_base64=base64_image_encoder(image_path)<br>            encoded_image_base64_all.append(encoded_image_base64)<br>            image_summaries.append(generate_image_summaries(encoded_image_base64, prompt))<br><br>    return encoded_image_base64_all, image_summaries<br><br># Image summaries<br>encoded_image_base64_all, image_summaries = image_summarizer_llm(saveImagesInFilePath)</pre><p><strong>Step 5: </strong>First, we define an embedding generator model ( thenlper/gte-large) using HuggingFaceEmbeddings and setting up the vector database ( chroma ). We then pass this information, along with text_chunks, table_chunks_summaries, table_chunks, andimage_summaries, to the create_multivector_retriever() function. This function converts the summaries and actual text into embeddings and stores them in chromaDB and returns the created retriever. Since we are storing multiple vectors per document, we use <strong>LangChain&#39;s</strong> MultiVectorRetriever for the retrieval process. Finally, our retriever is ready to be used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7BY6kItP1ktKMvUh0ZOQA.png" /><figcaption><strong>Saving summaries as vectors in vector DB</strong></figcaption></figure><pre>def add_documents(retriever, doc_summaries, doc_contents,id_key):<br>    &quot;&quot;&quot;<br>    Helper function to add documents to the vectorstore and docstore<br>    &quot;&quot;&quot;<br>    doc_ids = [str(uuid.uuid4()) for _ in doc_contents]<br>    summary_docs = [<br>        Document(page_content=s, metadata={id_key: doc_ids[i]})<br>        for i, s in enumerate(doc_summaries)<br>    ]<br>    retriever.vectorstore.add_documents(summary_docs)<br>    retriever.docstore.mset(list(zip(doc_ids, doc_contents)))<br>    return retriever<br><br>def create_multivector_retriever(<br>    vectorstore, text_chunks_summaries, text_chunks, table_chunks_summaries, table_chunks, image_summaries, images<br>):<br>    &quot;&quot;&quot;<br>    Create retriever that indexes summaries, but returns raw images or texts<br>    Input:<br>    vectorstore : any vector db that you want to use<br>    text_chunks_summaries:<br>    text_chunks:<br>    table_chunks_summaries:<br>    table_chunks:<br>    image_summaries:<br>    images:<br>    <br>    Output:<br>    Retriever:<br>    &quot;&quot;&quot;<br><br>    # Initialize the storage layer<br>    store = InMemoryStore()<br>    id_key = &quot;doc_id&quot;<br><br>    # Create the multi-vector retriever<br>    retriever = MultiVectorRetriever(<br>        vectorstore=vectorstore,<br>        docstore=store,<br>        id_key=id_key,<br>    )<br><br>    ### Adding different components of the document in retriever<br><br>    # Check that text_summaries is not empty before adding<br>    if text_chunks_summaries:<br>        retriever=add_documents(retriever, text_chunks_summaries, text_chunks,id_key)<br>    # Check that table_summaries is not empty before adding<br>    if table_chunks_summaries:<br>        retriever=add_documents(retriever, table_chunks_summaries, table_chunks,id_key)<br>    # Check that image_summaries is not empty before adding<br>    if image_summaries:<br>        retriever=add_documents(retriever, image_summaries, images, id_key)<br><br>    return retriever<br><br>### prepare huggingface embedding model<br>model_name = &quot;thenlper/gte-large&quot;<br>model_kwargs = {&quot;device&quot;: &quot;cpu&quot;}<br>encode_kwargs = {&quot;normalize_embeddings&quot;: False}<br>embeddings_huggingFace = HuggingFaceEmbeddings(<br>    model_name=model_name, model_kwargs=model_kwargs, encode_kwargs=encode_kwargs<br>)<br># The vectorstore to use to index the summaries<br>vectorstore = Chroma(<br>    collection_name=&quot;multi_modal_rag&quot;, embedding_function=embeddings_huggingFace<br>)<br><br># Create retriever<br>retriever = create_multivector_retriever(<br>    vectorstore, text_chunks_summaries, text_chunks, table_chunks_summaries, table_chunks, image_summaries, encoded_image_base64_all<br>)</pre><p><strong>Step 6: </strong>In this step, we integrate all the previously defined components to create a LangChain chain. The multi_modal_rag_chain takes the retriever as input and processes a query by first performing the retrieval. The retrieved documents may include images or text, which we categorize using the extractTextAndImage() function. We also utilize utility functions isBase64() to determine if a string is in base64 format and isImage() to verify if the base64 string represents an image. Next, we generate an LLM prompt with the retrieved components using createLLMPrompt(). Finally, the chain produces an answer to the user&#39;s query using the llava-llama3 model.</p><pre>def isBase64(sb):<br>    &quot;&quot;&quot;<br>    This is a helper function, it is used to check, given string is base64 string or not<br>    &quot;&quot;&quot;<br>    return re.match(&quot;^[A-Za-z0-9+/]+[=]{0,2}$&quot;, sb) is not None<br><br>def isImage(b64data):<br>    &quot;&quot;&quot;<br>    This is a helper function which is used to check whether the given base64 represents an image or not<br>    &quot;&quot;&quot;<br>    image_signatures = {<br>        b&quot;\xFF\xD8\xFF&quot;: &quot;jpg&quot;,<br>        b&quot;\x89\x50\x4E\x47\x0D\x0A\x1A\x0A&quot;: &quot;png&quot;,<br>        b&quot;\x47\x49\x46\x38&quot;: &quot;gif&quot;,<br>        b&quot;\x52\x49\x46\x46&quot;: &quot;webp&quot;,<br>    }<br>    try:<br>        header = base64.b64decode(b64data)[:8]  # Decode and get the first 8 bytes<br>        for signature, format in image_signatures.items():<br>            if header.startswith(signature):<br>                ### if header of the given base64 data matches with any signature of any of the above format,<br>                return True<br>        return False<br>    except Exception:<br>        return False<br><br>def resizeGivenImage(base64_string, size=(128, 128)):<br>    &quot;&quot;&quot;<br>    This function accepts a base64 image of any size , and returns a base64 image of given dimension.<br>    &quot;&quot;&quot;<br>    # Decode the Base64 string<br>    img_data = base64.b64decode(base64_string)<br>    img = Image.open(io.BytesIO(img_data))<br><br>    # Resize the image<br>    resized_img = img.resize(size, Image.LANCZOS)<br><br>    # Save the resized image to a bytes buffer<br>    buffered = io.BytesIO()<br>    resized_img.save(buffered, format=img.format)<br><br>    # Encode the resized image to Base64<br>    return base64.b64encode(buffered.getvalue()).decode(&quot;utf-8&quot;)<br><br>def extractTextAndImage(all_documents):<br>    &quot;&quot;&quot;<br>    Extract and segment, image and text data from retrieved document<br>    &quot;&quot;&quot;<br>    extracted_images_list = []<br>    extracted_texts_list = []<br>    for doc in all_documents:<br>        # Check if the document is of type Document and extract page_content if so<br>        if isinstance(doc, Document):<br>            doc = doc.page_content<br>        # Check if given doc string is image or not<br>        # if it is image resize them (1300, 600) dimension <br>        if isBase64(doc) and isImage(doc):<br>            doc = resizeGivenImage(doc, size=(1300, 600))<br>            extracted_images_list.append(doc)<br>        else:<br>            extracted_texts_list.append(doc)<br>    return {&quot;images&quot;: extracted_images_list, &quot;texts&quot;: extracted_texts_list}<br><br>def createLLMPrompt(retreived_documents):<br>    &quot;&quot;&quot;<br>    This function will create prompt for llm<br>    &quot;&quot;&quot;<br>    formatted_texts = &quot;\n&quot;.join(retreived_documents[&quot;context&quot;][&quot;texts&quot;])<br>    messages = []<br><br>    # Adding image(s) to the messages if present<br>    if retreived_documents[&quot;context&quot;][&quot;images&quot;]:<br>        for image in retreived_documents[&quot;context&quot;][&quot;images&quot;]:<br>            image_data = {<br>                &quot;type&quot;: &quot;image_url&quot;,<br>                &quot;image_url&quot;: {&quot;url&quot;: f&quot;data:image/jpeg;base64,{image}&quot;},<br>            }<br>            messages.append(image_data)<br><br>    # Adding the text for analysis<br>    text_data = {<br>        &quot;type&quot;: &quot;text&quot;,<br>        &quot;text&quot;: (<br>            &quot;You will be given a mixed of text, tables, and image(s) usually of charts or graphs.\n&quot;<br>            &quot;Use this information to answer related to the user question. \n&quot;<br>            f&quot;User-provided question: {retreived_documents[&#39;question&#39;]}\n\n&quot;<br>            &quot;Text and / or tables:\n&quot;<br>            f&quot;{formatted_texts}&quot;<br>        ),<br>    }<br>    messages.append(text_data)<br>    return [HumanMessage(content=messages)]<br><br>def multi_modal_rag_chain(retriever):<br>    &quot;&quot;&quot;<br>    Multi-modal RAG chain<br>    &quot;&quot;&quot;<br>    # Multi-modal LLM<br>    model= ChatOllama(model=&quot;llava-llama3&quot;)<br><br>    <br>    # RAG pipeline<br>    chain = (<br>        {<br>            &quot;context&quot;: retriever | RunnableLambda(extractTextAndImage),<br>            &quot;question&quot;: RunnablePassthrough(),<br>        }<br>        | RunnableLambda(createLLMPrompt)<br>        | model<br>        | StrOutputParser()<br>    )<br><br>    return chain<br><br># Create RAG chain<br>chain_multimodal_rag = multi_modal_rag_chain(retriever)</pre><p><strong>Step 7: </strong>Voilà! Your application is now ready, and you can start retrieving answers from your documents.</p><pre># Run RAG chain<br>query =   &quot;What is the Total revenues for FY&#39;24 Q4?&quot;<br>chain_multimodal_rag.invoke(query)</pre><pre>The total revenues for FY&#39;24 Q4 are $173.4 million, representing a 5.7% increase from the same period in FY&#39;23. </pre><p><strong>Step 8: </strong>If you want to see which components (image or text) are passed to your LLM, you can use this code.</p><pre>def plot_images(image_base64):<br>    &quot;&quot;&quot;Disply base64 encoded string as image&quot;&quot;&quot;<br>    # Create an HTML img tag with the base64 string as the source<br>    image_html = f&#39;&lt;img src=&quot;data:image/jpeg;base64,{image_base64}&quot; /&gt;&#39;<br>    # Display the image by rendering the HTML<br>    display(HTML(image_html))</pre><pre># Check retrieval<br>docs = retriever.get_relevant_documents(query, limit=6)<br># to get length of  retrieved documents<br>len(docs)</pre><pre># pass the docs index which contains image<br>plot_images(docs[0])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NVU9_qAsrgEmTY133mrvaw.png" /><figcaption>Output Image extracted from given pdf, when the query was &quot;What are the Total revenues for FY’24 Q4?&quot;</figcaption></figure><h3>Conclusion</h3><p>While we’ve covered the basics of implementing a Multimodal RAG pipeline, several challenges still need addressing. Scalability remains a significant issue, as processing large volumes of multimodal data requires substantial computational power. Maintaining data consistency across different modalities and crafting effective queries that utilize various data types adds to the complexity. Additionally, managing increased resource demands, optimizing latency, and developing appropriate evaluation metrics are essential for building a robust and efficient multimodal RAG system. Despite these challenges, ongoing research and development hold the potential to advance human-like communication between machines and humans.</p><h3>Written by</h3><ol><li><a href="https://www.linkedin.com/in/abh1jeet/">Abhijeet Pandey</a> — Data Scientist — Walmart Global Tech</li><li><a href="https://in.linkedin.com/in/himanshu-pant-ds">Himanshu Pant </a>— Senior Data Scientist — Walmart Global Tech</li></ol><h3>References</h3><ol><li><a href="https://python.langchain.com/v0.2/docs/tutorials/rag/">https://python.langchain.com/v0.2/docs/tutorials/rag/</a></li><li><a href="https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html">https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html</a></li><li><a href="https://huggingface.co/thenlper/gte-large">https://huggingface.co/thenlper/gte-large</a></li><li><a href="https://docs.unstructured.io/api-reference/api-services/document-elements">https://docs.unstructured.io/api-reference/api-services/document-elements</a></li><li><a href="https://llama.meta.com/llama2/">https://llama.meta.com/llama2/</a></li><li><a href="https://ollama.com/">https://ollama.com/</a></li><li><a href="https://ollama.com/library/llava-llama3">https://ollama.com/library/llava-llama3</a></li><li><a href="https://ai.google/discover/palm2/">https://ai.google/discover/palm2/</a></li><li><a href="https://ai.google.dev/gemini-api/docs">https://ai.google.dev/gemini-api/docs</a></li><li><a href="https://arxiv.org/abs/1810.04805">https://arxiv.org/abs/1810.04805</a></li><li><a href="https://platform.openai.com/docs/concepts">https://platform.openai.com/docs/concepts</a></li><li><a href="https://unstructured.io/blog/chunking-for-rag-best-practices">https://unstructured.io/blog/chunking-for-rag-best-practices</a></li><li><a href="https://arxiv.org/pdf/2312.06648">https://arxiv.org/pdf/2312.06648</a></li><li><a href="https://www.trychroma.com/">https://www.trychroma.com/</a></li><li><a href="https://github.com/explodinggradients/ragas">https://github.com/explodinggradients/ragas</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2200cb013a86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/decoding-multimodal-rag-advanced-techniques-for-seamless-document-interaction-part-2-2200cb013a86">Decoding Multimodal RAG: Advanced Techniques for Seamless Document Interaction (Part 2)</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Multimodal RAG: Advanced Techniques for Seamless Document Interaction (Part 1)]]></title>
            <link>https://medium.com/walmartglobaltech/decoding-multimodal-rag-advanced-techniques-for-seamless-document-interaction-part-1-bbc0b07d4703?source=rss-b34699dfb8e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbc0b07d4703</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <dc:creator><![CDATA[Abhijeet Pandey]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 04:14:16 GMT</pubDate>
            <atom:updated>2025-08-22T04:14:16.732Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5F5ZDMZZVUH0wWRjK5MQNw.jpeg" /><figcaption>Image generated using Stable Diffusion 2</figcaption></figure><h3>Introduction</h3><p>Retrieval-Augmented Generation (RAG) is an architectural strategy that enhances the effectiveness of large language model (LLM) applications by utilizing custom data. It achieves this by retrieving relevant data or documents for a specific question or task and using them as context for the LLM. In a RAG pipeline, when we provide it with documents, a knowledge base is created using those documents. The performance of the RAG pipeline primarily depends on four key questions:</p><blockquote><strong><em>How are we extracting different components from the document ?</em></strong></blockquote><blockquote><strong><em>How are we splitting the document into chunks ?</em></strong></blockquote><blockquote><strong><em>How are we indexing these chunks ?</em></strong></blockquote><blockquote><strong><em>How are we retrieving these chunks ?</em></strong></blockquote><p>This will be a <strong>multi-part series</strong>. In this <strong>part</strong>, we will discuss each of the questions mentioned above in more detail. In the second part, we will explore how to <strong>implement</strong> the proposed solutions.</p><p>It’s important to note that there are multiple ways to enhance your RAG pipeline. These suggestions can be tailored and optimized according to your specific use case, potentially leading to effective improvements.</p><h3>How are we extracting different components from the document?</h3><p>When we parse a PDF, HTML, or DOCX file, various components might be present in the document. These include regular text, tables, images, and more. The text itself can be subdivided into multiple categories, such as headers, footers, narrative text, titles, table captions, and figure captions. We cannot treat each of these components in the same way, which is why proper component extraction is necessary.</p><p>We can achieve this using the <strong>partition</strong> function from the <a href="https://docs.unstructured.io/open-source/core-functionality/partitioning"><strong>Unstructured</strong></a> library. Partitioning functions in Unstructured enable users to extract structured content from raw, unstructured documents. These functions break a document into elements like <strong>Title, NarrativeText, ListItem</strong>, and more (as shown in the figure below), allowing users to select the specific content they want to retain for their application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nkbWX8iCtYzYkfmUeVJ0Tw.png" /><figcaption>Examples of the element types supported by <a href="https://docs.unstructured.io/api-reference/api-services/document-elements">unstructured library</a></figcaption></figure><p>Unstructured provides capabilities to read and extract components from different types of files. The partition function is the simplest way to partition and extract components from any type of document. When you call the <strong>partition</strong> function, Unstructured will attempt to detect the file type and route it to the appropriate partitioning function. If you know the file type, you can call the appropriate partitioning function directly, such as <strong>partition_pdf</strong> for PDFs or <strong>partition_csv</strong> for CSV files.</p><h3>How are we splitting the document into chunks?</h3><p>We cannot pass unlimited data to our language model (LLM) due to constraints such as the context window and the signal-to-noise ratio:</p><blockquote><strong>Context Window:</strong> LLMs have a context limit, which is the maximum number of tokens that can be processed in one instance. Therefore, we must fit the input within this upper bound to ensure proper processing.</blockquote><blockquote><strong>Signal-to-Noise Ratio:</strong> Language models perform better when the signal-to-noise ratio is high. This means we should pass only the most relevant and useful information to the model, minimizing the inclusion of irrelevant data.</blockquote><p>For effective retrieval and prompt generation, vector databases require documents to be split into chunks. The way a document is chunked can significantly affect the content returned by the same query.</p><p>A straightforward method is to split the document into roughly equal-sized chunks. However, this approach can lead to similar content being split across multiple chunks. Due to the limited context window, it is often impractical to pass an entire retrieved component, and converting large paragraphs into embeddings may result in the loss of important details.</p><p>When creating chunks from various components of a document, consider the following points:</p><blockquote><strong>Context Preservation:</strong> Ensure chunks maintain coherent context, especially for narrative text, to avoid losing important information.</blockquote><blockquote><strong>Component Integrity:</strong> Keep tables, figures, and list items intact to preserve their structure and meaning. Avoid splitting these elements across chunks.</blockquote><blockquote><strong>Uniform Size:</strong> Strive for consistent chunk sizes to balance processing efficiency and context retention, but adapt based on component type.</blockquote><blockquote><strong>Relevance and Coherence:</strong> Ensure each chunk is relevant and cohesive, avoiding mixing unrelated content which could dilute the focus.</blockquote><blockquote><strong>Metadata Inclusion:</strong> Include relevant metadata (e.g., captions for tables and figures) within chunks to maintain context and enhance retrieval accuracy.</blockquote><p>These are a few strategies we can use while chunking</p><h3>1. Fixed Size Chunking</h3><p><strong>Method: </strong>Divide the document into chunks of a predetermined number of words or tokens (smallest units of text, such as words or subwords); we can use the optional overlapping tokens feature (a method where chunks overlap to ensure smooth transitions) which will blend together chunks so that the tail of Chunk(n)will be the same thing and the head of Chunk (n+1).</p><pre>from langchain.text_splitter import CharacterTextSplitter<br>splitter = CharacterTextSplitter(chunk_size = 35, chunk_overlap=10)<br>splitter.create_documents([text])</pre><p><strong>Limitations:</strong></p><ul><li><strong>Loss of Context:</strong> Content may be fragmented, leading to loss of important context, especially if chunks cut across logical boundaries.</li><li><strong>Relevance Issues:</strong> Important information might be split across chunks, reducing coherence.</li></ul><p><strong>Example:</strong></p><ul><li>A research paper divided into chunks of 500 words. A paragraph spanning 600 words would be split into two chunks, potentially breaking a narrative or argument.</li></ul><h3>2. Recursive chunking</h3><p><strong>Method:</strong> Split the document into chunks and recursively divide these chunks further based on context or structure. Start with larger chunks (e.g., sections or paragraphs) and recursively split into smaller units if they exceed a certain size or contain multiple topics. Like langchain’s RecursiveCharacterTextSplitteruses [&quot;\n\n&quot;, &quot;\n&quot;, &quot; &quot;, &quot;&quot;]as default splitters, it takes in the large text and then tries to split it by the first splitter \n\n. If the first split-by \n\n is still large, then it moves to the next character, which is \n and tries to split by it. If it is still larger than our specified chunk size, it moves to the next character in the set until we get a split that is less than our specified chunk size.</p><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>splitter = RecursiveCharacterTextSplitter(chunk_size = 35, chunk_overlap=10)<br>splitter.create_documents([text])</pre><p><strong>Limitations:</strong></p><ul><li><strong>Complexity:</strong> Recursive chunking can be complex to implement and manage, especially for large documents with varied structures.</li><li><strong>Overhead:</strong> The process may require multiple iterations, increasing processing time.</li></ul><p><strong>Example:</strong></p><ul><li><strong>Text:</strong> A book is first divided into chapters, then each chapter is recursively split into sections and then into paragraphs if needed. This ensures that chunks are contextually relevant.</li></ul><h3>3. Document chunking</h3><p><strong>Method:</strong> In the previous section, we discussed how to extract different components from our document. LangChain provides various functions to handle these extracted components differently. For example, we have MarkdownTextSplitter to split markdown content and PythonCodeTextSplitter to split Python code.</p><pre>from langchain.text_splitter import MarkdownTextSplitter<br>splitter = MarkdownTextSplitter(chunk_size = 35, chunk_overlap=0)<br>splitter.create_documents([any_mark_down_text])</pre><p><strong>Limitations:</strong></p><ul><li><strong>Inconsistency:</strong> Chunk sizes can vary widely, potentially resulting in some chunks being too large or too small.</li><li><strong>Irregular Structure:</strong> Documents with irregular or inconsistent structures may pose challenges.</li></ul><h3>4. Proposition-Based Chunking</h3><p>Let’s start with a simple example to illustrate proposition-based chunking.</p><blockquote><strong>Raw Paragraph:</strong>“Lionel Messi is an Argentina football player. He won the World Cup in 2022.”<br><strong>Normal Chunking: </strong>[Lionel Messi is an Argentina football player.] [He won the World Cup in 2022.]<br><strong>Proposition-Based Chunking:</strong>[Lionel Messi is an Argentina football player.]<br>[Lionel Messi won the World Cup in 2022.]</blockquote><p>In normal chunking, the second chunk, “[He won the World Cup in 2022.]” lacks clarity because it doesn’t specify who “he” refers to, thus losing important information.</p><p>Proposition-based chunking is used to address this issue. Instead of directly chunking the raw text, we pass the extracted text to a finetuned text generation model, which takes a passage as input and generates the list of propositions. The idea described in the <a href="https://arxiv.org/pdf/2312.06648">paper</a> is that each proposition should describe exactly one contextualized atomic fact. This makes propositions ideal as retrieval units for information-seeking questions.</p><p>In the paper, a proposition is described by these 3 principles.</p><blockquote><strong>Distinct Meaning:</strong> Each proposition corresponds to a unique piece of meaning in the text. Together, all propositions represent the entire text’s semantics.</blockquote><blockquote><strong>Minimalism: </strong>Each proposition should be minimal and not further splittable into separate propositions.</blockquote><blockquote><strong>Contextualization:</strong> Each proposition should be self-contained and include all necessary context (e.g., resolving coreferences) to interpret its meaning.</blockquote><p>We use proposition-based chunking to ensure that each chunk retains complete information, making it more effective for retrieval and processing by language models. The ultimate goal remains to group similar items together so they are ready when passed to the language model. This method enhances clarity and preserves the contextual integrity of the information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-580OkwoRaqyee8vK0IGA.png" /><figcaption>The architecture described in the paper “<a href="https://arxiv.org/pdf/2312.06648">Dense X Retrieval: What Retrieval Granularity Should We Use?</a>”</figcaption></figure><h3>How are we indexing these chunks?</h3><p>Now that we’ve discussed extracting relevant components from the document and splitting them, the next step is to save this information in our knowledge base for retrieval tasks. Indexing involves deciding how to represent your data in your database or knowledge base. But remember that we have different chunks, including text, images, and tables, which require specific indexing methods.</p><p><strong>1: Indexing the raw chunks</strong></p><p>A common technique involves converting each created chunk into embeddings. Many models are available for this, such as <strong>BERT</strong> and <strong>DistilBERT</strong> for text data and <strong>resnet50</strong>, <strong>vgg16</strong>, etc, for images. However, using separate models for text and images results in embeddings that are not in the same vector space. For example, the text “A red car is driving on the road” and an image of “a red car driving on the road” should be close in the vector space for effective retrieval. This is where <a href="https://github.com/openai/CLIP"><strong>CLIP</strong></a> comes in. CLIP (Contrastive Language-Image Pretraining) is a deep learning model whose generated embeddings for images and text share the same space, enabling direct comparisons between the two modalities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyCvREeeb-ZwQDvTUN6lvg.png" /><figcaption><strong>Multimodel RAG: indexing the raw chunks directly</strong></figcaption></figure><p><strong>2: Indexing the summaries of chunks</strong></p><blockquote>It’s important to note that creating a vector database is under our control. We can transform the extracted chunks into different forms before creating embeddings. This is where LangChain’s <strong>MultiVectorStore</strong> comes into play. MultiVectorStore is useful for performing semantic searches with vectors derived from sources other than raw text.</blockquote><p>Instead of passing chunks directly to an embedding model, consider creating summaries of these chunks first. Using multimodal LLMs, we can generate concise summaries that more effectively capture the essence of each chunk (text and image). Summaries often provide a clearer representation of the content, improving retrieval accuracy. Later in this blog, we’ll demonstrate how to create and embed these summaries for enhanced data retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9c2zZFF5F2BjpYl8eB5-g.png" /><figcaption><strong>Multimodel RAG: Indexing the summaries of chunks</strong></figcaption></figure><p><strong>3: Hypothetical questions</strong></p><p>An LLM can generate a list of hypothetical questions related to a particular chunk, which can then be embedded for improved retrieval. This technique particularly benefits sparse, unstructured data, such as chat messages. For example, if you’re developing a bot that uses Slack conversations as a knowledge base, directly performing semantic searches on raw chat messages may be ineffective. However, generating hypothetical questions that the messages could answer enhances the likelihood of retrieving relevant information when new queries are made.</p><blockquote>The underlying hypothesis in these approaches is that smaller chunks are more likely to match semantically with a query. However, you have the option to decide the context provided to the LLM. You can either send the smaller chunks directly to the LLM or use them for matching and provide the original, larger chunk (text or image) as context to the LLM.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*22y9ivarlxi5hy11cxFTPg.png" /><figcaption><strong>Multimodel RAG: Indexing hypothetical questions</strong></figcaption></figure><h3>How are we retrieving these chunks?</h3><p>A better retrieval strategy in a RAG pipeline enhances the accuracy, relevance, and efficiency of responses by effectively matching user queries with the most pertinent information. It ensures that the system retrieves highly relevant content, reducing <strong>noise</strong> and improving understanding of <strong>complex queries</strong>. This leads to faster, more accurate answers, greater user satisfaction, and efficient use of computational resources. By adapting to different data types and continuously learning from interactions, an optimized retrieval process can handle ambiguity and provide high-quality results, ultimately boosting the overall performance of the RAG pipeline.</p><p><strong>1: RAG Fusion</strong></p><p>RAG Fusion is an advanced extension of the traditional Retrieval-Augmented Generation (RAG) model, designed to better understand and respond to complex human queries. In this, we generate multiple queries similar to the user query using any LLM. The key innovation in RAG Fusion lies in its ability to handle multiple queries simultaneously and re-rank the retrieved results for more accurate and comprehensive responses. It employs techniques like <strong>Reciprocal Rank Fusion (RRF)</strong> and custom vector score weighting to combine and evaluate multiple search results, ensuring that the most relevant information is prioritized.</p><blockquote>Reciprocal Rank Fusion is a rank aggregation method that combines rankings from multiple sources into a single, unified ranking. In the context of RAG, these sources typically use different retrieval models or approaches.</blockquote><blockquote><strong>RRF(d) </strong>= Σ(r ∈ R) 1 / (k + r(d))</blockquote><blockquote>Where:<br>d is a document<br>R is the set of rankers (retrievers)<br>k is a constant (typically 60)<br> r(d) is the rank of document d in ranker r</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*mRNv2O6p4onE2ZmM23mPyQ.png" /><figcaption>RAG FUSION (<a href="https://towardsdatascience.com/forget-rag-the-future-is-rag-fusion-1147298d8ad1">Image Source</a>)</figcaption></figure><p>RAG Fusion significantly enhances the retrieval process by bridging the gap between what users explicitly ask and their underlying intent, leading to more precise and contextually appropriate responses. This approach not only improves the quality of answers but also brings the system closer to uncovering deeper, transformative insights from the data, making it particularly useful for handling complex queries in various applications.</p><p><strong>2: Synonym and Concept Expansion</strong></p><p>Rather than generating entirely new queries using an LLM, which can be costly, we can transform the current query to improve its retrieval performance.</p><ul><li><strong>Synonym Expansion</strong>: When a user submits a query, the system can automatically expand it by including synonyms or alternative phrases. For example, if the query is “car,” the system might expand it to include “automobile,” “vehicle,” or “sedan.” This expansion helps capture documents that might use different terms to describe the same concept, improving the likelihood of retrieving relevant results.</li><li><strong>Concept Expansion</strong>: Beyond simple synonyms, the system can recognize related concepts. For instance, a query about “heart attack” might be expanded to include terms like “myocardial infarction,” “cardiac arrest,” or “coronary thrombosis.” This is particularly useful in specialized domains like medicine or law, where terminology can vary widely.</li><li><strong>Context-Aware Expansion</strong>: CQE can consider the broader context in which a query is made. For example, if a user has been asking about “battery life” in previous interactions, a new query about “charging” could be expanded to include terms related to “battery efficiency,” “power management,” and “charging cycles.” This contextual awareness ensures that the expansion is relevant and aligned with the user’s overall intent.</li></ul><h3>RAG Pipeline Evaluation</h3><p>Evaluating RAG models involves a multi-faceted approach to ensure their effectiveness. Key strategies include measuring performance metrics such as <strong>accuracy</strong>, <strong>precision</strong>, <strong>recall</strong>, and <strong>F1 score</strong> to gauge response relevance. Qualitative assessments, including human evaluations and sample queries, provide insight into response quality. Evaluating retrieval quality through metrics like <strong>Precision</strong> and <strong>Mean Reciprocal Rank</strong> (MRR) helps assess the relevance of retrieved documents. Response generation can be evaluated using <strong>BLEU</strong>, <strong>ROUGE</strong>, and <strong>METEOR</strong> scores, while diversity metrics assess the variety of generated outputs. User feedback and real-world testing, along with system integration and performance monitoring through A/B testing and CI/CD integration, offer a comprehensive understanding of model performance and areas for improvement.</p><blockquote><a href="https://github.com/explodinggradients/ragas"><strong><em>RAGAS</em></strong></a> is a framework that helps you evaluate your RAG pipelines. Ragas provides you with the tools based on the latest research for evaluating LLM-generated text to give you insights about your RAG pipeline. Ragas can be integrated with your CI/CD to provide continuous checks to ensure performance.</blockquote><h3>Conclusion</h3><p>While effective, the RAG pipeline often faces challenges that hinder its production readiness. Key issues include scalability due to high computational demands, potential latency from its multi-step process, and integration complexity with existing systems. Additionally, managing resources like large language models and vector databases can be costly and demanding. Ensuring consistent performance, robust model training, and handling edge cases are also crucial for reliable production operation. In this <strong>Blog</strong>, we’ve explored several techniques that can help enhance the effectiveness of your RAG model, and in the next <a href="https://medium.com/@itsmeabhijeetpandey/2200cb013a86"><strong>part</strong></a>, we will implement the proposed solutions.</p><h3>Written by</h3><ol><li><a href="https://www.linkedin.com/in/abh1jeet/">Abhijeet Pandey</a> — Data Scientist — Walmart Global Tech</li><li><a href="https://in.linkedin.com/in/himanshu-pant-ds">Himanshu Pant </a>— Senior Data Scientist — Walmart Global Tech</li></ol><h3>References</h3><ol><li><a href="https://python.langchain.com/v0.2/docs/tutorials/rag/">https://python.langchain.com/v0.2/docs/tutorials/rag/</a></li><li><a href="https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html">https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html</a></li><li><a href="https://github.com/openai/CLIP">https://github.com/openai/CLIP</a></li><li><a href="https://huggingface.co/thenlper/gte-large">https://huggingface.co/thenlper/gte-large</a></li><li><a href="https://docs.unstructured.io/api-reference/api-services/document-elements">https://docs.unstructured.io/api-reference/api-services/document-elements</a></li><li><a href="https://llama.meta.com/llama2/">https://llama.meta.com/llama2/</a></li><li><a href="https://ollama.com/library/llava-llama3">https://ollama.com/library/llava-llama3</a></li><li><a href="https://unstructured.io/blog/chunking-for-rag-best-practices">https://unstructured.io/blog/chunking-for-rag-best-practices</a></li><li><a href="https://arxiv.org/pdf/2312.06648">https://arxiv.org/pdf/2312.06648</a></li><li><a href="https://www.trychroma.com/">https://www.trychroma.com/</a></li><li><a href="https://github.com/explodinggradients/ragas">https://github.com/explodinggradients/ragas</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbc0b07d4703" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/decoding-multimodal-rag-advanced-techniques-for-seamless-document-interaction-part-1-bbc0b07d4703">Decoding Multimodal RAG: Advanced Techniques for Seamless Document Interaction (Part 1)</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Web App For File Interactions Using RAG: A Developers Guide]]></title>
            <link>https://medium.com/walmartglobaltech/creating-web-app-for-file-interactions-using-rag-a-developers-guide-aeaed58de536?source=rss-b34699dfb8e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeaed58de536</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Abhijeet Pandey]]></dc:creator>
            <pubDate>Fri, 06 Sep 2024 08:15:58 GMT</pubDate>
            <atom:updated>2024-09-06T08:15:58.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pe4HYug8d48MOnYY.jpeg" /><figcaption>Image generated using Stable Diffusion 2</figcaption></figure><h3>Introduction</h3><p>Large language models (LLMs) are AI systems trained on vast datasets of text and code. They can produce text, translate languages, create diverse content, and provide informative responses to questions.</p><p>One of the biggest limitations of large language models (LLMs) is their reliance on outdated training data. Since LLMs are trained on datasets that may not be continuously updated, they lack information about recent events, trends, and advancements. This means they cannot provide accurate or relevant answers about the latest developments in various fields, making their responses less useful for real-time applications.</p><p>Additionally, LLMs can sometimes produce “hallucinations” — generating information that is incorrect, nonsensical, or entirely fabricated. This occurs because the models generate responses based on patterns in the data they were trained on rather than having a true understanding of the content. As a result, they might confidently provide wrong information or create plausible-sounding but false statements, which can be misleading and problematic in critical applications.</p><p>To mitigate these limitations, retrieval techniques are used. By combining the language understanding capabilities of LLMs with the provision of up-to-date and relevant data, we can reduce hallucinations and improve accuracy.</p><p>There are several ways to use retrieval-based techniques. In this article, we&#39;ll build a Flask web app using langchain, where users can upload their files. Using the above-described retrieval techniques along with LLMs, users can easily interact with their documents. This type of application can be used in various scenarios requiring Q&amp;A interactions with documents.</p><h3><strong>Prequisite &amp; Setup</strong></h3><ul><li><strong>LLM</strong>: Basic understanding of large language models. In this article, we will use OpenAI’s model API. You will need to generate an API key from <a href="https://openai.com/index/openai-api/"><strong>here</strong></a>, but you can also use LLaMA or any other LLM.</li><li><strong>Vector Embedings:</strong> Vector embeddings in the context of large language models (LLMs) are numerical representations of text data. They capture the semantic meaning of words, phrases, sentences, or entire documents in a continuous vector space. There are various models available to convert a textual document into vectors, in this article we will be using the General Text Embeddings (<a href="https://huggingface.co/thenlper/gte-large"><strong>gte-large</strong></a>) model which generates <strong>1024</strong> dimension vector embeddings.</li><li><strong>Vector Database</strong>: A vector database stores data as vectors, allowing for fast and accurate similarity searches. In this article, we will use <strong>FAISS</strong> to store vector embeddings of the documents, although other options like Chroma, Weaviate, Milvus, and Pinecone are available.</li><li><strong>LangChain</strong>: LangChain is an open-source framework that simplifies creating applications using large language models (LLMs).</li><li><strong>Flask</strong>: Flask is a web framework that provides libraries for building lightweight web applications in <strong>Python</strong>.</li></ul><p>Let’s start by installing all the required libraries</p><pre>pip install openai==1.35.14<br>pip install -U sentence-transformers==2.6.1<br>pip install langchain==0.1.16<br>pip install flask==3.0.3</pre><h3>Under the hood</h3><p>The concept involves providing the LLM with relevant content retrieved from the document, along with the user’s question. This approach ensures that the LLM generates responses based on the specific and accurate information contained in the document, thereby minimizing the risk of hallucinations or fabricating information. This method enhances the accuracy and trustworthiness of the responses, as the LLM relies on verified information rather than generating text solely based on its training data.</p><p>When the user uploads documents, we will first read all the files, split them into chunks, and convert these chunks into vector embeddings, which will then be stored in a vector database. When the user asks a question related to the document, we will perform a similarity search to extract only the relevant text. This relevant text will then be provided to the LLM along with the user’s query, which will then generate a response for the user (See below figure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1opQPL-l-Uf1RMDw270Ug.png" /><figcaption>Chat with documents architecture diagram</figcaption></figure><h3>Let’s Start building our application</h3><ol><li><strong>Set up the Flask app structure</strong>:</li></ol><pre>flask_app/<br>│<br>├── static/<br>│   ├── index.css<br>│   └── chat.css<br>│   └── chat.js<br>├── templates/<br>│   ├── index.html<br>│   └── chat.html<br>├── app.py<br>├── upload2vectorDb.py<br>├── talk2vectorDb.py</pre><p><strong>2. Create the home page for our application</strong></p><p><strong>templates/index.html</strong>:</p><p>This HTML page features a form where users can enter a unique username and select multiple files in various formats for upload. The username will be used to identify all files associated with a specific user. This will help us cache information at the user level, allowing users to interact with all their uploaded historical documents.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br> &lt;title&gt;Upload Files&lt;/title&gt;<br>  &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;../static/index.css&quot;&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;div class=&#39;App&#39;&gt;<br>  &lt;form id=&quot;upload-form&quot; method=&quot;post&quot; enctype=multipart/form-data action=&quot;{{ url_for(&#39;upload_file&#39;) }}&quot;&gt;<br>&lt;div class=&quot;form-container&quot;&gt;<br>  &lt;div class=&quot;form-group&quot;&gt;<br>  &lt;label for=&quot;username&quot;  &gt;Type a unique username::&lt;/label&gt;<br>  &lt;input type=&quot;text&quot; id=&quot;username&quot; name=&quot;username&quot; &gt;&lt;br&gt;<br>  &lt;/div&gt;<br>  &lt;div class=&quot;form-group&quot;&gt;<br>    &lt;label for=&quot;file&quot;  &gt;Upload all your files:: &lt;/label&gt;<br>    &lt;input type=&quot;file&quot; type=&quot;file&quot; name=&quot;file&quot; id=&quot;file&quot; multiple required&gt;&lt;br&gt;<br>  &lt;/div&gt;<br>    &lt;button type=&quot;submit&quot; id=&quot;submitButton&quot; &gt;Upload Files&lt;/button&gt;<br>  &lt;/div&gt;<br>  &lt;/form&gt;<br>  <br>&lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>Update <strong>static /index.css</strong>:</p><p>To add some styling to index.html page.</p><pre>html {<br>    text-align: center;<br>    background-color: #1d3469;<br>    color: white;<br>  } <br>  .App-header {<br>    background-color: #1d3469;<br>    min-height: 100vh;<br>    display: flex;<br>    flex-direction: column;<br>    align-items: center;<br>    justify-content: center;<br>    font-size: calc(10px + 2vmin);<br>    color: white;<br>  }<br>form {<br>    display: flex;<br>    flex-direction: column;<br>    align-items: flex-start;<br>    max-width: 1200px;<br>    margin: 0 auto;<br>    padding: 20px;<br>    background-color: #3d5c95;<br>    border-radius: 8px;<br>    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);<br>    font-size: 20px;<br>  }<br>.form-group {<br>display: flex;<br>align-items: center;<br>margin-bottom: 10px;<br>}<br>.form-container {<br>      display: flex;<br>      flex-direction: column;<br>      width: 800px; /* adjust the width as needed */<br>  } <br>  #submitButton {<br>    margin-top: 10px;<br>    padding: 8px 16px;<br>    background-color: #898f9c;<br>    color: white;<br>    border: none;<br>    border-radius: 4px;<br>    cursor: pointer;<br>  }<br><br>  #submitButton:hover {<br>    background-color:#bdc5d1;<br>  }<br>  <br>  #submitButton:active {<br>    background-color: #97a1af;<br>  }</pre><p>3. <strong>Setting up the helper functions</strong></p><p><strong>upload2vectorDb.py:</strong></p><p>This helper function is designed to read various file types, including <strong>csv</strong>, <strong>doc</strong>, <strong>html</strong>, <strong>pdf</strong>, <strong>py</strong>, and <strong>ipynb</strong> etc. As mentioned earlier, we first read the file and then split it into smaller chunks using LangChain&#39;s <strong>RecursiveCharacterTextSplitter</strong> function. Each chunk is set to a size of 1,000 characters, with an overlap of 200 characters between chunks. These hyperparameters can be adjusted based on the use case and the size of the uploaded documents. Once the document is divided into chunks, we use the <a href="https://huggingface.co/thenlper/gte-large"><strong>gte-large</strong></a>text embedding model to convert these chunks into vector embeddings of 1024 dimension, which are then saved in <strong>FAISS</strong>.</p><pre>import os<br>from langchain.embeddings import HuggingFaceEmbeddings<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.vectorstores import FAISS<br>from langchain_community.document_loaders import (<br>    CSVLoader,<br>    NotebookLoader,<br>    PyPDFLoader,<br>    PythonLoader,<br>    TextLoader,<br>    UnstructuredEPubLoader,<br>    UnstructuredFileLoader,<br>    UnstructuredHTMLLoader,<br>    UnstructuredMarkdownLoader,<br>    UnstructuredODTLoader,<br>    UnstructuredPowerPointLoader,<br>    UnstructuredWordDocumentLoader,<br>)<br>FILE_LOADER_MAPPING = {<br>    &quot;.csv&quot;: (CSVLoader, {&quot;encoding&quot;: &quot;utf-8&quot;}),<br>    &quot;.doc&quot;: (UnstructuredWordDocumentLoader, {}),<br>    &quot;.docx&quot;: (UnstructuredWordDocumentLoader, {}),<br>    &quot;.epub&quot;: (UnstructuredEPubLoader, {}),<br>    &quot;.html&quot;: (UnstructuredHTMLLoader, {}),<br>    &quot;.md&quot;: (UnstructuredMarkdownLoader, {}),<br>    &quot;.odt&quot;: (UnstructuredODTLoader, {}),<br>    &quot;.pdf&quot;: (PyPDFLoader, {}),<br>    &quot;.ppt&quot;: (UnstructuredPowerPointLoader, {}),<br>    &quot;.pptx&quot;: (UnstructuredPowerPointLoader, {}),<br>    &quot;.txt&quot;: (TextLoader, {&quot;encoding&quot;: &quot;utf8&quot;}),<br>    &quot;.ipynb&quot;: (NotebookLoader, {}),<br>    &quot;.py&quot;: (PythonLoader, {}),<br>}<br>### define page content output formatter<br>def format_docs(docs):<br>    return &quot;\n&quot;.join(doc.page_content for doc in docs)<br><br><br># Function to convert text to embeddings<br>def text_to_embedding(text, model):<br>    embedding = model.encode(text, convert_to_tensor=False)<br>    return embedding.tolist()<br><br><br>def read_file_and_save_to_faiss(directory_name):<br>    documents = []<br><br>    for file in os.listdir(directory_name):<br>        file_path = os.path.join(directory_name, file)<br>        file_type = os.path.splitext(file)[1]<br>        if file_type == &quot;&quot;:<br>            continue<br>        if file_type in FILE_LOADER_MAPPING:<br>            loader_class, loader_args = FILE_LOADER_MAPPING[file_type]<br>            loader = loader_class(file_path, **loader_args)<br>        else:<br>            loader = UnstructuredFileLoader(file_path)<br>        documents.extend(loader.load())<br><br>    print(&quot;files reading done&quot;)<br><br>    ### creating chunks<br>    text_splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=1000, chunk_overlap=200, add_start_index=True<br>    )<br>    all_splits = text_splitter.split_documents(documents)<br>    print(&quot;length of splits: &quot;, len(all_splits))<br><br>    ### prepare huggingface embedding model<br>    model_name = &quot;thenlper/gte-large&quot;<br>    model_kwargs = {&quot;device&quot;: &quot;cpu&quot;}<br>    encode_kwargs = {&quot;normalize_embeddings&quot;: False}<br>    embeddings_huggingFace = HuggingFaceEmbeddings(<br>        model_name=model_name, model_kwargs=model_kwargs, encode_kwargs=encode_kwargs<br>    )<br><br>    ### create vectorstore database and convert text into embeddings and save them<br>    vectorstore = FAISS.from_documents(<br>        documents=all_splits, embedding=embeddings_huggingFace<br>    )<br>    print(&quot;vector db created for uploaded documents&quot;)<br><br>    return vectorstore</pre><p><strong>talk2vectorDb.py:</strong></p><p>In this function, we utilize LangChain to retrieve documents similar to the user’s question. Using these documents, we create a prompt that is then passed to the OpenAI model, and the model’s response is returned to the calling function.</p><pre>import os<br>from langchain.embeddings import HuggingFaceEmbeddings<br>from langchain_core.output_parsers import StrOutputParser<br>from langchain_core.prompts import PromptTemplate<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_openai import OpenAI<br>from langchain.vectorstores import FAISS<br><br>### replace this with your open ai key<br>api_key = &quot;..&lt;your open ai key&gt;&quot;<br><br>### initalising the llm <br>llm = OpenAI(openai_api_key=api_key)<br><br><br>### define page content output formatter<br>def format_docs(docs):<br>    return &quot;\n\n&quot;.join(doc.page_content for doc in docs)<br><br><br>def retrieve_answers_from_faiss(vectorstore, question):<br><br>    ### create retriever from the database define its parameters<br>    retriever = vectorstore.as_retriever(<br>        search_type=&quot;similarity&quot;, search_kwargs={&quot;k&quot;: 6}<br>    )<br><br>    ### create prompt template<br>    template = &quot;&quot;&quot;Use the following pieces of context to answer the question at the end.<br>    If you don&#39;t know the answer, just say that you don&#39;t know, don&#39;t try to make up an answer.<br>    keep the answer as concise as possible.<br>    {context}<br>    Question: {question}<br>    Helpful Answer:&quot;&quot;&quot;<br><br>    custom_rag_prompt = PromptTemplate.from_template(template)<br>    print(&quot;prompt created&quot;)<br>    ### create chain of events<br>    ## retrive ,using question , pass the prompt, call the llm, parse the output<br>    rag_chain = (<br>        {&quot;context&quot;: retriever | format_docs, &quot;question&quot;: RunnablePassthrough()}<br>        | custom_rag_prompt<br>        | llm<br>        | StrOutputParser()<br>    )<br><br>    ### calling the chain using invoke method<br>    return rag_chain.invoke(question)</pre><p>4. Creating chat interface</p><p><strong>templates/chat.html</strong>:</p><p>This file contains HTML and JavaScript code to send and receive messages from the backend server. It also helps maintain and display the history of user messages.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br>&lt;head&gt;<br>    &lt;meta charset=&quot;UTF-8&quot;&gt;<br>    &lt;title&gt;Chat Page&lt;/title&gt;<br>    &lt;link rel=&quot;stylesheet&quot; href=&quot;{{ url_for(&#39;static&#39;, filename=&#39;chat.css&#39;) }}&quot;&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;div id=&quot;chat-header&quot;&gt;<br>        &lt;h1&gt;Chat with {{ username }}&lt;/h1&gt;<br>    &lt;/div&gt;<br>    &lt;div id=&quot;chat-container&quot;&gt;<br>        {% for message in chat_history %}<br>            {% if message.startswith(&#39;You:&#39;) %}<br>                &lt;div class=&quot;message user-message&quot;&gt;{{ message }}&lt;/div&gt;<br>            {% else %}<br>                &lt;div class=&quot;message app-message&quot;&gt;{{ message }}&lt;/div&gt;<br>            {% endif %}<br>        {% endfor %}<br>    &lt;/div&gt;<br>    &lt;div id=&quot;chat-input-container&quot;&gt;<br>        &lt;form id=&quot;chat-form&quot;&gt;<br>            &lt;input type=&quot;text&quot; id=&quot;chat-input&quot; placeholder=&quot;Type a message&quot; required&gt;<br>            &lt;button type=&quot;submit&quot;&gt;Send&lt;/button&gt;<br>        &lt;/form&gt;<br>    &lt;/div&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p><strong>static/chat.css</strong>:</p><p>Adding styling elements to give this chat app the look and feel of a standard chatting application.</p><pre>body {<br>  font-family: Arial, sans-serif;<br>  display: flex;<br>  flex-direction: column;<br>  height: 100vh;<br>  margin: 0;<br>}<br><br>#chat-header {<br>  background-color: #075E54;<br>  color: white;<br>  padding: 10px;<br>  text-align: center;<br>  font-size: 24px;<br>  flex-shrink: 0;<br>}<br><br>#chat-container {<br>  background-color: #E5DDD5;<br>  padding: 10px;<br>  flex-grow: 1;<br>  overflow-y: auto;<br>  display: flex;<br>  flex-direction: column;<br>}<br><br>.message {<br>  border-radius: 8px;<br>  padding: 10px;<br>  margin: 5px 0;<br>  max-width: 70%;<br>  display: inline-block;<br>  position: relative;<br>  word-wrap: break-word;<br>}<br><br>.user-message {<br>  background-color: #DCF8C6;<br>  align-self: flex-end;<br>  text-align: right;<br>}<br><br>.app-message {<br>  background-color: #FFF;<br>  align-self: flex-start;<br>}<br><br>.loading-message {<br>  background-color: #FFC107;<br>  align-self: center;<br>  text-align: center;<br>}<br><br>#chat-input-container {<br>  display: flex;<br>  align-items: center;<br>  background-color: #F1F1F1;<br>  padding: 10px;<br>  flex-shrink: 0;<br>}<br><br>#chat-form {<br>  display: flex;<br>  flex-grow: 1;<br>}<br><br>#chat-input {<br>  flex-grow: 1;<br>  border: none;<br>  padding: 10px;<br>  border-radius: 20px;<br>  margin-right: 10px;<br>  font-size: 16px;<br>}<br><br>#chat-input:focus {<br>  outline: none;<br>}<br><br>button {<br>  background-color: #25D366;<br>  color: white;<br>  border: none;<br>  border-radius: 50%;<br>  padding: 10px;<br>  font-size: 16px;<br>  cursor: pointer;<br>  display: flex;<br>  align-items: center;<br>  justify-content: center;<br>}<br><br>button:focus {<br>  outline: none;<br>}<br><br>#loading-message {<br>  color: #FF5733;<br>  padding-left: 10px;<br>}</pre><p><strong>static/chat.js</strong>:</p><pre>const chatForm = document.getElementById(&#39;chat-form&#39;);<br>const chatInput = document.getElementById(&#39;chat-input&#39;);<br>const chatContainer = document.getElementById(&#39;chat-container&#39;);<br><br>chatForm.addEventListener(&#39;submit&#39;, function(event) {<br>event.preventDefault();<br>const message = chatInput.value;<br>chatInput.value = &#39;&#39;;<br>// Add user&#39;s message to chat<br>const userMessage = document.createElement(&#39;div&#39;);<br>userMessage.classList.add(&#39;message&#39;, &#39;user-message&#39;);<br>userMessage.textContent = &#39;You: &#39; + message;<br>chatContainer.appendChild(userMessage);<br>chatContainer.scrollTop = chatContainer.scrollHeight;<br><br>// Show loading message<br>const loadingMessage = document.createElement(&#39;div&#39;);<br>loadingMessage.classList.add(&#39;message&#39;, &#39;loading-message&#39;);<br>loadingMessage.textContent = &#39;Loading...&#39;;<br>chatContainer.appendChild(loadingMessage);<br>chatContainer.scrollTop = chatContainer.scrollHeight;<br><br>fetch(&#39;{{ url_for(&#39;send_message&#39;) }}&#39;, {<br>    method: &#39;POST&#39;,<br>    headers: {<br>        &#39;Content-Type&#39;: &#39;application/x-www-form-urlencoded&#39;,<br>    },<br>    body: new URLSearchParams({<br>        &#39;message&#39;: message<br>    })<br>}).then(response =&gt; response.json()).then(data =&gt; {<br>    chatContainer.removeChild(loadingMessage); // Remove loading message<br>    data.chat_history.forEach(msg =&gt; {<br>        const newMessage = document.createElement(&#39;div&#39;);<br>        newMessage.classList.add(&#39;message&#39;);<br>        if (msg.startsWith(&#39;You:&#39;)) {<br>            newMessage.classList.add(&#39;user-message&#39;);<br>        } else {<br>            newMessage.classList.add(&#39;app-message&#39;);<br>        }<br>        newMessage.textContent = msg;<br>        chatContainer.appendChild(newMessage);<br>    });<br>    chatContainer.scrollTop = chatContainer.scrollHeight;<br>});<br>});</pre><p><strong>5. Flask Backend</strong></p><p>app.py:</p><p>This is the main backend function for <strong>Flask</strong> that integrates all components. It starts with the <strong>index.html</strong> page, and when the user uploads a document, it redirects them to <strong>chat.html</strong>, where they can interact with the document.</p><pre>import os<br><br>from flask import Flask, jsonify, redirect, render_template, request, session, url_for<br>from werkzeug.utils import secure_filename<br><br>## contains code to interact with vector DB<br>import talk2vectorDb<br><br>## contains code to upload file in vector DB<br>import upload2vectorDb<br><br>global vectorstore<br>app = Flask(__name__)<br><br># Set the secret key, which is needed for the session to work<br>app.secret_key = &quot;..&lt; choose any session key &gt;..&quot;<br><br>@app.route(&quot;/&quot;)<br>def index():<br>    return render_template(&quot;index.html&quot;)<br><br>### this function is used to save file upload folder<br>@app.route(&quot;/upload_file&quot;, methods=[&quot;POST&quot;])<br>def upload_file():<br><br>    username = request.form[&quot;username&quot;]<br>    files = request.files.getlist(&quot;file&quot;)<br><br>    directory_name = &quot;../&quot; + username<br>    # Set a session variable<br>    session[&quot;username&quot;] = username<br><br>    ## checking if folder for user exists or not if not create new folder<br>    if not os.path.exists(directory_name):<br>        os.makedirs(directory_name)<br><br>    for file in files:<br>        if file:<br>            filename = secure_filename(file.filename)<br>            file.save(os.path.join(directory_name, filename))<br>            print(file, &quot;is saved in &quot;, directory_name)<br><br>    ### calling function to create embedding and saving it to vectordb<br>    global vectorstore<br>    vectorstore = upload2vectorDb.read_file_and_save_to_faiss(directory_name)<br><br>    return redirect(url_for(&quot;chat&quot;))<br><br><br>### this function calls the chat.html page<br>@app.route(&quot;/chat&quot;)<br>def chat():<br>    username = session.get(&quot;username&quot;, &quot;Guest&quot;)<br>    chat_history = session.get(&quot;chat_history&quot;, [])<br>    return render_template(&quot;chat.html&quot;, username=username, chat_history=chat_history)<br><br><br>### this function is to provide an iteractive user interface for user to interact with vector DB<br>@app.route(&quot;/send_message&quot;, methods=[&quot;POST&quot;])<br>def send_message():<br>    message = request.form[&quot;message&quot;]<br>    chat_history = session.get(&quot;chat_history&quot;, [])<br>    chat_history.append(f&quot;You: {message}&quot;)<br>    print(&quot;send message to backend:&quot;, message)<br><br>    try:<br>        server_response = talk2vectorDb.retrieve_answers_from_faiss(<br>            vectorstore, message<br>        )<br>    except:<br>        server_response = &quot;*********file not uploaded properly********&quot;<br><br>    chat_history.append(server_response)<br>    session[&quot;chat_history&quot;] = chat_history<br>    return jsonify({&quot;chat_history&quot;: chat_history})<br><br><br>if __name__ == &quot;__main__&quot;:<br>    app.run(debug=True, host=&quot;0.0.0.0&quot;, port=8080)</pre><p><strong>6. Run the Flask server by executing:</strong></p><pre>python app.py</pre><p>After executing this command, open your web browser and go to <a href="http://127.0.0.1:5000/">http://127.0.0.1:8080/</a> select a unique username, upload your files, and wait for a few seconds; you will see a chatting interface like the one below, where you can interact with your file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dcguHeaYeGYCCHJNfi2LA.png" /><figcaption>The landing page of our created app where users can select a username and upload their files.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUT-uPEIYhic5los95Xgdg.png" /><figcaption>Example output of our created app, when account statement file is uploaded by the user.</figcaption></figure><h3>Conclusion</h3><p>This is a simple app that showcases the importance of the RAG pipeline. This article will also help you seamlessly integrate various components, such as LangChain, LLM, Vector DB and Flask. You can develop many such solutions using LLMs. Happy coding!</p><h3><strong>Written by</strong></h3><ol><li><a href="https://www.linkedin.com/in/abh1jeet/">Abhijeet Pandey</a> — Data Scientist — Walmart Global Tech</li><li><a href="https://in.linkedin.com/in/himanshu-pant-ds">Himanshu Pant </a>— Senior Data Scientist — Walmart Global Tech</li></ol><h3>References</h3><ol><li><a href="https://python.langchain.com/v0.2/docs/tutorials/rag/">https://python.langchain.com/v0.2/docs/tutorials/rag/</a></li><li><a href="https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html">https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html</a></li><li><a href="https://huggingface.co/thenlper/gte-large">https://huggingface.co/thenlper/gte-large</a></li><li><a href="https://python-adv-web-apps.readthedocs.io/en/latest/flask.html">https://python-adv-web-apps.readthedocs.io/en/latest/flask.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeaed58de536" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/creating-web-app-for-file-interactions-using-rag-a-developers-guide-aeaed58de536">Creating Web App For File Interactions Using RAG: A Developers Guide</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End to End pipeline for building a semantic search engine]]></title>
            <link>https://medium.com/@itsmeabhijeetpandey/end-to-end-pipeline-for-building-a-semantic-search-engine-fe4e845d892b?source=rss-b34699dfb8e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4e845d892b</guid>
            <category><![CDATA[nearest-neighbor-search]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[search-engine-ranking]]></category>
            <dc:creator><![CDATA[Abhijeet Pandey]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 15:22:41 GMT</pubDate>
            <atom:updated>2024-08-20T15:23:02.540Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A0M0aAHwamN8VjnqBGnTlQ.png" /><figcaption>Image generated using Stable Diffusion 2</figcaption></figure><p>Searching for a piece of a sentence in a vast corpus of sentences can be quickly done by fuzzy matching or pattern matching. Suppose you have a query that says “captain of the Indian cricket team” You will get all the articles that contain this exact query, but this is not what we always want because, in some cases, we might search for “Rohit Sharma “, but still we would want to see results related to the Indian cricket team. Simple pattern matching cannot do this, so &quot;Semantic Search&quot; comes to the rescue for this use case.</p><p>So, what exactly is semantic search? How to implement it? How to make it work for a vast corpus of data? Be patient, and all of these questions will be answered in this blog.</p><h3><strong>Semantic Search</strong></h3><p>Firstly, <strong>a semantic search</strong> is a data-searching procedure that denotes a search with meaning. It is different from lexical search, where the search engine looks for keywords of the query or variants of them without understanding the overall meaning of the query. In semantic search, we try to understand the intent and contextual purpose of the words. Semantic search improves search accuracy by understanding the content of the search query, in contrast to conventional search engines, which only find records based on lexical matches.</p><h3><strong>The NLP behind search engine</strong></h3><p>While working with textual data, one of the most significant steps is to convert words or sentences into vectors. There are multiple techniques like “Bag of words” and “Tf-IDF”, but these techniques do not give much importance to the meaning of words and also, with the increase in the size of vocabulary, they might face the curse of dimensionality. So, in our use case to convert sentences into embeddings, we will use state-of-the-art sentence transformers.</p><p>Sentence transformers are the models used to generate dense vector embeddings for words, sentences, or even paragraphs. Multiple transformer models are available (like BERT, RoBERTa, DistilBERT etc.). Out of which, we will use DistilBERT</p><h3><strong>Training Pipeline</strong></h3><p>The first step in any NLP project usually is preprocessing of the text, this step is use-case specific, and thus I will leave it. Some common preprocessing steps include removing stop-words, removing numbers, removing abbreviations, and converting to lowercase. Etc.</p><p>Once we have preprocessed text, we will pass it to a transformer network to convert these sentences into embeddings.</p><p>We are using a pre-trained “distilbert-base-nli-mean-tokens” transformer from the hugging face “sentence_transformer” library it converts given sentences into a 768-dimensional vector. Once the entire corpus of sentences has been converted to vector embeddings, we will save it in any database.</p><h3><strong>Inference Pipeline</strong></h3><p>All the training steps have been completed. Now, whenever a new query comes, it will go through similar preprocessing steps, and then it will pass through the same transformer model, which will spit out a query vector of 768 Dimension.</p><h3><strong>The Search Pipeline</strong></h3><p>To find the vector from the saved corpus that is most similar to the query vector, we will calculate the cosine similarity between all the vectors in the corpus and the query vector. In data analysis, cosine similarity is a measure of similarity between two vectors; its values lie between [-1,1]. Two exactly similar vectors have a similarity of 1, and two exactly opposite vectors have a cosine similarity of -1.</p><p>With the above-described linear search method, we can easily find the most similar vector for the query vector. But is it efficient?</p><p>When dealing with a huge corpus of data (~10 M records), linear search is inefficient for exact matching by scanning the whole repository. If we are trying to get output in real-time, linear search cannot fulfill our demands due to the curse of dimensionality.</p><p>These problems make linear search almost impractical, and approximate nearest neighbour (ANN) comes into the picture.</p><h3><strong>What is ANN?</strong></h3><p>ANNs are a set of algorithmic techniques that aims to return the approximate nearest neighbour as fast as possible.</p><p>The neighbour returned by ANN might not be the exact nearest neighbour, but it is the optimal neighbour given the time constraint. The difference between KNN and ANN is that instead of searching all the training data points in the prediction phase, we only search for a small subset of candidate points.</p><h3><strong>How does ANN work?</strong></h3><p>With the increase in internet usage these days, a lot of information is consumed by users daily, so finding efficient ways of searching a query has always been a hot topic for search engines like Google and e-commerce giants like Amazon. Query searching plays an important role. Therefore, finding the nearest neighbour has become a hot research topic increasing the chance of users finding the information they seek in a reasonable time.</p><p>There are multiple techniques available for finding ANN. But the core of all the methods is they speed up the search by preprocessing the data into an efficient data structure, then dividing the entire vector space into multiple regions and then searching only in some of the areas instead of the entire vector space.</p><p>Some of the common preprocessing steps for ANN include.</p><p>· <strong>Vector transformation</strong> — Before indexing vector goes through different transformation techniques, e.g., dimensionality reduction and vector rotation.</p><p>· <strong>Vector Encoding </strong>— This step converts the vector into the actual index that can be searched. These techniques include data structure-based techniques like Trees, LSH and quantisation-based techniques to encode vectors to a much more compact form</p><p>· <strong>Non-Exhaustive Search</strong> — All the ANN technique uses this step in one way or another while searching index files. Instead of searching all the data points, we only searched in the probable region where there is the possibility of finding the nearest neighbour.</p><p>ANN reduces the search space by only comparing with a small subset of candidate points. This can be done by using Tree-Based algorithms (Like Annoy, ScaNN ) or LSH-based algorithms (like FAISS).</p><p>ANNOY is one of the most famous ANNs in the market, and it was developed by a team of researchers at Spotify. <strong>ANNOY</strong> is the algorithm that provides us with music recommendations based on songs we have previously played. You can read more about ANNOY <a href="https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html"><strong>here</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIZqUDGbv-z4Tm7POCNJKA.png" /><figcaption>Source: <a href="https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html">https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auInYkelIlJg0f2aczXieQ.png" /><figcaption>Source: <a href="https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html">https://erikbern.com/2015/10/01/nearest-neighbors-and-vector-models-part-2-how-to-search-in-high-dimensional-spaces.html</a></figcaption></figure><p>Basically, ANNOY works by creating many trees (forests) by picking two points at random and splitting the space into two parts by using hyperplane; we recursively keep splitting into subspaces until the number of data points in the node is small enough.</p><p>FAISS is another ANN technique developed by the Facebook AI team. The core of FAISS is Location-sensitive hashing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IfRRFOfiztl3i08UXF-aag.png" /><figcaption>Source: <a href="https://brc7.github.io/2019/09/19/Visual-LSH.html">https://brc7.github.io/2019/09/19/Visual-LSH.html</a></figcaption></figure><p>In LSH, to construct the index, we apply multiple hash functions to map data points into buckets so that data points near each other are in the same buckets with high probability. In contrast, data points far from each other will likely fall into different buckets.</p><h3><strong>How to select which ANN algorithm to use?</strong></h3><p>The answer to this question varies from use case to use case. These points should be considered while selecting which ANN algorithm to use.</p><blockquote>· Algorithm should give results as close as possible to the actual neighbour</blockquote><blockquote>· There might be use cases where the addition of new indexes and removing old indexes are important.</blockquote><blockquote>· Some algorithms don’t support user-defined IDs, which might be important for some use cases.</blockquote><blockquote>· There are multiple ways to calculate similarity scores, like Manhattan distance, L2 distance, cosine distance etc. Based on the use case, this might be important.</blockquote><blockquote>· GPU-based implementations are much faster than Only CPU-based ANN. Thus, depending on the availability, you can select GPU-based algorithms.</blockquote><h3><strong>Performance Benchmarking</strong></h3><p>For Benchmarking purposes, we used a dataset of 1M records. Based on our use case, we will use FAISS-GPU for our semantic search engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*XO5yf1-fzGLgNl9NPP8J_A.png" /><figcaption>Performance benchmarking of different ANN algorithms</figcaption></figure><h3><strong>Final Search Architecture</strong></h3><p>The final architecture of our search engine is going to look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*qBYBxPB46GDXfCchUXWyzA.png" /><figcaption>The architecture of a basic search engine</figcaption></figure><p>As everything is final, let&#39;s code it.</p><p>We will start with importing and installing the required libraries</p><pre><br>!pip install faiss-cpu<br>!pip install -U sentence-transformers<br>import numpy as np<br>import torch<br>import os<br>import pandas as pd<br>import faiss<br>import time<br>from sentence_transformers import SentenceTransformer</pre><p>Now import any text data, and apply some preprocessing on it.</p><pre>df=pd.read_csv(&quot;any_data.csv&quot;)<br>df=preprocessing(df)</pre><p>As per the architecture, convert the textual data into vector embeddings using a sentence transformer.</p><pre>model = SentenceTransformer(&#39;distilbert-base-nli-mean-tokens&#39;)<br>encoded_data = model.encode(data)</pre><p>Once embeddings have been generated, we will convert them into the FAISS index file and save them for future reference.</p><pre>## dimension of embeddings<br>vector_dimension=768<br><br>## number of clusters in which data will be partitined<br>ncluster=5<br><br>quantizer = faiss.IndexFlatL2(vector_dimension)<br><br>## creating partition index<br>index = faiss.IndexIVFFlat(quantizer,vector_dimension,ncluster)<br><br>## always search in nearest 2 clusters<br>index.nprobe=2<br><br>index.train(encoded_data)<br>index.add(encoded_data)<br><br>## saving the index in local for future use<br>faiss.write_index(index,&#39;index_file_name&#39;)</pre><p>Now our training pipeline is done; whenever a new query comes, it will pass through similar steps to give us the most similar results.</p><pre>## reading the saved index file<br>index = faiss.read_index(&#39;index_file_name&#39;)<br><br>## converting query into vector<br>search_query= preprocessing(search_query)<br>query_vector= model.encode([search_query])<br><br>## searching for nearest 5 neighbours in the index file<br>k= 5<br><br>## indices will contain index of nearest 5 neighbour<br>## distances will contain the similarity score of correspoding index<br>distances, indices = index.search(query_vector, k)</pre><p><strong>Voila!!</strong> Our Semantic Search engine is ready.</p><h4><strong>References</strong></h4><p>1-<a href="https://www.elastic.co/what-is/semantic-search">https://www.elastic.co/what-is/semantic-search</a></p><p>2-<a href="https://github.com/facebookresearch/faiss">https://github.com/facebookresearch/faiss</a></p><p>3-<a href="https://github.com/spotify/annoy">https://github.com/spotify/annoy</a></p><p>4-<a href="https://huggingface.co/thenlper/gte-large">https://huggingface.co/thenlper/gte-large</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4e845d892b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>