<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Milind Choudhary  on Medium]]></title>
        <description><![CDATA[Stories by Milind Choudhary  on Medium]]></description>
        <link>https://medium.com/@milovate?source=rss-48bfc10d9262------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XwuskmvNpv3xqi0DqjT31g.jpeg</url>
            <title>Stories by Milind Choudhary  on Medium</title>
            <link>https://medium.com/@milovate?source=rss-48bfc10d9262------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:06:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@milovate/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ReaderQA: Vision-Based RAG System with ColPali and Vespa]]></title>
            <link>https://milovate.medium.com/readerqa-vision-based-rag-system-with-colpali-and-vespa-096b58803198?source=rss-48bfc10d9262------2</link>
            <guid isPermaLink="false">https://medium.com/p/096b58803198</guid>
            <category><![CDATA[vespa]]></category>
            <category><![CDATA[colpali]]></category>
            <category><![CDATA[colbert]]></category>
            <dc:creator><![CDATA[Milind Choudhary ]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 08:06:32 GMT</pubDate>
            <atom:updated>2024-08-18T08:06:32.085Z</atom:updated>
            <content:encoded><![CDATA[<p>Document retrieval is an essential aspect of modern information systems, aiming to efficiently match user queries with the most relevant documents from extensive databases. As digital content becomes increasingly diverse, including text, images, tables, and diagrams, the challenge of retrieving accurate information grows. Traditional document retrieval systems have excelled in handling textual data but often fall short when it comes to integrating visual elements, which are crucial for a complete understanding of many documents.</p><p>ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features. It is a <a href="https://huggingface.co/google/paligemma-3b-mix-448">PaliGemma-3B</a> extension that generates <a href="https://arxiv.org/abs/2004.12832">ColBERT</a>- style multi-vector representations of text and images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*wgyn-TtX1IOMFkVY.png" /></figure><p><strong>ReaderQA</strong> utilizes ColPali and Vespa to enable conversational interactions with PDFs. By appending a specific prefix to a URL, the embedding process is triggered, which utilizes ColPali to generate and store embeddings for text and titles using the Vespa framework and Vespa Cloud. When a user poses a question, documents are ranked using a hybrid rank profile that combines BM25 for text and titles with similarity scores for embeddings.</p><p>For each page of a PDF, <strong>ReaderQA</strong> generates an image using pdf2image and extracts the text using pypdf. ColPali processes these images to obtain visual multi-vector embeddings, which are then stored in Vespa. The <a href="https://blog.vespa.ai/announcing-long-context-colbert-in-vespa/">long-context variant</a> of <strong>ColBERT</strong> is employed, representing each page’s embeddings with the tensor tensor(page{}, patch{}, v[128]). This method treats the entire PDF as a document, with page embeddings stored within it.</p><p>Additionally, base64-encoded images and page metadata, such as titles and URLs, are stored. This metadata is utilized not only for displaying results but also for RAG (retrieval-augmented generation) with powerful language models that possess vision capabilities.</p><p>At query time, <strong>ReaderQA</strong> retrieves documents using <a href="https://docs.vespa.ai/en/reference/bm25.html">BM25</a> for all text across pages. The results are then refined using ColPali embeddings to maximize page scores, ensuring that the most relevant information is presented to the user.</p><p>most relevent documents are utilized to answer the question using a LLM with vision capabilities. we input the best matching PDF <em>_page_</em> image and the question.</p><p>Our application largely builds upon the following excellent resources:</p><ul><li><a href="https://github.com/illuin-tech/colpali/blob/main/demo/app.py">https://github.com/illuin-tech/colpali/blob/main/demo/app.py</a></li><li><a href="https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models.html">https://pyvespa.readthedocs.io/en/latest/examples/colpali-document-retrieval-vision-language-models.html</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5H0KLZdnGUcFkU0F7AndA.png" /><figcaption>ReaderQA :Architecture</figcaption></figure><p><strong>Results</strong></p><ul><li>GOVERNMENT OF INDIA BUDGET 2024–2025</li><li><a href="https://www.indiabudget.gov.in/doc/Budget_Speech.pdf">https://www.indiabudget.gov.in/doc/Budget_Speech.pdf</a></li></ul><pre>query    : <br>----------------------------------------------------------------------------<br>&quot;custom duty rates changes in roasted nuts and seeds&quot;<br><br><br><br><br>response :<br>----------------------------------------------------------------------------<br><br>&quot;Based on the information provided in the image, there are changes to the custom duty rates for roasted nuts and seeds:\n\n&quot;<br>            &quot;1. The tariff rate for &#39;Other roasted nuts and seeds, including areca nuts&#39; is increasing from 30% to 150%.\n\n&quot;<br>            &quot;2. This change is part of section D.3. titled &#39;Increase in tariff rate with no change in effective duty rate&#39;.\n\n&quot;<br>            &quot;3. The new rates will take effect from 01.10.2024.\n\n&quot;<br>            &quot;4. This change applies to two categories:\n&quot;<br>            &quot;   a) &#39;Other roasted nuts and seeds, including areca nuts&#39;\n&quot;<br>            &quot;   b) &#39;Other nuts, otherwise prepared or preserved, including areca nuts&#39;\n\n&quot;<br>            &quot;5. For both categories, the rate is increasing from 30% (per cent) to 150% (per cent).\n\n&quot;<br>            &quot;It&#39;s important to note that while the tariff rate is increasing significantly, the title of the section suggests that there is &#39;no change in effective duty rate&#39;. &quot;<br>            &quot;This implies that while the official tariff is increasing, the actual amount paid may remain the same due to other factors or calculations not shown in this table.&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JI9y_kI4_mnP0p4U4v-Kg.png" /><figcaption>GOVERNMENT OF INDIA BUDGET 2024–2025</figcaption></figure><ul><li>Rajasthan Medical &amp; Health Service Rule, 1963</li><li><a href="https://rajswasthya.nic.in/Amendments%20rules.pdf">https://rajswasthya.nic.in/Amendments%20rules.pdf</a></li></ul><pre>query    : <br>----------------------------------------------------------------------------<br>&quot; what are the methods of recruitment ?&quot;<br><br>response :<br>----------------------------------------------------------------------------<br>&quot;&quot;&quot;<br>Based on the information provided in the image, the methods of recruitment to the Rajasthan Medical &amp; Health Service are:<br><br>1. Direct recruitment in accordance with Part IV of these rules<br>2. Promotion in accordance with Part V of these rules<br>3. Transfer from amongst persons holding substantively post of Junior Specialist in any other speciality, provided they meet certain qualifications and experience requirements<br><br>Additionally, the image mentions:<br><br>4. Recruitment to the service by the aforementioned methods shall be made in a manner that adheres to certain proportions and cadre strength guidelines.<br><br>The image also notes some provisions for exceptional circumstances:<br><br>5. If the Government is satisfied in consultation with the Commission that suitable persons are not available for appointment by either method, appointments may be made by other methods in relaxation of the prescribed proportion.<br><br>6. The Government retains the ability to appoint officers previously in the employment of pre-reorganization of States of Ajmer, Bombay and Madhya Bharat to suitable posts in the Schedule.<br><br>7. The rules do not preclude the Government from re-employing persons having retired from the Service.<br><br>These methods provide a comprehensive framework for recruitment into the Rajasthan Medical &amp; Health Service, allowing for various entry points and flexibility when needed.<br>&quot;&quot;&quot;<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMcg4AIqf-ZDgfOffEkTeQ.png" /><figcaption>Rajasthan Medical &amp; Health Service Rule, 1963</figcaption></figure><p><strong>Conclusion</strong> :</p><p><strong>ReaderQA</strong> demonstrates the powerful capabilities of ColPali in document retrieval, particularly its ability to handle complex visual data without relying on advanced OCR or layout detection techniques. By integrating ColPali with Vespa and leveraging vision-capable language models, ReaderQA offers a robust solution for conversational interactions with PDF documents.</p><p><strong>Contributers </strong>:</p><ul><li><a href="http://www.linkedin.com/in/milind-choudhary42">www.linkedin.com/in/milind-choudhary42</a></li><li><a href="https://www.linkedin.com/in/aryaman-vishnoi-827502229?">www.linkedin.com/in/aryaman-vishnoi-827502229</a></li><li><a href="https://www.linkedin.com/in/kushal-garg-238051234">www.linkedin.com/in/kushal-garg-238051234</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=096b58803198" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid Agentic RAG with Neo4j and Weaviate using LlamaAgents]]></title>
            <link>https://milovate.medium.com/hybrid-agentic-rag-with-neo4j-and-weaviate-using-llamaagent-b09a517e949e?source=rss-48bfc10d9262------2</link>
            <guid isPermaLink="false">https://medium.com/p/b09a517e949e</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[weaviate]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[neo4j]]></category>
            <dc:creator><![CDATA[Milind Choudhary ]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 04:46:02 GMT</pubDate>
            <atom:updated>2024-07-14T06:03:22.838Z</atom:updated>
            <content:encoded><![CDATA[<p>In the ever-evolving landscape of AI and information retrieval, Hybrid Agentic RAG (Retrieval-Augmented Generation) emerges as a powerful approach. This article explores an innovative implementation using Neo4j for knowledge graphs and Weaviate for vector storage, using newly launched llama agents Framework . this innovative system excels in handling multi-hop queries by leveraging knowledge graphs to navigate complex entity relationships, while also optimising for detailed answers using vector databases . By integrating these technologies, enabling more accurate and resource-efficient responses to a wide range of query complexities. We’ll dive into the architecture and implementation details, showcasing how this hybrid system enhances information retrieval and generation capabilities.</p><h4>Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQpjhwl02wBCsLK4WRpYvA.png" /><figcaption>Agentic RAG architecture</figcaption></figure><p>The architecture of our Hybrid Agentic RAG system comprises several key components:</p><p><strong><em>Agent Services:</em></strong></p><p>KG_agent: Utilizes Neo4jGraphStore for knowledge graph operations<br>Vector_agent: Employs Weaviate as a Vectorstore for efficient vector-based retrieval</p><p><strong><em>Language Models:</em></strong></p><p>LLM: NousResearch/Nous-Hermes-2-Mistral-7B-DPO (via TogetherAI)<br>Embeddings: togethercomputer/m2-bert-80M-8k-retrieval</p><p><strong><em>llama-agents Components:</em></strong></p><p><em>Message Queue</em>: Manages communication between services and the control plane<br><em>Control Plane: </em>Acts as the central gateway, tracking tasks and registered services<br><em>Orchestrator</em>: Handles task delegation and result processing.<br><em>Services</em>: Perform specific tasks, including the KG_agent and Vector_agent <em>Dataset:</em> “The Hound of the Baskervilles” by Sir Arthur Conan Doyle</p><h4>Implementation</h4><pre>%pip install llama-index-llms-together<br>%pip install llama_agents<br>%pip install neo4j<br>%pip install llama-index-graph-stores-neo4j<br>%pip install llama-index-vector-stores-weaviate<br>%pip install llama-index-embeddings-together</pre><pre>import logging<br>import sys<br>import os<br>import nest_asyncio<br>from llama_index.core import Settings<br>from IPython.display import Markdown, display<br>nest_asyncio.apply()<br><br>logging.basicConfig(<br>    stream=sys.stdout, level=logging.INFO<br>)<br><br>logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))</pre><pre>from llama_index.llms.together import TogetherLLM<br>from llama_index.embeddings.together import TogetherEmbedding<br><br><br>llm = TogetherLLM(<br>    model=&quot;NousResearch/Nous-Hermes-2-Mistral-7B-DPO&quot;,<br>    api_key=&quot;&quot;<br>)<br>embed_model = TogetherEmbedding(<br>    model_name=&quot;togethercomputer/m2-bert-80M-8k-retrieval&quot;,<br>    api_key=&quot;&quot;<br>)<br><br>Settings.embed_model = embed_model<br>Settings.llm = llm</pre><p>We set up our LLM and embedding model using TogetherAI by installing the necessary packages and configuring the models. We use the TogetherLLM class to set up the LLM model Nous-Hermes-2-Mistral-7B-DPOand the TogetherEmbeddingclass to set up the embedding model with m2-bert-80M-8k-retrieval.</p><p>load document</p><pre>from llama_index.core import SimpleDirectoryReader<br><br>!mkdir -p &#39;data/&#39;<br>!wget &#39;https://www.gutenberg.org/cache/epub/2852/pg2852.txt&#39; -O &#39;data/The Hound of the Baskervilles.txt&#39;<br><br><br>documents = SimpleDirectoryReader(<br>    input_files=[&#39;data/The Hound of the Baskervilles.txt&#39;]<br>).load_data()<br><br></pre><p>Setting up Knowledge Graph Store</p><pre><br>from llama_index.core import KnowledgeGraphIndex<br>from llama_index.core import StorageContext<br>from llama_index.graph_stores.neo4j import Neo4jGraphStore<br><br><br>graph_store = Neo4jGraphStore(<br>    username=neo4j_user,<br>    password=neo4j_password,<br>    url=neo4j_uri,<br>    database=neo4j_database,<br>)<br><br>storage_context = StorageContext.from_defaults(graph_store=graph_store)<br><br># NOTE: can take a while!<br>index = KnowledgeGraphIndex.from_documents(<br>    documents,<br>    storage_context=storage_context,<br>    max_triplets_per_chunk=2,<br>    include_embeddings=True,<br>)<br><br><br><br>query_engine = index.as_query_engine(<br>    include_text=False,<br>    response_mode=&quot;tree_summarize&quot;,<br>    similarity_top_k=5,<br>    verbose=True,<br>)</pre><p>Setting up Vector Store</p><pre>from llama_index.core import VectorStoreIndex<br>from llama_index.vector_stores.weaviate import WeaviateVectorStore<br>from llama_index.core import StorageContext<br>import weaviate<br><br># cloud<br>cluster_url = &quot;&quot;<br>api_key = &quot;&quot;<br><br>client = weaviate.connect_to_wcs(<br>    cluster_url=cluster_url,<br>    auth_credentials=weaviate.auth.AuthApiKey(api_key),<br>)<br><br># local<br># client = connect_to_local()<br><br>vector_store = WeaviateVectorStore(<br>    weaviate_client=client, index_name=&quot;LlamaIndex&quot; ,embed_model=embed_model<br>)<br>storage_context_vector = StorageContext.from_defaults(vector_store=vector_store)<br>index_vector = VectorStoreIndex.from_documents(<br>    documents, storage_context=storage_context_vector , embeddings=embed_model<br>)<br><br>query_engine_vector = index_vector.as_query_engine(verbose=True)</pre><pre><br>from llama_index.core.tools import QueryEngineTool, ToolMetadata<br>import pandas as pd<br><br>from llama_agents import (<br>    AgentService,<br>    ToolService,<br>    LocalLauncher,<br>    MetaServiceTool,<br>    ControlPlaneServer,<br>    SimpleMessageQueue,<br>    AgentOrchestrator,<br>)<br><br>from llama_index.core.agent import FunctionCallingAgentWorker<br>from llama_index.llms.openai import OpenAI<br><br><br># TODO: Try optimize descriptions <br>tool_KG = QueryEngineTool(<br>    query_engine=query_engine,<br>    metadata=ToolMetadata(<br>        name=&quot;NEO4J&quot;,<br>        description=(<br>            &quot;This is a knowledge graph query engine. Users can input queries with specific keywords or patterns, &quot;<br>            &quot;which the agent translates into Cypher queries to search and retrieve information from the Neo4j graph database. &quot;<br>            &quot;Use a detailed plain text question as input to this tool.&quot;<br>        ),<br>    ),<br>)<br><br>tool_vector = QueryEngineTool(<br>    query_engine=query_engine_vector,<br>    metadata=ToolMetadata(<br>        name=&quot;VECTOR&quot;,<br>        description=(<br>            &quot;This is a vector store query engine. Users can input queries with specific keywords or patterns, &quot;<br>            &quot;which the agent translates into vector representations to perform similarity searches within the vector database. &quot;<br>            &quot;Use a detailed plain text question as input to this tool.&quot;<br>        ),<br>    ),<br>)<br><br><br># create our multi-agent framework components<br>message_queue = SimpleMessageQueue()<br>control_plane = ControlPlaneServer(<br>    message_queue=message_queue,<br>    orchestrator=AgentOrchestrator( llm=OpenAI(model=&quot;gpt-3.5-turbo-0125&quot;, api_key=&quot;sk-&quot;),),<br>)<br><br># define Tool Service<br>tool_service = ToolService(<br>    message_queue=message_queue,<br>    tools=[tool_KG,tool_vector],<br>    running=True,<br>    step_interval=0.5,<br>)<br><br># define meta-tools here<br>meta_tool_KG = await MetaServiceTool.from_tool_service(<br>        tool_KG.metadata.name,<br>        message_queue=message_queue,<br>        tool_service=tool_service,<br>    )<br><br><br>meta_tool_vector = await MetaServiceTool.from_tool_service(<br>        tool_vector.metadata.name,<br>        message_queue=message_queue,<br>        tool_service=tool_service,<br>    )<br><br># define Agent and agent service<br><br># TODO: optimize agent description <br>worker1 = FunctionCallingAgentWorker.from_tools(<br>    [tool_KG],<br>    llm=OpenAI(model=&quot;gpt-3.5-turbo-0125&quot;, api_key=&quot;sk-&quot;),<br>)<br>agent1 = worker1.as_agent()<br>agent_server_1 = AgentService(<br>    agent=agent1,<br>    message_queue=message_queue,<br>    description=&quot;The Knowledge Graph Agent Service .Use it for exploring entity relationships hierarchical data queries, and multi-hop queries. Perfect for detailed document searches based on content relationships.&quot;,<br>    service_name=&quot;KG_AGENT&quot;,<br>)<br><br><br>worker2 = FunctionCallingAgentWorker.from_tools(<br>    [meta_tool_vector],<br>    llm=OpenAI(model=&quot;gpt-3.5-turbo-0125&quot;, api_key=&quot;sk-&quot;),<br>)<br>agent2 = worker2.as_agent()<br>agent_server_2 = AgentService(<br>    agent=agent2,<br>    message_queue=message_queue,<br>    description=&quot;The Vector Store Agent Service Ideal for finding similar documents,and provide generalized answers&quot;,<br>    service_name=&quot;VECTOR_AGENT&quot;,<br>)<br><br></pre><p>Launching agents</p><pre>import logging<br><br># change logging level to enable or disable more verbose logging<br>logging.getLogger(&quot;llama_agents&quot;).setLevel(logging.INFO)<br><br>## Define Launcher<br>launcher = LocalLauncher(<br>    [agent_server_1,agent_server_2,tool_service],<br>    control_plane,<br>    message_queue,<br>)</pre><pre>question =&quot;&quot;&quot;What key evidence does Sherlock Holmes find in the portrait of Hugo Baskerville?&quot;&quot;&quot;<br>result = launcher.launch_single(question)</pre><pre>OUTPUT:<br>============================================================================<br><br>The key evidence in the portrait of Hugo Baskerville lies in the resemblance between Stapleton and the portrait of Hugo Baskerville. Sherlock Holmes observes this similarity, particularly in the facial features, which leads him to deduce that Stapleton is actually a member of the Baskerville family, specifically a descendant of Hugo Baskerville.</pre><pre>question =&quot;&quot;&quot;How does Sherlock Holmes use both the discovery of Sir Charles Baskerville&#39;s footprints and the portrait of Hugo Baskerville to deduce the identity of the murderer and uncover their motive?&quot;&quot;&quot;<br>result = launcher.launch_single(question)</pre><pre>OUTPUT:<br>============================================================================<br><br>Sherlock Holmes uses the discovery of Sir Charles Baskerville&#39;s footprints and the portrait of Hugo Baskerville to deduce the identity of the murderer and uncover their motive by carefully observing and analyzing the evidence. By examining the footprints, Holmes is able to determine specific details about the murderer&#39;s physical attributes, such as their gait or limp, which helps narrow down the list of suspects. The portrait of Hugo Baskerville, on the other hand, provides Holmes with historical context and potential motives for the crime. By combining the information gathered from the footprints and the portrait, Holmes is able to piece together the identity of the murderer and their motive, ultimately solving the case.</pre><h4>Conclusion</h4><p>Although a naive half baked approach to RAG , Hybrid Agentic RAG system can leverage the strengths of both knowledge graphs and vector stores to provide comprehensive and accurate information retrieval. we can create a flexible and system capable of handling complex queries with improved context understanding and relevance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b09a517e949e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>